From: jongfoster <jongfoster@users.sourceforge.net>
Date: Sat, 22 Sep 2001 14:55:03 +0000 (+0000)
Subject: Telling robots not to touch /config/ - if there's a JunkBuster
X-Git-Tag: v_2_9_9~49
X-Git-Url: http://www.privoxy.org/gitweb/?a=commitdiff_plain;h=475c90e2459bd79c5e76bfd12ec173c75d63b9b1;p=privoxy.git

Telling robots not to touch /config/ - if there's a JunkBuster
proxy between the robot and the web server, then accessing
/config can affect the proxy.
---

diff --git a/doc/webserver/robots.txt b/doc/webserver/robots.txt
new file mode 100644
index 00000000..790f3ffc
--- /dev/null
+++ b/doc/webserver/robots.txt
@@ -0,0 +1,15 @@
+# This is the Internet Junkbuster web site.
+#
+# If the robot is connecting through a JunkBuster, then the
+# control interface is at /config.   It isn't very useful
+# to index it, and you're likely to break stuff.  So go away!
+#
+# Even if you're not connected through a JunkBuster, the 
+# one "JunkBuster's not working" page there is not very
+# interesting.
+
+User-agent: *
+Disallow: /config/
+
+
+