7 CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+
10 TITLE="Privoxy User Manual"
11 HREF="index.html"><LINK
14 HREF="seealso.html"><LINK
17 HREF="../p_doc.css"></HEAD
28 SUMMARY="Header navigation table"
37 >Privoxy User Manual</TH
69 NAME="APPENDIX">14. Appendix</H1
75 NAME="REGEX">14.1. Regular Expressions</H2
80 > uses Perl-style <SPAN
85 HREF="actions-file.html"
89 HREF="filter-file.html"
93 HREF="http://www.pcre.org/"
98 HREF="http://www.oesterhelt.org/pcrs/"
103 > If you are reading this, you probably don't understand what <SPAN
107 > are, or what they can do. So this will be a very brief
108 introduction only. A full explanation would require a <A
109 HREF="http://www.oreilly.com/catalog/regex/"
114 > Regular expressions provide a language to describe patterns that can be
115 run against strings of characters (letter, numbers, etc), to see if they
116 match the string or not. The patterns are themselves (sometimes complex)
117 strings of literal characters, combined with wild-cards, and other special
118 characters, called meta-characters. The <SPAN
120 >"meta-characters"</SPAN
122 special meanings and are used to build complex patterns to be matched against.
123 Perl Compatible Regular Expressions are an especially convenient
127 > of the regular expression language.</P
129 > To make a simple analogy, we do something similar when we use wild-card
130 characters when listing files with the <B
137 > matches all filenames. The <SPAN
141 character here is the asterisk which matches any and all characters. We can be
142 more specific and use <TT
145 > to match just individual
148 >"dir file?.text"</SPAN
156 >, etc. We are pattern
157 matching, using a similar technique to <SPAN
159 >"regular expressions"</SPAN
162 > Regular expressions do essentially the same thing, but are much, much more
163 powerful. There are many more <SPAN
165 >"special characters"</SPAN
167 building complex patterns however. Let's look at a few of the common ones,
168 and then some examples:</P
183 > - Matches any single character, e.g. <SPAN
221 > - The preceding character or expression is matched ZERO or ONE
244 > - The preceding character or expression is matched ONE or MORE
267 > - The preceding character or expression is matched ZERO or MORE
293 > character denotes that
294 the following character should be taken literally. This is used where one of the
295 special characters (e.g. <SPAN
298 >) needs to be taken literally and
299 not as a special meta-character. Example: <SPAN
301 >"example\.com"</SPAN
303 sure the period is recognized only as a period (and not expanded to its
304 meta-character meaning of any single character).
326 > - Characters enclosed in brackets will be matched if
327 any of the enclosed characters are encountered. For instance, <SPAN
331 matches any numeric digit (zero through nine). As an example, we can combine
335 > to match any digit one of more times: <SPAN
360 > - parentheses are used to group a sub-expression,
361 or multiple sub-expressions.
386 > character works like an
390 > conditional statement. A match is successful if the
391 sub-expression on either side of <SPAN
394 > matches. As an example:
397 >"/(this|that) example/"</SPAN
398 > uses grouping and the bar character
399 and would match either <SPAN
401 >"this example"</SPAN
415 > These are just some of the ones you are likely to use when matching URLs with
419 >, and is a long way from a definitive
420 list. This is enough to get us started with a few simple examples which may
421 be more illuminating:</P
433 that uses the common combination of <SPAN
440 denote any character, zero or more times. In other words, any string at all.
441 So we start with a literal forward slash, then our regular expression pattern
445 >) another literal forward slash, the string
449 >, another forward slash, and lastly another
454 a directory path here. This will match any file with the path that has a
455 directory named <SPAN
462 any characters, and this could conceivably be more forward slashes, so it
463 might expand into a much longer looking path. For example, this could match:
466 >"/eye/hate/spammers/banners/annoy_me_please.gif"</SPAN
470 >"/banners/annoying.html"</SPAN
471 >, or almost an infinite number of other
472 possible combinations, just so it has <SPAN
478 > A now something a little more complex:</P
486 >/.*/adv((er)?ts?|ertis(ing|ements?))?/</TT
490 We have several literal forward slashes again (<SPAN
494 building another expression that is a file path statement. We have another
498 >, so we are matching against any conceivable sub-path, just so
499 it matches our expression. The only true literal that <SPAN
506 > our pattern is <SPAN
510 the forward slashes. What comes after the <SPAN
514 interesting part. </P
519 > means the preceding expression (either a
520 literal character or anything grouped with <SPAN
524 can exist or not, since this means either zero or one match. So
527 >"((er)?ts?|ertis(ing|ements?))"</SPAN
528 > is optional, as are the
529 individual sub-expressions: <SPAN
535 >"(ing|ements?)"</SPAN
546 >. We have two of those. For instance,
549 >"(ing|ements?)"</SPAN
550 >, can expand to match either <SPAN
563 >. What is being done here, is an
564 attempt at matching as many variations of <SPAN
566 >"advertisement"</SPAN
568 similar, as possible. So this would expand to match just <SPAN
584 >"advertisement"</SPAN
588 >"advertisements"</SPAN
589 >. You get the idea. But it would not match
592 >"advertizements"</SPAN
596 >). We could fix that by
597 changing our regular expression to:
600 >"/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/"</SPAN
601 >, which would then match
610 >/.*/advert[0-9]+\.(gif|jpe?g)</TT
614 another path statement with forward slashes. Anything in the square brackets
618 > can be matched. This is using <SPAN
622 shorthand expression to mean any digit one through nine. It is the same as
626 >. So any digit matches. The <SPAN
630 means one or more of the preceding expression must be included. The preceding
631 expression here is what is in the square brackets -- in this case, any digit
632 one through nine. Then, at the end, we have a grouping: <SPAN
636 This includes a <SPAN
639 >, so this needs to match the expression on
640 either side of that bar character also. A simple <SPAN
643 > on one side, and the other
644 side will in turn match either <SPAN
654 > means the letter <SPAN
658 can be matched once or not at all. So we are building an expression here to
659 match image GIF or JPEG type image file. It must include the literal
663 >, then one or more digits, and a <SPAN
667 (which is now a literal, and not a special character, since it is escaped
671 >), and lastly either <SPAN
681 >. Some possible matches would
684 >"//advert1.jpg"</SPAN
688 >"/nasty/ads/advert1234.gif"</SPAN
692 >"/banners/from/hell/advert99.jpg"</SPAN
693 >. It would not match
697 > (no leading slash), or
700 >"/adverts232.jpg"</SPAN
701 > (the expression does not include an
707 >"/advert1.jsp"</SPAN
712 in the expression anywhere).</P
714 > We are barely scratching the surface of regular expressions here so that you
715 can understand the default <SPAN
719 configuration files, and maybe use this knowledge to customize your own
720 installation. There is much, much more that can be done with regular
721 expressions. Now that you know enough to get started, you can learn more on
724 > More reading on Perl Compatible Regular expressions:
726 HREF="http://www.perldoc.com/perl5.6/pod/perlre.html"
728 >http://www.perldoc.com/perl5.6/pod/perlre.html</A
731 > For information on regular expression based substitutions and their applications
732 in filters, please see the <A
733 HREF="filter-file.html"
734 >filter file tutorial</A
743 NAME="AEN3384">14.2. <SPAN
746 >'s Internal Pages</H2
751 > proxies each requested
752 web page, it is easy for <SPAN
756 trap certain special URLs. In this way, we can talk directly to
761 configured, see how our rules are being applied, change these
762 rules and other configuration options, and even turn
766 > filtering off, all with
767 a web browser. </P
769 > The URLs listed below are the special ones that allow direct access
777 > must be running to access these. If
778 not, you will get a friendly error message. Internet access is not
790 NAME="AEN3399"><BLOCKQUOTE
795 HREF="http://config.privoxy.org/"
797 >http://config.privoxy.org/</A
802 > There is a shortcut: <A
807 doesn't provide a fall-back to a real page, in case the request is not
817 Show information about the current configuration, including viewing and
818 editing of actions files:
821 NAME="AEN3407"><BLOCKQUOTE
826 HREF="http://config.privoxy.org/show-status"
828 >http://config.privoxy.org/show-status</A
836 Show the source code version numbers:
839 NAME="AEN3412"><BLOCKQUOTE
844 HREF="http://config.privoxy.org/show-version"
846 >http://config.privoxy.org/show-version</A
854 Show the browser's request headers:
857 NAME="AEN3417"><BLOCKQUOTE
862 HREF="http://config.privoxy.org/show-request"
864 >http://config.privoxy.org/show-request</A
872 Show which actions apply to a URL and why:
875 NAME="AEN3422"><BLOCKQUOTE
880 HREF="http://config.privoxy.org/show-url-info"
882 >http://config.privoxy.org/show-url-info</A
890 Toggle Privoxy on or off. In this case, <SPAN
894 to run, but only as a pass-through proxy, with no actions taking place:
897 NAME="AEN3428"><BLOCKQUOTE
902 HREF="http://config.privoxy.org/toggle"
904 >http://config.privoxy.org/toggle</A
909 > Short cuts. Turn off, then on:
912 NAME="AEN3432"><BLOCKQUOTE
917 HREF="http://config.privoxy.org/toggle?set=disable"
919 >http://config.privoxy.org/toggle?set=disable</A
924 NAME="AEN3435"><BLOCKQUOTE
929 HREF="http://config.privoxy.org/toggle?set=enable"
931 >http://config.privoxy.org/toggle?set=enable</A
939 > These may be bookmarked for quick reference. See next. </P
945 NAME="BOOKMARKLETS">14.2.1. Bookmarklets</H3
947 > Below are some <SPAN
949 >"bookmarklets"</SPAN
950 > to allow you to easily access a
954 > version of some of <SPAN
958 special pages. They are designed for MS Internet Explorer, but should work
959 equally well in Netscape, Mozilla, and other browsers which support
960 JavaScript. They are designed to run directly from your bookmarks - not by
961 clicking the links below (although that should work for testing).</P
963 > To save them, right-click the link and choose <SPAN
965 >"Add to Favorites"</SPAN
969 >"Add Bookmark"</SPAN
970 > (Netscape). You will get a warning that
973 >"may not be safe"</SPAN
974 > - just click OK. Then you can run the
975 Bookmarklet directly from your favorites/bookmarks. For even faster access,
976 you can put them on the <SPAN
979 > bar (IE) or the <SPAN
983 > (Netscape), and run them with a single click. </P
991 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=enabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1000 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=disabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1002 >Privoxy - Disable</A
1009 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=toggle','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1011 >Privoxy - Toggle Privoxy</A
1012 > (Toggles between enabled and disabled)
1018 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y','ijbstatus','width=250,height=2,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1020 >Privoxy- View Status</A
1027 HREF="javascript:w=Math.floor(screen.width/2);h=Math.floor(screen.height*0.9);void(window.open('http://www.privoxy.org/actions/index.php?url='+escape(location.href),'Feedback','screenx='+w+',width='+w+',height='+h+',scrollbars=yes,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1029 >Privoxy - Submit Actions File Feedback</A
1036 HREF="javascript:void(window.open('http://config.privoxy.org/show-url-info?url='+escape(location.href),'Why').focus());"
1045 > Credit: The site which gave us the general idea for these bookmarklets is
1047 HREF="http://www.bookmarklets.com"
1049 >www.bookmarklets.com</A
1051 have more information about bookmarklets. </P
1059 NAME="CHAIN">14.3. Chain of Events</H2
1061 > Let's take a quick look at the basic sequence of events when a web page is
1062 requested by your browser and <SPAN
1072 > First, your web browser requests a web page. The browser knows to send
1073 the request to <SPAN
1076 >, which will in turn,
1077 relay the request to the remote web server after passing the following
1086 > traps any request for its own internal CGI
1087 pages (e.g http://p.p/) and sends the CGI page back to the browser.
1095 > checks to see if the URL
1097 HREF="actions-file.html#BLOCK"
1103 so, the URL is then blocked, and the remote web server will not be contacted.
1105 HREF="actions-file.html#HANDLE-AS-IMAGE"
1108 >"+handle-as-image"</SPAN
1111 is then checked and if it does not match, an
1115 > page is sent back. Otherwise, if it does match,
1116 an image is returned. The type of image depends on the setting of <A
1117 HREF="actions-file.html#SET-IMAGE-BLOCKER"
1120 >"+set-image-blocker"</SPAN
1123 (blank, checkerboard pattern, or an HTTP redirect to an image elsewhere).
1128 > Untrusted URLs are blocked. If URLs are being added to the
1132 > file, then that is done.
1137 > If the URL pattern matches the <A
1138 HREF="actions-file.html#FAST-REDIRECTS"
1141 >"+fast-redirects"</SPAN
1144 it is then processed. Unwanted parts of the requested URL are stripped.
1149 > Now the rest of the client browser's request headers are processed. If any
1150 of these match any of the relevant actions (e.g. <A
1151 HREF="actions-file.html#HIDE-USER-AGENT"
1154 >"+hide-user-agent"</SPAN
1157 etc.), headers are suppressed or forged as determined by these actions and
1163 > Now the web server starts sending its response back (i.e. typically a web page and related
1169 > First, the server headers are read and processed to determine, among other
1170 things, the MIME type (document type) and encoding. The headers are then
1171 filtered as deterimined by the
1173 HREF="actions-file.html#CRUNCH-INCOMING-COOKIES"
1176 >"+crunch-incoming-cookies"</SPAN
1180 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1183 >"+session-cookies-only"</SPAN
1187 HREF="actions-file.html#DOWNGRADE-HTTP-VERSION"
1190 >"+downgrade-http-version"</SPAN
1199 HREF="actions-file.html#KILL-POPUPS"
1202 >"+kill-popups"</SPAN
1205 action applies, and it is an HTML or JavaScript document, the popup-code in the
1206 response is filtered on-the-fly as it is received.
1212 HREF="actions-file.html#FILTER"
1219 HREF="actions-file.html#DEANIMATE-GIFS"
1222 >"+deanimate-gifs"</SPAN
1225 action applies (and the document type fits the action), the rest of the page is
1226 read into memory (up to a configurable limit). Then the filter rules (from
1230 >) are processed against the buffered
1231 content. Filters are applied in the order they are specified in the
1235 > file. Animated GIFs, if present, are
1236 reduced to either the first or last frame, depending on the action
1237 setting.The entire page, which is now filtered, is then sent by
1241 > back to your browser.
1245 HREF="actions-file.html#FILTER"
1252 HREF="actions-file.html#DEANIMATE-GIFS"
1255 >"+deanimate-gifs"</SPAN
1261 > passes the raw data through
1262 to the client browser as it becomes available.
1267 > As the browser receives the now (probably filtered) page content, it
1268 reads and then requests any URLs that may be embedded within the page
1269 source, e.g. ad images, stylesheets, JavaScript, other HTML documents (e.g.
1270 frames), sounds, etc. For each of these objects, the browser issues a new
1271 request. And each such request is in turn processed as above. Note that a
1272 complex web page may have many such embedded URLs.
1283 NAME="ACTIONSANAT">14.4. Anatomy of an Action</H2
1290 HREF="actions-file.html#ACTIONS"
1293 HREF="actions-file.html#FILTER"
1296 to any given URL can be complex, and not always so
1297 easy to understand what is happening. And sometimes we need to be able to
1308 doing. Especially, if something <SPAN
1312 is causing us a problem inadvertently. It can be a little daunting to look at
1313 the actions and filters files themselves, since they tend to be filled with
1315 HREF="appendix.html#REGEX"
1316 >regular expressions</A
1317 > whose consequences are not
1318 always so obvious. </P
1320 > One quick test to see if <SPAN
1323 > is causing a problem
1324 or not, is to disable it temporarily. This should be the first troubleshooting
1326 HREF="appendix.html#BOOKMARKLETS"
1327 >the Bookmarklets</A
1328 > section on a quick
1329 and easy way to do this (be sure to flush caches afterward!).</P
1336 HREF="http://config.privoxy.org/show-url-info"
1338 >http://config.privoxy.org/show-url-info</A
1340 page that can show us very specifically how <SPAN
1344 are being applied to any given URL. This is a big help for troubleshooting.</P
1346 > First, enter one URL (or partial URL) at the prompt, and then
1351 how the current configuration will handle it. This will not
1352 help with filtering effects (i.e. the <A
1353 HREF="actions-file.html#FILTER"
1362 > file since this is handled very
1363 differently and not so easy to trap! It also will not tell you about any other
1364 URLs that may be embedded within the URL you are testing. For instance, images
1365 such as ads are expressed as URLs within the raw page source of HTML pages. So
1366 you will only get info for the actual URL that is pasted into the prompt area
1367 -- not any sub-URLs. If you want to know about embedded URLs like ads, you
1368 will have to dig those out of the HTML source. Use your browser's <SPAN
1372 > option for this. Or right click on the ad, and grab the
1375 > Let's try an example, <A
1376 HREF="http://google.com"
1380 and look at it one section at a time:</P
1390 > Matches for http://google.com:
1392 In file: default.action <SPAN
1402 -crunch-outgoing-cookies
1403 -crunch-incoming-cookies
1404 +deanimate-gifs{last}
1405 -downgrade-http-version
1409 -filter{shockwave-flash}
1410 -filter{crude-parental}
1411 +filter{html-annoyances}
1412 +filter{js-annoyances}
1413 +filter{content-cookies}
1415 +filter{refresh-tags}
1417 +filter{banners-by-size}
1418 +hide-forwarded-for-headers
1419 +hide-from-header{block}
1420 +hide-referer{forge}
1425 +prevent-compression
1428 +session-cookies-only
1429 +set-image-blocker{pattern} }
1432 { -session-cookies-only }
1438 In file: user.action <SPAN
1445 (no matches in this file) </PRE
1451 > This tells us how we have defined our
1453 HREF="actions-file.html#ACTIONS"
1459 which ones match for our example, <SPAN
1462 >. The first listing
1463 is any matches for the <TT
1465 >standard.action</TT
1470 >. Then next is <SPAN
1477 > file. The large, multi-line listing,
1478 is how the actions are set to match for all URLs, i.e. our default settings.
1479 If you look at your <SPAN
1482 > file, this would be the section
1483 just below the <SPAN
1486 > section near the top. This will apply to
1487 all URLs as signified by the single forward slash at the end of the listing
1493 > But we can define additional actions that would be exceptions to these general
1494 rules, and then list specific URLs (or patterns) that these exceptions would
1495 apply to. Last match wins. Just below this then are two explicit matches for
1498 >".google.com"</SPAN
1499 >. The first is negating our previous cookie setting,
1501 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1504 >"+session-cookies-only"</SPAN
1507 (i.e. not persistent). So we will allow persistent cookies for google. The
1516 HREF="actions-file.html#FAST-REDIRECTS"
1519 >"+fast-redirects"</SPAN
1522 action, allowing this to take place unmolested. Note that there is a leading
1525 >".google.com"</SPAN
1526 >. This will match any hosts and
1527 sub-domains, in the google.com domain also, such as
1530 >"www.google.com"</SPAN
1531 >. So, apparently, we have these two actions
1532 defined somewhere in the lower part of our <TT
1539 > is referenced somewhere in these latter
1545 > file, we again have no hits.</P
1547 > And finally we pull it all together in the bottom section and summarize how
1551 > is applying all its <SPAN
1568 > Final results:
1572 -crunch-outgoing-cookies
1573 -crunch-incoming-cookies
1574 +deanimate-gifs{last}
1575 -downgrade-http-version
1579 -filter{shockwave-flash}
1580 -filter{crude-parental}
1581 +filter{html-annoyances}
1582 +filter{js-annoyances}
1583 +filter{content-cookies}
1585 +filter{refresh-tags}
1587 +filter{banners-by-size}
1588 +hide-forwarded-for-headers
1589 +hide-from-header{block}
1590 +hide-referer{forge}
1595 +prevent-compression
1598 -session-cookies-only
1599 +set-image-blocker{pattern} </PRE
1605 > Notice the only difference here to the previous listing, is to
1608 >"fast-redirects"</SPAN
1611 >"session-cookies-only"</SPAN
1614 > Now another example, <SPAN
1616 >"ad.doubleclick.net"</SPAN
1627 > { +block +handle-as-image }
1630 { +block +handle-as-image }
1633 { +block +handle-as-image }
1634 .doubleclick.net</PRE
1640 > We'll just show the interesting part here, the explicit matches. It is
1641 matched three different times. Each as an <SPAN
1643 >"+block +handle-as-image"</SPAN
1645 which is the expanded form of one of our aliases that had been defined as:
1648 >"+imageblock"</SPAN
1650 HREF="actions-file.html#ALIASES"
1656 the first section of the actions file and typically used to combine more
1657 than one action.)</P
1659 > Any one of these would have done the trick and blocked this as an unwanted
1660 image. This is unnecessarily redundant since the last case effectively
1661 would also cover the first. No point in taking chances with these guys
1662 though ;-) Note that if you want an ad or obnoxious
1663 URL to be invisible, it should be defined as <SPAN
1665 >"ad.doubleclick.net"</SPAN
1667 is done here -- as both a <A
1668 HREF="actions-file.html#BLOCK"
1682 HREF="actions-file.html#HANDLE-AS-IMAGE"
1685 >"+handle-as-image"</SPAN
1688 The custom alias <SPAN
1690 >"+imageblock"</SPAN
1691 > just simplifies the process and make
1692 it more readable.</P
1694 > One last example. Let's try <SPAN
1696 >"http://www.rhapsodyk.net/adsl/HOWTO/"</SPAN
1698 This one is giving us problems. We are getting a blank page. Hmmm ...</P
1708 > Matches for http://www.rhapsodyk.net/adsl/HOWTO/:
1710 In file: default.action <SPAN
1720 -crunch-incoming-cookies
1721 -crunch-outgoing-cookies
1723 -downgrade-http-version
1725 +filter{html-annoyances}
1726 +filter{js-annoyances}
1727 +filter{kill-popups}
1730 +filter{banners-by-size}
1733 +hide-forwarded-for-headers
1734 +hide-from-header{block}
1735 +hide-referer{forge}
1739 +prevent-compression
1742 +session-cookies-only
1743 +set-image-blocker{blank} }
1746 { +block +handle-as-image }
1760 we did not want this at all! Now we see why we get the blank page. We could
1761 now add a new action below this that explicitly does <SPAN
1775 various ways to handle such exceptions. Example:</P
1792 > Now the page displays ;-) Be sure to flush your browser's caches when
1793 making such changes. Or, try using <TT
1798 > But now what about a situation where we get no explicit matches like
1809 > { +block +handle-as-image }
1816 > That actually was very telling and pointed us quickly to where the problem
1817 was. If you don't get this kind of match, then it means one of the default
1818 rules in the first section is causing the problem. This would require some
1819 guesswork, and maybe a little trial and error to isolate the offending rule.
1820 One likely cause would be one of the <SPAN
1824 adding the URL for the site to one of aliases that turn off <SPAN
1839 .worldpay.com # for quietpc.com
1857 >"{ -filter -session-cookies-only }"</SPAN
1859 Or you could do your own exception to negate filtering: </P
1876 > This would probably be most appropriately put in <TT
1880 for local site exceptions.</P
1885 > is an alias that disables most actions. This can be
1886 used as a last resort for problem sites. Remember to flush caches! If this
1887 still does not work, you will have to go through the remaining actions one by
1888 one to find which one(s) is causing the problem.</P
1896 SUMMARY="Footer navigation table"