7 CONTENT="Modular DocBook HTML Stylesheet Version 1.64
10 TITLE="Privoxy User Manual"
11 HREF="index.html"><LINK
14 HREF="seealso.html"><LINK
17 HREF="../p_doc.css"></HEAD
36 >Privoxy User Manual</TH
76 >14.1. Regular Expressions</A
84 >"regular expressions"</SPAN
86 in various config files. Assuming support for <SPAN
90 Compatible Regular Expressions) is compiled in, which is the default. Such
91 configuration directives do not require regular expressions, but they can be
92 used to increase flexibility by matching a pattern with wild-cards against
95 > If you are reading this, you probably don't understand what <SPAN
99 > are, or what they can do. So this will be a very brief
100 introduction only. A full explanation would require a book ;-)</P
104 >"Regular expressions"</SPAN
105 > is a way of matching one character
106 expression against another to see if it matches or not. One of the
110 > is a literal string of readable characters
111 (letter, numbers, etc), and the other is a complex string of literal
112 characters combined with wild-cards, and other special characters, called
113 meta-characters. The <SPAN
115 >"meta-characters"</SPAN
116 > have special meanings and
117 are used to build the complex pattern to be matched against. Perl Compatible
118 Regular Expressions is an enhanced form of the regular expression language
119 with backward compatibility.</P
121 > To make a simple analogy, we do something similar when we use wild-card
122 characters when listing files with the <B
129 > matches all filenames. The <SPAN
133 character here is the asterisk which matches any and all characters. We can be
134 more specific and use <TT
137 > to match just individual
140 >"dir file?.text"</SPAN
148 >, etc. We are pattern
149 matching, using a similar technique to <SPAN
151 >"regular expressions"</SPAN
154 > Regular expressions do essentially the same thing, but are much, much more
155 powerful. There are many more <SPAN
157 >"special characters"</SPAN
159 building complex patterns however. Let's look at a few of the common ones,
160 and then some examples:</P
172 > - Matches any single character, e.g. <SPAN
207 > - The preceding character or expression is matched ZERO or ONE
227 > - The preceding character or expression is matched ONE or MORE
247 > - The preceding character or expression is matched ZERO or MORE
270 > character denotes that
271 the following character should be taken literally. This is used where one of the
272 special characters (e.g. <SPAN
275 >) needs to be taken literally and
276 not as a special meta-character. Example: <SPAN
278 >"example\.com"</SPAN
280 sure the period is recognized only as a period (and not expanded to its
281 meta-character meaning of any single character).
300 > - Characters enclosed in brackets will be matched if
301 any of the enclosed characters are encountered. For instance, <SPAN
305 matches any numeric digit (zero through nine). As an example, we can combine
309 > to match any digit one of more times: <SPAN
331 > - parentheses are used to group a sub-expression,
332 or multiple sub-expressions.
354 > character works like an
358 > conditional statement. A match is successful if the
359 sub-expression on either side of <SPAN
362 > matches. As an example:
365 >"/(this|that) example/"</SPAN
366 > uses grouping and the bar character
367 and would match either <SPAN
369 >"this example"</SPAN
392 >s/string1/string2/g</I
393 > - This is used to rewrite strings of text.
397 > is replaced by <SPAN
401 example. There must of course be a match on <SPAN
413 > These are just some of the ones you are likely to use when matching URLs with
417 >, and is a long way from a definitive
418 list. This is enough to get us started with a few simple examples which may
419 be more illuminating:</P
428 that uses the common combination of <SPAN
435 denote any character, zero or more times. In other words, any string at all.
436 So we start with a literal forward slash, then our regular expression pattern
440 >) another literal forward slash, the string
444 >, another forward slash, and lastly another
449 a directory path here. This will match any file with the path that has a
450 directory named <SPAN
457 any characters, and this could conceivably be more forward slashes, so it
458 might expand into a much longer looking path. For example, this could match:
461 >"/eye/hate/spammers/banners/annoy_me_please.gif"</SPAN
465 >"/banners/annoying.html"</SPAN
466 >, or almost an infinite number of other
467 possible combinations, just so it has <SPAN
473 > A now something a little more complex:</P
479 >/.*/adv((er)?ts?|ertis(ing|ements?))?/</TT
482 We have several literal forward slashes again (<SPAN
486 building another expression that is a file path statement. We have another
490 >, so we are matching against any conceivable sub-path, just so
491 it matches our expression. The only true literal that <I
495 > our pattern is <SPAN
499 the forward slashes. What comes after the <SPAN
503 interesting part. </P
508 > means the preceding expression (either a
509 literal character or anything grouped with <SPAN
513 can exist or not, since this means either zero or one match. So
516 >"((er)?ts?|ertis(ing|ements?))"</SPAN
517 > is optional, as are the
518 individual sub-expressions: <SPAN
524 >"(ing|ements?)"</SPAN
535 >. We have two of those. For instance,
538 >"(ing|ements?)"</SPAN
539 >, can expand to match either <SPAN
549 >. What is being done here, is an
550 attempt at matching as many variations of <SPAN
552 >"advertisement"</SPAN
554 similar, as possible. So this would expand to match just <SPAN
570 >"advertisement"</SPAN
574 >"advertisements"</SPAN
575 >. You get the idea. But it would not match
578 >"advertizements"</SPAN
582 >). We could fix that by
583 changing our regular expression to:
586 >"/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/"</SPAN
587 >, which would then match
594 >/.*/advert[0-9]+\.(gif|jpe?g)</TT
597 another path statement with forward slashes. Anything in the square brackets
601 > can be matched. This is using <SPAN
605 shorthand expression to mean any digit one through nine. It is the same as
609 >. So any digit matches. The <SPAN
613 means one or more of the preceding expression must be included. The preceding
614 expression here is what is in the square brackets -- in this case, any digit
615 one through nine. Then, at the end, we have a grouping: <SPAN
619 This includes a <SPAN
622 >, so this needs to match the expression on
623 either side of that bar character also. A simple <SPAN
626 > on one side, and the other
627 side will in turn match either <SPAN
637 > means the letter <SPAN
641 can be matched once or not at all. So we are building an expression here to
642 match image GIF or JPEG type image file. It must include the literal
646 >, then one or more digits, and a <SPAN
650 (which is now a literal, and not a special character, since it is escaped
654 >), and lastly either <SPAN
664 >. Some possible matches would
667 >"//advert1.jpg"</SPAN
671 >"/nasty/ads/advert1234.gif"</SPAN
675 >"/banners/from/hell/advert99.jpg"</SPAN
676 >. It would not match
680 > (no leading slash), or
683 >"/adverts232.jpg"</SPAN
684 > (the expression does not include an
690 >"/advert1.jsp"</SPAN
695 in the expression anywhere).</P
701 >s/microsoft(?!.com)/MicroSuck/i</TT
704 a substitution. <SPAN
707 > will replace any occurrence of
714 > at the end of the expression
715 means ignore case. The <SPAN
719 the match should fail if <SPAN
726 >. In other words, this acts like a <SPAN
730 modifier. In case this is a hyperlink, we don't want to break it ;-).</P
732 > We are barely scratching the surface of regular expressions here so that you
733 can understand the default <SPAN
737 configuration files, and maybe use this knowledge to customize your own
738 installation. There is much, much more that can be done with regular
739 expressions. Now that you know enough to get started, you can learn more on
742 > More reading on Perl Compatible Regular expressions:
744 HREF="http://www.perldoc.com/perl5.6/pod/perlre.html"
746 >http://www.perldoc.com/perl5.6/pod/perlre.html</A
758 >'s Internal Pages</A
764 > proxies each requested
765 web page, it is easy for <SPAN
769 trap certain special URLs. In this way, we can talk directly to
774 configured, see how our rules are being applied, change these
775 rules and other configuration options, and even turn
779 > filtering off, all with
780 a web browser. </P
782 > The URLs listed below are the special ones that allow direct access
790 > must be running to access these. If
791 not, you will get a friendly error message. Internet access is not
810 HREF="http://config.privoxy.org/"
812 >http://config.privoxy.org/</A
817 > Alternately, this may be reached at <A
822 variation may not work as reliably as the above in some configurations.
828 Show information about the current configuration, including viewing and
829 editing of actions files:
839 HREF="http://config.privoxy.org/show-status"
841 >http://config.privoxy.org/show-status</A
849 Show the source code version numbers:
859 HREF="http://config.privoxy.org/show-version"
861 >http://config.privoxy.org/show-version</A
869 Show the browser's request headers:
879 HREF="http://config.privoxy.org/show-request"
881 >http://config.privoxy.org/show-request</A
889 Show which actions apply to a URL and why:
899 HREF="http://config.privoxy.org/show-url-info"
901 >http://config.privoxy.org/show-url-info</A
909 Toggle Privoxy on or off. In this case, <SPAN
913 to run, but only as a pass-through proxy, with no actions taking place:
923 HREF="http://config.privoxy.org/toggle"
925 >http://config.privoxy.org/toggle</A
930 > Short cuts. Turn off, then on:
940 HREF="http://config.privoxy.org/toggle?set=disable"
942 >http://config.privoxy.org/toggle?set=disable</A
954 HREF="http://config.privoxy.org/toggle?set=enable"
956 >http://config.privoxy.org/toggle?set=enable</A
964 > These may be bookmarked for quick reference. See next. </P
971 >14.2.1. Bookmarklets</A
974 > Below are some <SPAN
976 >"bookmarklets"</SPAN
977 > to allow you to easily access a
981 > version of some of <SPAN
985 special pages. They are designed for MS Internet Explorer, but should work
986 equally well in Netscape, Mozilla, and other browsers which support
987 JavaScript. They are designed to run directly from your bookmarks - not by
988 clicking the links below (although that should work for testing).</P
990 > To save them, right-click the link and choose <SPAN
992 >"Add to Favorites"</SPAN
996 >"Add Bookmark"</SPAN
997 > (Netscape). You will get a warning that
1000 >"may not be safe"</SPAN
1001 > - just click OK. Then you can run the
1002 Bookmarklet directly from your favorites/bookmarks. For even faster access,
1003 you can put them on the <SPAN
1006 > bar (IE) or the <SPAN
1010 > (Netscape), and run them with a single click. </P
1018 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=enabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1020 >Privoxy - Enable</A
1027 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=disabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1029 >Privoxy - Disable</A
1036 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=toggle','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1038 >Privoxy - Toggle Privoxy</A
1039 > (Toggles between enabled and disabled)
1045 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y','ijbstatus','width=250,height=2,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1047 >Privoxy- View Status</A
1054 HREF="javascript:w=Math.floor(screen.width/2);h=Math.floor(screen.height*0.9);void(window.open('http://www.privoxy.org/actions','Feedback','screenx='+w+',width='+w+',height='+h+',scrollbars=yes,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1056 >Privoxy - Submit Filter Feedback</A
1063 > Credit: The site which gave me the general idea for these bookmarklets is
1065 HREF="http://www.bookmarklets.com"
1067 >www.bookmarklets.com</A
1069 have more information about bookmarklets. </P
1078 >14.3. Chain of Events</A
1081 > Let's take a quick look at the basic sequence of events when a web page is
1082 requested by your browser and <SPAN
1092 > First, your web browser requests a web page. The browser knows to send
1093 the request to <SPAN
1096 >, which will in turn,
1097 relay the request to the remote web server after passing the following
1106 > traps any request for its own internal CGI
1107 pages (e.g http://p.p/) and sends the CGI page back to the browser.
1115 > checks to see if the URL
1117 HREF="actions-file.html#BLOCK"
1124 so, the URL is then blocked, and the remote web server will not be contacted.
1126 HREF="actions-file.html#HANDLE-AS-IMAGE"
1130 >"+handle-as-image"</SPAN
1133 is then checked and if it does not match, an
1137 > page is sent back. Otherwise, if it does match,
1138 an image is returned. The type of image depends on the setting of <A
1139 HREF="actions-file.html#SET-IMAGE-BLOCKER"
1143 >"+set-image-blocker"</SPAN
1146 (blank, checkerboard pattern, or an HTTP redirect to an image elsewhere).
1151 > Untrusted URLs are blocked. If URLs are being added to the
1155 > file, then that is done.
1160 > If the URL pattern matches the <A
1161 HREF="actions-file.html#FAST-REDIRECTS"
1165 >"+fast-redirects"</SPAN
1168 it is then processed. Unwanted parts of the requested URL are stripped.
1173 > Now the rest of the client browser's request headers are processed. If any
1174 of these match any of the relevant actions (e.g. <A
1175 HREF="actions-file.html#HIDE-USER-AGENT"
1179 >"+hide-user-agent"</SPAN
1182 etc.), headers are suppressed or forged as determined by these actions and
1188 > Now the web server starts sending its response back (i.e. typically a web page and related
1194 > First, the server headers are read and processed to determine, among other
1195 things, the MIME type (document type) and encoding. The headers are then
1196 filtered as deterimed by the
1198 HREF="actions-file.html#PREVENT-SETTING-COOKIES"
1202 >"+prevent-setting-cookies"</SPAN
1206 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1210 >"+session-cookies-only"</SPAN
1214 HREF="actions-file.html#DOWNGRADE-HTTP-VERSION"
1218 >"+downgrade-http-version"</SPAN
1227 HREF="actions-file.html#KILL-POPUPS"
1231 >"+kill-popups"</SPAN
1234 action applies, and it is an HTML or JavaScript document, the popup-code in the
1235 response is filtered on-the-fly as it is received.
1241 HREF="actions-file.html#FILTER"
1249 HREF="actions-file.html#DEANIMATE-GIFS"
1253 >"+deanimate-gifs"</SPAN
1256 action applies (and the document type fits the action), the rest of the page is
1257 read into memory (up to a configurable limit). Then the filter rules (from
1261 >) are processed against the buffered
1262 content. Filters are applied in the order they are specified in the
1266 > file. Animated GIFs, if present, are
1267 reduced to either the first or last frame, depending on the action
1268 setting.The entire page, which is now filtered, is then sent by
1272 > back to your browser.
1276 HREF="actions-file.html#FILTER"
1284 HREF="actions-file.html#DEANIMATE-GIFS"
1288 >"+deanimate-gifs"</SPAN
1294 > passes the raw data through
1295 to the client browser as it becomes available.
1300 > As the browser receives the now (probably filtered) page content, it
1301 reads and then requests any URLs that may be embedded within the page
1302 source, e.g. ad images, stylesheets, JavaScript, other HTML documents (e.g.
1303 frames), sounds, etc. For each of these objects, the browser issues a new
1304 request. And each such request is in turn processed as above. Note that a
1305 complex web page may have many such embedded URLs.
1317 >14.4. Anatomy of an Action</A
1325 HREF="actions-file.html#ACTIONS"
1333 HREF="actions-file.html#FILTER"
1340 to any given URL can be complex, and not always so
1341 easy to understand what is happening. And sometimes we need to be able to
1349 doing. Especially, if something <SPAN
1353 is causing us a problem inadvertently. It can be a little daunting to look at
1354 the actions and filters files themselves, since they tend to be filled with
1357 >"regular expressions"</SPAN
1358 > whose consequences are not always
1361 > One quick test to see if <SPAN
1364 > is causing a problem
1365 or not, is to disable it temporarily. This should be the first troubleshooting
1367 HREF="appendix.html#BOOKMARKLETS"
1368 >the Bookmarklets</A
1369 > section on a quick
1370 and easy way to do this (be sure to flush caches afterward!).</P
1377 HREF="http://config.privoxy.org/show-url-info"
1379 >http://config.privoxy.org/show-url-info</A
1381 page that can show us very specifically how <SPAN
1385 are being applied to any given URL. This is a big help for troubleshooting.</P
1387 > First, enter one URL (or partial URL) at the prompt, and then
1392 how the current configuration will handle it. This will not
1393 help with filtering effects (i.e. the <A
1394 HREF="actions-file.html#FILTER"
1404 > file since this is handled very
1405 differently and not so easy to trap! It also will not tell you about any other
1406 URLs that may be embedded within the URL you are testing. For instance, images
1407 such as ads are expressed as URLs within the raw page source of HTML pages. So
1408 you will only get info for the actual URL that is pasted into the prompt area
1409 -- not any sub-URLs. If you want to know about embedded URLs like ads, you
1410 will have to dig those out of the HTML source. Use your browser's <SPAN
1414 > option for this. Or right click on the ad, and grab the
1417 > Let's try an example, <A
1418 HREF="http://google.com"
1422 and look at it one section at a time:</P
1432 > Matches for http://google.com:
1434 --- File standard ---
1435 (no matches in this file)
1437 --- File default ---
1439 { -add-header -block +deanimate-gifs{last} -downgrade-http-version +fast-redirects
1440 -filter{popups} -filter{fun} -filter{shockwave-flash} -filter{crude-parental}
1441 +filter{html-annoyances} +filter{js-annoyances} +filter{content-cookies}
1442 +filter{webbugs} +filter{refresh-tags} +filter{nimda} +filter{banners-by-size}
1443 +hide-forwarded-for-headers +hide-from-header{block} +hide-referer{forge}
1444 -hide-user-agent -handle-as-image +set-image-blocker{pattern} -limit-connect
1445 +prevent-compression +session-cookies-only -prevent-reading-cookies
1446 -prevent-setting-cookies -kill-popups -send-vanilla-wafer -send-wafer }
1449 { -session-cookies-only }
1456 (no matches in this file) </PRE
1462 > This tells us how we have defined our
1464 HREF="actions-file.html#ACTIONS"
1471 which ones match for our example, <SPAN
1474 >. The first listing
1475 is any matches for the <TT
1477 >standard.action</TT
1482 >. Then next is <SPAN
1489 > file. The large, multi-line listing,
1490 is how the actions are set to match for all URLs, i.e. our default settings.
1491 If you look at your <SPAN
1494 > file, this would be the section
1495 just below the <SPAN
1498 > section near the top. This will apply to
1499 all URLs as signified by the single forward slash at the end of the listing
1505 > But we can define additional actions that would be exceptions to these general
1506 rules, and then list specific URLs (or patterns) that these exceptions would
1507 apply to. Last match wins. Just below this then are two explicit matches for
1510 >".google.com"</SPAN
1511 >. The first is negating our previous cookie setting,
1513 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1517 >"+session-cookies-only"</SPAN
1520 (i.e. not persistent). So we will allow persistent cookies for google. The
1526 HREF="actions-file.html#FAST-REDIRECTS"
1530 >"+fast-redirects"</SPAN
1533 action, allowing this to take place unmolested. Note that there is a leading
1536 >".google.com"</SPAN
1537 >. This will match any hosts and
1538 sub-domains, in the google.com domain also, such as
1541 >"www.google.com"</SPAN
1542 >. So, apparently, we have these two actions
1543 defined somewhere in the lower part of our <TT
1550 > is referenced somewhere in these latter
1556 > file, we again have no hits.</P
1558 > And finally we pull it all together in the bottom section and summarize how
1562 > is applying all its <SPAN
1579 > Final results:
1580 -add-header -block +deanimate-gifs{last} -downgrade-http-version -fast-redirects
1581 -filter{popups} -filter{fun} -filter{shockwave-flash} -filter{crude-parental}
1582 +filter{html-annoyances} +filter{js-annoyances} +filter{content-cookies}
1583 +filter{webbugs} +filter{refresh-tags} +filter{nimda} +filter{banners-by-size}
1584 +hide-forwarded-for-headers +hide-from-header{block} +hide-referer{forge}
1585 -hide-user-agent -handle-as-image +set-image-blocker{pattern} -limit-connect
1586 +prevent-compression -session-cookies-only -prevent-reading-cookies
1587 -prevent-setting-cookies -kill-popups -send-vanilla-wafer -send-wafer</PRE
1593 > Notice the only difference here to the previous listing, is to
1596 >"fast-redirects"</SPAN
1599 >"session-cookies-only"</SPAN
1602 > Now another example, <SPAN
1604 >"ad.doubleclick.net"</SPAN
1615 > { +block +handle-as-image }
1618 { +block +handle-as-image }
1621 { +block +handle-as-image }
1622 .doubleclick.net</PRE
1628 > We'll just show the interesting part here, the explicit matches. It is
1629 matched three different times. Each as an <SPAN
1631 >"+block +handle-as-image"</SPAN
1633 which is the expanded form of one of our aliases that had been defined as:
1636 >"+imageblock"</SPAN
1638 HREF="actions-file.html#ALIASES"
1645 the first section of the actions file and typically used to combine more
1646 than one action.)</P
1648 > Any one of these would have done the trick and blocked this as an unwanted
1649 image. This is unnecessarily redundant since the last case effectively
1650 would also cover the first. No point in taking chances with these guys
1651 though ;-) Note that if you want an ad or obnoxious
1652 URL to be invisible, it should be defined as <SPAN
1654 >"ad.doubleclick.net"</SPAN
1656 is done here -- as both a <A
1657 HREF="actions-file.html#BLOCK"
1669 HREF="actions-file.html#HANDLE-AS-IMAGE"
1673 >"+handle-as-image"</SPAN
1676 The custom alias <SPAN
1678 >"+imageblock"</SPAN
1679 > just simplifies the process and make
1680 it more readable.</P
1682 > One last example. Let's try <SPAN
1684 >"http://www.rhapsodyk.net/adsl/HOWTO/"</SPAN
1686 This one is giving us problems. We are getting a blank page. Hmmm...</P
1696 > Matches for http://www.rhapsodyk.net/adsl/HOWTO/:
1698 { -add-header -block +deanimate-gifs -downgrade-http-version +fast-redirects
1699 +filter{html-annoyances} +filter{js-annoyances} +filter{kill-popups}
1700 +filter{webbugs} +filter{nimda} +filter{banners-by-size} +filter{hal}
1701 +filter{fun} +hide-forwarded-for-headers +hide-from-header{block}
1702 +hide-referer{forge} -hide-user-agent -handle-as-image +set-image-blocker{blank}
1703 +prevent-compression +session-cookies-only -prevent-setting-cookies
1704 -prevent-reading-cookies +kill-popups -send-vanilla-wafer -send-wafer }
1707 { +block +handle-as-image }
1721 we did not want this at all! Now we see why we get the blank page. We could
1722 now add a new action below this that explicitly does <I
1733 various ways to handle such exceptions. Example:</P
1750 > Now the page displays ;-) Be sure to flush your browser's caches when
1751 making such changes. Or, try using <TT
1756 > But now what about a situation where we get no explicit matches like
1767 > { +block +handle-as-image }
1774 > That actually was very telling and pointed us quickly to where the problem
1775 was. If you don't get this kind of match, then it means one of the default
1776 rules in the first section is causing the problem. This would require some
1777 guesswork, and maybe a little trial and error to isolate the offending rule.
1778 One likely cause would be one of the <SPAN
1782 adding the URL for the site to one of aliases that turn off <SPAN
1797 .worldpay.com # for quietpc.com
1815 >"{ -filter -session-cookies-only }"</SPAN
1817 Or you could do your own exception to negate filtering: </P
1834 > This would probably be most appropriately put in <TT
1838 for local site exceptions.</P
1843 > is an alias that disables most actions. This can be
1844 used as a last resort for problem sites. Remember to flush caches! If this
1845 still does not work, you will have to go through the remaining actions one by
1846 one to find which one(s) is causing the problem.</P