| Privoxy User Manual | Privoxy 3.0.4 User Manual||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Prev | 15. Appendix14. Appendix15.1. Regular Expressions14.1. Regular ExpressionsPrivoxy can use "regular expressions" - in various config files. Assuming support for uses Perl-style "pcre" (Perl - Compatible Regular Expressions) is compiled in, which is the default. Such - configuration directives do not require regular expressions, but they can be - used to increase flexibility by matching a pattern with wild-cards against - URLs. "regular + expressions" in its actions + files and filter file, + through the PCRE and + PCRS libraries.If you are reading this, you probably don't understand what "regular expressions" are, or what they can do. So this will be a very brief - introduction only. A full explanation would require a book ;-) book ;-)Regular expressions provide a language to describe patterns that can be + run against strings of characters (letter, numbers, etc), to see if they + match the string or not. The patterns are themselves (sometimes complex) + strings of literal characters, combined with wild-cards, and other special + characters, called meta-characters. The "Regular expressions" is a way of matching one character - expression against another to see if it matches or not. One of the +>"meta-characters" have + special meanings and are used to build complex patterns to be matched against. + Perl Compatible Regular Expressions are an especially convenient "expressions" is a literal string of readable characters - (letter, numbers, etc), and the other is a complex string of literal - characters combined with wild-cards, and other special characters, called - meta-characters. The "meta-characters" have special meanings and - are used to build the complex pattern to be matched against. Perl Compatible - Regular Expressions is an enhanced form of the regular expression language - with backward compatibility. "dialect" of the regular expression language. To make a simple analogy, we do something similar when we use wild-card
characters when listing files with the These are just some of the ones you are likely to use when matching URLs with
/.*/banners/.* - A simple example
that uses the common combination of in the path
somewhere. A now something a little more complex: /.*/adv((er)?ts?|ertis(ing|ements?))?/ -
We have several literal forward slashes again (".*", so we are matching against any conceivable sub-path, just so
- it matches our expression. The only true literal that must
match our pattern is adv"ing"
- OR "ements?", which would then match
either spelling. /.*/advert[0-9]+\.(gif|jpe?g) - Again
another path statement with forward slashes. Anything in the square brackets
"[]""[ ]" can be matched. This is using "0-9" is not
in the expression anywhere). s/microsoft(?!.com)/MicroSuck/i - This is
- a substitution. "MicroSuck" will replace any occurrence of
- "microsoft". The "i" at the end of the expression
- means ignore case. The "(?!.com)" means
- the match should fail if "microsoft" is followed by
- ".com". In other words, this acts like a "NOT"
- modifier. In case this is a hyperlink, we don't want to break it ;-). We are barely scratching the surface of regular expressions here so that you
can understand the default More reading on Perl Compatible Regular expressions:
http://www.perldoc.com/perl5.6/pod/perlre.htmlhttp://perldoc.perl.org/perlre.html For information on regular expression based substitutions and their applications
+ in filters, please see the filter file tutorial
+ in this manual. Since Alternately, this may be reached at There is a shortcut: http://p.p/, but this
- variation may not work as reliably as the above in some configurations.
+> (But it
+ doesn't provide a fall-back to a real page, in case the request is not
+ sent through Privoxy)
Below are some Privoxy - Submit Filter FeedbackPrivoxy - Why?
Credit: The site which gave me the general idea for these bookmarklets is
+> Credit: The site which gave us the general idea for these bookmarklets is
www.bookmarklets.com. They
@@ -1075,8 +1073,8 @@ CLASS="SECT2"
CLASS="SECT2"
>15.3. Chain of Events14.3. Chain of Events Let's take a quick look at the basic sequence of events when a web page is
requested by your browser and Privoxy traps any request for its own internal CGI
- pages (e.g http://p.p/) and sends the CGI page back to the browser.
+ pages (e.g http://p.p/) and sends the CGI page back to the browser.
If the URL pattern matches the "+fast-redirects" Now the rest of the client browser's request headers are processed. If any
of these match any of the relevant actions (e.g. "+hide-user-agent" First, the server headers are read and processed to determine, among other
things, the MIME type (document type) and encoding. The headers are then
- filtered as deterimed by the
+ filtered as determined by the
"+prevent-setting-cookies""+crunch-incoming-cookies",
"+session-cookies-only",
and "+downgrade-http-version" If the "+kill-popups" If a "+filter"
or "+deanimate-gifs"default.filter) are processed against the buffered
- content. Filters are applied in the order they are specified in the
- default.filter file. Animated GIFs, if present, are
- reduced to either the first or last frame, depending on the action
+> and any other filter files) are
+ processed against the buffered content. Filters are applied in the order
+ they are specified in one of the filter files. Animated GIFs, if present,
+ are reduced to either the first or last frame, depending on the action
setting.The entire page, which is now filtered, is then sent by
If neither "+filter"
or "+deanimate-gifs" As the browser receives the now (probably filtered) page content, it
+> As the browser receives the now (possibly filtered) page content, it
reads and then requests any URLs that may be embedded within the page
source, e.g. ad images, stylesheets, JavaScript, other HTML documents (e.g.
frames), sounds, etc. For each of these objects, the browser issues a new
@@ -1314,8 +1300,8 @@ CLASS="SECT2"
CLASS="SECT2"
>15.4. Anatomy of an Action14.4. Anatomy of an Action The way applies
"actions"
- and actions and "filters"filters
to any given URL can be complex, and not always so
easy to understand what is happening. And sometimes we need to be able to
- see just what Privoxy is doing
is causing us a problem inadvertently. It can be a little daunting to look at
the actions and filters files themselves, since they tend to be filled with
- "regular expressions" whose consequences are not always
- so obvious. One quick test to see if the Bookmarklets section on a quick
- and easy way to do this (be sure to flush caches afterward!). "+filter" action) from
- the default.filter file since this is handled very
+ one of the filter files since this is handled very
differently and not so easy to trap! It also will not tell you about any other
URLs that may be embedded within the URL you are testing. For instance, images
such as ads are expressed as URLs within the raw page source of HTML pages. So
@@ -1419,7 +1396,8 @@ HREF="http://google.com"
TARGET="_top"
>google.com,
- and look at it one section at a time: This tells us how we have defined our
+> This is telling us how we have defined our
"actions", and
- which ones match for our example, "google.com". The first listing
+>.
+ Displayed is all the actions that are available to us. Remember,
+ the + sign denotes "on". -
+ denotes "off". So some are "on" here, but many
+ are "off". Each example we try may provide a slightly different
+ end result, depending on our configuration directives. The first listing
is any matches for the standard.action. The first is negating our previous cookie setting,
which was for "+session-cookies-only"
- (i.e. not persistent). So we will allow persistent cookies for google. The
- second turns off any
"+fast-redirects" Then, for our user.action file, we again have no hits. And finally we pull it all together in the bottom section and summarize how
|
Now another example, "+imageblock". ("Aliases"
is done here -- as both a "+block"
- and an
"+handle-as-image" One last example. Let's try "http://www.rhapsodyk.net/adsl/HOWTO/""http://www.example.net/adsl/HOWTO/".
- This one is giving us problems. We are getting a blank page. Hmmm... This would probably be most appropriately put in This would turn off all filtering for that site. This would probably be most
+ appropriately put in user.action,
- for local site exceptions. Images that are inexplicably being blocked, may well be hitting the
+ "+filter{banners-by-size}" rule, which assumes
+ that images of certain sizes are ad banners (works well most of the time
+ since these tend to be standardized).
Matches for http://www.rhapsodyk.net/adsl/HOWTO/:
+>
Matches for http://www.example.net/adsl/HOWTO/:
- { -add-header -block +deanimate-gifs -downgrade-http-version +fast-redirects
- +filter{html-annoyances} +filter{js-annoyances} +filter{kill-popups}
- +filter{webbugs} +filter{nimda} +filter{banners-by-size} +filter{hal}
- +filter{fun} +hide-forwarded-for-headers +hide-from-header{block}
- +hide-referer{forge} -hide-user-agent -handle-as-image +set-image-blocker{blank}
- +prevent-compression +session-cookies-only -prevent-setting-cookies
- -prevent-reading-cookies +kill-popups -send-vanilla-wafer -send-wafer }
+ In file: default.action [ View ] [ Edit ]
+
+ {-add-header
+ -block
+ -content-type-overwrite
+ -crunch-client-header
+ -crunch-if-none-match
+ -crunch-incoming-cookies
+ -crunch-outgoing-cookies
+ -crunch-server-header
+ +deanimate-gifs
+ -downgrade-http-version
+ +fast-redirects{check-decoded-url}
+ +filter{html-annoyances}
+ +filter{js-annoyances}
+ +filter{kill-popups}
+ +filter{webbugs}
+ +filter{nimda}
+ +filter{banners-by-size}
+ +filter{hal}
+ +filter{fun}
+ -filter-client-headers
+ -filter-server-headers
+ -force-text-mode
+ -handle-as-empty-document
+ -handle-as-image
+ -hide-accept-language
+ -hide-content-disposition
+ +hide-forwarded-for-headers
+ +hide-from-header{block}
+ +hide-referer{forge}
+ -hide-user-agent
+ -inspect-jpegs
+ +kill-popups
+ -overwrite-last-modified
+ +prevent-compression
+ -redirect
+ -send-vanilla-wafer
+ -send-wafer
+ +session-cookies-only
+ +set-image-blocker{blank}
+ -treat-forbidden-connects-like-blocks }
/
{ +block +handle-as-image }
@@ -1717,20 +1861,24 @@ CLASS="QUOTE"
> is matching "/ads"! But
- we did not want this at all! Now we see why we get the blank page. We could
- now add a new action below this that explicitly does in our
+ configuration! But we did not want this at all! Now we see why we get the
+ blank page. We could now add a new action below this that explicitly
+ not
- block (un blocks ("{-block}") paths with ) paths with
+ "adsl". There are
- various ways to handle such exceptions. Example: in them (remember, last match in the configuration wins).
+ There are various ways to handle such exceptions. Example:"{+filter}" actions. Try
- adding the URL for the site to one of aliases that turn off actions. These
+ tend to be harder to troubleshoot. Try adding the URL for the site to one of
+ aliases that turn off "+filter":