| Privoxy 3.0.25 User Manual | +Privoxy 3.0.27 User Manual | ||||
|---|---|---|---|---|---|
| Prev | -- | ||||
Privoxy uses Perl-style "regular expressions" in its actions files and filter file, through the PCRE and PCRS libraries.
-If you are reading this, you probably don't understand what "regular expressions" are, or what they can do. So this will be a very brief introduction only. A full explanation would require a book ;-)
-Regular expressions provide a language to describe patterns that can be run against strings of characters (letter, numbers, etc), to see if they match the string or not. The patterns are themselves (sometimes @@ -63,7 +53,6 @@ build complex patterns to be matched against. Perl Compatible Regular Expressions are an especially convenient "dialect" of the regular expression language.
-To make a simple analogy, we do something similar when we use wild-card characters when listing files with the dir command in DOS. *.* matches @@ -75,12 +64,10 @@ "QUOTE">"file2.txt", etc. We are pattern matching, using a similar technique to "regular expressions"!
-Regular expressions do essentially the same thing, but are much, much more powerful. There are many more "special characters" and ways of building complex patterns however. Let's look at a few of the common ones, and then some examples:
-These are just some of the ones you are likely to use when matching URLs with Privoxy, and is a long way from a definitive list. This is enough to get us started with a few simple examples which may be more illuminating:
-/.*/banners/.* - A simple example that uses the common combination of "." and @@ -202,9 +180,7 @@ just "/banners/annoying.html", or almost an infinite number of other possible combinations, just so it has "banners" in the path somewhere.
-And now something a little more complex:
-/.*/adv((er)?ts?|ertis(ing|ements?))?/ - We have several literal forward slashes again (adv, together with the forward slashes. What comes after the "adv" string is the interesting part.
-Remember the "?" means the preceding expression (either a literal character or anything grouped with "(...)" in this case) can exist or not, @@ -241,7 +216,6 @@ expression to: "/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/", which would then match either spelling.
-/.*/advert[0-9]+\.(gif|jpe?g) - Again another path statement with forward slashes. Anything in the square brackets @@ -275,27 +249,22 @@ include an "s"), or "/advert1.jsp" ("jsp" is not in the expression anywhere).
-We are barely scratching the surface of regular expressions here so that you can understand the default Privoxy configuration files, and maybe use this knowledge to customize your own installation. There is much, much more that can be done with regular expressions. Now that you know enough to get started, you can learn more on your own :/
-More reading on Perl Compatible Regular expressions: http://perldoc.perl.org/perlre.html
-For information on regular expression based substitutions and their applications in filters, please see the filter file tutorial in this manual.
Since Privoxy proxies each requested web page, it is easy for Privoxy to trap certain special URLs. In this way, @@ -304,90 +273,73 @@ these rules and other configuration options, and even turn Privoxy's filtering off, all with a web browser.
-The URLs listed below are the special ones that allow direct access to Privoxy. Of course, Privoxy must be running to access these. If not, you will get a friendly error message. Internet access is not necessary either.
-Privoxy main page:
- +Privoxy main page:
-
There is a shortcut: http://p.p/ (But it doesn't provide a fall-back to a real page, in case the request is not sent through Privoxy)
Show information about the current configuration, including - viewing and editing of actions files:
- + viewing and editing of actions files:Show the source code version numbers:
- +Show the source code version numbers:
Show the browser's request headers:
- +Show the browser's request headers:
Show which actions apply to a URL and why:
- +Show which actions apply to a URL and why:
Toggle Privoxy on or off. This feature can be turned off/on in the main config file. When toggled "off", "Privoxy" continues to run, but only as a pass-through proxy, with no actions taking place:
- + "AEN6098" id="AEN6098">- -
Short cuts. Turn off, then on:
- +Short cuts. Turn off, then on:
-- +
@@ -395,16 +347,13 @@
Let's take a quick look at how some of Privoxy's core features are triggered, and the ensuing sequence of events when a web page is requested by your browser:
-First, your web browser requests a web page. The browser knows @@ -412,14 +361,12 @@ which will in turn, relay the request to the remote web server after passing the following tests:
Privoxy traps any request for its own internal CGI pages (e.g http://p.p/) and sends the CGI page back to the browser.
Next, Privoxy checks to see if the URL matches any "+set-image-blocker" (blank, checkerboard pattern, or an HTTP redirect to an image elsewhere).
Untrusted URLs are blocked. If URLs are being added to the trust file, then that is done.
If the URL pattern matches the "+fast-redirects" action, it is then processed. Unwanted parts of the requested URL are stripped.
Now the rest of the client browser's request headers are processed. If any of these match any of the relevant actions (e.g. @@ -459,12 +403,10 @@ suppressed or forged as determined by these actions and their parameters.
Now the web server starts sending its response back (i.e. typically a web page).
First, the server headers are read and processed to determine, among other things, the MIME type (document type) and encoding. The @@ -476,7 +418,6 @@ "actions-file.html#DOWNGRADE-HTTP-VERSION">"+downgrade-http-version" actions.
If any "+filter" action or Privoxy back to your browser.
-If neither a "+filter" action or Privoxy passes the raw data through to the client browser as it becomes available.
As the browser receives the now (possibly filtered) page content, it reads and then requests any URLs that may be embedded @@ -513,17 +452,14 @@ very differing set of actions is triggered.
NOTE: This is somewhat of a simplistic overview of what happens with each URL request. For the sake of brevity and simplicity, we have focused on Privoxy's core features only.
The way Privoxy applies actions and filters to any given URL can be complex, @@ -536,7 +472,6 @@ actions and filters files themselves, since they tend to be filled with regular expressions whose consequences are not always so obvious.
-One quick test to see if Privoxy is causing a problem or not, is to disable it temporarily. This should be the first troubleshooting step (be sure to flush caches afterward!). @@ -544,20 +479,17 @@ feature and logging are enabled via config file settings, and may need to be turned "on".)
-Another easy troubleshooting step to try is if you have done any customization of your installation, revert back to the installed defaults and see if that helps. There are times the developers get complaints about one thing or another, and the problem is more related to a customized configuration issue.
-Privoxy also provides the http://config.privoxy.org/show-url-info page that can show us very specifically how actions are being applied to any given URL. This is a big help for troubleshooting.
-First, enter one URL (or partial URL) at the prompt, and then Privoxy will tell us how the current configuration will handle it. This will not help with filtering effects @@ -572,16 +504,13 @@ those out of the HTML source. Use your browser's "View Page Source" option for this. Or right click on the ad, and grab the URL.
-Let's try an example, google.com, and look at it one section at a time in a sample configuration (your real configuration may vary):
-
- - Matches for http://www.google.com: +Matches for http://www.google.com: In file: default.action [ View ] [ Edit ] @@ -609,12 +538,10 @@ In file: user.action [ View ] [ Edit ] -(no matches in this file) -+(no matches in this file) |
This is telling us how we have defined our "actions", and which ones match for our test case, [ View ] "on" here, but many are "off". Each example we try may provide a slightly different end result, depending on our configuration directives.
-The first listing is for our default.action file. The large, multi-line listing, is how the actions are set to match for all URLs, i.e. our default @@ -634,7 +560,6 @@ In file: user.action [ View ] "aliases" section near the top. This will apply to all URLs as signified by the single forward slash at the end of the listing -- " / ".
-But we have defined additional actions that would be exceptions to these general rules, and then we list specific URLs (or patterns) that these exceptions would apply to. Last match wins. Just below this then @@ -657,7 +582,6 @@ In file: user.action [ View ] default.action file, and "google.com" is referenced somewhere in these latter sections.
-Then, for our user.action file, we again have no hits. So there is nothing google-specific that we might have added to our own, local configuration. If there was, those actions @@ -665,17 +589,15 @@ In file: user.action [ View ] default.action. user.action typically has the last word. This is the best place to put hard and fast exceptions,
-And finally we pull it all together in the bottom section and summarize how Privoxy is applying all its "actions" to "google.com":
-
- Final results: + Final results: -add-header -block @@ -732,38 +654,32 @@ In file: user.action [ View ] + +set-image-blocker {pattern} |
Notice the only difference here to the previous listing, is to "fast-redirects" and "session-cookies-only", which are activated specifically for this site in our configuration, and thus show in the "Final Results".
-Now another example, "ad.doubleclick.net":
-
-
{ +block{Domains starts with "ad"} }
+ { +block{Domains starts with "ad"} }
ad*.
{ +block{Domain contains "ad"} }
.ad.
{ +block{Doubleclick banner server} +handle-as-image }
- .[a-vx-z]*.doubleclick.net
-
+ .[a-vx-z]*.doubleclick.net
|
We'll just show the interesting part here - the explicit matches. It is matched three different times. Two "+block{}" sections, and a "+block{} @@ -773,7 +689,6 @@ In file: user.action [ View ] "Aliases" are defined in the first section of the actions file and typically used to combine more than one action.)
-Any one of these would have done the trick and blocked this as an unwanted image. This is unnecessarily redundant since the last case effectively would also cover the first. No point in taking chances with @@ -786,16 +701,15 @@ In file: user.action [ View ] "+handle-as-image". The custom alias "+block-as-image" just simplifies the process and make it more readable.
-One last example. Let's try "http://www.example.net/adsl/HOWTO/". This one is giving us problems. We are getting a blank page. Hmmm ...
-
- Matches for http://www.example.net/adsl/HOWTO/: + + Matches for http://www.example.net/adsl/HOWTO/: In file: default.action [ View ] [ Edit ] @@ -857,12 +771,10 @@ In file: user.action [ View ] + /ads |
Ooops, the "/adsl/" is matching "/ads" in our configuration! But we did not want this at all! Now we see why we get the blank page. It is actually @@ -877,37 +789,31 @@ In file: user.action [ View ] "adsl" in them (remember, last match in the configuration wins). There are various ways to handle such exceptions. Example:
-
-
{ -block }
- /adsl
-
+ { -block }
+ /adsl
|
Now the page displays ;-) Remember to flush your browser's caches when making these kinds of changes to your configuration to insure that you get a freshly delivered page! Or, try using Shift+Reload.
-But now what about a situation where we get no explicit matches like we did with:
-
-
{ +block{Path starts with "ads".} +handle-as-image }
- /ads
-
+
+ { +block{Path starts with "ads".} +handle-as-image }
+ /ads
|
That actually was very helpful and pointed us quickly to where the problem was. If you don't get this kind of match, then it means one of the default rules in the first section of [ View ] "+filter":
-
-
{ shop }
+ { shop }
.quietpc.com
.worldpay.com # for quietpc.com
.jungle.com
.scan.co.uk
- .forbes.com
-
+ .forbes.com
|
"{ shop }" is an "alias" that expands to "{ -filter -session-cookies-only }". Or you could do your own exception to negate filtering:
-
-
{ -filter }
+ { -filter }
# Disable ALL filter actions for sites in this section
.forbes.com
developer.ibm.com
- localhost
-
+ localhost
|
This would turn off all filtering for these sites. This is best put in user.action, for local site exceptions. Note that when a simple domain pattern is used by itself (without the subsequent path portion), all sub-pages within that domain are included automatically in the scope of the action.
-Images that are inexplicably being blocked, may well be hitting the "+filter{banners-by-size}" rule, which assumes that images of certain sizes are ad banners (works well most of the time since these tend to be standardized).
-"{ fragile }" is an alias that disables most actions that are the most likely to cause trouble. This can be used as a last resort for problem sites.
-
-
{ fragile }
+ { fragile }
# Handle with care: easy to break
mail.google.
- mybank.example.com
-
+ mybank.example.com
|
Remember to flush caches! Note that the mail.google reference lacks the TLD portion (e.g. ".com"). This will effectively match any TLD with google in it, such as mail.google.de., just as an example.
-If this still does not work, you will have to go through the remaining actions one by one to find which one(s) is causing the problem.