>The Filter File</TITLE
><META
NAME="GENERATOR"
-CONTENT="Modular DocBook HTML Stylesheet Version 1.64
-"><LINK
+CONTENT="Modular DocBook HTML Stylesheet Version 1.60"><LINK
REL="HOME"
TITLE="Privoxy User Manual"
HREF="index.html"><LINK
>9. The Filter File</A
></H1
><P
-> Any web page can be dynamically modified with the filter file. This
- modification can be removal, or re-writing, of any web page content,
- including tags and non-visible content. The default filter file is
- oddly enough <TT
+> All text substitutions that can be invoked through the
+ <TT
+CLASS="LITERAL"
+><A
+HREF="actions-file.html#FILTER"
+>filter</A
+></TT
+> action
+ must first be defined in the filter file, which is typically
+ called <TT
CLASS="FILENAME"
>default.filter</TT
->, located in the config
- directory. </P
+> and which can be
+ selected through the <TT
+CLASS="LITERAL"
+> <A
+HREF="config.html#FILTERFILE"
+>filterfile</A
+></TT
+> config
+ option.</P
><P
-> This is potentially a very powerful feature, and requires knowledge of both
- <SPAN
+> Typical reasons for doing such substitutions are to eliminate
+ common annoyances in HTML and JavaScript, such as pop-up windows,
+ exit consoles, crippled windows without navigation tools, the
+ infamous <BLINK> tag etc, to suppress images with certain
+ width and height attributes (standard banner sizes or web-bugs),
+ or just to have fun. The possibilities are endless.</P
+><P
+> Filtering works on any text-based document type, including plain
+ text, HTML, JavaScript, CSS etc. (all <TT
+CLASS="LITERAL"
+>text/*</TT
+>
+ MIME types). Substitutions are made at the source level, so if
+ you want to <SPAN
+CLASS="QUOTE"
+>"roll your own"</SPAN
+> filters, you should be
+ familiar with HTML syntax.</P
+><P
+> Just like the <A
+HREF="actions-file.html"
+>actions files</A
+>, the
+ filter file is organized in sections, which are called <I
+CLASS="EMPHASIS"
+>filters</I
+>
+ here. Each filter consists of a heading line, that starts with the
+ <I
+CLASS="EMPHASIS"
+>keyword</I
+> <TT
+CLASS="LITERAL"
+>FILTER:</TT
+>, followed by
+ the filter's <I
+CLASS="EMPHASIS"
+>name</I
+>, and a short (one line)
+ <I
+CLASS="EMPHASIS"
+>description</I
+> of what it does. Below that line
+ come the <I
+CLASS="EMPHASIS"
+>jobs</I
+>, i.e. lines that define the actual
+ text substitutions. By convention, the name of a filter
+ should describe what the filter <I
+CLASS="EMPHASIS"
+>eliminates</I
+>. The
+ comment is used in the <A
+HREF="http://config.privoxy.org/"
+TARGET="_top"
+>web-based
+ user interface</A
+>.</P
+><P
+> Once a filter called <TT
+CLASS="REPLACEABLE"
+><I
+>name</I
+></TT
+> has been defined
+ in the filter file, it can be invoked by using an action of the form
+ +<TT
+CLASS="LITERAL"
+><A
+HREF="actions-file.html#FILTER"
+>filter</A
+>{<TT
+CLASS="REPLACEABLE"
+><I
+>name</I
+></TT
+>}</TT
+>
+ in any <A
+HREF="actions-file.html"
+>actions file</A
+>.</P
+><P
+> A filter header line for a filter called <SPAN
+CLASS="QUOTE"
+>"foo"</SPAN
+> could look
+ like this:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+>FILTER: foo Replace all "foo" with "bar"</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> Below that line, and up to the next header line, come the jobs that
+ define what text replacements the filter executes. They are specified
+ in a syntax that imitates <A
+HREF="http://www.perl.org/"
+TARGET="_top"
+>Perl</A
+>'s
+ <TT
+CLASS="LITERAL"
+>s///</TT
+> operator. If you are familiar with Perl, you
+ will find this to be quite intuitive, and may want to look at the
+ <A
+HREF="http://www.oesterhelt.org/pcrs/pcrs.1.html"
+TARGET="_top"
+>PCRS man page</A
+>
+ for the subtle differences to Perl behaviour. Most notably, the non-standard
+ option letter <TT
+CLASS="LITERAL"
+>U</TT
+> is supported, which turns the default
+ to ungreedy matching.</P
+><P
+> If you are new to regular expressions, you might want to take a look at
+ the <A
+HREF="appendix.html#REGEX"
+>Appendix on regular expressions</A
+>, and
+ see the <A
+HREF="http://perldoc.com/perl5.6.1/pod/perl.html"
+TARGET="_top"
+>Perl
+ manual</A
+> for
+ <A
+HREF="http://perldoc.com/perl5.6.1/pod/perlop.html#s-PATTERN-REPLACEMENT-egimosx"
+TARGET="_top"
+>the
+ <TT
+CLASS="LITERAL"
+>s///</TT
+> operator's syntax</A
+> and <A
+HREF="http://perldoc.com/perl5.6.1/pod/perlre.html"
+TARGET="_top"
+>Perl-style regular
+ expressions</A
+> in general.
+ The below examples might also help to get you started.</P
+><DIV
+CLASS="SECT2"
+><H2
+CLASS="SECT2"
+><A
+NAME="AEN2778"
+>9.1. Filter File Tutorial</A
+></H2
+><P
+> Now, let's complete our <SPAN
CLASS="QUOTE"
->"regular expression"</SPAN
-> and HTML in order create custom
- filters. But, there are a number of useful filters included with
+>"foo"</SPAN
+> filter. We have already defined
+ the heading, but the jobs are still missing. Since all it does is to replace
<SPAN
-CLASS="APPLICATION"
->Privoxy</SPAN
-> for many common situations.</P
+CLASS="QUOTE"
+>"foo"</SPAN
+> with <SPAN
+CLASS="QUOTE"
+>"bar"</SPAN
+>, there is only one (trivial) job
+ needed:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+>s/foo/bar/</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> But wait! Didn't the comment say that <I
+CLASS="EMPHASIS"
+>all</I
+> occurrences
+ of <SPAN
+CLASS="QUOTE"
+>"foo"</SPAN
+> should be replaced? Our current job will only take
+ care of the first <SPAN
+CLASS="QUOTE"
+>"foo"</SPAN
+> on each page. For global substitution,
+ we'll need to add the <TT
+CLASS="LITERAL"
+>g</TT
+> option:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+>s/foo/bar/g</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> Our complete filter now looks like this:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+>FILTER: foo Replace all "foo" with "bar"
+s/foo/bar/g</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> Let's look at some real filters for more interesting examples. Here you see
+ a filter that protects against some common annoyances that arise from JavaScript
+ abuse. Let's look at its jobs one after the other:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+>FILTER: js-annoyances Get rid of particularly annoying JavaScript abuse
+
+# Get rid of JavaScript referrer tracking. Test page: http://www.randomoddness.com/untitled.htm
+#
+s|(<script.*)document\.referrer(.*</script>)|$1"Not Your Business!"$2|Usg</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> Following the header line and a comment, you see the job. Note that it uses
+ <TT
+CLASS="LITERAL"
+>|</TT
+> as the delimiter instead of <TT
+CLASS="LITERAL"
+>/</TT
+>, because
+ the pattern contains a forward slash, which would otherwise have to be escaped
+ by a backslash (<TT
+CLASS="LITERAL"
+>\</TT
+>).</P
><P
-> The included example file is divided into sections. Each section begins
- with the <TT
+> Now, let's examine the pattern: it starts with the text <TT
CLASS="LITERAL"
->FILTER</TT
-> keyword, followed by the identifier
- for that section, e.g. <SPAN
+><script.*</TT
+>
+ enclosed in parentheses. Since the dot matches any character, and <TT
+CLASS="LITERAL"
+>*</TT
+>
+ means: <SPAN
CLASS="QUOTE"
->"FILTER: webbugs"</SPAN
->. Each section performs
- a similar type of filtering, such as <SPAN
+>"Match an arbitrary number of the element left of myself"</SPAN
+>, this
+ matches <SPAN
CLASS="QUOTE"
->"html-annoyances"</SPAN
->.</P
+>"<script"</SPAN
+>, followed by <I
+CLASS="EMPHASIS"
+>any</I
+> text, i.e.
+ it matches the whole page, from the start of the first <script> tag.</P
><P
-> This file uses regular expressions to alter or remove any string in the
- target page. The expressions can only operate on one line at a time. Some
- examples from the included default <TT
-CLASS="FILENAME"
->default.filter</TT
->:</P
+> That's more than we want, but the pattern continues: <TT
+CLASS="LITERAL"
+>document\.referrer</TT
+>
+ matches only the exact string <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+>. The dot needed to
+ be <I
+CLASS="EMPHASIS"
+>escaped</I
+>, i.e. preceded by a backslash, to take away its
+ special meaning as a joker, and make it just a regular dot. So far, the meaning is:
+ Match from the start of the first <script> tag in a the page, up to, and including,
+ the text <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+>, if <I
+CLASS="EMPHASIS"
+>both</I
+> are present
+ in the page (and appear in that order).</P
><P
-> Stop web pages from displaying annoying messages in the status bar by
- deleting such references:</P
+> But there's still more pattern to go. The next element, again enclosed in parentheses,
+ is <TT
+CLASS="LITERAL"
+>.*</script></TT
+>. You already know what <TT
+CLASS="LITERAL"
+>.*</TT
+>
+ means, so the whole pattern translates to: Match from the start of the first <script>
+ tag in a page to the end of the last <script> tag, provided that the text
+ <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+> appears somewhere in between.</P
><P
-> <TT
+> This is still not the whole story, since we have ignored the options and the parentheses:
+ The portions of the page matched by sub-patterns that are enclosed in parentheses, will be
+ remembered and be available through the variables <TT
CLASS="LITERAL"
-> <P
-CLASS="LITERALLAYOUT"
-> FILTER: html-annoyances<br>
-<br>
- # New browser windows should be resizeable and have a location and status<br>
- # bar. Make it so.<br>
- #<br>
- s/resizable="?(no|0)"?/resizable=1/ig s/noresize/yesresize/ig<br>
- s/location="?(no|0)"?/location=1/ig s/status="?(no|0)"?/status=1/ig<br>
- s/scrolling="?(no|0|Auto)"?/scrolling=1/ig<br>
- s/menubar="?(no|0)"?/menubar=1/ig <br>
-<br>
- # The <BLINK> tag was a crime!<br>
- #<br>
- s*<blink>|</blink>**ig<br>
-<br>
- # Is this evil? <br>
- #<br>
- #s/framespacing="?(no|0)"?//ig<br>
- #s/margin(height|width)=[0-9]*//gi<br>
- </P
->
- </TT
-></P
+>$1, $2, ...</TT
+> in
+ the substitute. The <TT
+CLASS="LITERAL"
+>U</TT
+> option switches to ungreedy matching, which means
+ that the first <TT
+CLASS="LITERAL"
+>.*</TT
+> in the pattern will only <SPAN
+CLASS="QUOTE"
+>"eat up"</SPAN
+> all
+ text in between <SPAN
+CLASS="QUOTE"
+>"<script"</SPAN
+> and the <I
+CLASS="EMPHASIS"
+>first</I
+> occurrence
+ of <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+>, and that the second <TT
+CLASS="LITERAL"
+>.*</TT
+> will
+ only span the text up to the <I
+CLASS="EMPHASIS"
+>first</I
+> <SPAN
+CLASS="QUOTE"
+>"</script>"</SPAN
+>
+ tag. Furthermore, the <TT
+CLASS="LITERAL"
+>s</TT
+> option says that the match may span
+ multiple lines in the page, and the <TT
+CLASS="LITERAL"
+>g</TT
+> option again means that the
+ substitution is global.</P
><P
-> Just for kicks, replace any occurrence of <SPAN
+> So, to summarize, the pattern means: Match all scripts that contain the text
+ <SPAN
CLASS="QUOTE"
->"Microsoft"</SPAN
-> with
+>"document.referrer"</SPAN
+>. Remember the parts of the script from
+ (and including) the start tag up to (and excluding) the string
<SPAN
CLASS="QUOTE"
->"MicroSuck"</SPAN
->, and have a little fun with topical buzzwords: </P
+>"document.referrer"</SPAN
+> as <TT
+CLASS="LITERAL"
+>$1</TT
+>, and the part following
+ that string, up to and including the closing tag, as <TT
+CLASS="LITERAL"
+>$2</TT
+>.</P
><P
-> <TT
+> Now the pattern is deciphered, but wasn't this about substituting things? So
+ lets look at the substitute: <TT
CLASS="LITERAL"
-> <P
-CLASS="LITERALLAYOUT"
-> FILTER: fun<br>
-<br>
- s/microsoft(?!.com)/MicroSuck/ig<br>
-<br>
- # Buzzword Bingo:<br>
- #<br>
- s/industry-leading|cutting-edge|award-winning/<font color=red><b>BINGO!</b></font>/ig<br>
- </P
->
- </TT
-></P
+>$1"Not Your Business!"$2</TT
+> is
+ easy to read: The text remembered as <TT
+CLASS="LITERAL"
+>$1</TT
+>, followed by
+ <TT
+CLASS="LITERAL"
+>"Not Your Business!"</TT
+> (<I
+CLASS="EMPHASIS"
+>including</I
+>
+ the quotation marks!), followed by the text remembered as <TT
+CLASS="LITERAL"
+>$2</TT
+>.
+ This produces an exact copy of the original string, with the middle part
+ (the <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+>) replaced by <TT
+CLASS="LITERAL"
+>"Not Your
+ Business!"</TT
+>.</P
><P
-> Kill those pesky little web-bugs:</P
+> The whole job now reads: Replace <SPAN
+CLASS="QUOTE"
+>"document.referrer"</SPAN
+> by
+ <TT
+CLASS="LITERAL"
+>"Not Your Business!"</TT
+> wherever it appears inside a
+ <script> tag. Note that this job won't break JavaScript syntax,
+ since both the original and the replacement are syntactically valid
+ string objects. The script just won't have access to the referrer
+ information anymore.</P
+><P
+> We'll show you two other jobs from the JavaScript taming department, but
+ this time only point out the constructs of special interest:</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+># The status bar is for displaying link targets, not pointless blahblah
+#
+s/window\.status\s*=\s*['"].*?['"]/dUmMy=1/ig</PRE
+></TD
+></TR
+></TABLE
+></P
><P
> <TT
CLASS="LITERAL"
-> <P
-CLASS="LITERALLAYOUT"
-> # webbugs: Squish WebBugs (1x1 invisible GIFs used for user tracking)<br>
- FILTER: webbugs<br>
-<br>
- s/<img\s+[^>]*?(width|height)\s*=\s*['"]?1\D[^>]*?(width|height)\s*=\s*['"]?1(\D[^>]*?)?>/<!-- Squished WebBug -->/sig<br>
- </P
->
- </TT
-></P
-><DIV
-CLASS="SECT2"
-><H2
-CLASS="SECT2"
-><A
-NAME="AEN2432"
->9.1. The <I
+>\s</TT
+> stands for whitespace characters (space, tab, newline,
+ carriage return, form feed), so that <TT
+CLASS="LITERAL"
+>\s*</TT
+> means: <SPAN
+CLASS="QUOTE"
+>"zero
+ or more whitespace"</SPAN
+>. The <TT
+CLASS="LITERAL"
+>?</TT
+> in <TT
+CLASS="LITERAL"
+>.*?</TT
+>
+ makes this matching of arbitrary text ungreedy. (Note that the <TT
+CLASS="LITERAL"
+>U</TT
+>
+ option is not set). The <TT
+CLASS="LITERAL"
+>['"]</TT
+> construct means: <SPAN
+CLASS="QUOTE"
+>"a single
+ <I
CLASS="EMPHASIS"
->+filter</I
-> Action</A
-></H2
+>or</I
+> a double quote"</SPAN
+>.</P
><P
-> Filters are enabled with the <A
-HREF="actions-file.html#FILTER"
+> So what does this job do? It replaces assignments of single- or double-quoted
+ strings to the <SPAN
+CLASS="QUOTE"
+>"window.status"</SPAN
+> object with a dummy assignment
+ (using a variable name that is hopefully odd enough not to conflict with
+ real variables in scripts). Thus, it catches many cases where e.g. pointless
+ descriptions are displayed in the status bar instead of the link target when
+ you move your mouse over links.</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+># Kill OnUnload popups. Yummy. Test: http://www.zdnet.com/zdsubs/yahoo/tree/yfs.html
+#
+s/(<body .*)onunload(.*>)/$1never$2/iU</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> Including the
+ <A
+HREF="http://www.w3.org/TR/2000/REC-DOM-Level-2-Events-20001113/events.html#Events-eventgroupings-htmlevents"
TARGET="_top"
-><SPAN
+>OnUnload
+ event binding</A
+> in the HTML DOM was a <I
+CLASS="EMPHASIS"
+>CRIME</I
+>.
+ When I close a browser window, I want it to close and die. Basta.
+ This job replaces the <SPAN
CLASS="QUOTE"
->"+filter"</SPAN
-></A
-> action from within
- one of the actions files. <SPAN
+>"onunload"</SPAN
+> attribute in
+ <SPAN
CLASS="QUOTE"
->"+filter"</SPAN
-> requires one parameter, which
- should match one of the section identifiers in the filter file itself. Example:</P
-><TABLE
+>"<body>"</SPAN
+> tags with the dummy word <TT
+CLASS="LITERAL"
+>never</TT
+>.
+ Note that the <TT
+CLASS="LITERAL"
+>i</TT
+> option makes the pattern matching
+ case-insensitive.</P
+><P
+> The last example is from the fun department:</P
+><P
+> <TABLE
BORDER="0"
BGCOLOR="#E0E0E0"
WIDTH="100%"
><TD
><PRE
CLASS="SCREEN"
-> +filter{html-annoyances}</PRE
+>FILTER: fun Fun text replacements
+
+# Spice the daily news:
+#
+s/microsoft(?!\.com)/MicroSuck/ig</PRE
></TD
></TR
></TABLE
+></P
><P
-> This would activate that particular filter. Similarly, <SPAN
-CLASS="QUOTE"
->"+filter"</SPAN
->
- can be turned off for selected sites as:
+> Note the <TT
+CLASS="LITERAL"
+>(?!\.com)</TT
+> part (a so-called negative lookahead)
+ in the job's pattern, which means: Don't match, if the string
<SPAN
CLASS="QUOTE"
->"-filter{html-annoyances}"</SPAN
->. Remember too, all actions are off by
- default, unless they are explicity enabled in one of the actions files.</P
+>".com"</SPAN
+> appears directly following <SPAN
+CLASS="QUOTE"
+>"microsoft"</SPAN
+>
+ in the page. This prevents links to microsoft.com from being messed, while
+ still replacing the word everywhere else.</P
+><P
+> <TABLE
+BORDER="0"
+BGCOLOR="#E0E0E0"
+WIDTH="100%"
+><TR
+><TD
+><PRE
+CLASS="SCREEN"
+># Buzzword Bingo (example for extended regex syntax)
+#
+s* industry[ -]leading \
+| cutting[ -]edge \
+| award[ -]winning # Comments are OK, too! \
+| high[ -]performance \
+| solutions[ -]based \
+| unmatched \
+| unparalleled \
+| unrivalled \
+*<font color="red"><b>BINGO!</b></font> \
+*igx</PRE
+></TD
+></TR
+></TABLE
+></P
+><P
+> The <TT
+CLASS="LITERAL"
+>x</TT
+> option in this job turns on extended syntax, and allows for
+ e.g. the liberal use of (non-interpreted!) whitespace for nicer formatting.</P
+><P
+> You get the idea?</P
></DIV
></DIV
><DIV
projects / privoxy.git / blobdiff