Junkbuster User Manual$Id: user-manual.sgml,v 1.47 2002/03/11 13:13:27 swa Exp $By: Junkbuster Developers
The user manual gives users information on how to install, configure and use
Internet Junkbuster. Internet
Junkbuster is a web proxy with advanced filtering capabilities
for protecting privacy, filtering web page content, managing cookies,
controlling access, and removing ads, banners, pop-ups and other obnoxious
Internet Junk. Junkbuster has a very flexible configuration and can be
customized to suit individual needs and tastes. Internet
Junkbuster has application for both stand-alone systems and
multi-user networks.
You can find the latest version of the user manual at http://ijbswa.sourceforge.net/user-manual/.
IntroductionInternet Junkbuster is a web proxy with advanced
filtering capabilities for protecting privacy, filtering and modifying web
page content, managing cookies, controlling access, and removing ads,
banners, pop-ups and other obnoxious Internet Junk.
Junkbuster has a very flexible configuration and
can be customized to suit individual needs and tastes. Internet
Junkbuster has application for both stand-alone systems and
multi-user networks.
This documentation is included with the current BETA version of
Internet Junkbuster and is mostly complete at this
point. The most up to date reference for the time being is still the comments
in the source files and in the individual configuration files. Development
of version 3.0 is currently nearing completion, and includes many significant
changes and enhancements over earlier versions. The target release date for
stable v3.0 is soon ;-)
Since this is a BETA version, not all new features are well tested. This
documentation may be slightly out of sync as a result (especially with
CVS sources). And there may be bugs, though hopefully
not many!
New Features
In addition to Junkbuster's traditional features
of ad and banner blocking and cookie management, this is a list of new
features currently under development:
Integrated browser based configuration and control utility (http://i.j.b). Browser-based tracing of rule
and filter effects.
Modularized configuration that will allow for system wide settings, and
individual user settings. (not implemented yet, probably a 3.1 feature)
Blocking of annoying pop-up browser windows.
HTTP/1.1 compliant (most, but not all 1.1 features are supported).
Support for Perl Compatible Regular Expressions in the configuration files, and
generally a more sophisticated and flexible configuration syntax over
previous versions.
GIF de-animation.
Web page content filtering (removes banners based on size,
invisible web-bugs, JavaScript, pop-ups, status bar abuse,
etc.)
Bypass many click-tracking scripts (avoids script redirection).
Multi-threaded (POSIX and native threads).
Auto-detection and re-reading of config file changes.
User-customizable HTML templates (e.g. 404 error page).
Improved cookie management features (e.g. session based cookies).
Builds from source on most UNIX-like systems. Packages available for: Linux
(RedHat, SuSE, or Debian), Windows, Sun Solaris, Mac OSX, OS/2, HP-UX 11 and AmigaOS.
In addition, the configuration is much more powerful and versatile over-all.
InstallationJunkbuster is available as raw source code, or
pre-compiled binaries. See the Junkbuster Home Page
for binaries and current release info. Junkbuster
is also available via CVS.
This is the recommended approach at this time. But please be aware that CVS
is constantly changing, and it may break in mysterious ways.
Source
For gzipped tar archives, unpack the source:
tar xzvf ijb_source_* [.tgz or .tar.gz]
cd ijb_source_2.9.11_beta
For retrieving the current CVS sources, you'll need the CVS
package installed first. To download CVS source:
cvs -d:pserver:anonymous@cvs.ijbswa.sourceforge.net:/cvsroot/ijbswa login
cvs -z3 -d:pserver:anonymous@cvs.ijbswa.sourceforge.net:/cvsroot/ijbswa co current
cd current
This will create a directory named current/, which will
contain the source tree.
Then, in either case, to build from tarball/CVS source:
./configure (--help to see options)
make (the make from gnu, gmake for *BSD)
su
make -n install (to see where all the files will go)
make install (to really install)
For Redhat and SuSE Linux RPM packages, see below.
Red Hat
To build Redhat RPM packages, install source as above. Then:
autoheader [suggested for CVS source]
autoconf [suggested for CVS source]
./configure
make redhat-dist
This will create both binary and src RPMs in the usual places. Example:
/usr/src/redhat/RPMS/i686/junkbuster-2.9.11-1.i686.rpm
/usr/src/redhat/SRPMS/junkbuster-2.9.11-1.src.rpm
To install, of course:
rpm -Uvv /usr/src/redhat/RPMS/i686/junkbuster-2.9.11-1.i686.rpm
This will place the Junkbuster configuration
files in /etc/junkbuster/, and log files in
/var/log/junkbuster/.
SuSE
To build SuSE RPM packages, install source as above. Then:
autoheader [suggested for CVS source]
autoconf [suggested for CVS source]
./configure
make suse-dist
This will create both binary and src RPMs in the usual places. Example:
/usr/src/packages/RPMS/i686/junkbuster-2.9.11-1.i686.rpm
/usr/src/packages/SRPMS/junkbuster-2.9.11-1.src.rpm
To install, of course:
rpm -Uvv /usr/src/packages/RPMS/i686/junkbuster-2.9.11-1.i686.rpm
This will place the Junkbuster configuration
files in /etc/junkbuster/, and log files in
/var/log/junkbuster/.
OS/2Junkbuster is packaged in a WarpIN self-
installing archive. The self-installing program will be named depending
on the release version, something like:
ijbos2_setup_1.2.3.exe. In order to install it, simply
run this executable or double-click on its icon and follow the WarpIN
installation panels. A shadow of the Junkbuster
executable will be placed in your startup folder so it will start
automatically whenever OS/2 starts.
The directory you choose to install Junkbuster
into will contain all of the configuration files.
If you would like to build binary images on OS/2 yourself, you will need
a few Unix-like tools: autoconf, autoheader and sh. These tools will be
used to create the required config.h file, which is not part of the
source distribution because it differs based on platform. You will also
need a compiler.
The distribution has been created using IBM VisualAge compilers, but you
can use any compiler you like. GCC/EMX has the disadvantage of needing
to be single-threaded due to a limitation of EMX's implementation of the
select() socket call.
In addition to needing the source code distribution as outlined earlier,
you will want to extract the os2seutp directory from CVS:
cvs -d:pserver:anonymous@cvs.ijbswa.sourceforge.net:/cvsroot/ijbswa login
cvs -z3 -d:pserver:anonymous@cvs.ijbswa.sourceforge.net:/cvsroot/ijbswa co os2setup
This will create a directory named os2setup/, which will contain the
Makefile.vac makefile and os2build.cmd
which is used to completely create the binary distribution. The sequence
of events for building the executable for yourself goes something like this:
cd current
autoheader
autoconf
sh configure
cd ..\os2setup
nmake -f Makefile.vac
You will see this sequence laid out in os2build.cmd.
WindowsClick-click. (I need help on this. Not a clue here. Also for
configuration section below. HB.)
Other
Some quick notes on other Operating Systems.
For FreeBSD (and other *BSDs?), the build will require gmake
instead of the included make. gmake is
available from http://www.gnu.org.
The rest should be the same as above for Linux/Unix.
JunkBuster Configuration
All JunkBuster configuration is kept
in text files. These files can be edited with a text editor.
Many important aspects of JunkBuster can
also be controlled easily with a web browser.
Controlling Junkbuster with Your Web BrowserJunkBuster can be reached by the special
URL http://i.j.b/ (or alternately
http://ijbswa.sourceforge.net/config/,
which is an internal page. You will see the following section:
Please choose from the following options:
* Show information about the current configuration
* Show the source code version numbers
* Show the client's request headers.
* Show which actions apply to a URL and why
* Toggle JunkBuster on or off
* Edit the actions list
This should be self-explanatory. Note the last item is an editor for the
actions list, which is where much of the ad, banner, cookie,
and URL blocking magic is configured as well as other advanced features of
Junkbuster. This is an easy way to adjust various
aspects of Junkbuster configuration. The actions
file, and other configuration files, are explained in detail below.
Junkbuster will automatically detect any changes
to these files.
Toggle JunkBuster On or Off is handy for sites that might
have problems with your current actions and filters, or just to test if
a site misbehaves, whether it is JunkBuster
causing the problem or not. Junkbuster continues
to run as a proxy in this case, but all filtering is disabled.
Configuration Files Overview
For Unix, *BSD and Linux, all configuration files are located in
/etc/junkbuster/ by default. For MS Windows, OS/2, and
AmigaOS these are all in the same directory as the
Junkbuster executable. The name and number of
configuration files has changed from previous versions, and is subject to
change as development progresses.
The installed defaults provide a reasonable starting point, though possibly
aggressive by some standards. For the time being, there are only three
default configuration files (this will change in time):
The main configuration file is named config
on Linux, Unix, BSD, OS/2, and AmigaOS and config.txt
on Windows.
The ijb.action file is used to define various
actions relating to images, banners, pop-ups, access
restrictions, banners and cookies. There is a CGI based editor for this
file that can be accessed via http://i.j.b. (Other actions
files are included as well with differing levels of filtering
and blocking, e.g. ijb-basic.action.)
The re_filterfile file can be used to re-write the raw
page content, including viewable text as well as embedded HTML and JavaScript,
and whatever else lurks on any given web page.
ijb.action and re_filterfile
can use Perl style regular expressions for maximum flexibility. All files use
the # character to denote a comment. Such
lines are not processed by Junkbuster. After
making any changes, there is no need to restart
Junkbuster in order for the changes to take
effect. Junkbuster should detect such changes
automatically.
While under development, the configuration content is subject to change.
The below documentation may not be accurate by the time you read this.
Also, what constitutes a default setting, may change, so
please check all your configuration files on important issues.
The Main Configuration File
Again, the main configuration file is named config on
Linux/Unix/BSD and OS/2, and config.txt on Windows.
Configuration lines consist of an initial keyword followed by a list of
values, all separated by whitespace (any number of spaces or tabs). For
example:
blockfile blocklist.ini
Indicates that the blockfile is named blocklist.ini.
A # indicates a comment. Any part of a
line following a # is ignored, except if
the # is preceded by a
\.
Thus, by placing a # at the start of an
existing configuration line, you can make it a comment and it will be treated
as if it weren't there. This is called commenting out an
option and can be useful to turn off features: If you comment out the
logfile line, junkbuster will not
log to a file at all. Watch for the default: section in each
explanation to see what happens if the option is left unset (or commented
out).
Long lines can be continued on the next line by using a
\ as the very last character.
There are various aspects of Junkbuster behavior
that can be tuned.
Defining Other Configuration FilesJunkbuster can use a number of other files to tell it
what ads to block, what cookies to accept, etc. This section of the
configuration file tells Junkbuster where to find
all those other files.
On Windows and AmigaOS,
Junkbuster looks for these files in the same
directory as the executable. On Unix and OS/2,
Junkbuster looks for these files in the current
working directory. In either case, an absolute path name can be used to
avoid problems.
When development goes modular and multi-user, the blocker, filter, and
per-user config will be stored in subdirectories of confdir.
For now, only confdir/templates is used for storing HTML
templates for CGI results.
The location of the configuration files:
confdir /etc/junkbuster # No trailing /, please.
The directory where all logging (i.e. logfile and
jarfile) takes place. No trailing
/, please:
logdir /var/log/junkbuster
Note that all file specifications below are relative to
the above two directories!
The ijb.action file contains patterns to specify the actions to
apply to requests for each site. Default: Cookies to and from all
destinations are kept only during the current browser session (i.e. they
are not saved to disk). Pop-ups are disabled for all sites. All sites are
filtered through selected sections of re_filterfile. No sites
are blocked. The JunkBuster logo is displayed for filtered ads and other
images . The syntax of this file is explained in detail below.
actionsfile ijb.action
The re_filterfile file contains content modification rules
that use regular expressions. These rules permit powerful
changes on the content of Web pages, e.g., you could disable your favorite
JavaScript annoyances, re-write the actual displayed text, or just have some
fun replacing Microsoft with MicroSuck wherever
it appears on a Web page. Default: whatever the developers are playing with
:-/
Filtering requires buffering the page content, which may appear to slow down
page rendering since nothing is displayed until all content has passed
the filters. (It does not really take longer, but seems that way since
the page is not incrementally displayed.) This effect will be more noticeable
on slower connections.
re_filterfile re_filterfile
The logfile is where all logging and error messages are written. The logfile
can be useful for tracking down a problem with
Junkbuster (e.g., it's not blocking an ad you
think it should block) but in most cases you probably will never look at it.
Your logfile will grow indefinitely, and you will probably want to
periodically remove it. On Unix systems, you can do this with a cron job
(see man cron). For Redhat, a logrotate
script has been included.
On SuSE Linux systems, you can place a line like /var/log/junkbuster.*
+1024k 644 nobody.nogroup in /etc/logfiles, with
the effect that cron.daily will automatically archive, gzip, and empty the
log, when it exceeds 1M size.
Default: Log to the a file named logfile.
Comment out to disable logging.
logfile logfile
The jarfile defines where
Junkbuster stores the cookies it intercepts. Note
that if you use a jarfile, it may grow quite large. Default:
Don't store intercepted cookies.
#jarfile jarfile
If you specify a trustfile,
Junkbuster will only allow access to sites that
are named in the trustfile. You can also mark sites as trusted referrers,
with the effect that access to untrusted sites will be granted, if a link
from a trusted referrer was used. The link target will then be added to the
trustfile. This is a very restrictive feature that typical
users most probably want to leave disabled. Default: Disabled, don't use the
trust mechanism.
#trustfile trust
If you use the trust mechanism, it is a good idea to write up some on-line
documentation about your blocking policy and to specify the URL(s) here. They
will appear on the page that your users receive when they try to access
untrusted content. Use multiple times for multiple URLs. Default: Don't
display links on the untrusted info page.
trust-info-url http://www.your-site.com/why_we_block.htmltrust-info-url http://www.your-site.com/what_we_allow.htmlOther Configuration Options
This part of the configuration file contains options that control how
Junkbuster operates.
Admin-address should be set to the email address of the proxy
administrator. It is used in many of the proxy-generated pages. Default:
fill@me.in.please.
#admin-address fill@me.in.pleaseProxy-info-url can be set to a URL that contains more info
about this Junkbuster installation, it's
configuration and policies. It is used in many of the proxy-generated pages
and its use is highly recommended in multi-user installations, since your
users will want to know why certain content is blocked or modified. Default:
Don't show a link to on-line documentation.
proxy-info-url http://www.your-site.com/proxy.htmlListen-address specifies the address and port where
Junkbuster will listen for connections from your
Web browser. The default is to listen on the localhost port 8118, and
this is suitable for most users. (In your web browser, under proxy
configuration, list the proxy server as localhost and the
port as 8118).
If you already have another service running on port 8118, or if you want to
serve requests from other machines (e.g. on your local network) as well, you
will need to override the default. The syntax is
listen-address [<ip-address>]:<port>. If you leave
out the IP address, junkbuster will bind to all
interfaces (addresses) on your machine and may become reachable from the
Internet. In that case, consider using access control lists (acl's) (see
aclfile above), or a firewall.
For example, suppose you are running Junkbuster on
a machine which has the address 192.168.0.1 on your local private network
(192.168.0.0) and has another outside connection with a different address.
You want it to serve requests from inside only:
listen-address 192.168.0.1:8118
If you want it to listen on all addresses (including the outside
connection):
listen-address :8118
If you do this, consider using ACLs (see aclfile above). Note:
you will need to point your browser(s) to the address and port that you have
configured here. Default: localhost:8118 (127.0.0.1:8118).
The debug option sets the level of debugging information to log in the
logfile (and to the console in the Windows version). A debug level of 1 is
informative because it will show you each request as it happens. Higher
levels of debug are probably only of interest to developers.
debug 1 # GPC = show each GET/POST/CONNECT request
debug 2 # CONN = show each connection status
debug 4 # IO = show I/O status
debug 8 # HDR = show header parsing
debug 16 # LOG = log all data into the logfile
debug 32 # FRC = debug force feature
debug 64 # REF = debug regular expression filter
debug 128 # = debug fast redirects
debug 256 # = debug GIF de-animation
debug 512 # CLF = Common Log Format
debug 1024 # = debug kill pop-ups
debug 4096 # INFO = Startup banner and warnings.
debug 8192 # ERROR = Non-fatal errors
It is highly recommended that you enable ERROR
reporting (debug 8192), at least until the next stable release.
The reporting of FATAL errors (i.e. ones which crash
JunkBuster) is always on and cannot be disabled.
If you want to use CLF (Common Log Format), you should set debug
512 ONLY, do not enable anything else.
Multiple debug directives, are OK - they're logical-OR'd
together.
debug 15 # same as setting the first 4 listed above
Default:
debug 1 # URLsdebug 4096 # Infodebug 8192 # Errors - *we highly recommended enabling this*Junkbuster normally uses
multi-threading, a software technique that permits it to
handle many different requests simultaneously. In some cases you may wish to
disable this -- particularly if you're trying to debug a problem. The
single-threaded option forces
Junkbuster to handle requests sequentially.
Default: Multi-threaded mode.
#single-threadedtoggle allows you to temporarily disable all
Junkbuster's filtering. Just set toggle
0.
The Windows version of Junkbuster puts an icon in
the system tray, which also allows you to change this option. If you
right-click on that icon (or select the Options menu), one
choice is Enable. Clicking on enable toggles
Junkbuster on and off. This is useful if you want
to temporarily disable Junkbuster, e.g., to access
a site that requires cookies which you would otherwise have blocked. This can also
be toggled via a web browser at the Junkbuster
internal address of http://i.j.b on
any platform.
toggle 1 means Junkbuster runs
normally, toggle 0 means that
Junkbuster becomes a non-anonymizing non-blocking
proxy. Default: 1 (on).
toggle 1
For content filtering, i.e. the +filter and
+deanimate-gif actions, it is necessary that
Junkbuster buffers the entire document body.
This can be potentially dangerous, since a server could just keep sending
data indefinitely and wait for your RAM to exhaust. With nasty consequences.
The buffer-limit option lets you set the maximum
size in Kbytes that each buffer may use. When the documents buffer exceeds
this size, it is flushed to the client unfiltered and no further attempt to
filter the rest of it is made. Remember that there may multiple threads
running, which might require increasing the buffer-limit
Kbytes each, unless you have enabled
single-threaded above.
buffer-limit 4069
To enable the web-based ijb.action file editor set
enable-edit-actions to 1, or 0 to disable. Note
that you must have compiled JunkBuster with
support for this feature, otherwise this option has no effect. This
internal page can be reached at http://i.j.b.
Security note: If this is enabled, anyone who can use the proxy
can edit the actions file, and their changes will affect all users.
For shared proxies, you probably want to disable this. Default: enabled.
enable-edit-actions 1
Allow JunkBuster to be toggled on and off
remotely, using your web browser. Set enable-remote-toggleto
1 to enable, and 0 to disable. Note that you must have compiled
JunkBuster with support for this feature,
otherwise this option has no effect.
Security note: If this is enabled, anyone who can use the proxy can toggle
it on or off (see http://i.j.b), and
their changes will affect all users. For shared proxies, you probably want to
disable this. Default: enabled.
enable-remote-toggle 1Access Control List (ACL)
Access controls are included at the request of some ISPs and systems
administrators, and are not usually needed by individual users. Please note
the warnings in the FAQ that this proxy is not intended to be a substitute
for a firewall or to encourage anyone to defer addressing basic security
weaknesses.
If no access settings are specified, the proxy talks to anyone that
connects. If any access settings file are specified, then the proxy
talks only to IP addresses permitted somewhere in this file and not
denied later in this file.
Summary -- if using an ACL:
Client must have permission to receive service.
LAST match in ACL wins.
Default behavior is to deny service.
The syntax for an entry in the Access Control List is:
ACTION SRC_ADDR[/SRC_MASKLEN] [ DST_ADDR[/DST_MASKLEN] ]
Where the individual fields are:
ACTION = permit-access or deny-accessSRC_ADDR = client hostname or dotted IP address
SRC_MASKLEN = number of bits in the subnet mask for the source
DST_ADDR = server or forwarder hostname or dotted IP address
DST_MASKLEN = number of bits in the subnet mask for the target
The field separator (FS) is whitespace (space or tab).
IMPORTANT NOTE: If the junkbuster is using a
forwarder (see below) or a gateway for a particular destination URL, the
DST_ADDR that is examined is the address of the forwarder
or the gateway and NOT the address of the ultimate
target. This is necessary because it may be impossible for the local
Junkbuster to determine the address of the
ultimate target (that's often what gateways are used for).
Here are a few examples to show how the ACL features work:
localhost is OK -- no DST_ADDR implies that
ALL destination addresses are OK:
permit-access localhost
A silly example to illustrate permitting any host on the class-C subnet with
Junkbuster to go anywhere:
permit-access www.junkbusters.com/24
Except deny one particular IP address from using it at all:
deny-access ident.junkbusters.com
You can also specify an explicit network address and subnet mask.
Explicit addresses do not have to be resolved to be used.
permit-access 207.153.200.0/24
A subnet mask of 0 matches anything, so the next line permits everyone.
permit-access 0.0.0.0/0
Note, you cannot say:
permit-access .org
to allow all *.org domains. Every IP address listed must resolve fully.
An ISP may want to provide a Junkbuster that is
accessible by the world and yet restrict use of some of their
private content to hosts on its internal network (i.e. its own subscribers).
Say, for instance the ISP owns the Class-B IP address block 123.124.0.0 (a 16
bit netmask). This is how they could do it:
permit-access 0.0.0.0/0 0.0.0.0/0 # other clients can go anywhere
# with the following exceptions:
deny-access 0.0.0.0/0 123.124.0.0/16 # block all external requests for
# sites on the ISP's network
permit 0.0.0.0/0 www.my_isp.com # except for the ISP's main
# web site
permit 123.124.0.0/16 0.0.0.0/0 # the ISP's clients can go
# anywhere
Note that if some hostnames are listed with multiple IP addresses,
the primary value returned by DNS (via gethostbyname()) is used. Default:
Anyone can access the proxy.
Forwarding
This feature allows chaining of HTTP requests via multiple proxies.
It can be used to better protect privacy and confidentiality when
accessing specific domains by routing requests to those domains
to a special purpose filtering proxy such as lpwa.com. Or to use
a caching proxy to speed up browsing.
It can also be used in an environment with multiple networks to route
requests via multiple gateways allowing transparent access to multiple
networks without having to modify browser configurations.
Also specified here are SOCKS proxies. Junkbuster
SOCKS 4 and SOCKS 4A. The difference is that SOCKS 4A will resolve the target
hostname using DNS on the SOCKS server, not our local DNS client.
The syntax of each line is:
forward target_domain[:port] http_proxy_host[:port]forward-socks4 target_domain[:port] socks_proxy_host[:port] http_proxy_host[:port]forward-socks4a target_domain[:port] socks_proxy_host[:port] http_proxy_host[:port]
If http_proxy_host is ., then requests are not forwarded to a
HTTP proxy but are made directly to the web servers.
Lines are checked in sequence, and the last match wins.
There is an implicit line equivalent to the following, which specifies that
anything not finding a match on the list is to go out without forwarding
or gateway protocol, like so:
forward .* . # implicit
In the following common configuration, everything goes to Lucent's LPWA,
except SSL on port 443 (which it doesn't handle):
forward .* lpwa.com:8000forward :443 .
See the FAQ for instructions on how to automate the login procedure for LPWA.
Some users have reported difficulties related to LPWA's use of
. as the last element of the domain, and have said that this
can be fixed with this:
forward lpwa. lpwa.com:8000
(NOTE: the syntax for specifying target_domain has changed since the
previous paragraph was written -- it will not work now. More information
is welcome.)
In this fictitious example, everything goes via an ISP's caching proxy,
except requests to that ISP:
forward .* caching.myisp.net:8000forward myisp.net .
For the @home network, we're told the forwarding configuration is this:
forward .* proxy:8080
Also, we're told they insist on getting cookies and JavaScript, so you should
add home.com to the cookie file. We consider JavaScript a security risk.
Java need not be enabled.
In this example direct connections are made to all internal
domains, but everything else goes through Lucent's LPWA by way of the
company's SOCKS gateway to the Internet.
forward-socks4 .* lpwa.com:8000 firewall.my_company.com:1080forward my_company.com .
This is how you could set up a site that always uses SOCKS but no forwarders:
forward-socks4a .* . firewall.my_company.com:1080
An advanced example for network administrators:
If you have links to multiple ISPs that provide various special content to
their subscribers, you can configure forwarding to pass requests to the
specific host that's connected to that ISP so that everybody can see all
of the content on all of the ISPs.
This is a bit tricky, but here's an example:
host-a has a PPP connection to isp-a.com. And host-b has a PPP connection to
isp-b.com. host-a can run a Junkbuster proxy with
forwarding like this:
forward .* .forward isp-b.com host-b:8118
host-b can run a Junkbuster proxy with forwarding
like this:
forward .* .forward isp-a.com host-a:8118
Now, anyone on the Internet (including users on host-a
and host-b) can set their browser's proxy to either
host-a or host-b and be able to browse the content on isp-a or isp-b.
Here's another practical example, for University of Kent at
Canterbury students with a network connection in their room, who
need to use the University's Squid web cache.
forward *. ssbcache.ukc.ac.uk:3128 # Use the proxy, except for:
forward .ukc.ac.uk . # Anything on the same domain as us
forward * . # Host with no domain specified
forward 129.12.*.* . # A dotted IP on our /16 network.
forward 127.*.*.* . # Loopback address
forward localhost.localdomain . # Loopback address
forward www.ukc.mirror.ac.uk . # Specific host
If you intend to chain Junkbuster and
squid locally, then chain as
browser -> squid -> junkbuster is the recommended way.
Your squid configuration could then look like this:
# Define junkbuster as parent cache
cache_peer 127.0.0.1 parent 8118 0 no-query
# Define ACL for protocol FTP
acl FTP proto FTP
# Do not forward ACL FTP to junkbuster
always_direct allow FTP
# Do not forward ACL CONNECT (https) to junkbuster
always_direct allow CONNECT
# Forward the rest to junkbuster
never_direct allow all
Windows GUI OptionsJunkbuster has a number of options specific to the
Windows GUI interface:
If activity-animation is set to 1, the
Junkbuster icon will animate when
Junkbuster is active. To turn off, set to 0.
activity-animation 1
If log-messages is set to 1,
Junkbuster will log messages to the console
window:
log-messages 1
If log-buffer-size is set to 1, the size of the log buffer,
i.e. the amount of memory used for the log messages displayed in the
console window, will be limited to log-max-lines (see below).
Warning: Setting this to 0 will result in the buffer to grow infinitely and
eat up all your memory!
log-buffer-size 1log-max-lines is the maximum number of lines held
in the log buffer. See above.
log-max-lines 200
If log-highlight-messages is set to 1,
Junkbuster will highlight portions of the log
messages with a bold-faced font:
log-highlight-messages 1
The font used in the console window:
log-font-name Comic Sans MS
Font size used in the console window:
log-font-size 8show-on-task-bar controls whether or not
Junkbuster will appear as a button on the Task bar
when minimized:
show-on-task-bar 0
If close-button-minimizes is set to 1, the Windows close
button will minimize Junkbuster instead of closing
the program (close with the exit option on the File menu).
close-button-minimizes 1
The hide-console option is specific to the MS-Win console
version of JunkBuster. If this option is used,
Junkbuster will disconnect from and hide the
command console.
#hide-console
The Actions File
The ijb.action file (formerly
actionsfile) is used to define what actions
Junkbuster takes, and thus determines how images,
cookies and various other aspects of HTTP content and transactions are
handled. Images can be anything you want, including ads, banners, or just
some obnoxious image that you would rather not see. Cookies can be accepted
or rejected, or accepted only during the current browser session (i.e.
not written to disk). Changes to ijb.action should
be immediately visible to Junkbuster without
the need to restart.
To determine which actions apply to a request, the URL of the request is
compared to all patterns in this file. Every time it matches, the list of
applicable actions for the URL is incrementally updated. You can trace
this process by visiting http://i.j.b/show-url-info.
The actions file can be edited with a browser by loading
http://i.j.b/, and then select
Edit Actions.
There are four types of lines in this file: comments (begin with a
# character), actions, aliases and patterns, all of which are
explained below, as well as the configuration file syntax that
Junkbuster understands.
URL Domain and Path Syntax
Generally, a pattern has the form <domain>/<path>, where both the
<domain> and <path> part are optional. If you only specify a
domain part, the / can be left out:
www.example.com - is a domain only pattern and will match any request to
www.example.com.
www.example.com/ - means exactly the same.
www.example.com/index.html - matches only the single
document /index.html on www.example.com.
/index.html - matches the document /index.html, regardless of
the domain.
index.html - matches nothing, since it would be
interpreted as a domain name and there is no top-level domain called
.html.
The matching of the domain part offers some flexible options: if the
domain starts or ends with a dot, it becomes unanchored at that end.
For example:
.example.com - matches any domain that ENDS in
.example.com.
www. - matches any domain that STARTS with
www.
Additionally, there are wild-cards that you can use in the domain names
themselves. They work pretty similar to shell wild-cards: *
stands for zero or more arbitrary characters, ? stands for
any single character. And you can define character classes in square
brackets and they can be freely mixed:
ad*.example.com - matches adserver.example.com,
ads.example.com, etc but not sfads.example.com.
*ad*.example.com - matches all of the above, and then some.
.?pix.com - matches www.ipix.com,
pictures.epix.com, a.b.c.d.e.upix.com, etc.
www[1-9a-ez].example.com - matches www1.example.com,
www4.example.com, wwwd.example.com,
wwwz.example.com, etc., but notwwww.example.com.
If Junkbuster was compiled with
pcre support (default), Perl compatible regular expressions
can be used. See the pcre/docs/ directory or man
perlre (also available on http://www.perldoc.com/perl5.6/pod/perlre.html)
for details. A brief discussion of regular expressions is in the
Appendix. For instance:
/.*/advert[0-9]+\.jpe?g - would match a URL from any
domain, with any path that includes advert followed
immediately by one or more digits, then a . and ending in
either jpeg or jpg. So we match
example.com/ads/advert2.jpg, and
www.example.com/ads/banners/advert39.jpeg, but not
www.example.com/ads/banners/advert39.gif (no gifs in the
example pattern).
Please note that matching in the path is case
INSENSITIVE by default, but you can switch to case
sensitive at any point in the pattern by using the
(?-i) switch:
www.example.com/(?-i)PaTtErN.* - will match only
documents whose path starts with PaTtErN in
exactly this capitalization.
Actions
Actions are enabled if preceded with a +, and disabled if
preceded with a -. Actions are invoked by enclosing the
action name in curly braces (e.g. {+some_action}), followed by a list of
URLs to which the action applies. There are three classes of actions:
Boolean (e.g. +/-block):
{+name} # enable this action
{-name} # disable this action
parameterized (e.g. +/-hide-user-agent):
{+name{param}} # enable action and set parameter to param{-name} # disable action
Multi-value (e.g. {+/-add-header{Name: value}}, {+/-wafer{name=value}}):
{+name{param}} # enable action and add parameter param{-name{param}} # remove the parameter param{-name} # disable this action totally
If nothing is specified in this file, no actions are taken.
So in this case JunkBuster would just be a
normal, non-blocking, non-anonymizing proxy. You must specifically
enable the privacy and blocking features you need (although the
provided default ijb.action file will
give a good starting point).
Later defined actions always over-ride earlier ones. For multi-valued
actions, the actions are applied in the order they are specified.
The list of valid Junkbusteractions are:
Add the specified HTTP header, which is not checked for validity.
You may specify this many times to specify many different headers:
+add-header{Name: value}
Block this URL totally.
+block
De-animate all animated GIF images, i.e. reduce them to their last frame.
This will also shrink the images considerably (in bytes, not pixels!). If
the option first is given, the first frame of the animation
is used as the replacement. If last is given, the last frame
of the animation is used instead, which probably makes more sense for most
banner animations, but also has the risk of not showing the entire last
frame (if it is only a delta to an earlier frame).
+deanimate-gifs{last}+deanimate-gifs{first}+downgrade will downgrade HTTP/1.1 client requests to
HTTP/1.0 and downgrade the responses as well. Use this action for servers
that use HTTP/1.1 protocol features that
Junkbuster doesn't handle well yet. HTTP/1.1
is only partially implemented. Default is not to downgrade requests.
+downgrade
Many sites, like yahoo.com, don't just link to other sites. Instead, they
will link to some script on their own server, giving the destination as a
parameter, which will then redirect you to the final target. URLs resulting
from this scheme typically look like:
http://some.place/some_script?http://some.where-else.
Sometimes, there are even multiple consecutive redirects encoded in the
URL. These redirections via scripts make your web browsing more traceable,
since the server from which you follow such a link can see where you go to.
Apart from that, valuable bandwidth and time is wasted, while your browser
ask the server for one redirect after the other. Plus, it feeds the
advertisers.
The +fast-redirects option enables interception of these
requests by Junkbuster, who will cut off all but
the last valid URL in the request and send a local redirect back to your
browser without contacting the remote site.
+fast-redirects
Apply the filters in the section_header
section of the re_filterfile file to the site(s).
Re_filterfile sections are grouped according to like
functionality.
+filter{section_header}
Filter sections that are pre-defined in the supplied
re_filterfile include:
html-annoyances: Get rid of particularly annoying HTML abuse.
js-annoyances: Get rid of particularly annoying JavaScript abuse
no-poups: Kill all popups in JS and HTML
frameset-borders: Give frames a border
webbugs: Squish WebBugs (1x1 invisible GIFs used for user tracking)
no-refresh: Automatic refresh sucks on auto-dialup lines
fun: Text replacements for subversive browsing fun!
nimda: Remove (virus) Nimda code.
banners-by-size: Kill banners by size
crude-parental: Kill all web pages that contain the words "sex" or "warez"
Block any existing X-Forwarded-for header, and do not add a new one:
+hide-forwarded
If the browser sends a From: header containing your e-mail
address, this either completely removes the header (block), or
changes it to the specified e-mail address.
+hide-from{block}+hide-from{spam@sittingduck.xqq}
Don't send the Referer: (sic) header to the web site. You
can block it, forge a URL to the same server as the request (which is
preferred because some sites will not send images otherwise) or set it to a
constant string of your choice.
+hide-referer{block}+hide-referer{forge}+hide-referer{http://nowhere.com}
Alternative spelling of +hide-referer. It has the same
parameters, and can be freely mixed with, +hide-referer.
(referrer is the correct English spelling, however the HTTP
specification has a bug - it requires it to be spelled referer.)
+hide-referrer{...}
Change the User-Agent: header so web servers can't tell your
browser type. Warning! This breaks many web sites. Specify the
user-agent value you want. Example, pretend to be using Netscape on
Linux:
+hide-user-agent{Mozilla (X11; I; Linux 2.0.32 i586)}
Treat this URL as an image. This only matters if it's also +blocked,
in which case a blocked image can be sent rather than a HTML page.
See +image-blocker{} below for the control over what is actually sent.
+image
Decides what to do with URLs that end up tagged with {+block
+image}, e.g an advertizement. There are five options.
-image-blocker will send a HTML blocked page,
usually resulting in a broken image icon.
+image-blocker{logo} will send a JunkBuster
logo image. +image-blocker{blank} will send a 1x1
transparent GIF image. And finally,
+image-blocker{http://xyz.com} will send a HTTP temporary
redirect to the specified image. This has the advantage of the icon being
being cached by the browser, which will speed up the display.
+image-blocker{pattern} will send a checkboard type pattern,
which scales better than the logo (which can get blocky if the browser
enlarges it too much).
+image-blocker{logo}+image-blocker{blank}+image-blocker{pattern}+image-blocker{http://i.j.b/send-banner}
By default (i.e. in the absence of a +limit-connect
action), Junkbuster will only allow CONNECT
requests to port 443, which is the standard port for https as a
precaution.
The CONNECT methods exists in HTTP to allow access to secure websites
(https:// URLs) through proxies. It works very simply: the proxy
connects to the server on the specified port, and then short-circuits
its connections to the client and to the remote proxy.
This can be a big security hole, since CONNECT-enabled proxies can
be abused as TCP relays very easily.
If you want to allow CONNECT for more ports than this, or want to forbid
CONNECT altogether, you can specify a comma separated list of ports and
port ranges (the latter using dashes, with the minimum defaulting to 0 and
max to 65K):
+limit-connect{443} # This is the default and need no be specified.+limit-connect{80,443} # Ports 80 and 443 are OK.+limit-connect{-3, 7, 20-100, 500-} # Port less than 3, 7, 20 to 100 #and above 500 are OK.+no-compression prevents the website from compressing the
data. Some websites do this, which can be a problem for
Junkbuster, since +filter,
+no-popup and +gif-deanimate will not work on
compressed data. This will slow down connections to those websites,
though. Default is nocompression is turned on.
+nocompression
If the website sets cookies, no-cookies-keep will make sure
they are erased when you exit and restart your web browser. This makes
profiling cookies useless, but won't break sites which require cookies so
that you can log in for transactions. Default: on.
+no-cookies-keep
Prevent the website from reading cookies:
+no-cookies-read
Prevent the website from setting cookies:
+no-cookies-set
Filter the website through a built-in filter to disable those obnoxious
JavaScript pop-up windows via window.open(), etc. The two alternative
spellings are equivalent.
+no-popup+no-popups
This action only applies if you are using a jarfile
for saving cookies. It sends a cookie to every site stating that you do not
accept any copyright on cookies sent to you, and asking them not to track
you. Of course, this is a (relatively) unique header they could use to
track you.
+vanilla-wafer
This allows you to add an arbitrary cookie. It can be specified multiple
times in order to add as many cookies as you like.
+wafer{name=value}
The meaning of any of the above is reversed by preceding the action with a
-, in place of the +.
Some examples:
Turn off cookies by default, then allow a few through for specified sites:
# Turn off all persistent cookies
{ +no-cookies-read }
{ +no-cookies-set }
# Allow cookies for this browser session ONLY
{ +no-cookies-keep }
# Exceptions to the above, sites that benefit from persistent cookies
{ -no-cookies-read }
{ -no-cookies-set }
{ -no-cookies-keep }
.javasoft.com
.sun.com
.yahoo.com
.msdn.microsoft.com
.redhat.com
# Alternative way of saying the same thing
{-no-cookies-set -no-cookies-read -no-cookies-keep}
.sourceforge.net
.sf.net
Now turn off fast redirects, and then we allow two exceptions:
# Turn them off!
{+fast-redirects}
# Reverse it for these two sites, which don't work right without it.
{-fast-redirects}
www.ukc.ac.uk/cgi-bin/wac\.cgi\?
login.yahoo.com
Turn on page filtering according to rules in the defined sections
of refilterfile, and make one exception for
sourceforge:
# Run everything through the filter file, using only the
# specified sections:
+filter{html-annoyances} +filter{js-annoyances} +filter{no-popups}\
+filter{webbugs} +filter{nimda} +filter{banners-by-size}
# Then disable filtering of code from sourceforge!
{-filter}
.cvs.sourceforge.net
Now some URLs that we want blocked, ie we won't see them.
Many of these use regular expressions that will expand to match multiple
URLs:
# Blocklist:
{+block}
/.*/(.*[-_.])?ads?[0-9]?(/|[-_.].*|\.(gif|jpe?g))
/.*/(.*[-_.])?count(er)?(\.cgi|\.dll|\.exe|[?/])
/.*/(ng)?adclient\.cgi
/.*/(plain|live|rotate)[-_.]?ads?/
/.*/(sponsor)s?[0-9]?/
/.*/_?(plain|live)?ads?(-banners)?/
/.*/abanners/
/.*/ad(sdna_image|gifs?)/
/.*/ad(server|stream|juggler)\.(cgi|pl|dll|exe)
/.*/adbanners/
/.*/adserver
/.*/adstream\.cgi
/.*/adv((er)?ts?|ertis(ing|ements?))?/
/.*/banner_?ads/
/.*/banners?/
/.*/banners?\.cgi/
/.*/cgi-bin/centralad/getimage
/.*/images/addver\.gif
/.*/images/marketing/.*\.(gif|jpe?g)
/.*/popupads/
/.*/siteads/
/.*/sponsor.*\.gif
/.*/sponsors?[0-9]?/
/.*/advert[0-9]+\.jpg
/Media/Images/Adds/
/ad_images/
/adimages/
/.*/ads/
/bannerfarm/
/grafikk/annonse/
/graphics/defaultAd/
/image\.ng/AdType
/image\.ng/transactionID
/images/.*/.*_anim\.gif # alvin brattli
/ip_img/.*\.(gif|jpe?g)
/rotateads/
/rotations/
/worldnet/ad\.cgi
/cgi-bin/nph-adclick.exe/
/.*/Image/BannerAdvertising/
/.*/ad-bin/
/.*/adlib/server\.cgi
/autoads/
Aliases
Custom actions, known to Junkbuster
as aliases, can be defined by combining other actions.
These can in turn be invoked just like the built-in actions.
Currently, an alias can contain any character except space, tab, =,
{ or }. But please use only a-
z, 0-9, +, and
-. Alias names are not case sensitive, and
must be defined before anything else in the
ijb.actionfile ! And there can only be one set of
aliases defined.
Now let's define a few aliases:
# Useful customer aliases we can use later. These must come first!
{{alias}}
+no-cookies = +no-cookies-set +no-cookies-read
-no-cookies = -no-cookies-set -no-cookies-read
fragile = -block -no-cookies -filter -fast-redirects -hide-referer -no-popups
shop = -no-cookies -filter -fast-redirects
+imageblock = +block +image
#For people who don't like to type too much: ;-)
c0 = +no-cookies
c1 = -no-cookies
c2 = -no-cookies-set +no-cookies-read
c3 = +no-cookies-set -no-cookies-read
#... etc. Customize to your heart's content.
Some examples using our shop and fragile
aliases from above:
# These sites are very complex and require
# minimal interference.
{fragile}
.office.microsoft.com
.windowsupdate.microsoft.com
.nytimes.com
# Shopping sites - still want to block ads.
{shop}
.quietpc.com
.worldpay.com # for quietpc.com
.jungle.com
.scan.co.uk
# These shops require pop-ups
{shop -no-popups}
.dabs.com
.overclockers.co.uk
The Filter File
Any web page can be dynamically modified with the filter file. This
modification can be removal, or re-writing, of any web page content,
including tags and non-visible content. The default filter file is
re_filterfile, located in the config directory.
The included example file is divided into sections. Each section begins
with the FILTER keyword, followed by the identifier
for that section, e.g. FILTER: webbugs. Each section performs
a similar type of filtering, such as html-annoyances.
This file uses regular expressions to alter or remove any string in the
target page. The expressions can only operate on one line at a time. Some
examples from the included default re_filterfile:
Stop web pages from displaying annoying messages in the status bar by
deleting such references:
FILTER: html-annoyances
# New browser windows should be resizeable and have a location and status
# bar. Make it so.
#
s/resizable="?(no|0)"?/resizable=1/ig s/noresize/yesresize/ig
s/location="?(no|0)"?/location=1/ig s/status="?(no|0)"?/status=1/ig
s/scrolling="?(no|0|Auto)"?/scrolling=1/ig
s/menubar="?(no|0)"?/menubar=1/ig
# The <BLINK> tag was a crime!
#
s*<blink>|</blink>**ig
# Is this evil?
#
#s/framespacing="?(no|0)"?//ig
#s/margin(height|width)=[0-9]*//gi
Just for kicks, replace any occurrence of Microsoft with
MicroSuck, and have a little fun with topical buzzwords:
FILTER: fun
s/microsoft(?!.com)/MicroSuck/ig
# Buzzword Bingo:
#
s/industry-leading|cutting-edge|award-winning/<font color=red><b>BINGO!</b></font>/ig
Kill those pesky little web-bugs:
# webbugs: Squish WebBugs (1x1 invisible GIFs used for user tracking)
FILTER: webbugs
s/<img\s+[^>]*?(width|height)\s*=\s*['"]?1\D[^>]*?(width|height)\s*=\s*['"]?1(\D[^>]*?)?>/<!-- Squished WebBug -->/sig
Templates
When Junkbuster displays one of its internal
pages, such as a 404 Not Found error page, it uses the appropriate template.
On Linux, BSD, and Unix, these are located in
/etc/junkbuster/templates by default. These may be
customized, if desired.
Quickstart to Using Junkbuster
Install package, then run and enjoy! JunkBuster
is typically started by specifying the main configuration file to be
used on the command line. Example Unix startup command:
# /usr/sbin/junkbuster /etc/junkbuster/config
An init script is provided for SuSE and Redhat.
For for SuSE: /etc/rc.d/junkbuster start
For RedHat: /etc/rc.d/init.d/junkbuster start
If no configuration file is specified on the command line,
Junkbuster will look for a file named
config in the current directory. Except on Win32 where
it will try config.txt. If no file is specified on the
command line and no default configuration file can be found,
Junkbuster will fail to start.
Be sure your browser is set to use the proxy which is by default at
localhost, port 8118. With Netscape (and
Mozilla), this can be set under Edit
-> Preferences -> Advanced -> Proxies -> HTTP Proxy.
For Internet Explorer: Tools >
Internet Properties -> Connections -> LAN Setting. Then,
check Use Proxy and fill in the appropriate info (Address:
localhost, Port: 8118). Include if HTTPS proxy support too.
The included default configuration files should give a reasonable starting
point, though may be somewhat aggressive in blocking junk. You will probably
want to keep an eye out for sites that require persistent cookies, and add these to
ijb.action as needed. By default, most of these will
be accepted only during the current browser session, until you add them to
the configuration. If you want the browser to handle this instead, you will
need to edit ijb.action and disable this feature. If you
use more than one browser, it would make more sense to let
Junkbuster handle this. In which case, the
browser(s) should be set to accept all cookies.
If a particular site shows problems loading properly, try adding it
to the {fragile} section of
ijb.action. This will turn off most actions for
this site.
Junkbuster is HTTP/1.1 compliant, but not all 1.1
features are as yet implemented. If browsers that support HTTP/1.1 (like
Mozilla or recent versions of I.E.) experience
problems, you might try to force HTTP/1.0 compatibility. For Mozilla, look
under Edit -> Preferences -> Debug -> Networking.
Or set the +downgrade config option in
ijb.action.
After running Junkbuster for a while, you can
start to fine tune the configuration to suit your personal, or site,
preferences and requirements. There are many, many aspects that can
be customized. Actions (as specified in ijb.action)
can be adjusted by pointing your browser to
http://i.j.b/,
and then follow the link to edit the actions list.
(This is an internal page and does not require Internet access.)
In fact, various aspects of Junkbuster
configuration can be viewed from this page, including
current configuration parameters, source code version numbers,
the browser's request headers, and actions that apply
to a given URL. In addition to the ijb.action file
editor mentioned above, Junkbuster can also
be turned on and off from this page.
If you encounter problems, please verify it is a
Junkbuster bug, by disabling
Junkbuster, and then trying the same page.
Also, try another browser if possible to eliminate browser or site
problems. Before reporting it as a bug, see if there is not a configuration
option that is enabled that is causing the page not to load. You can
then add an exception for that page or site. If a bug, please report it to
the developers (see below).
Command Line OptionsJunkBuster may be invoked with the following
command-line options:
--version
Print version info and exit, Unix only.
--help
Print a short usage info and exit, Unix only.
--no-daemon
Don't become a daemon, i.e. don't fork and become process group
leader, don't detach from controlling tty. Unix only.
--pidfile FILE
On startup, write the process ID to FILE. Delete the
FILE on exit. Failiure to create or delete the
FILE is non-fatal. If no FILE
option is given, no PID file will be used. Unix only.
--user USER[.GROUP]
After (optionally) writing the PID file, assume the user ID of
USER, and if included the GID of GROUP. Exit if the
privileges are not sufficient to do so. Unix only.
configfile
If no configfile is included on the command line,
JunkBuster will look for a file named
config in the current directory (except on Win32
where it will look for config.txt instead). Specify
full path to avoid confusion.
Contacting the Developers, Bug Reporting and Feature
Requests
We value your feedback. However, to provide you with the best support,
please note:
Use the Sourceforge support forum to get
help.Submit bugs only thru our Sourceforge bug
forum.
Make sure that the bug has not already been submitted. Please try to
verify that it is a Junkbuster bug, and not
a browser or site bug first. If you are using your own custom configuration,
please try the stock configs to see if the problem is a configuration
related bug. And if not using the latest development snapshot, please
try the latest one. Or even better, CVS sources.Submit feature requests only thru our Sourceforge feature request forum.
For any other issues, feel free to use the mailing lists.
Anyone interested in actively participating in development and related
discussions can join the appropriate mailing list
here.
Archives are available here too.
Copyright and HistoryLicenseInternet Junkbuster is free software; you can
redistribute it and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
details, which is available from the Free Software Foundation,
Inc, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
HistoryJunkbuster was originally written by Anonymous
Coders and Junkbuster's
Corporation, and was released as free open-source software under the
GNU GPL. Stefan
Waldherr made many improvements, and started the SourceForge project to
rekindle development. There are now several active developers contributing.
The last stable release was v2.0.2, which has now grown whiskers ;-).
See alsohttp://sourceforge.net/projects/ijbswahttp://ijbswa.sourceforge.net/http://i.j.b/http://www.junkbusters.com/ht/en/cookies.htmlhttp://www.waldherr.org/junkbuster/http://privacy.net/analyze/http://www.squid-cache.org/AppendixRegular ExpressionsJunkbuster can use regular expressions
in various config files. Assuming support for pcre (Perl
Compatible Regular Expressions) is compiled in, which is the default. Such
configuration directives do not require regular expressions, but they can be
used to increase flexibility by matching a pattern with wild-cards against
URLs.
If you are reading this, you probably don't understand what regular
expressions are, or what they can do. So this will be a very brief
introduction only. A full explanation would require a book ;-)
Regular expressions is a way of matching one character
expression against another to see if it matches or not. One of the
expressions is a literal string of readable characters
(letter, numbers, etc), and the other is a complex string of literal
characters combined with wild-cards, and other special characters, called
meta-characters. The meta-characters have special meanings and
are used to build the complex pattern to be matched against. Perl Compatible
Regular Expressions is an enhanced form of the regular expression language
with backward compatibility.
To make a simple analogy, we do something similar when we use wild-card
characters when listing files with the dir command in DOS.
*.* matches all filenames. The special
character here is the asterisk which matches any and all characters. We can be
more specific and use ? to match just individual
characters. So dir file?.text would match
file1.txt, file2.txt, etc. We are pattern
matching, using a similar technique to regular expressions!
Regular expressions do essentially the same thing, but are much, much more
powerful. There are many more special characters and ways of
building complex patterns however. Let's look at a few of the common ones,
and then some examples:
. - Matches any single character, e.g. a,
A, 4, :, or @.
? - The preceding character or expression is matched ZERO or ONE
times. Either/or.
+ - The preceding character or expression is matched ONE or MORE
times.
* - The preceding character or expression is matched ZERO or MORE
times.
\ - The escape character denotes that
the following character should be taken literally. This is used where one of the
special characters (e.g. .) needs to be taken literally and
not as a special meta-character.
[] - Characters enclosed in brackets will be matched if
any of the enclosed characters are encountered.
() - parentheses are used to group a sub-expression,
or multiple sub-expressions.
| - The bar character works like an
or conditional statement. A match is successful if the
sub-expression on either side of | matches.
s/string1/string2/g - This is used to rewrite strings of text.
string1 is replaced by string2 in this
example.
These are just some of the ones you are likely to use when matching URLs with
Junkbuster, and is a long way from a definitive
list. This is enough to get us started with a few simple examples which may
be more illuminating:
/.*/banners/.* - A simple example
that uses the common combination of . and * to
denote any character, zero or more times. In other words, any string at all.
So we start with a literal forward slash, then our regular expression pattern
(.*) another literal forward slash, the string
banners, another forward slash, and lastly another
.*. We are building
a directory path here. This will match any file with the path that has a
directory named banners in it. The .* matches
any characters, and this could conceivably be more forward slashes, so it
might expand into a much longer looking path. For example, this could match:
/eye/hate/spammers/banners/annoy_me_please.gif, or just
/banners/annoying.html, or almost an infinite number of other
possible combinations, just so it has banners in the path
somewhere.
A now something a little more complex:
/.*/adv((er)?ts?|ertis(ing|ements?))?/ -
We have several literal forward slashes again (/), so we are
building another expression that is a file path statement. We have another
.*, so we are matching against any conceivable sub-path, just so
it matches our expression. The only true literal that must
match our pattern is adv, together with
the forward slashes. What comes after the adv string is the
interesting part.
Remember the ? means the preceding expression (either a
literal character or anything grouped with (...) in this case)
can exist or not, since this means either zero or one match. So
((er)?ts?|ertis(ing|ements?)) is optional, as are the
individual sub-expressions: (er),
(ing|ements?), and the s. The |
means or. We have two of those. For instance,
(ing|ements?), can expand to match either ingORements?. What is being done here, is an
attempt at matching as many variations of advertisement, and
similar, as possible. So this would expand to match just adv,
or advert, or adverts, or
advertising, or advertisement, or
advertisements. You get the idea. But it would not match
advertizements (with a z). We could fix that by
changing our regular expression to:
/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/, which would then match
either spelling.
/.*/advert[0-9]+\.(gif|jpe?g) - Again
another path statement with forward slashes. Anything in the square brackets
[] can be matched. This is using 0-9 as a
shorthand expression to mean any digit one through nine. It is the same as
saying 0123456789. So any digit matches. The +
means one or more of the preceding expression must be included. The preceding
expression here is what is in the square brackets -- in this case, any digit
one through nine. Then, at the end, we have a grouping: (gif|jpe?g).
This includes a |, so this needs to match the expression on
either side of that bar character also. A simple gif on one side, and the other
side will in turn match either jpeg or jpg,
since the ? means the letter e is optional and
can be matched once or not at all. So we are building an expression here to
match image GIF or JPEG type image file. It must include the literal
string advert, then one or more digits, and a .
(which is now a literal, and not a special character, since it is escaped
with \), and lastly either gif, or
jpeg, or jpg. Some possible matches would
include: //advert1.jpg,
/nasty/ads/advert1234.gif,
/banners/from/hell/advert99.jpg. It would not match
advert1.gif (no leading slash), or
/adverts232.jpg (the expression does not include an
s), or /advert1.jsp (jsp is not
in the expression anywhere).
s/microsoft(?!.com)/MicroSuck/i - This is
a substitution. MicroSuck will replace any occurrence of
microsoft. The i at the end of the expression
means ignore case. The (?!.com) means
the match should fail if microsoft is followed by
.com. In other words, this acts like a NOT
modifier. In case this is a hyperlink, we don't want to break it ;-).
We are barely scratching the surface of regular expressions here so that you
can understand the default Junkbuster
configuration files, and maybe use this knowledge to customize your own
installation. There is much, much more that can be done with regular
expressions. Now that you know enough to get started, you can learn more on
your own :/
More reading on Perl Compatible Regular expressions:
http://www.perldoc.com/perl5.6/pod/perlre.htmlJunkBuster's Internal Pages
Since JunkBuster proxies each requested
web page, it is easy for JunkBuster to
trap certain URLs. In this way, we can talk directly to
JunkBuster, and see how it is
configured, see how our rules are being applied, change these
rules and other configuration options, and even turn
JunkBuster's filtering off, all with
a web browser.
The URLs listed below are the special ones that allow direct access
to JunkBuster. Of course,
JunkBuster must be running to access these. If
not, you will get a friendly error message.
Junkbuster main page:
http://ijbswa.sourceforge.net/config/
Alternately, this may be reached at http://i.j.b/,
but this variation may not work as reliably as the above in some
configurations.
Show information about the current configuration: