X-Git-Url: http://www.privoxy.org/gitweb/?p=privoxy.git;a=blobdiff_plain;f=doc%2Fsource%2Fdeveloper-manual.sgml;h=b56d6beeafe381b06b6f06d240887e8a582eb3f1;hp=6b1333fb162bce92497c882ba7292ffe4850a1e6;hb=3faadc6490219a0db4d617918145d540e4affb98;hpb=fdbc2647e7be299218036c36c9179b6f5adb2349
diff --git a/doc/source/developer-manual.sgml b/doc/source/developer-manual.sgml
index 6b1333fb..b56d6bee 100644
--- a/doc/source/developer-manual.sgml
+++ b/doc/source/developer-manual.sgml
@@ -5,7 +5,7 @@
-
+
@@ -19,7 +19,7 @@
Purpose : developer manual
- Copyright (C) 2001-2018 Privoxy Developers https://www.privoxy.org/
+ Copyright (C) 2001-2020 Privoxy Developers https://www.privoxy.org/
See LICENSE.
========================================================================
@@ -39,7 +39,7 @@
Copyright
- &my-copy; 2001-2018 by
+ &my-copy; 2001-2020 by
Privoxy Developers
@@ -110,8 +110,8 @@ Hal.
-->
Privoxy, as an heir to
- Junkbuster, is a Free Software project
- and the code is licensed under the GNU General Public License version 2.
+ Junkbuster, is a Free Software project.
As such, Privoxy development is potentially open
to anyone who has the time, knowledge, and desire to contribute
in any capacity. Our goals are simply to continue the mission,
@@ -128,13 +128,13 @@ Hal.
The first step is to join the privoxy-devel mailing list.
- You can submit your ideas or, even better, patches. Patches are best
- submitted to the Sourceforge tracker set up for this purpose, but
- can be sent to the list for review too.
+ You can submit your ideas or, even better, patches.
+ Patches can also be submitted to the
+ Sourceforge patch tracker.
- You will also need to have a git package installed, which will
- entail having ssh installed as well, in order to access the git repository.
+ You will also need to have a git package installed,
+ in order to access the git repository.
Having the GNU build tools is also going to be important (particularly,
autoconf and gmake).
@@ -157,11 +157,17 @@ Hal.
Access to Git
- The project's Git repository is hosted at the
- Privoxy site.
- The Git repository URL is
- ssh://git@git.privoxy.org:23/git/privoxy.git,
- the central repository is called privoxy, and the
+ The project's Git repository is hosted on the
+ Privoxy webserver.
+ For Privoxy team members with push privileges the Git repository URL is
+ ssh://git@git.privoxy.org:23/git/privoxy.git.
+
+
+ Contributors without push privileges can
+ git clone https://www.privoxy.org/git/privoxy.git
.
+
+
+ The central repository is called privoxy, and the
source branch is called master. Subfolders exist
within the project for target-dependent build and packaging tools, each
including the name of the target operating system in their name (e.g.
@@ -182,6 +188,60 @@ Hal.
such changes are fully tested ought they be pushed back to the central
repository master branch.
+
+ Before pushing stuff, please rebase it on a current master so we get
+ an uncomplicated commit history. Avoid merges where possible.
+
+
+ Here's an example git sesssion that should result in a merge-free history:
+
+
+fk@t520 ~/git/privoxy $git checkout master
+Switched to branch 'master'
+Your branch is up to date with 'origin/master'.
+# Make sure you have the latest changes
+fk@t520 ~/git/privoxy $git pull
+Already up to date.
+# Create a local banch for changes
+fk@t520 ~/git/privoxy $git checkout -b local-branch
+Switched to a new branch 'local-branch'
+# Create some change
+fk@t520 ~/git/privoxy $gmake dok dok-tidy
+[...]
+# Review your change
+fk@t520 ~/git/privoxy $git diff
+[...]
+# Commit your changes if they look goood
+fk@t520 ~/git/privoxy $git commit -m "developer-manual: Regenerate" doc/webserver/
+[local-branch 1abb7316] developer-manual: Regenerate
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+# Review your commit
+fk@t520 ~/git/privoxy $git show
+[...]
+# Go to the master branch
+fk@t520 ~/git/privoxy $git checkout master
+Switched to branch 'master'
+Your branch is up to date with 'origin/master'.
+# Make sure you are still in sync
+fk@t520 ~/git/privoxy $git pull
+[...]
+Already up to date.
+# Apply the commit you made to the local-branch
+fk@t520 ~/git/privoxy $git cherry-pick local-branch
+[master 046e85e2] developer-manual: Regenerate
+ Date: Tue Dec 15 05:10:07 2020 +0100
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+# Make sure the history looks as expected
+fk@t520 ~/git/privoxy $git log -p
+# Finally push your change to the Privoxy repository
+fk@t520 ~/git/privoxy $git push
+[...]
+# Go back to the local branch
+fk@t520 ~/git/privoxy $git checkout local-branch
+# Rebase on top of master and continue hacking
+fk@t520 ~/git/privoxy $git rebase master
+Successfully rebased and updated refs/heads/local-branch.
+
- Privoxy Custom Entities
+ Privoxy Custom Entities
Privoxy documentation is using
a number of customized entities
to facilitate
@@ -651,6 +709,7 @@ Hal.
the finished doc at that point.
+
Commonly used internal entities
:
@@ -1963,6 +2022,48 @@ Install the rpm. Any error messages?
+
+ Testing with <application>Privoxy-Regression-Test</application>
+
+ If you compiled, packaged or merely installed Privoxy, it is recommended to run
+ Privoxy-Regression-Test to verify that at least
+ the tested parts of Privoxy are working as expected.
+
+
+ This is actually pretty easy. For details, please see
+ perldoc privoxy-regression-test.pl.
+
+
+ Here is an example of what Privoxy-Regression-Test can do for you:
+
+
+# Run all the tests
+fk@t520 ~ $privoxy-regression-test.pl
+2020-12-14 12:16:32: Asking Privoxy for the number of action files available ...
+2020-12-14 12:16:32: Gathering regression tests from 9 action file(s) delivered by Privoxy 3.0.30.
+2020-12-14 12:16:32: Executing regression tests ...
+2020-12-14 12:16:41: Ooops. Expected removal but: 'Referer: https://p.p/' is still there.
+2020-12-14 12:16:41: Failure for test 785. Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-block}'
+2020-12-14 12:16:41: Ooops. Got: 'Referer: https://p.p/' while expecting: 'Referer: http://p.p/'
+2020-12-14 12:16:41: Failure for test 791. Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-forge}'
+2020-12-14 12:16:44: Executed 1087 regression tests. Skipped 115. 1085 successes, 2 failures.
+# Repeat one of the failing tests and get a curl command to quickly reproduce the problem
+# without causing too much log noise.
+fk@t520 ~ $privoxy-regression-test.pl --test-number 785 --verbose --debug 4
+2020-12-14 12:17:55: Asking Privoxy for the number of action files available ...
+[...]
+2020-12-14 12:17:56: Executing regression tests ...
+2020-12-14 12:17:56: Executing: curl --include -H 'Proxy-Connection:' -H 'Connection: close' -s -S --user-agent 'Privoxy-Regression-Test 0.7.2' --max-time '5' --globoff -H 'X-Privoxy-Control: hide-referrer{conditional-block}' -H 'Referer: https://p.p/' http://p.p/show-request 2>&1
+2020-12-14 12:17:56: Ooops. Expected removal but: 'Referer: https://p.p/' is still there.
+2020-12-14 12:17:56: Failure for test 785 (0/13/5). Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-block}'
+2020-12-14 12:17:56: Executed 1 regression tests. Skipped 1201. 0 successes, 1 failures.
+
+
+ Use the if the --privoxy-address option if the
+ http_proxy environment variable isn't configured.
+
+
+
Fuzzing Privoxy
@@ -2045,8 +2146,9 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla.
The following programs are required to follow this process:
- ncftpput (ncftp), scp, ssh (ssh),
- gmake (GNU's version of make), autoconf, cvs.
+ ssh,
+ gmake (GNU's version of make), autoconf, git,
+ a web browser.
@@ -2063,12 +2165,13 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla.
X, the version major, is rarely ever changed. It is increased by one if
turning a development branch into stable substantially changes the functionality,
user interface or configuration syntax. Majors 1 and 2 were
- Junkbuster, and 3 will be the first stable
+ Junkbuster, and 3 is the first stable
Privoxy release.
+
Y, the version minor, represents the branch within the major version.
At any point in time, there are two branches being maintained:
The stable branch, with an even minor, say, 2N, in which no functionality is
@@ -2086,13 +2189,15 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla.
Z, the point or sub version, represents a release of the software within a branch.
- It is therefore incremented immediately before each code freeze.
+ It is therefore incremented immediately after each software release.
+
The point version is reset to zero when the minor changes.
@@ -2135,32 +2240,39 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla.
- Before the Release: Freeze
+ Before the Release
The following must be done by one of the
developers prior to each new release.
-
-
-
+
+
+
Make sure that everybody who has worked on the code in the last
couple of days has had a chance to yell no!
in case
they have pending changes/fixes in their pipelines. Announce the
freeze so that nobody will interfere with last minute changes.
-
+
+
+
+
+ Update the code status (CODE_STATUS="xxx") in configure.in to one of
+ "alpha", "beta" or "stable".
+
- Increment the version number (point from odd to even in development
- branches!) in configure.in and update the code
- status (CODE_STATUS="xxx") to one of "alpha", "beta" or "stable".
- Rebuild configure and GNUMakefile to make sure the updated values are
- being used.
+ Rebuild configure and GNUMakefile to make sure the updated values are being used.
+
+
+$ autoheader && autoconf # rebuild configure
+$ ./configure # rebuild GNUmakefile
+
- Use the dok-release target to update the sgml documentation source files.
+ make dok-release to update the sgml documentation source files.
@@ -2178,15 +2290,70 @@ for-privoxy-version=3.0.11
- All documentation should be rebuild after the version bump.
+ Create the change log:
+
+
+ $ git tag
+ # to see the tags
+ $ git log [last release tag]..master > /tmp/log
+ # get the commit log since the last release
+ $ utils/makeChangeLog /tmp/log > /tmp/change.log
+ # reformat the commit log
+
+
+ Edit /tmp/change.log to remove trivial
+ changes and group the changes under general headings like:
+
+
+- Bug fixes:
+- Action file improvements:
+- Filter file improvements:
+- General improvements:
+- Documentation improvements:
+- Build system improvements:
+- Code cleanups:
+- Privoxy-Log-Parser:
+- Privoxy-Regression-Test:
+
+
+ Add the contents of /tmp/change.log to the
+ start of ChangeLog and re-create
+ doc/source/changelog.sgml:
+
+
+ $ utils/changelog2doc.pl /tmp/change.log >| doc/source/changelog.sgml
+
+
+
+
+ All developers should look at the ChangeLog and
+ make sure noteworthy changes are referenced.
+
+
+
+
+ All documentation should be rebuilt:
+
+ $ make man
+ $ make dok
+ $ make dok-man
+ $ make dok-tidy
+ $ make config-file
+
Finished docs should be then be committed to Git (for those
without the ability to build these). Some docs may require
rather obscure processing tools. config,
the man page (and the html version of the man page)
fall in this category. README, the man page, AUTHORS, and config
should all also be committed to Git for other packagers. The
- formal docs should be uploaded to the webserver. See the
- Section "Updating the webserver" in this manual for details.
+ formal docs should be uploaded to the webserver. See the section
+ "Updating the webserver"
+ in this manual for details.
+
+
+
+
+ Commit all files that were changed in the above steps!
@@ -2198,36 +2365,18 @@ for-privoxy-version=3.0.11
(this is in addition to the main User Manual
link from the main page since we need to keep manuals for various
versions available). The CGI pages will link to something like
- http://privoxy.org/$(VERSION)/user-manual/. This
- will need to be updated for each new release. There is no Makefile
- target for this at this time!!! It needs to be done manually.
-
-
-
-
- All developers should look at the ChangeLog and
- make sure noteworthy changes are referenced.
-
-
-
-
- Commit all files that were changed in the above steps!
+ https://www.privoxy.org/$(VERSION)/user-manual/. This
+ needs to be updated for each new release and is done with the
+ webserver
target.
Tag all files in Git with the version number with
- cvs tag v_X_Y_Z
.
+ git tag -s v_X_Y_Z
.
Don't use vX_Y_Z, ver_X_Y_Z, v_X.Y.Z (won't work) etc.
-
-
- If the release was in a development branch, increase the point version
- from even to odd (X.Y.(Z+1)) again in configure.in and
- commit your change.
-
-
On the webserver, copy the user manual to a new top-level directory
@@ -2258,8 +2407,9 @@ for-privoxy-version=3.0.11
mkdir dist # delete or choose different name if it already exists
cd dist
- cvs -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa login
- cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa export -r v_X_Y_Z current
+ git clone https://www.privoxy.org/git/privoxy.git
+ cd privoxy
+ git checkout v_X_Y_Z
@@ -2340,9 +2490,7 @@ for-privoxy-version=3.0.11
Also suggested: Developer Manual
(doc/webserver/developer-manual) and ChangeLog
(top-level directory). FAQ and the manuals are
- HTML docs. There are also text versions in
- doc/text/ which could conceivably also be
- included.
+ HTML docs.
The documentation has been designed such that the manuals are linked
@@ -2372,7 +2520,8 @@ for-privoxy-version=3.0.11
- Other configuration files (default.action and
+ Other configuration files (default.action,
+ regression-tests.action and
default.filter) should be installed as the new
defaults, but all previously installed configuration files should be
preserved as backups. This is just good manners :-) These files are
@@ -2407,10 +2556,9 @@ for-privoxy-version=3.0.11
First, make sure that you have freshly exported the right
version into an empty directory. (See "Building and releasing
- packages" above). Then run:
+ packages" above). Then run from that directory:
- cd current
autoheader && autoconf && ./configure
@@ -2419,17 +2567,6 @@ for-privoxy-version=3.0.11
make tarball-dist
-
- To upload the package to Sourceforge, simply issue
-
-
- make tarball-upload
-
-
- Go to the displayed URL and release the file publicly on Sourceforge.
- For the change log field, use the relevant section of the
- ChangeLog file.
-
SuSE, Conectiva or Red Hat RPM
@@ -2450,7 +2587,7 @@ for-privoxy-version=3.0.11
be reset to one if this is the first RPM for
dist which is built from version
X.Y.Z. Check the
- file
+ file
list if unsure. Else, it must be set to the highest already available RPM
release number for that version plus one.
@@ -2467,67 +2604,6 @@ for-privoxy-version=3.0.11
make dist-dist
-
- To upload the package to Sourceforge, simply issue
-
-
- make dist-upload rpm_packagerev
-
-
- where rpm_packagerev is the
- RPM release number as determined above.
- Go to the displayed URL and release the file publicly on Sourceforge.
- Use the release notes and change log from the source tarball package.
-
-
-
- OS/2
-
- First, make sure that you have freshly exported the right
- version into an empty directory. (See "Building and releasing
- packages" above). Then get the OS/2 Setup module:
-
-
- cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co os2setup
-
-
- You will need a mix of development tools.
- The main compilation takes place with IBM Visual Age C++.
- Some ancillary work takes place with GNU tools, available from
- various sources like hobbes.nmsu.edu.
- Specificially, you will need autoheader,
- autoconf and sh tools.
- The packaging takes place with WarpIN, available from various sources, including
- its home page: xworkplace.
-
-
- Change directory to the os2setup directory.
- Edit the os2build.cmd file to set the final executable filename.
- For example,
-
-
- installExeName='privoxyos2_setup_X.Y.Z.exe'
-
-
- Next, edit the IJB.wis file so the release number matches
- in the PACKAGEID section:
-
-
- PACKAGEID="Privoxy Team\Privoxy\Privoxy Package\X\Y\Z"
-
-
- You're now ready to build. Run:
-
-
- os2build
-
-
- You will find the WarpIN-installable executable in the
- ./files directory. Upload this anonymously to
- uploads.sourceforge.net/incoming, create a release
- for it, and you're done. Use the release notes and Change Log from the
- source tarball package.
-
Solaris
@@ -2562,33 +2638,50 @@ for-privoxy-version=3.0.11
- Windows
+ Windows
+
- Use the
- Cygwin Time Machine to install the last 1.5 version of Cygwin.
- Run the following commands from within the Cygwin 1.5 bash shell.
+ Note that the docbook generated files might need some hand editing,
+ so the Windows build makefile does not rebuild the docs.
+
First, make sure that you have freshly exported the right
version into an empty directory. (See "Building and releasing
- packages" above). Then get the Windows setup module:
+ packages" above).
+
-
- cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co winsetup
-
Then you can build the package. This is fully automated, and is
- controlled by winsetup/GNUmakefile.
+ controlled by windows/GNUmakefile.
All you need to do is:
- cd winsetup
+ cd windows
make
Now you can manually rename privoxy_setup.exe to
- privoxy_setup_X_Y_Z.exe, and upload it to
- SourceForge. When releasing the package on SourceForge, use the release notes
+ privoxy_setup_X.Y.Z.exe, and the build
+ directory to privoxy_X.Y.Z.
+ Create a .zip file of the newly renamed privoxy_X.Y.Z directory,
+ GPG sign the installer and zip file,
+
+
+ $ gpg --armor --detach --sign privoxy_setup_X.Y.Z.exe
+ $ gpg --armor --detach --sign privoxy_X.Y.Z.zip
+
+
+ and upload the files to SourceForge.
+
+
+
+ When releasing the package on SourceForge, use the release notes
and Change Log from the source tarball package.
@@ -2613,12 +2706,8 @@ for-privoxy-version=3.0.11
This will create
../privoxy_&p-version;-&p-status;-1_i386.deb
- which can be uploaded. To upload the package to Sourceforge, simply
- issue
+ which can be uploaded.
-
- make debian-upload
-
Mac OS X
@@ -2628,15 +2717,15 @@ for-privoxy-version=3.0.11
packages" above).
- There are three modules available in the Git repository for use on Mac
+ There are three modules available in the CVS repository backups for use on Mac
OS X, though technically only two of them generate a release (the other
can be used to install from source).
- OSXPackageBuilder module
+ OSXPackageBuilder module (Documentation out of date)
The OSXPackageBuilder module generates OS X installer packages
- supporting all Macs running OS X 10.4 and above. Obtain it from Git as
+ supporting all Macs running OS X 10.4 and above. Obtain it from CVS as
follows into a folder parallel to the exported privoxy source:
@@ -2656,14 +2745,14 @@ for-privoxy-version=3.0.11
- osxsetup module (DEPRECATED)
+ osxsetup module (DEPRECATED) (Documentation out of date)
This module is deprecated since the installer it generates
places all Privoxy files in one folder in a non-standard location, and
supports only Intel Macs running OS X 10.6 or higher.
- Check out the module from Git as follows into a folder parallel to the
+ Check out the module from CVS as follows into a folder parallel to the
exported privoxy source:
@@ -2701,13 +2790,13 @@ for-privoxy-version=3.0.11
- macsetup module
+ macsetup module (Documentation out of date)
The macsetup module is ideal if you wish to build and install Privoxy
from source on a single machine.
- Check out the module from Git as follows into a folder parallel to the
+ Check out the module from CVS as follows into a folder parallel to the
exported privoxy source:
@@ -2733,38 +2822,10 @@ for-privoxy-version=3.0.11
Uploading and Releasing Your Package
After the package is ready, it is time to upload it
- to SourceForge, and go through the release steps. The upload
- is done via FTP:
-
-
-
-
- Upload to: ftp://upload.sourceforge.net/incoming
-
-
-
-
- user: anonymous
-
-
-
-
- password: ijbswa-developers@lists.sourceforge.net
-
-
-
-
- Or use the make targets as described above.
-
-
- Once this done go to https://sourceforge.net/project/admin/editpackages.php?group_id=11118,
- making sure you are logged in. Find your target platform in the
- second column, and click Add Release. You will
- then need to create a new release for your package, using the format
- of $VERSION ($CODE_STATUS), e.g. &p-version;
- (beta).
+ and go through the release steps. The upload
+ is done at
+ SourceForge
+ after logging in.
Now just follow the prompts. Be sure to add any appropriate Release
@@ -2789,15 +2850,44 @@ for-privoxy-version=3.0.11
After the Release
When all (or: most of the) packages have been uploaded and made available,
- send an email to the announce
- mailing list, Subject: "Version X.Y.Z available for download". Be sure to
+ send an email to the
+ announce mailing
+ list, Subject: "Version X.Y.Z available for download". Be sure to
include the
- download
- location, the release notes and the Changelog. Also, post an
+
+ download location, the release notes and the Changelog. Also, post an
updated News item on the project page Sourceforge, and update the Home
page and docs linked from the Home page (see below). Other news sites
and release oriented sites, such as Freshmeat, should also be notified.
+
+ Then update the source code for the next version to be released:
+
+
+
+
+ Increment the version number and change the code status to "UNRELEASED"
+ in configure.in
+
+
+
+
+ Rebuild configure (autoheader && autoconf
)
+ and GNUMakefile (./configure
)
+
+
+
+
+ make dok-release
to update the sgml documentation source files.
+
+
+
+
+ Commit all your changes.
+
+
+
+
@@ -2843,9 +2933,9 @@ for-privoxy-version=3.0.11
make webserver
- This will do the upload to the
- webserver (www.privoxy.org) and ensure all files and directories
- there are group writable.
+ This will do the upload to the SourceForge webserver (which is manually
+ syncronized with www.privoxy.org)
+ and ensure all files and directories there are group writable.
Please do NOT use any other means of transferring