X-Git-Url: http://www.privoxy.org/gitweb/?p=privoxy.git;a=blobdiff_plain;f=doc%2Fsource%2Fdeveloper-manual.sgml;h=79e0a7e02350f3a54d00ace8bd9eeaefda232b29;hp=4bfde091b1c682c0f686af004205731318ae5845;hb=34a6a841e529579e2be4457ea0d4cb1befbc840a;hpb=db4dcd1706dc2aef86138452feb37e1496aa4ecd diff --git a/doc/source/developer-manual.sgml b/doc/source/developer-manual.sgml index 4bfde091..79e0a7e0 100644 --- a/doc/source/developer-manual.sgml +++ b/doc/source/developer-manual.sgml @@ -5,7 +5,7 @@ - + @@ -110,8 +110,8 @@ Hal. --> Privoxy, as an heir to - Junkbuster, is a Free Software project - and the code is licensed under the GNU General Public License version 2. + Junkbuster, is a Free Software project. As such, Privoxy development is potentially open to anyone who has the time, knowledge, and desire to contribute in any capacity. Our goals are simply to continue the mission, @@ -128,13 +128,13 @@ Hal. The first step is to join the privoxy-devel mailing list. - You can submit your ideas or, even better, patches. Patches are best - submitted to the Sourceforge tracker set up for this purpose, but - can be sent to the list for review too. + You can submit your ideas or, even better, patches. + Patches can also be submitted to the + Sourceforge patch tracker. - You will also need to have a git package installed, which will - entail having ssh installed as well, in order to access the git repository. + You will also need to have a git package installed, + in order to access the git repository. Having the GNU build tools is also going to be important (particularly, autoconf and gmake). @@ -157,8 +157,8 @@ Hal. Access to Git - The project's Git repository is hosted at the - Privoxy website. + The project's Git repository is hosted on the + Privoxy webserver. For Privoxy team members with push privileges the Git repository URL is ssh://git@git.privoxy.org:23/git/privoxy.git. @@ -188,6 +188,60 @@ Hal. such changes are fully tested ought they be pushed back to the central repository master branch. + + Before pushing stuff, please rebase it on a current master so we get + an uncomplicated commit history. Avoid merges where possible. + + + Here's an example git sesssion that should result in a merge-free history: + + +fk@t520 ~/git/privoxy $git checkout master +Switched to branch 'master' +Your branch is up to date with 'origin/master'. +# Make sure you have the latest changes +fk@t520 ~/git/privoxy $git pull +Already up to date. +# Create a local banch for changes +fk@t520 ~/git/privoxy $git checkout -b local-branch +Switched to a new branch 'local-branch' +# Create some change +fk@t520 ~/git/privoxy $gmake dok dok-tidy +[...] +# Review your change +fk@t520 ~/git/privoxy $git diff +[...] +# Commit your changes if they look goood +fk@t520 ~/git/privoxy $git commit -m "developer-manual: Regenerate" doc/webserver/ +[local-branch 1abb7316] developer-manual: Regenerate + 1 file changed, 2 insertions(+), 2 deletions(-) +# Review your commit +fk@t520 ~/git/privoxy $git show +[...] +# Go to the master branch +fk@t520 ~/git/privoxy $git checkout master +Switched to branch 'master' +Your branch is up to date with 'origin/master'. +# Make sure you are still in sync +fk@t520 ~/git/privoxy $git pull +[...] +Already up to date. +# Apply the commit you made to the local-branch +fk@t520 ~/git/privoxy $git cherry-pick local-branch +[master 046e85e2] developer-manual: Regenerate + Date: Tue Dec 15 05:10:07 2020 +0100 + 1 file changed, 2 insertions(+), 2 deletions(-) +# Make sure the history looks as expected +fk@t520 ~/git/privoxy $git log -p +# Finally push your change to the Privoxy repository +fk@t520 ~/git/privoxy $git push +[...] +# Go back to the local branch +fk@t520 ~/git/privoxy $git checkout local-branch +# Rebase on top of master and continue hacking +fk@t520 ~/git/privoxy $git rebase master +Successfully rebased and updated refs/heads/local-branch. + - Privoxy Custom Entities + Privoxy Custom Entities Privoxy documentation is using a number of customized entities to facilitate @@ -1970,6 +2022,49 @@ Install the rpm. Any error messages? + + Testing with <application>Privoxy-Regression-Test</application> + + If you compiled, packaged or merely installed Privoxy, it is recommended to run + Privoxy-Regression-Test to verify that at least + the tested parts of Privoxy are working as expected. + + + This is actually pretty easy. For details, please see + perldoc privoxy-regression-test.pl. + + + Here is an example of what Privoxy-Regression-Test can do for you: + + +# Run all the tests +fk@t520 ~ $privoxy-regression-test.pl +2020-12-14 12:16:32: Asking Privoxy for the number of action files available ... +2020-12-14 12:16:32: Gathering regression tests from 9 action file(s) delivered by Privoxy 3.0.30. +2020-12-14 12:16:32: Executing regression tests ... +2020-12-14 12:16:41: Ooops. Expected removal but: 'Referer: https://p.p/' is still there. +2020-12-14 12:16:41: Failure for test 785. Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-block}' +2020-12-14 12:16:41: Ooops. Got: 'Referer: https://p.p/' while expecting: 'Referer: http://p.p/' +2020-12-14 12:16:41: Failure for test 791. Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-forge}' +2020-12-14 12:16:44: Executed 1087 regression tests. Skipped 115. 1085 successes, 2 failures. +# Repeat one of the failing tests and get a curl command to quickly reproduce the problem +# without causing too much log noise. +fk@t520 ~ $privoxy-regression-test.pl --test-number 785 --verbose --debug 4 +2020-12-14 12:17:55: Asking Privoxy for the number of action files available ... +[...] +2020-12-14 12:17:56: Executing regression tests ... +2020-12-14 12:17:56: Executing: curl --include -H 'Proxy-Connection:' -H 'Connection: close' -s -S --user-agent 'Privoxy-Regression-Test 0.7.2' --max-time '5' --globoff -H 'X-Privoxy-Control: hide-referrer{conditional-block}' -H 'Referer: https://p.p/' http://p.p/show-request 2>&1 +2020-12-14 12:17:56: Ooops. Expected removal but: 'Referer: https://p.p/' is still there. +2020-12-14 12:17:56: Failure for test 785 (0/13/5). Header 'Referer: https://p.p/' and tag 'hide-referrer{conditional-block}' +2020-12-14 12:17:56: Executed 1 regression tests. Skipped 1201. 0 successes, 1 failures. + + + Use the if the --privoxy-address option if the + http_proxy environment variable isn't configured and you don't want + to use the default (http://127.0.0.1:8118/). + + + Fuzzing Privoxy @@ -2052,8 +2147,9 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla. The following programs are required to follow this process: - ncftpput (ncftp), scp, ssh (ssh), - gmake (GNU's version of make), autoconf, cvs. + ssh, + gmake (GNU's version of make), autoconf, git, + a web browser. @@ -2070,7 +2166,7 @@ $ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla. X, the version major, is rarely ever changed. It is increased by one if turning a development branch into stable substantially changes the functionality, user interface or configuration syntax. Majors 1 and 2 were - Junkbuster, and 3 will be the first stable + Junkbuster, and 3 is the first stable Privoxy release. @@ -2200,7 +2296,7 @@ for-privoxy-version=3.0.11 $ git tag # to see the tags - $ git log [last release tag]..HEAD > /tmp/log + $ git log [last release tag]..master > /tmp/log # get the commit log since the last release $ utils/makeChangeLog /tmp/log > /tmp/change.log # reformat the commit log @@ -2270,15 +2366,15 @@ for-privoxy-version=3.0.11 (this is in addition to the main User Manual link from the main page since we need to keep manuals for various versions available). The CGI pages will link to something like - http://privoxy.org/$(VERSION)/user-manual/. This - will need to be updated for each new release. There is no Makefile - target for this at this time!!! It needs to be done manually. + https://www.privoxy.org/$(VERSION)/user-manual/. This + needs to be updated for each new release and is done with the + webserver target. Tag all files in Git with the version number with - cvs tag v_X_Y_Z. + git tag -s v_X_Y_Z. Don't use vX_Y_Z, ver_X_Y_Z, v_X.Y.Z (won't work) etc. @@ -2312,8 +2408,9 @@ for-privoxy-version=3.0.11 mkdir dist # delete or choose different name if it already exists cd dist - cvs -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa login - cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa export -r v_X_Y_Z current + git clone https://www.privoxy.org/git/privoxy.git + cd privoxy + git checkout v_X_Y_Z @@ -2394,9 +2491,7 @@ for-privoxy-version=3.0.11 Also suggested: Developer Manual (doc/webserver/developer-manual) and ChangeLog (top-level directory). FAQ and the manuals are - HTML docs. There are also text versions in - doc/text/ which could conceivably also be - included. + HTML docs. The documentation has been designed such that the manuals are linked @@ -2426,7 +2521,8 @@ for-privoxy-version=3.0.11 - Other configuration files (default.action and + Other configuration files (default.action, + regression-tests.action and default.filter) should be installed as the new defaults, but all previously installed configuration files should be preserved as backups. This is just good manners :-) These files are @@ -2461,10 +2557,9 @@ for-privoxy-version=3.0.11 First, make sure that you have freshly exported the right version into an empty directory. (See "Building and releasing - packages" above). Then run: + packages" above). Then run from that directory: - cd current autoheader && autoconf && ./configure @@ -2473,147 +2568,6 @@ for-privoxy-version=3.0.11 make tarball-dist - - To upload the package to Sourceforge, simply issue - - - make tarball-upload - - - Go to the displayed URL and release the file publicly on Sourceforge. - For the change log field, use the relevant section of the - ChangeLog file. - - - - SuSE, Conectiva or Red Hat RPM - - In following text, replace dist - with either rh for Red Hat or suse for SuSE. - - - First, make sure that you have freshly exported the right - version into an empty directory. (See "Building and releasing - packages" above). - - - As the only exception to not changing anything after export from Git, - now examine the file privoxy-dist.spec - and make sure that the version information and the RPM release number are - correct. The RPM release numbers for each version start at one. Hence it must - be reset to one if this is the first RPM for - dist which is built from version - X.Y.Z. Check the - file - list if unsure. Else, it must be set to the highest already available RPM - release number for that version plus one. - - - Then run: - - - cd current - autoheader && autoconf && ./configure - - - Then do - - - make dist-dist - - - To upload the package to Sourceforge, simply issue - - - make dist-upload rpm_packagerev - - - where rpm_packagerev is the - RPM release number as determined above. - Go to the displayed URL and release the file publicly on Sourceforge. - Use the release notes and change log from the source tarball package. - - - - OS/2 - - First, make sure that you have freshly exported the right - version into an empty directory. (See "Building and releasing - packages" above). Then get the OS/2 Setup module: - - - cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co os2setup - - - You will need a mix of development tools. - The main compilation takes place with IBM Visual Age C++. - Some ancillary work takes place with GNU tools, available from - various sources like hobbes.nmsu.edu. - Specificially, you will need autoheader, - autoconf and sh tools. - The packaging takes place with WarpIN, available from various sources, including - its home page: xworkplace. - - - Change directory to the os2setup directory. - Edit the os2build.cmd file to set the final executable filename. - For example, - - - installExeName='privoxyos2_setup_X.Y.Z.exe' - - - Next, edit the IJB.wis file so the release number matches - in the PACKAGEID section: - - - PACKAGEID="Privoxy Team\Privoxy\Privoxy Package\X\Y\Z" - - - You're now ready to build. Run: - - - os2build - - - You will find the WarpIN-installable executable in the - ./files directory. Upload this anonymously to - uploads.sourceforge.net/incoming, create a release - for it, and you're done. Use the release notes and Change Log from the - source tarball package. - - - - Solaris - - Login to Sourceforge's compilefarm via ssh: - - - ssh cf.sourceforge.net - - - Choose the right operating system (not the Debian one). - When logged in, make sure that you have freshly exported the right - version into an empty directory. (See "Building and releasing - packages" above). Then run: - - - cd current - autoheader && autoconf && ./configure - - - Then run - - - gmake solaris-dist - - - which creates a gzip'ed tar archive. Sadly, you cannot use make - solaris-upload on the Sourceforge machine (no ncftpput). You now have - to manually upload the archive to Sourceforge's ftp server and release - the file publicly. Use the release notes and Change Log from the - source tarball package. - Windows @@ -2670,13 +2624,13 @@ for-privoxy-version=3.0.11 right version into an empty directory. (See "Building and releasing packages" above). Then add a log entry to debian/changelog, if it is not - already there, for example by running: + already there, for example by running (from the debian directory): debchange -v &p-version;-&p-status;-1 "New upstream version" - Then, run: + Then, run (from the root directory): dpkg-buildpackage -rfakeroot -us -uc -b @@ -2684,12 +2638,172 @@ for-privoxy-version=3.0.11 This will create ../privoxy_&p-version;-&p-status;-1_i386.deb - which can be uploaded. To upload the package to Sourceforge, simply - issue + which can be uploaded. - - make debian-upload - + + Using git-buildpackage we start with a clone of the last Debian version: + + + gbp clone https://salsa.debian.org/debian/privoxy.git + cd privoxy + + + or if the repository is already there + + + cd privoxy + gbp pull + + + Now import the newly released upstream tarball via debian/watch file: + + + gbp import-orig --uscan + + + Next update all Debian quilt patches to the new version: + + + while quilt push; do quilt refresh; done + + + If some patch is no longer required (because it is already merged + upstream), it can be removed using + + + quilt delete XX_patchname.patch + git rm debian/patches/XX_patchname.patch + + + If the patch needs modification, you can apply, edit and update it with + + + quilt push -f + quilt edit some_file + quilt refresh + + + until + + + while quilt push; do quilt refresh; done + + + succeeds. Then you can + + + quilt pop -a + + + Now add a new entry to the debian/changelog representing the new + version: + + + dch -v &p-version;-1 + + + and describe what you did before and don't forget to git commit all + changes. + + + Now you can build the package on the local machine using + + + gbp buildpackage -us -uc + + + You should check for warnings using + + + lintian -iI ../build-area/privoxy_&p-version;-1_amd64.changes + + + Maybe rebuild the package in different defined cowbuilder environments + like + + + sudo cowbuilder --build --basepath /var/cache/pbuilder/base.cow ../build-area/privoxy_&p-version;-1.dsc + + + And try to run autopackage testing suite on the result: + + + autopkgtest /var/cache/pbuilder/result/privoxy_&p-version;-1_amd64.changes -s -- schroot sid + + + Or just push the changes to salsa.debian.org, where a CI pipeline is + defined for the package, that builds and tests it. + + + If everything is okay, run cowbuilder with i386 and amd64 environments + for current Debian stable release and build + privoxy_&p-version;-1_i386.deb and privoxy_&p-version;-1_amd64.deb. + Then sign both files: + + + gpg --detach-sign --armor privoxy_&p-version;-1_i386.deb + gpg --detach-sign --armor privoxy_&p-version;-1_amd64.deb + + + Create a README file containing the recent block from debian/changelog + and upload the two packages, the two signatures and the README to a + freshly created folder below + https://sourceforge.net/projects/ijbswa/files/Debian/ + + + For building just a git snapshot build the following workflow may be + useful. First create a build environment, for this you may have to + run the following commands: + + + sudo apt install build-essential devscripts + sudo apt-get build-dep privoxy + + + After this enter the checked out privoxy git tree and check that all + (new) build dependencies are met: + + + dpkg-checkbuilddeps + + + If something is missing, just add it using + + + sudo apt install foobar + + + Now you may update debian/changelog, especially the version number + using + + + dch + + + and finally build the package: + + + debuild -us -uc -b + + + If everything went okay, you may find the resulting Debian package in + the parent directory. + + + You may want to clean up the build tree using + + + debian/rules clean + + + And maybe repair some artefacts using one or both of the following + commands: + + + git reset --hard + git clean -fd + + Mac OS X @@ -2699,15 +2813,15 @@ for-privoxy-version=3.0.11 packages" above). - There are three modules available in the Git repository for use on Mac + There are three modules available in the CVS repository backups for use on Mac OS X, though technically only two of them generate a release (the other can be used to install from source). - OSXPackageBuilder module + OSXPackageBuilder module (Documentation out of date) The OSXPackageBuilder module generates OS X installer packages - supporting all Macs running OS X 10.4 and above. Obtain it from Git as + supporting all Macs running OS X 10.4 and above. Obtain it from CVS as follows into a folder parallel to the exported privoxy source: @@ -2727,14 +2841,14 @@ for-privoxy-version=3.0.11 - osxsetup module (DEPRECATED) + osxsetup module (DEPRECATED) (Documentation out of date) This module is deprecated since the installer it generates places all Privoxy files in one folder in a non-standard location, and supports only Intel Macs running OS X 10.6 or higher. - Check out the module from Git as follows into a folder parallel to the + Check out the module from CVS as follows into a folder parallel to the exported privoxy source: @@ -2772,13 +2886,13 @@ for-privoxy-version=3.0.11 - macsetup module + macsetup module (Documentation out of date) The macsetup module is ideal if you wish to build and install Privoxy from source on a single machine. - Check out the module from Git as follows into a folder parallel to the + Check out the module from CVS as follows into a folder parallel to the exported privoxy source: @@ -2804,38 +2918,10 @@ for-privoxy-version=3.0.11 Uploading and Releasing Your Package After the package is ready, it is time to upload it - to SourceForge, and go through the release steps. The upload - is done via FTP: - - - - - Upload to: ftp://upload.sourceforge.net/incoming - - - - - user: anonymous - - - - - password: ijbswa-developers@lists.sourceforge.net - - - - - Or use the make targets as described above. - - - Once this done go to - - https://sourceforge.net/project/admin/editpackages.php?group_id=11118, - making sure you are logged in. Find your target platform in the - second column, and click Add Release. You will - then need to create a new release for your package, using the format - of $VERSION ($CODE_STATUS), e.g. &p-version; - (beta). + and go through the release steps. The upload + is done at + SourceForge + after logging in. Now just follow the prompts. Be sure to add any appropriate Release @@ -2862,9 +2948,9 @@ for-privoxy-version=3.0.11 When all (or: most of the) packages have been uploaded and made available, send an email to the announce mailing - list, Subject: "Version X.Y.Z available for download". Be sure to + list, Subject: "Announcing Privoxy X.Y.Z $CODE_STATUS". Be sure to include the - + download location, the release notes and the Changelog. Also, post an updated News item on the project page Sourceforge, and update the Home page and docs linked from the Home page (see below). Other news sites @@ -2943,9 +3029,9 @@ for-privoxy-version=3.0.11 make webserver - This will do the upload to the - webserver (www.privoxy.org) and ensure all files and directories - there are group writable. + This will do the upload to the SourceForge webserver (which is manually + syncronized with www.privoxy.org) + and ensure all files and directories there are group writable. Please do NOT use any other means of transferring