X-Git-Url: http://www.privoxy.org/gitweb/?p=privoxy.git;a=blobdiff_plain;f=doc%2Fsource%2Fdeveloper-manual.sgml;h=6b1333fb162bce92497c882ba7292ffe4850a1e6;hp=84fd17c7003538d058d86217815bd6eed22e2dd7;hb=fdbc2647e7be299218036c36c9179b6f5adb2349;hpb=3411b7fcb261ee443e24ededaec0e1ea22505038 diff --git a/doc/source/developer-manual.sgml b/doc/source/developer-manual.sgml index 84fd17c7..6b1333fb 100644 --- a/doc/source/developer-manual.sgml +++ b/doc/source/developer-manual.sgml @@ -5,7 +5,7 @@ - + @@ -15,15 +15,11 @@ ]> Copyright - &my-copy; 2001-2016 by + &my-copy; 2001-2018 by Privoxy Developers - - $Id: developer-manual.sgml,v 2.72 2016/07/26 10:45:49 fabiankeil Exp $ - - The CVS Repository + The Git Repository If you become part of the active development team, you will eventually - need write access to our holy grail, the CVS repository. One of the + need write access to our holy grail, the Git repository. One of the team members will need to set this up for you. Please read - this chapter completely before accessing via CVS. + this chapter completely before accessing via Git. - Access to CVS + Access to Git - The project's CVS repository is hosted on - SourceForge. - For historical reasons, the CVS server is - called ijbswa.cvs.sourceforge.net, the repository is - called ijbswa, and the source tree module is called - current. + The project's Git repository is hosted at the + Privoxy site. + The Git repository URL is + ssh://git@git.privoxy.org:23/git/privoxy.git, + the central repository is called privoxy, and the + source branch is called master. Subfolders exist + within the project for target-dependent build and packaging tools, each + including the name of the target operating system in their name (e.g. + Windows, OSXPackageBuilder, debian). There is a webview of the Git + hierarchy at + + https://www.privoxy.org/gitweb/?p=privoxy.git;a=tree, + which might help with visualizing how these pieces fit together. - + Branches - Within the CVS repository, there are modules and branches. As - mentioned, the sources are in the current - module. Other modules are present for platform specific - issues. There is a webview of the CVS hierarchy at http://ijbswa.cvs.sourceforge.net/viewvc/ijbswa/, - which might help with visualizing how these pieces fit together. + Whilst the central repository contains only the master branch, developers + are of course free to create branches in their local repositories as they + develop features, fixes, or update the target-dependent tools. Only once + such changes are fully tested ought they be pushed back to the central + repository master branch. - CVS Commit Guidelines + Git Commit Guidelines The source tree is the heart of every software project. Every effort must be made to ensure that it is readable, compilable and consistent at all times. We expect anyone with CVS access to strictly + main development trunk, and --> We expect anyone with Git access to strictly adhere to the following guidelines: Basic Guidelines, for all branches: - Please don't commit even @@ -255,7 +252,7 @@ Hal. If necessary, prepare the commit messages in advance. - Before changing things on CVS, make sure that your changes are in line + Before changing things on Git, make sure that your changes are in line with the team's general consensus on what should be done. @@ -267,7 +264,6 @@ Hal. - + + Writing Documentation Using DocBook - A Crash Course useful. @@ -505,7 +503,6 @@ Hal. Here it is: - @@ -516,21 +513,24 @@ Hal. Tags delimiting a block of text (even small blocks) should be on their own line. Like: + <para> Some text goes here. </para> + Tags marking individual words, or few words, should be in-line: + Just to <emphasis>emphasize</emphasis>, some text goes here. - Tags should be nested and step indented for block text like: (except in-line tags) + <para> <itemizedlist> @@ -542,6 +542,7 @@ Hal. </itemizedlist> </para> + This makes it easier to find the text amongst the tags ;-) @@ -602,7 +603,6 @@ Hal. - @@ -634,7 +634,6 @@ Hal. custom entities are listed below. See any of the main docs for examples. - @@ -678,7 +677,6 @@ Hal. - There are others in various places that are defined for a specific purpose. Read the source! @@ -946,11 +944,9 @@ int ms_iis5_hack = 0; Instead of: - int msiis5hack = 0; int msIis5Hack = 0; - @@ -973,12 +969,10 @@ int load_some_file(struct client_state *csp) Instead of: - int loadsomefile(struct client_state *csp) int loadSomeFile(struct client_state *csp) - @@ -997,13 +991,12 @@ int loadSomeFile(struct client_state *csp) (.h) extern int load_aclfile(struct client_state *csp); (.c) int load_aclfile(struct client_state *csp) - Instead of: + Instead of: (.h) extern int load_aclfile(struct client_state *); or (.h) extern int load_aclfile(); (.c) int load_aclfile(struct client_state *csp) - @@ -1055,7 +1048,6 @@ int loadSomeFile(struct client_state *csp) Instead of: - #define USE_IMG_LST 1 or #define _USE_IMAGE_LIST 1 or @@ -1063,7 +1055,6 @@ int loadSomeFile(struct client_state *csp) #define use_image_list 1 or #define UseImageList 1 - @@ -1464,12 +1455,10 @@ for (size_t cnt = 0; cnt < len; cnt++) Exception: - /* This is not a local include, but requires a path element. */ #include <sys/fileName.h> - Note: Please! do not add "-I." to the Makefile without a _very_ good reason. This duplicates the #include @@ -1795,10 +1784,9 @@ static void unload_re_filterfile(void *f) { ... } Example for file comments: -const char FILENAME_rcs[] = "$I<!-- Break CVS Substitution -->d$"; /********************************************************************* * - * File : $S<!-- Break CVS Substitution -->ource$ + * File : $Source * * Purpose : (Fill me in with a good description!) * @@ -1835,7 +1823,7 @@ const char FILENAME_h_rcs[] = FILENAME_H_VERSION; Note: This declares the rcs variables that should be - added to the "show-proxy-args" page. If this is a brand new + added to the "show-version" page. If this is a brand new creation by you, you are free to change the "Copyright" section to represent the rights you wish to maintain. @@ -1849,10 +1837,9 @@ const char FILENAME_h_rcs[] = FILENAME_H_VERSION; #ifndef _FILENAME_H #define _FILENAME_H -#define FILENAME_H_VERSION "$I<!-- Break CVS Substitution -->d$" /********************************************************************* * - * File : $S<!-- Break CVS Substitution -->ource$ + * File : $Source * * Purpose : (Fill me in with a good description!) * @@ -1948,13 +1935,14 @@ int FUNCTION_NAME(void *param1, const char *x) Testplan for releases Explain release numbers. major, minor. developer releases. etc. - + Remove any existing rpm with rpm -e Remove any file that was left over. This includes (but is not limited to) + /var/log/privoxy /etc/privoxy @@ -1962,7 +1950,7 @@ Remove any file that was left over. This includes (but is not limited to) /etc/init.d/privoxy /usr/doc/privoxy* - + Install the rpm. Any error messages? @@ -1972,9 +1960,72 @@ Install the rpm. Any error messages? Start browsing. Does Privoxy work? Logfile written? Remove the rpm. Any error messages? All files removed? - + + + Fuzzing Privoxy + + To make fuzzing more convenient, Privoxy can be configured + with --enable-fuzz which will result in the --fuzz option + becoming available. + + + Example (tested on ElectroBSD): + + +# Compile Privoxy with instrumentation for afl +$ export CC=afl-clang +$ export CFLAGS="-fsanitize=address -ggdb" +$ export CPPFLAGS=-I/usr/local/include/ +$ export LDFLAGS="-fsanitize=address -L/usr/local/lib" +$ export AFL_USE_ASAN=1 +$ export AFL_HARDEN=1 +$ ./configure --with-debug --enable-extended-host-patterns --enable-accept-filter --enable-no-gifs --enable-compression --enable-strptime-sanity-checks --enable-external-filters --enable-fuzz + +$ ./privoxy --fuzz +Privoxy version 3.0.24 (http://www.privoxy.org/) +Usage: ./privoxy [--config-test] [--chroot] [--help] [--no-daemon] [--pidfile pidfile] [--pre-chroot-nslookup hostname] [--user user[.group]] [--version] [configfile] + ./privoxy --fuzz fuzz-mode ./path/to/fuzzed/input [--stfu] + +Supported fuzz modes and the expected input: + action: Text to parse as action file. + client-request: Client request to parse. Currently incomplete + client-header: Client header to parse. + chunked-transfer-encoding: Chunk-encoded data to dechunk. + deflate: deflate-compressed data to decompress. + filter: Text to parse as filter file. + gif: gif to deanimate. + gzip: gzip-compressed data to decompress. + pcrs-substitute: A pcrs-substitute to compile. Not a whole pcrs job! Example: Bla $1 bla C $3 blah. + server-header: Server header to parse. + server-response: Server response to parse. + +The following fuzz modes read data from stdin if the 'file' is '-' + client-request + client-header + chunked-transfer-encoding + deflate + gif + gzip + pcrs-substitute + server-header + server-response + +Aborting + +$ export ASAN_OPTIONS='abort_on_error=1' +$ mkdir input output +$ echo '$1 bla fasel $2' > input/pcrs +$ afl-fuzz -i input -o output -m none ~/git/privoxy/privoxy --fuzz pcrs-substitute - --stfu + +$ cat >input/pcrs.txt +FILTER: bla fasel +s@(.{1})[432](\d+)@$1$2$hostname@UgisT + +$ afl-fuzz -i input/ -o output/ -f bla.filter -m none privoxy --fuzz filter bla.filter --stfu + + @@ -2005,6 +2056,7 @@ Install the rpm. Any error messages? First you need to determine which version number the release will have. Privoxy version numbers consist of three numbers, separated by dots, like in X.Y.Z (e.g. 3.0.0), where: + @@ -2036,11 +2088,11 @@ Install the rpm. Any error messages? Z, the point or sub version, represents a release of the software within a branch. It is therefore incremented immediately before each code freeze. In development branches, only the even point versions correspond to actual releases, - while the odd ones denote the evolving state of the sources on CVS in between. - It follows that Z is odd on CVS in development branches most of the time. There, it gets + while the odd ones denote the evolving state of the sources on Git in between. + It follows that Z is odd on Git in development branches most of the time. There, it gets increased to an even number immediately before a code freeze, and is increased to an odd number again immediately thereafter. - This ensures that builds from CVS snapshots are easily distinguished from released versions. + This ensures that builds from Git snapshots are easily distinguished from released versions. The point version is reset to zero when the minor changes. @@ -2056,9 +2108,8 @@ Install the rpm. Any error messages? - - In summary, the main CVS trunk is the development branch where new + In summary, the main Git trunk is the development branch where new features are being worked on for the next stable series. This should almost always be where the most activity takes place. There is always at least one stable branch from the trunk, e.g now it is @@ -2075,7 +2126,7 @@ Install the rpm. Any error messages? Developers should remember too that if they commit a bugfix to the stable branch, this will more than likely require a separate submission to the - main trunk, since these are separate development trees within CVS. If you + main trunk, since these are separate development trees within Git. If you are working on both, then this would require at least two separate check outs (i.e main trunk, and the stable release branch, which is v_3_0_branch at the moment). @@ -2089,7 +2140,6 @@ Install the rpm. Any error messages? The following must be done by one of the developers prior to each new release. - @@ -2102,35 +2152,39 @@ Install the rpm. Any error messages? Increment the version number (point from odd to even in development - branches!) in configure.in. (RPM spec files - will need to be incremented as well.) + branches!) in configure.in and update the code + status (CODE_STATUS="xxx") to one of "alpha", "beta" or "stable". + Rebuild configure and GNUMakefile to make sure the updated values are + being used. - If default.action has changed since last - release (i.e. software release or standalone actions file release), - bump up its version info to A.B in this line: + Use the dok-release target to update the sgml documentation source files. + + + If action file processing has changed and is not backward-compatible, + make sure the "for-privoxy-version=x.y.z" minimum version number in + default.action.master has been updated: + - {+add-header{X-Actions-File-Version: A.B} -filter -no-popups} +{{settings}} +############################################################################# +#MASTER# COMMENT: The minimum Privoxy version: +for-privoxy-version=3.0.11 - - - Then change the version info in doc/webserver/actions/index.php, - line: '$required_actions_file_version = "A.B";' - All documentation should be rebuild after the version bump. - Finished docs should be then be committed to CVS (for those + Finished docs should be then be committed to Git (for those without the ability to build these). Some docs may require rather obscure processing tools. config, the man page (and the html version of the man page) - fall in this category. REAMDE, the man page, AUTHORS, and config - should all also be committed to CVS for other packagers. The + fall in this category. README, the man page, AUTHORS, and config + should all also be committed to Git for other packagers. The formal docs should be uploaded to the webserver. See the Section "Updating the webserver" in this manual for details. @@ -2162,7 +2216,7 @@ Install the rpm. Any error messages? - Tag all files in CVS with the version number with + Tag all files in Git with the version number with cvs tag v_X_Y_Z. Don't use vX_Y_Z, ver_X_Y_Z, v_X.Y.Z (won't work) etc. @@ -2185,7 +2239,6 @@ Install the rpm. Any error messages? - @@ -2198,22 +2251,20 @@ Install the rpm. Any error messages? For all types of packages, including the source tarball, you must make sure that you build from clean sources by exporting - the right version from CVS into an empty directory (just press return when + the right version from Git into an empty directory (just press return when asked for a password): - mkdir dist # delete or choose different name if it already exists cd dist cvs -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa login cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa export -r v_X_Y_Z current - Do NOT change a single bit, including, but not limited to - version information after export from CVS. This is to make sure that + version information after export from Git. This is to make sure that all release packages, and with them, all future bug reports, are based on exactly the same code. @@ -2241,7 +2292,6 @@ Install the rpm. Any error messages? Please keep these general guidelines in mind when putting together your package. These apply to all platforms! - @@ -2302,7 +2352,7 @@ Install the rpm. Any error messages? interest (and possibly renamed to index.html). This should be one level up from the manuals. There is a link also on this page to an HTMLized version of the man page. To avoid 404 for - this, it is in CVS as + this, it is in Git as doc/webserver/man-page/privoxy-man-page.html, and should be included along with the manuals. There is also a css stylesheets that can be included for better presentation: @@ -2350,7 +2400,6 @@ Install the rpm. Any error messages? - @@ -2360,28 +2409,22 @@ Install the rpm. Any error messages? version into an empty directory. (See "Building and releasing packages" above). Then run: - cd current autoheader && autoconf && ./configure - Then do: - make tarball-dist - To upload the package to Sourceforge, simply issue - make tarball-upload - Go to the displayed URL and release the file publicly on Sourceforge. For the change log field, use the relevant section of the @@ -2400,7 +2443,7 @@ Install the rpm. Any error messages? packages" above). - As the only exception to not changing anything after export from CVS, + As the only exception to not changing anything after export from Git, now examine the file privoxy-dist.spec and make sure that the version information and the RPM release number are correct. The RPM release numbers for each version start at one. Hence it must @@ -2414,28 +2457,22 @@ Install the rpm. Any error messages? Then run: - cd current autoheader && autoconf && ./configure - Then do - make dist-dist - To upload the package to Sourceforge, simply issue - make dist-upload rpm_packagerev - where rpm_packagerev is the RPM release number as determined above. @@ -2450,11 +2487,9 @@ Install the rpm. Any error messages? version into an empty directory. (See "Building and releasing packages" above). Then get the OS/2 Setup module: - cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co os2setup - You will need a mix of development tools. The main compilation takes place with IBM Visual Age C++. @@ -2470,28 +2505,22 @@ Install the rpm. Any error messages? Edit the os2build.cmd file to set the final executable filename. For example, - installExeName='privoxyos2_setup_X.Y.Z.exe' - Next, edit the IJB.wis file so the release number matches in the PACKAGEID section: - PACKAGEID="Privoxy Team\Privoxy\Privoxy Package\X\Y\Z" - You're now ready to build. Run: - os2build - You will find the WarpIN-installable executable in the ./files directory. Upload this anonymously to @@ -2505,31 +2534,25 @@ Install the rpm. Any error messages? Login to Sourceforge's compilefarm via ssh: - ssh cf.sourceforge.net - Choose the right operating system (not the Debian one). When logged in, make sure that you have freshly exported the right version into an empty directory. (See "Building and releasing packages" above). Then run: - cd current autoheader && autoconf && ./configure - Then run - gmake solaris-dist - which creates a gzip'ed tar archive. Sadly, you cannot use make solaris-upload on the Sourceforge machine (no ncftpput). You now have @@ -2550,22 +2573,18 @@ Install the rpm. Any error messages? version into an empty directory. (See "Building and releasing packages" above). Then get the Windows setup module: - cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co winsetup - Then you can build the package. This is fully automated, and is controlled by winsetup/GNUmakefile. All you need to do is: - cd winsetup make - Now you can manually rename privoxy_setup.exe to privoxy_setup_X_Y_Z.exe, and upload it to @@ -2582,30 +2601,24 @@ Install the rpm. Any error messages? entry to debian/changelog, if it is not already there, for example by running: - debchange -v &p-version;-&p-status;-1 "New upstream version" - Then, run: - dpkg-buildpackage -rfakeroot -us -uc -b - This will create ../privoxy_&p-version;-&p-status;-1_i386.deb which can be uploaded. To upload the package to Sourceforge, simply issue - make debian-upload - Mac OS X @@ -2615,7 +2628,7 @@ Install the rpm. Any error messages? packages" above). - There are three modules available in the CVS repository for use on Mac + There are three modules available in the Git repository for use on Mac OS X, though technically only two of them generate a release (the other can be used to install from source). @@ -2623,12 +2636,12 @@ Install the rpm. Any error messages? OSXPackageBuilder module The OSXPackageBuilder module generates OS X installer packages - supporting all Macs running OS X 10.4 and above. Obtain it from CVS as + supporting all Macs running OS X 10.4 and above. Obtain it from Git as follows into a folder parallel to the exported privoxy source: + cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co OSXPackageBuilder - The module contains complete instructions on its usage in the file OS X Package Builder HOWTO.txt. @@ -2650,21 +2663,19 @@ Install the rpm. Any error messages? supports only Intel Macs running OS X 10.6 or higher. - Check out the module from CVS as follows into a folder parallel to the + Check out the module from Git as follows into a folder parallel to the exported privoxy source: + cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co osxsetup - Then run: - cd osxsetup build - This will run autoheader, autoconf and configure as well as make. @@ -2677,11 +2688,9 @@ Install the rpm. Any error messages? package" button. If you specify ./Privoxy.pkg as the output package name, you can then create the distributable zip file with the command: - zip -r privoxyosx_setup_x.y.z.zip Privoxy.pkg - You can then upload this file directly to the Files section of the Sourceforge project in the Macintosh (OS X) folder. Each new version @@ -2698,12 +2707,12 @@ Install the rpm. Any error messages? from source on a single machine. - Check out the module from CVS as follows into a folder parallel to the + Check out the module from Git as follows into a folder parallel to the exported privoxy source: + cvs -z3 -d:pserver:anonymous@ijbswa.cvs.sourceforge.net:/cvsroot/ijbswa co macsetup - The module contains complete instructions on its usage in its README file. The end result will be the @@ -2727,7 +2736,6 @@ Install the rpm. Any error messages? to SourceForge, and go through the release steps. The upload is done via FTP: - @@ -2745,7 +2753,6 @@ Install the rpm. Any error messages? - Or use the make targets as described above. @@ -2807,11 +2814,9 @@ Install the rpm. Any error messages? If you have changed anything in the stable-branch documentation source SGML files, do: - make dok - That will generate doc/webserver/user-manual, doc/webserver/developer-manual, @@ -2823,7 +2828,7 @@ Install the rpm. Any error messages? doc/webserver/man-page/privoxy-man-page.html by running make man. (This is a separate target due to dependencies on some obscure perl scripts - [now in CVS, but not well tested]. See comments in GNUmakefile.) + [now in Git, but not well tested]. See comments in GNUmakefile.) If you want to add new files to the webserver, create them locally in @@ -2831,14 +2836,12 @@ Install the rpm. Any error messages? create new directories under doc/webserver). - Next, commit any changes from the above steps to CVS. All set? + Next, commit any changes from the above steps to Git. All set? If these are docs in the stable branch, then do: - make webserver - This will do the upload to the webserver (www.privoxy.org) and ensure all files and directories