Last Update: April 19 2005


Last Changes :

* ---------- released version 0.9.33 (2005-09-27)
* fixed 64bit problems (BUG #1226863)
* updated German locale, fixes done by Debian developers (Hey, please inform
  us about errors. Scanning the net and all distributions for possible fixes
  is not very helpful.)
* ---------- released version 0.9.34 (2006-01-09)
* security fixes
* some minor bug fixes
* reworked build system a lot, fixed RPM spec file
* now builds fine using most of the possibilities pavuk provides
* RPM builds on openSUSE build service for SUSE since version 9.3, Fedora
  since version 4 and Mandriva since version 2006
* RPM packages can be found here:
* ---------- released version 0.9.35 (2007-02-21)
* added -persistent/-nopersistent option

2007-april-30 [notes taken from old work back in 2005/2006 merged into pavuk mainstream source tree]

* bufio has seen a MAJOR overhaul. It is now capable of pushing text &
  binary data to the file system at unprecedented rates. This is done by
  adding a variable sized (and possibly large) memory cache, resulting in
  large size I/O operations. These perform very much faster than the regular
  RTL I/O calls. (tested on quad CPU UNIX Dell servers)

  the new bufio was required as I needed to log/track a huge amount of data
  in the shortest possible time / lowest possible CPU load.

* cookie handling has been fixed/augmented. pavuk can now have the initial
  cookie values that go with a certain web request preconfigured on the
  commandline. Also, several bugs in handling the cookies have been fixed.
  (tested on a wicked ASP.NET intranet site which 'assumed' the use of a
  special web client (a TV set top box) which would transmit it's serial #
  as a client-side created(!) cookie to the web server. This site/client
  combo thus actually transmitted cookies which would first show up in a web
  _request_ instead of the usual: a server-side _response_.)

* several portability items have been changed (h_errno, ...) to make the
  code compile and work on the odd-flavored UNIX box. A native Win32 port is
  under way: it now works, inclusing zlib and OpenSSL, though the latter has
  not been tested recently.

  Note that the changes may have broken GTK support, as I was not able to
  build the code with GTK on my UNIX boxes.

* socket I/O (IP traffic) has been fixed to properly cope with user breaks
  (a user hitting Ctrl+C). Several locations in the software where the
  unexpected signal would cause an infinite loop have been identified and

* added several lines of DEBUG_xxx to aid both developer and user in
  tracking down hard to diagnose issues inside pavuk while scanning a site.

* Accepted-Encoding (more specifically: the handling of x-gzip/gzip/x-
  compress/compress encoding) has been changed to allow for better
  portability: data is expanded in-memory, without the need for an external
  'gzip' tool and/or OS-specific forks & pipes.

  (Win32 wouldn't know a fork if ever it saw one.)

* ALL stdio is now handled through the new bufio system. This not only
  improves performance when you've got -debug and -debuglevel dialed all the
  way up, but also corrected several spots where, depending on your C RTL,
  stdio/stderr traffic would arrive at different moments on your console
  (some of it was written through the FILE I/O, some through direct I/O,
  causing blurbs of output to pass one another along the way to the actual

* buffer overrun protection has been improved. Note also that every
  snprintf() and derivative thereof is now 'augmented' by an additional line
  of code which ensures that the last character in the buffer is guaranteed
  to be a NUL sentinel, thus ensuring that the buffer will always present
  data in correct C string format (NUL-terminated). (This is an old habit of
  mine as some C RTLs have shown to be kinda flaky on the subject of NUL
  sentinels when snprintf() et al are writing data up to the edge of their
  output buffers: some C RTLs 'forget' to put a NUL there under particular
  circumstances (some commercial Watcom compiler releases come to mind).

* multithreading pavuk has been tested on an high perf MP UNIX box and it
  was like the documentation/notes state somewhere: instable. The thread
  interlocking has now been fixed; one of the hardest to fix proved to be
  the lockup at the end of a pavuk run. The fix also includes the use of
  semaphores and some additional code changes to make the code thread safe;
  critical sections are now handled as such. This includes placing several
  non-threadsafe C RTL calls (e.g. ctime()) inside critical sections!

* auto-form-filling (the feature which led me to select pavuk over wget et
  al when I started the hammer/chunky project) has been fixed for those
  special pages where you have an empty form to submit: the site I had to
  test included such a form, which was submitted using javascript, but did
  not contain _any_ input fields (but cookies were expected to come with
  that request, thank you). Before, pavuk crashed on such a page. This has
  now been fixed.

* added a 'reindent' target to the makefile, using GNU indent to reformat
  the code. (When you're working several weeks on end in crunch time, you
  want to see some proper and consistent looking source code, even when you
  just made it a mess yourself...)

  Also extended the cleanup makefile target to help me in cleaning up any
  backup and/or temporary files created by vi and some log diagnostic

  [edit may/2007: wasn't this already in the makefiles before - see
  ChangeLog entry in 2003?]

* added several commandline parameter types, which allow you to instruct
  pavuk to use OS file handles or file names for logging activity, while you
  can now also specify whether a log file should be overwritten (default) or
  appended to (new feature) by adding another '@' prefix to the file path.

  TODO: document this properly.

* added hammer/crunchy modes: several ways to scan a web site and than
  rescan it. The higher (later) hammer mode has been specifically written to
  use pavuk as a 'replay attack' based DoS tool for testing high performance
  web servers. (bufio was overhauled to allow us to log all I/O data +
  diagnostics to disc while hammering the server while the pavuk system
  _must_ perform better (= faster) than the web server when running both on
  equivalent hardware.)

* The native Win32 port has been overhauled (previous code was never
  released to the public) to make sure I did not have to look for OS-
  specific path elements _everywhere_ in the code (it was becomes a code-
  wise maintainance nightmare while fixing up/down all those 'absolute path'
  and 'path expansion' code sections to handle Win32 drive letters (root is
  '[A-Z]:[\\/]' instead of simply '/').

  This has been fixed by using the cygwin 'path hack' for the native Win32
  port too: root is '/cygdrive/[a-z]/' so it looks exactly like a UNIX path.

  Any places in the codes which need to address the OS while passing an OS-
  specific path are now handled almost invisibly: all relevant C RTL calls
  (fopen/open/stat/lstat/symlink/link/unlink/rename/mkdir/rmdir/opendir) are
  now encapsulated in tl_[sysname] wrapper functions where these
  /cygdrive/[x]/ paths are converted back to native Win32 paths before the
  actual C RTL function is called. Also any debug/print statement, which is
  used to report a file path, is fixed to convert file paths to the native
  representation with a minimum of fuss: see the new tl_native() call for a
  description how this was done. This code has not been tested in a UNIX/MP
  environment, but the design is such that this should not cause any trouble
  (pthread port for Win32 is in progress ATM).

* added -debug_level modes: all/trace/dev/bufio/cookie/htmlform. Also added
  a feature where you can now specify a set of debug levels and have some of
  those levels _removed_, e.g. 'all,!dev' will show anything _except_ 'dev'
  level debug output: note the new '!' prefix.

* -debug_level output is now prefixed with its level in caps and square
  brackets, e.g. '[PROCE]' to aid in filtering the debug output (for
  instance by piping it through sed/grep).

* unified debug output handling in the code: -debug_levels are now only
  active when you specify -debug too.

* inflate_decode() and gzip_decode() have been fixed to suit a multithreaded
  environment. gzip_decode() now has an in-memory implementation, using the
  zlib library, for those systems which do not support UNIX pipes/forks.

* Fixed deflate/compress handling: the MJF Accept-Encoding deflate hack has
  been removed and the request header extended. (tested on a Wikipedia
  HTTP/1.1 compliant server)

  You may wish to permanently disable the code within

  in decode.c if you do not wish to depend on the external gzip tool any

* _all_ system header file #include's have been removed from the sources and
  integrated into config.h to allow for better portable source code. and have been extended to include several more OS-
  dependent system call and header file checks.

  A seperate native Win32 version of the header file is also provided (used
  by the MSVC2005 native Win32 build).

* several hardcoded buffer sizes in the software have been made configurable
  (but remain hardcoded). See for instance dinfo.c: 12 -->
  PAVUK_INFO_DIRNAME and 1024-and-other-fixed-buf-sizes -->

* fixed several cases where dangling (i.e. free()d but not NULL-ed) pointers
  caused havok. Code has been quickly reviewed to locate and fix additional
  spots that did not yet cause pavuk to go 'crazy Ivan' (Hunt for the Red
  October, anyone? ;-) )

* hardcoded lock filenames have been converted to #define's to allow these
  to be changed in a single spot (config.h), improving portability. e.g.:

    '._lock' --> PAVUK_LOCK_FILENAME

* UNIX-specific octal privs have been changed to their proper #define's to
  allow for maximum portability (Win32 doesn't know '0644' but can cope with


  though maybe in a odd way).

* fixed quite a few spots where an unidentified form encoding method would
  lead to _very_ instable bahaviour, including crashes/core dumps. Look for

    fi->method = FORM_M_UNKNOWN

  assignments and additonal FORM_M_UNKNOWN checks.

* added -no_dns support for those who have to work in an environment with
  flaky or no DNS support (I had to as I was working on a box in a specially
  configured, partially walled-off DMZ zone while developing and testing
  pavuk against a web server.)

* fixed typos in the text as I came along them.

* the bufio overhaul also lead to a overhaul of the -dumpxxx code,
  removing/fixing several spots in the code which caused incorrect/instable
  behaviour. (e.g. code in doc.c)

* Fixed handling of compressed data for any text-based server response;
  pavuk now correctly handles any gzipped/deflated text, including, for
  instance, any 'text/javascript' content sent over the wire in compressed
  form (tested on a Wikipedia-based HTTP/1.1 compliant server).

* added -progress_mode: several choices in progress verbosity.

* added -no_disc_io: test a grab/scan without writing anything to disc.
  Mostly useful in combination with the earlier -hammer modes.

* fixed/updated HTTP error response handling in accordance with RFC2616 so I
  can better see what a HTTP/1.1 compliant target is reporting back to
  pavuk. (errcode.c et al)

* unified timing units to fix a few timing oddities: instead of minutes,
  etc. the code uses seconds everywhere (apart, of course, from the few
  locations where we use milleseconds ;-) )

  -timeout is now in milliseconds!

* Added -rtimeout and -wtimeout command line parameters.
  (unit: milliseocnds)

* added -allow_persistent / -noallow_persistent commandline arguments to
  allow/disallow the use of HTTP/1.1 persistent connections.

* added -dumpcmd and -dumpdir commandline arguments.

* added -bad_content commandline argument for use with the hammer/chunky

* added -report_url_on_err commandline argument: report the URL which was
  processed while the error occurred.

* added -test_id commandline argument: this is included in the timing report
  so reports can be better automatically processed / combined.

* added -page_sfx commandline argument to help pavuk identify what suffixes
  are to be considered web pages (useful for scanning ASP and ASP.NET sites
  which present unusual mime types with their pages).

* added -tlogfile4sum commandline argument: specify a log file where timing
  info is stored. Handy when pavuk is not only used to grab the info off a
  site but also scan & report site performance.

* added -encode commandline parameter as the counterpart of -noencode.

* added -nohtDig, -noquiet and -noverbose commandline parameters as
  counterparts of -htDig, -quiet and -verbose respectively.

* added filepath support to -dumpfd and -dump_urlfd: by specifying the
  option prefixed with a '@' character, pavuk will treat the option value as
  filepath specification instead of a OS file handle and subsequently open
  the specific file internally. Note that adding yet another '@' character
  as a prefix signals pavuk to _append_ to the specified file, instead of
  _overwriting_ it.

  This is useful when you wish to have those dumps but are working in an
  environment where you cannot pass valid file handles through the

* added -dump_request and -nodeump_request commandline arguments for use
  with -dumpfd: when -dump_request is specified, the log file will include
  complete dump of each request sent to the server by pavuk. Thus you can
  produce a complete audit trail of the exchange.

* replaced the DUMP_URLLIST macros in stats.c by two functions. Code is a
  bit cleaner that way.

* fixed times.c which barfed on timestamps beyond 2037 (signed int wrap
  around for time_t).

* added assert() checks at several locations in the code to help track down
  unexpected behaviour which could lead to crashes (like it did till now).

* unified the proliferation of HEX2ASC-alike macros with and without off-by-
  one offsets inside. Now there's one macro for each of 'em in tools.h.

* changed the option to --disable-threads to keep the pattern
  consistent (--disable-xxx series of options in configure), but the default
  behaviour remains the same.

* as --disable-debug removes any debug-_related_ features from
  the pavuk build, these options have been added: --disable-debugging will
  create a default build with all debugging removed from the compiled
  binaries. --disable-prof and --disable-gprof have been added to remove any
  profile info from the default compiled binaries.

* added checks in for socklen_t, pid_t and a bunch of system
  calls and header files that do not live in each environment.


* included pthreads-Win32 based multithreading support in the native Win32

* included EXPERIMENTAL tre (regex) support in the native Win32 build.

* fixed several lurking bugs (buffer overruns, etc.) which only showed in a
  multithreaded environment.

* fixed locking bugs in the new bufio implementation.

* added Win32 memory leak + heap checking for the DEBUG build: many memory
  leaks have been tracked and fixed. (MSVC <ctrdbg.h> based)

* fixed memory leak due to wrong scope in report_error() code.

* added DBGxxx macro's to aid heap tracking for the debug build. See
  DBGdecl/DBGpass/DBGvars usage.

* removed a very nasty memleak in html_parser_get_url() which would leak at
  least 3 blocks for each rejected local anchor URL - and those come quite a
  few! Took me a day to track it down. :-(

* added filtering so gzipped/compressed files on the server are not
  decompressed unintentionally while the server supports Accept-
  Encoding:gzip or compress.

  ( doc_download_helper() in doc.c )


* renamed function should_leave_persistent() to the more appropriately named

* Updated 'chunky' source to the state of the latest pavuk CVS contents (as
  of today) as this code has not yet been merged into CVS itself.

* fixed bugs in -scenario handling, when scanrio files produced by pavuk are
  re-used in the Win32 environment

* fixed bugs in path & file type commandline arguments for the native Win32

* fixed bug in retrying/resuming download for RFC2616 (HTTP/1.1) 'chunked'
  content download handling.

* merged -allow_persistent / -noallow_persistent commandline arguments with
  the equivalent -persistent/-nopersistent feature from the official pavuk
  CVS sources.

  Also improved the code a bit: added the 'Connection: close' header for
  requests over -nopersistent connections, so the server will close the
  connection for us.

* added the -ignore_chunk_bug commandline argument to allow pavuk to handle
  RFC2616 'chunked' downloads from buggy (IIS) web servers.

  ( See also:


* recompiled in 64-bit Linux (SuSe 10.2) and fixed a few items in the, and files. Also added the tests\
  and www\ directories to the distro.

* fixed a few 64-bit compile warnings; at least the test cases in tests\
  perform OK now on a 64-bit Linux system.

* updated the man page a bit; still a lot more to do. Where is that 'nroff
  for dummies' cheatsheet when you need it?  ;-(

* listed -use_http11 as 'on' by default now.

* moved MODE_MIRROR unescape code section up in url.c to line 1682 in
  url_get_local_name_real() as this code would otherwise have no effect at
  all in any environment where the '%' percent character is included in the
  FS_UNSAFE_CHARACTERS charset (for example: Win32).

* PARAM_DOUBLE default values are now fixed point values in 'long' integer
  format; the current values in the program (all 0.0) are clearly within
  range _and_ it 'saves' on compiler warnings quite a bit. (We've still some
  way to go before we get anywhere near a '[almost-]zero-warning cross
  platform portable build: few int to pointer and vice versa casts remain.)

* fixed bug in cfg_get_num_params() which would access uninitialized memory
  out there in NirvanaLand when a PARAM_UNSUPPORTED option was passed to

* Fixed to include 'debug' build handling for KDevelop (which
  would pass '--enable-debug=full' to ./configure).

* updated the script to increase portability (opendir/closedir:
  dirent.h et al)

* included a few aufoconf macros in the m4 directory for easier/proper
  portability support using autoconf et al.

* bugs fixed from BUGS list: multithreaded mode is not as stable as single
  threaded (fixed at least for the CLI version of pavuk; the GTK GUI version
  is in a rather bad shape)

* bugs fixed from BUGS list: signal handling / timeout does not really work
  (at least not in multi threaded downloads). After a SIGINT pavuk just
  hangs.) This has also been fixed for the CLI version of pavuk at least.

* Win32 port now includes JavaScript support (using the statically linked
  Mozilla js library).

* fixed short option definitions in options.h: -tp / -tsp et al

* 'fixed' GUI for Javascript enabled builds (GTK2) - WARNING: it compiles
  now, but has NOT been tested, so expect bugs here!

* merged the 'chunky' code with the pavuk main source tree. Now 'chunky' is
  equivalent to building pavuk with './configure --enable-hammer'.

* set default from -leave_site to -dont_leave_site to prevent 'blown up' web
  crawls when this filter parameter has not been specified.

  This change includes a fix for the cfg/command line handling of pavuk for
  the conditions section (see condition.h + config.c) as pavuk assumed
  sizeof(long)==sizeof(int) in these code sections.

* Now the proper GPL license (GPL, not LGPL) is included in the file


* fixed processing of zero byte length files (robot.txt at,
  etc.): no more crash/assertion failure due to NULLed docu->contents.

* fixed a few memleaks.

* added extra error checking for file rename operations as some issues were
  found with the Win32 build when using a SAMBA-shared filesystem for
  storing the spidered data/files. (It turned out that the same issues
  existed when using native (NTFS, FAT32) filesystems.)

* dialed down the number of default threads from 3 to 1 (see BUGS) to
  prevent a hail of (legitimate) rename error reports.

* added flock() implementation for Win32: when built with multithreading
  support, having no valid flock() implementation is very dangerous!

* changed to detect both flock() and fcntl() file locking
  mechanisms so pavuk will be able to support writing spidered content to
  network shares on both Win32 and UNIX systems: flock() does not support
  network shares locks, fcntl() does, at least on the latest Linux kernels,
  see man flock(2)

* added error reporting/checking for undesirable use of invalid flock()
  implementation. (Useful when porting pavuk to other non-Unix platforms.)

* Fixed content/file size treatment code for items which are already
  available locally (i.e. pavuk finds the item at the remote has not changed
  from when the last time it fetched the item into local cache).

* Fixed the conditions for when to display certain informational messages:
  less screen clutter when not running in '-verbose' mode OR when running in
  '-progress' modes.

* Fixed several error/info messages in the code section for decompressing
  gzip/compress transmitted HTTP content.

* Fixed handling of gzip/compress transmitted content when retrieved from
  local store instead (when pavuk discovers that the file at the remote site
  has not changed since the last time it was fetched and stored on your
  local disc).

* Fixed a few memleaks.

* Changed the DBGvars/DBGpass/DBGargs macros used for tracing memory
  allocations in debug mode to make these macros look more like regular 'C'
  functions to 'demented' code formatters and analysis tools. The drawback
  is that these still look 'weird' in function prototypes, but that causes
  quite a few less errors/warnings than the old style.

* Fixed bugs in get_abs_file_path() directory detection and Win32 abs path

  Also fixed code which produced double slashes in file paths on occasion,
  causing trouble on Win32 platforms. (Fix applied generally.)

* Fixed mk_native() allocated string management pool to support printf() et
  al where up to 3 mk_native() calls are made in the argument list. This is
  important to prevent spurious crashes in multithreaded mode when the worst
  case scenario for mk_native() applies: all threads are executing printf()-
  style statement which has multiple calls to mk_native() in the argument

  Currently overdimensioned a bit as the actual code only has two
  simultaneous calls while the pool now is dimensioned to tolerate 3
  simultaneous calls per thread.

* No more _strfindnchr() and strfindnchr(): strfindnchr() - and its use -
  has now been fixed to match the (proper working) _strfindnchr().
  [fnmatch.c/tools.c et al]

* Fixed const-correctness of several functions.

* Added '-mime_type_file' commandline option to help pavuk support an up-to-
  date list of mime types and their filename extensions, using, for example,
  the UNIX mime.types(5) config file as a source of MIME type information.

  If the user does not specify the '-mime_type_file' option, the original
  built-in defaults will be used instead.

  This feature has been added to provide better support for the pavuk -
  fnrules %M macro: this macro now will use this configuration to produce a
  suitable filename extension for each MIME type: the first extension listed
  in the '-mime_type_file' config file for the given MIME type will be used
  as extension for the %M macro.

* Changed the GTK GUI macros to become functions for ease of debugging. The
  added (tiny) call overhead won't be a performance hit anyway.

* Fixed -fnrules handling: the generated path is cleaned up before it is
  returned to pavuk for use.

  Cleanup actions:
  - duplicate '/' slashes are removed
  - filenames and directory names which end in a '.' dot, get the dot

* Added '%X' to the -fnrules formatted processing to allow reformatting of
  filenames using an optional mimetype-derived extension. This is useful
  when grabbing Wiki (MediaWiki et al) sites when you'd like to store the
  grabbed content using default mimetype-related filename extensions, so
  instead of storing a file like


  that would transform into


  while pages like


  would remain as is.

  (Note: this might be considered shorthand for a -fnrules (...) expression
   which compares both %e and %E. The intent of %X, however, is to only
   allow %e extensions to pass which are 'valid' for the given MIME type and
   force the %E mimetype based extension for all other cases.)

  CAVEAT: %e/%E/%X/%Y will print the extension WITHOUT the leading '.' dot in
          both simple mode and extended LISP mode.

* Added '%Y', '%A' and '%B' to the -fnrules macros: '%Y' uses the MIME type
  prefered filename extension if the URL/filename doesn't have an extension
  yet (while the rather similar '%X' will OVERRIDE the existing extension if
  it is not listed with the specified MIME type).

  '%B' prints the 'basic MIME type', i.e. the MIME type without the ';'
  semicolon separated MIME attributes such as language, etc., while '%A' will
  print these extensions (if they were passed to us by the server).

  CAVEAT: %e/%E/%X/%Y will print the extension WITHOUT the leading '.' dot in
          both simple mode and extended LISP mode.

  All this allows for pavuk -fnrules commandline arguments like this:

    -fnrules F '*' '%h:%r/%d/%b%s.%Y'
    -mime_types_file ./mime.types
    -tr_chr_chr ':\\!&=?' '_'

  so we'll be able to grab a [Media]Wiki site while storing those pages as
  regular 'abc_php_xyz.html', instead of 'abc.php?xyz' page/filenames.

* Added -fnrules 'fnseq' operator to the extended rules: compares a
  wildcard pattern and a string a la fnmatch(3).

* Checked and updated manpage for the -fnrules operators (added 'ud' and
  'sp' operators to the manpage).

* Added -fnrules 'sn' operator to the extended rules as counterpart of 'ns'.
  'sn' uses strtol() to convert a string to a number, while 'ns' uses
  printf() to format a number to a string. (See the man page.)

* Updated the man page a bit regarding '-fnrules'.

* sanitized escape_str(); a quick code review led us to a lurking bug in
  uconfig.c@309, which has been fixed implicitly.

* Added/updates source code documentation: tools.c/tr.c soure code comments.

* Added some sanity checks in the code (tools.c/tr.c/lfname.c)

* Added debug_level 'rules' to allow debugging of both simple and 'extended'
  -fnrules expressions and '-fnrules' URL F/R matching.

* Different boxes exhibit different mktime() behaviour, especially when
  handling out of range tm value sets. Besides, mktime() works in 'local
  time' while some parts of the code require a robust UTC mkgmtime() (not
  available on many boxes) --> ripped & introduced as tl_mkgmtime(). A local
  time-aware equivalent with excellent out-of-range handling is available as

* Added additional error handling around calls which try to parse time
  stamps using tl_mkgmtime() and tl_mktime() (times.c).

  Basically, now both HTTP and FTP benefit from the new code which should
  now proces timestamps like the UTC timestamps they are, while 'out of UNIX
  time_t bounds' timestamps (beyond the range 1970..2038 A.D.) are handled
  in a more sane manner:

  - out of bounds timestamps are reported by pavuk

  - out of bounds timestamps are then 'sanitized', i.e. restricted to the
    1/1/1970..31/12/2037 date range, i.e. a timestamp beyond the horizon,
    like '1/4/2051' will be 'sanitized' (= restricted) to the upper bound:
    31/12/2037. The same goes for te from antiquity like '11/3/1969' (the
    birthday of a certain person), which will be 'sanitized' towards

* Split up DEBUG into developer related stuff, such as memory/heap checking,
  ASSERT/VERIFY, etc. and user related stuff (the -debug and -debug_level
  command line arguments): ./configure is now fitted with an extra


  which will turn on/off -debug/-debug_level user level debugging support in
  pavuk, while the existing


  adds/removes additional developer checks, such as heap allocated checks
  and ASSERT and VERIFY macros.

  In the code, -debug/-debug_level related code is located within the
  'HAVE_DEBUG_FEATURES' sections, while the developer debug/release builds
  are still related to the standard 'DEBUG' #define.

  This now results in three ./configure options that determine the (debug)
  feature set of your binary:

  --enable/disable-debugging --> compile a binary with source level debug
                                 info included and all optimizations
                                 DISabled for improved debugging (by using
                                 gdb or another debugger of your choice)

  --enable/disable-debug     --> include/exclude additional run time checks
                                 in your binary. Most important are the
                                 ASSERT and VERIFY pre/post-condition
                                 validation methods located throughout the
                                 code. The use of these is advised, though
                                 these may cause a performance hit.

                             --> include/exclude user level -debug/-
                                 debug_level command line features, which
                                 help you as a pavuk user to 'debug' pavuk
                                 during the run. Using -debug, pavuk will be
                                 EXTREMELY verbose, which can be toned down
                                 by applying a -debug_level restriction
                                 filter. For example:

                                   -debug -debug_level all,!devel

                                 will be VERY verbose, but will NOT log any
                                 DEVEL level debug info, while:

                                   -debug -debug_level !all,rules

                                 will ONLY produce additional output for the
                                 RULES level, i.e. when pavuk processes -
                                 fnrules and/or JavaScript macros.

* Fixed crash when non-RFC compliant website was grabbed: see testcase 7a.

* Added targeted help: when options cannot be parsed correctly,
  short_usage() will try to help the user by printing the full help for the
  abusing commandline option only. (Of course, I screwed up while using
  debug_level flag sets _again_ :-( [Ger])

* Some improvements for network connectivity error handling and reporting.
  (xvherror() added.) This is the result of some FTP tests with pavuk (tests

* Don't yak about 'Checking "robots.txt"' anymore when doing a FTP grab when
  robots.txt is NOT applicable anyway.

* FTP: added crude 'autodetect/retry' mechanism for FTP servers which do not
  like NLST (==> response code 550) but report correct directory content for
  LIST (or vice versa). (ftp.c)

* FTP/HTTP: at debug level 'protoD' pavuk will now dump RAW data/content
  received from the server before preprocessing (i.e. converting to HTML or

* Added command line option integer sizing support: byte sizes can now be
  specified in K, M or G. Other integer values can also be postfixed with K,
  M or G, but then these will be treated like the ISO values 1000, 1E6 and

* Addition memory leak fixes in case pavuk is fed an invalid commandline.

* NTLM support code: fixed a few glaring bugs.

* Added O_SHORT_LIVED to lock file open() flags for better Win32 behaviour.

* Fixed code to load the pavuk configuration settings from, in order of


  which matches the description in the manual.
  (see also man page)


* Added 'js' flag to '-debug_level', which is used to dump a lot of detail
  about the pattern matching and transformation applied to JavaScript code
  using the '-js_pattern' and '-js_transform / -js_transform2' commandline

* Added sanity check for '-js_pattern' and '-js_transform[2]' regexes, which
  MUST contain a subexpression for them to 'work' as expected.

* removed re_pmatch_sub() and changed the code where it was used to work
  with the available re_pmatch_subs() call, which allows for more elaborate
  validation anyway. See htmlparser.c.

* Removed a regex handling bug in the -js_transform[2] code, which would
  crash pavuk when using regexes where the first subexpression might be

  The crash is due to the fact that the regex parser would return indexes '-
  1' for these empty subexpression(s), resulting in out-of-bounds memory
  writes in the rewrite code. This in turn would nuke the heap, so after
  that is was only a matter of time for pavuk to fail dramatically.

2008 feb 04

* Added DEBUG_MISC() lines to solve issue: [ 1852885 ] to
  improve manipulation by locally stored files

* Included provisional fix (I don't have a working sample run to reproduce
  the issue (yet)) for issue: 1852884 ] infinite loop on
  unexpected responses

* Cleaned up the mess that was -progress_mode.

* Cleaned up several DEBUG_xxx macro mistakes

* Added a little description to the 'hidden' -htDig commandline option,
  which can be used to dump the server-transmitted MIME headers for each
  URL, similar to the htdig tool.

* Added a bit of documentation for the -rollback option (which was

2008 mar 20

* GNU gettext tools don't like '\r' in i18n strings --> fixed by changing
  the related printf() statements in src/doc.c

* started update of configure scripts to the latest autoconf/automake.

  Also reordered the NEWS file so it will work with the new, stricter

    ./bootstrap && ./configure && make distcheck

  distro test cycle.

2008 jul 10

* fixed ';' semicolon bug in http.c near line 2074 which caused incorrect
  decoding of the HTTP/1.x response code header.

* fixed gzip/compress/... content compression support (HTTP/1.1 Accept-
  Encoding); the previous code was a valliant attempt to 'fix' the client
  side (pavuk) to cope with buggy web servers which send the wrong encoding
  type for already compressed files, but this would screw up particular
  responses by *well-behaving* web servers. Of course this would only happen
  in rare circumstances so it was kinda hard to track down.

  Documentation for -Enc/-noEnc has been updated to reflect this situation
  and the code now (hopefully properly) finally supports compressed data
  transmission for RFC2616-complaint web servers.

  If you find that your 'downloaded' compressed files are already
  /incorrectly/ DEcompressed by pavuk, this is NOT the fault of the client
  (pavuk) but evidence that your server is behaving inappropriately and the
  proper remedy for this is the use of the option '-noEnc' which turns this
  feature off so the server is not allowed to screw up in this way any more.

  Also made sure one can check if pavuk has been built with compression
  support by calling 'pavuk --version' and looking at the feature list.

* autoconf/configure script: using the highly undocumented v_cflags or other
  x_* variables as environment variables to hack the configure script (you
  could do that, especially with v_cflags) has been obsoleted while the
  configure and m4/* scripts have been upgraded to support autoconf
  2.62/automake 1.10 and use ONLY *documented* AC.*/etc. macros from now on.

  Note: thanks to the JavaScript library issues on SuSe10.2/AMD64 (older JS
        lib version and seemingly partial header install), I may have failed
        to eradicate all undocumented macros.

* Extra note about bash, at least on SuSe10.2/64-bit, handles
  'if eval test ...' just ever so slightly different than 'if test ...',
  especially where it comes to 'test -n'. As these styles were mixed rather
  arbitrarily before, the 'if eval test ...' style has been completely
  removed from the configure script, as this would sometimes render quite
  unexpected (and incorrect!) results.

* has been updated to ensure important Microsoft Visual Studio
  files are not damaged by having their CRLF sequences converted to UNIX LF
  line endings: this kind of thing will make MSVC spit you in the face and
  reject everything you try until you give it back those CRLF line endings
  in there. So much for XML as project file format and MSVC...

* extra fixes to ensure 'make distcheck' does not barf up a hairball. This
  includes enforcing the permanent inclusion of the 'po' subdirectory in the
  Makefile set for multilingual support.

* configure/Makefile(s): if you don't have one or more of the
  archiving/compression tools compress/lzma/gzip/tar/7z(7zip) installed on
  your system, we don't go belly up at config ~ nor at 'make dist' time
  anymore. This, of course, includes correct behaviour at 'make distcheck'
  time: only use/test those 'GNU standard' formats, which can be created on
  your box.

* Added the 'bootstrap' shell script, next to ''. I know they
  serve the (almost) same purpose, but 'bootstrap' is far more sophisticated
  than and I didn't wish to overwrite ''. Besides, IDEs
  on UNIX boxen expect either the one or the other (there's no single
  'standard' for this), so we might as well provide both.

  At a later time, we might probably point to bootstrap.

* Updated the mime.types MIME 'hint' file: currently, it's a mix of

  1) all properly registered MIME types ( )

  2) the mime.types file provided with the latest Apache/XAMPP

  3) my (Ger Hobbelt) additional file extension hints as used on my own
     servers. This is mostly about professional graphics ~ and modern
     'scene' audio/video container formats, such as Matroska. This only adds
     extensions for otherwise already existing MIME types.

* Updated the DocBook-based documentation for several options (-End/-noEnc, ...)

* 'pavuk --version' now also reports if ZLIB support is included in the
  binary. This is important for '-Enc'.

* Fixed the '-Enc' compressed transmission and HTTP header processing code
  to act properly with fully RFC2616-compliant web servers, discarding the
  old 'hack/fix' attempt to solve a non-complaint server issue at the
  client, as this would break things for fully compliant servers in the rare
  (but extremely annoying) use case:

  - pavuk with '-Enc' option

  - webserver is fully RFC2616 compliant

  - pavuk issues request for file in a .tar.Z or other gzip/compress
    compressed format, where the file on the server is only slightly
    compressed (fastest compression).

  - webserver will transmit file to pavuk, but due to pavuk reporting it is
    able to handle compressed transmission AND the server discovering that
    the content can be compressed quite some more than it already was, the
    file will be transmitted after a server-side just-in-time compression

  - pavuk receives the data. The old hacked code would NOT decompress the
    data. However it SHOULD because the server PROPERLY reported 'Content-
    Encoding: gzip' to pavuk. End result: grabbed data which you cannot
    process nor trust to be in the same format as stored on the server as it
    all 'depends' on arbitrary conditions which you cannot control: is the
    web server able to compress the data before transmission? Is the web
    server configured to allow compression? Etc.

  This use case has now been fixed.

  The effect of BADLY behaving web servers (which send 'Content-Encoding:
  gzip' for any .Z, .z or .gz files (IIS x.x and other servers which are not
  configured to /properly/ handle files and MIME types) is described in the
  DocBook manual page now, including the fix for this (specify the '-noEnc'
  commandline with pavuk).

* active FTP: timeout and stop/break handling slightly improved: now pavuk
  should always terminate under all circumstances while a break or stop has
  been signalled.

* Changed the default for '-url_strategy' from 'level' to 'leveli' to make
  pavuk behave more like your regular web browser (with a user clicking
  through web pages).

* Initial fix for NTLM support for 64-bit Windows. (Only lightly tested.)

  This includes converting that bit of code to support the C99 intNN_t types
  (where NN e {8,16,32}), while the configure script takes care about
  providing the proper types for not-fully-C99-compliant environments.

* The TRE regex package would barf up a hairball due to the incorrect header
  file being loaded. ./configure now recognizes TRE specifics a bit better
  and the code now loads the proper header file (<tre/regex.h> instead of
  <regex.h>). This is important on systems which have multiple, ever so
  slightly incompatible regex processing libraries installed.

* Improved diagnostics a little bit by adding reporting support for
  URL_PARENT_REWRITING, i.e. the situation where a parent page of a grabbed
  page is loaded for the sake of adjusting (rewriting) the URLs in its

* Fixed code so it would compile in full (-DDEBUG) debug mode on UNIX.

* autoconf/configure: ran into some weird issues due to inconsistent M4 []
  quoting: quite a few lines did without it. Turns out that this is a BIG
  No!No! as adding the AX_ADD_OPTION() macro turned this lurking mess into a
  true disaster.

  Fixed by applying [] quoting throughout. The only place where I didn't do
  it, is in the first and second args of AC_DEFINE() -- which should be used
  instead of AC_DEFINE_UNQUOTED when you don't need the latters extra
  functionality anyway -- and the first arg of AC_DEFINE_UNQUOTED(). Any
  other spot where [] quotes are missing in the M4 macros and/or Consider that a bug and please report so I can fix it.

* Finally got the configure system to recognize my JavaScript libraries and
  all. Tugged and tweaked a few items in the bindings to allow maximum
  flexibility for the JS code when it is used to filter URLs (e.g.
  JavaScript pavuk_url_cond_check() function).

* Updated jsbind.c to use latest SpiderMonkey 1.8.x (tested on Win32)

* Changed man/Makefile to ensure HTML is not recreated every 'make' run, but
  only when manpage changes. This should really copy the results from
  ./doc/, but that's for later...

* DocBook documentation: tweaked man page generation to mimic original
  manpage title exactly.

* DocBook documentation: updated '-version' info (important to see at run-
  time what abilities you've got with /your/ pavuk.

* Win32/MSVC: all project files have been updated to produce next to
  Win32/x86: Win64/AMD64 and Win64/Itanium binaries. These project files
  assume the existence of all optional libraries: OpenSSL, SpiderMonkey
  (JavaScript), zlib.

  Where to get those, prefered directory layout, etc. to be published, so
  others can build from source on Win32/64 too and get the same results.

2008 jul 20

* tweaked configure+makefiles so that a 'make dist' from CVS becomes
  possible: there were quite a few references to yet unpublishable files in
  my makefiles (Ger Hobbelt).

* config section: improved adherence to C standards: no more potentially
  dangerous mixed use of function and data pointers by typecasting function
  pointers into data pointers and vice versa.

  This has been resolved by an added layer of indirection, which makes it
  all very legal C again. It goes somewhat like this:

    function_pointer_type ptr = &function;
    data_pointer_type d = &ptr;

  then use (d[0])(...) to call the function.

  This contrasts the old code:

    data_pointer_type d = (data_pointer_type)&function;

  and function invocation using:


* Added support for parsing 'hidden' CSS and JavaScript in HTML. The support
  is also extended to generally parse inside HTML comments PLUS Microsoft IE
  CC's (Conditional Comments): <!--[if...]><![endif]-->


  These are all enabled by default; documentation has been updated for these
  as well.

* Fixed CSS and [Java]Script handling in the HTML tokenizer/parser, which
  was feeding the filters and URL extractors (htmlparser.c).

  Now the code can cope better with incorrectly formatted pages / files.

* Reordered the HTML tags in htmltags.c in a preparatory move to check the
  list for missing attributes (onXXX JavaScript items for one! several are
  missing) and HTML 3/4 tags. (htmltags.c)

2008 aug 13

* updated the -debug_level related code; DEBUG_DEVEL() and a few others now
  'automagically' report the sourcefile+lineno without the need to specify
  these explicitly + some DEVEL_*() calls have been shifted to other
  '-debug_devel' levels (net, mtthr, htmlform, ...)

* completed the -debug_level tracing for multithreaded runs: now all
  semaphore accesses can be traced using the -debug_devel mtthr

* Major fix for bufio+socket code: no more lockup for pavuk due to delayed
  reception of response data (tl_selectr() would incorrectly lock
  indefinitely -- which proved to be a generic coding mistake in both
  tl_selectr() and tl_selectw() -- PLUS better error condition handling in
  an attempt to improve handling of all sorts of 'spurious error conditions'
  which may occur when your network suffers from packet loss or other
  undesirable effects.

* -mode remind code fix for multithreaded use to make it match recurse and
  other modes better; not severely tested so YMMV! (The old code wouldn't
  work anyway, so it's an improvement anyhow).

* few code cleanups (#if 0 ... #endif)

* DocBook manual updated: now all return codes from pavuk are documented.

* minor code fixes for SSL/SFTP.

* updated configure and code to assist in compiling with both latest
  SiderMonkey and older Mozilla JavaScript libraries (Win32/64 and UNIX

* Some unused error checks replaced by ASSERT() and some ASSERT()s replaced
  by error reports as those errors /can/ happen in actual use (though

* Fix for parsing malformed URLs (with multiple '#' and/or '?': bookmarks
  and query string parts would not be stripped/detached correctly as the
  last '#'/'?' instead of the FIRST occurrence of '#'/'?' would be picked as
  a separation point.

* Ran the gettext files through pot/pox/po again. Lots of 'fuzzies'... These
  need to be fixed.

* EXPERIMENTAL: added preliminary code for extended JavaScript support:
  hooks to process HTML and CSS just like you can process embedded <SCRIPT>s
  now. The new hooks are still 'nulls', i.e. do not have any effect.

  This is a work in progress; it compiles & runs (tested on UNIX and Win32
  in multithreaded mode) but the new hooks still need to be implemented.

  The goal here is that all grabbed (parsable) content should be processable
  by custom JavaScript script functions AND when more than one URL is found,
  the JavaScript code should be allowed to add those extra URLs to the pavuk
  queue (using the new url.queue() JavaScript PavukUrl object method --
  currently a 'nil' member function as it still must be fully implemented).

* isatty() fixes which check for error conditions and do /not/ provide
  special 'console oriented' features when isatty(0) produces an error (may
  happen on Win32/UNIX).

* Checked and updated all header files (after I ran into a cyclic dependency
  when changing a bit of code): no .h files will #include "config.h"; all .c
  files /do/ #include "config.h" as the first header.

  System-dependent stuff (TRUE/FALSE definitions and a few other bits) have
  been moved to config.h (where they below IMO) and removed from tools.h

  This is a change required for the gzip fix [SF bug #2050527].

* Preliminary fix for CSS url grabbing and rewriting bug [SF bug #2050537].

  The new code will now try to keep these three styles of <url> formatting
  in CSS intact -- this is done so as to keep particular CSS browser hacks
  intact as much as possible:

    @import "<url>"
    @import url(<url>)
    @import url('<url>')
    @import url("<url>")

  and of course the use of 'url()' elsewhere in any CSS is treated like the
  three examples above, i.e. NONE of these should be changed regarding <url>
  delimiters (quotes or braces) when rewritten by pavuk.

  The ONLY situation where pavuk will CHANGE the quotes is when a <url> is
  found to contain the delimiter quote itself: in that case the quotes are
  changed from ' to " and vice versa.

2008 aug 18

* minor fixes to the includes mime.types file

* configure: added support/auto-detection for the GNU GDB extended debug
  output (-ggdb -g3) for when building a debug build.

* NTLM: fixed code for Win64 and other 64-bit platforms which do or do not
  support structure packing.

* documentation update: -[no]chunk_bug commandline argument finally
  documented (was in there already for a longer time; is a special fix for
  badly behaving IIS web servers which transmit data in 'chunked mode'.

  Also upgraded the documentation for the -tr_str_str/tr_chr_chr options so
  one can finally read how to use [:print:] and other definitions in there
  for -tr_chr_chr and be able to determine up front what the bugger will do
  for you.

  For example:

  Why does -tr_chr_chr '[hexnum:]' '0123456789abcdef' *not* do what you
  expect when the filename has any of the a..f characters? (Answer: they all
  become 'f' as [:hexnum:] actually expands to


  itself, so it is longer than the destination set and by definition any
  'overflow' will be replaced by the last character in the target set.)

* HTML/CSS/JavaScript parent rewriting was sometimes flaky; this has been
  fixed by fixing several bits of antiquated code in pavuk: now all code
  sections are equaly aware of URL_ISHTML, URL_ISSTYLE and/or URL_ISSCRIPT.

  Several functions have been adapted to mirror the new awareness:

  ext_is_html() has been enhanced and has been renamed to actually show its
  intended function: ext_is_parsable() -- which can be a HTML, CSS *or*
  JavaScript file! (not only HTML can be parent of other URLs and need
  updating ('URL parent rewriting').

  [ SF bug #2050537 ] CSS @import bad / HTML corrupted --> fixed

* On SuSe10.2/AMD64 glibc6 dumped core when running pavuk in full-out '-
  debug -debug_level all' (the latter is implicit when you use '-debug')
  mode. This was caused by glibc()'s printf() functions *sensibly* executing
  a strlen() operation on the data fed to one of several '%.*s' printf()
  formatting parameters, while those data series had NOT been NUL

  This would happen when debugging pavuk while fetching data from a gzip-
  enabled web server: the gzip/inflate code would NOT append a new NUL

* Several other '%.*s' and '%s' related core dump spots in the DEBUG_XYZ()
  code which would dump downloaded content have been fixed by feeding the
  data through an enhanced asciidump function -- which will switch to HEX
  dumping when the content to be shown for scutiny contains a large amount
  of non-ASCII data (> 10% is the current heuristic to switch over).

* glibc6 on SuSe10.2/AMD64 would also dump core when being fed a 110K string
  to a printf '%s' statement. This has been fixed by always limiting the
  amount of content to be displayed when debug-printing downloaded data
  (various '-debug_level's)

* gzip/inflate would fail to perform on 'non-parsable' content, i.e. plain
  text files downloaded from a gzip-enabled web server. This has been fixed.

  CAVEAT: The current gzip/inflate code does not deliver when it is fed very
          large files. Hence, when downloading VMware images and/or multi-GB
          ISO files, a workaround is to specify -noEnc. This will be fixed
          at a later date.

  [SF bug #2050527] nonparsed files saved in (wrong) compressed when using 
	            HTTP --> fixed

* Parent rewriting would try to treat all parents as HTML, which is VERY
  wrong when the actual parent is a CSS stylesheet or a JavaScript script
  file. Fixed.

* unified variable names for 'struct doc' variables: it is *QUITE*
  irritating to loose your display of 'docu' contents just because this call
  uses 'docp' for the same (or 'html_doc') while trying to track down
  lurking parent rewriting and file URL parsing bugs.

  Updated all sourcefiles to the use of varname 'docu' for the current
  document. 'docp' and 'html_doc' have been renamed.

* two bugfixes for the tr() code: (1) when using X-Y character ranges, the
  size estimator would allocate way too less space. This has been fixed. (2)
  the documentation says it well: you cannot include a NUL in a tr()
  character set. In one case (a range at the start of the spec like this: '-
  z' would actually attempt to insert such a NUL anyhow, causing subtle bug.
  Fixed. And a minor code cleanup.

* fixed argument quoting for external app invocation, which is particularly
  important for Windows machines: they treat '-quoting quite different from
  "-quoting. Fixed by using "-quotes instead of the original '-quotes.

* -enable_js is now turned ON by default - just like the documentation
  already said.

  KNOWN ISSUE: empty lines in JavaScript code and files gets stripped by
               pavuk on rewriting; this will be fixed at a later date.

* fix in mime.types file for CVS file extension + added mime types for 
  Microsoft Office 2007

* fixed heap corruption in ainterface.c when calling append_starting_url()
  when url has been specified in the extended '-request' format, including
  a predefined local filename. (Would dump core on some systems.)

* moved the url2diag and info2diag functions from recurse.c to where they should
  have been: url.c -- to resolve a cyclic dependency.

* fixed up the '-request' format url parser/decoder url_parse() call: several
  types of input specification error would be silently rejected (now pavuk
  prints a suitable error message to tell the user what [s]he did wrong and what
  was expected) + a few tugs & tweaks to fix behavior for parsing extended 
  URL specifications (including cookies, predefined local filenames, etc.) and
  an extra '-debug' (level: URL) line to help you diagnose how the '-request's
  have been parsed/decoded.

* now you can use the extended '-request' URL format anywhere on the
  commandline and/or your pavuk configuration files -- as long as you keep
  it within quotes on the commandline of course, e.g.

   pavuk "URL: LFNAME:example.html"

* fix: config files generated by pavuk now properly select the 'short format'
  (URL:....) instead of the 'long url spec fomat' (Request:....): previously
  pavuk would loose information about web forms, cookies, local filenames, etc.
  for some types of requested url.

* quickfix for issue reported on the mailing list regarding JavaScript
  interface functions causing the build to fail - which happened when no
  JavaScript library could be found.

  NOTE: on Linux, the JS libraries and headerfiles seem to get installed in
        various places. The current ./configure script looks for the
        header file in the directory
        unless you specify the '--with-js-includes=<dir>' option when running

        The same goes for the js library itself: the current configure script
        looks for either libjs or libmozjs in any of these directories:
        unless you specify the ./configure --with-js-libraries=<dir> option
        to point to your specific libjs.a / libmozjs.a

* added an advanced example of use to the pavuk DocBook documentation
  which will end up in the manpage (where it's a bit too much, but then
  at least the users have an extended example of actual use) -- example
  shows how to grab the up-to-date content from a MediaWiki-based web 

* added S/M/H/D unit support for the time argument decoder function

* Updated the manual regarding:

  - all missing 'hammer mode' options

  - the missing -rtimeout and -wtimeout options

  - checked first few options in options.h and made sure those were all
    documented. (This is a work in progress...)

* All timeouts are now in milliseconds, except the -max_time one, which is
  in minutes.

  All timeout arguments (except -max_time) now recognize the alternative
  units for specifying time: s/m/h/d/S/M/H/D: second, minute, hour, day.

  When no unit has been specified, the unit 'milliseconds' is assumed.

* Fix for bug report #2158794: now all DEBUG_*() functions are called 
  using the proper number of arguments.

  The code has been further enhanced for all printf()-like functions 
  (such as the DEBUF_*() and x*printf() functions) to enable GCC and MSVC
  to check the format specification strings and parameter count and 
  type (GCC).
  This led to the discovery of a multitude of errors, which have been 
  fixed (wrong integer sizes, etc.).

* Preliminary code move to allow downloading extremely large entities
  (larger than 2GB) such as DVD ISO images: this has been done by more
  judicious use of the size_t and ssize_t types instead of simply 'int'.
  On 64-bit platforms, size_t/ssize_t can handle 64-bit sizes, while
  'int' cannot (as GCC still uses 32-bit ints on most common hardware
  64-bit architectures (Intel, ...)).  Further effort will need to be
  spent to adapt the system (and OpenSSL) calls to enable the complete
  datapath for >2GB entity sizes (at least when compiled on 64-bit).

* Small documentation fix: regex overview of characterset changed in DocBook
  source so it appears as a simple list, instead of just one long paragraph
  full of concatenated items --> improved readability.

* const-ified the source code and fixed a few comment typos and a
  lurking bug in FTP (found thanks to constification): filename
  for directory index urls could be damaged in particular circumstances.

* fixed makefiles for environments without any DocBook tools. Also fixed
  configure script to help detect the absence of mandatory DocBook template
  files. Plus added DocBook produce to the distro as we cannot expect everyone
  to have the DocBook tools; nevertheless, everybody /should/ receive a full
  set of documentation.

* Bugfix in GET_NUMLIST(): now original numlist is properly removed (would only
  be noticable before when specifying multiple port numbers).

* memleak fix for _free_httphdr(): now also the httphdr struct itself gets 

* Fixed lockups in debug logging code when running in '-x' GUI mode; overhauled the
  'recursive invocation' detection code within, which is mandatory to prevent
  recursive calls to debug/log functions to blow up the stack and dump core while
  running in ultra verbose debug/diag mode (-debug -debug_level all). This is the
  second part of the fix for bug #2184196.

* Bugfix for #2023089: new code is introduced for '-lmax' depth level checks:
  the 'depth' (a.k.a. 'level') will always be taken from the non-inline parent URL
  which has the lowest level.

  This should fix situations where 'inline' URLs have 'inline' *parent* URLs, such
  as style sheets, which are referenced non-inline URLs (HTML files).

  Seeking out the lowest level non-inline parent should also take care of situations
  where multiple HTML files at different levels themselves, all (directly!) reference the same
  stylesheet/inline URL.

* Attempt at fixing a GUI semaphore lockup, caused by LOCK_CFG_URLSTACK being used
  for different purposes (was a quick hack once to create a 'critical section' there)
  in recurse.c @ 1129. Same hack, but now we use LOCK_GHBN which should cause much less trouble

* Bit of code cleanup.

* Code review checks to see if URLT_FTPS and URLT_GOPHER are used consistently where
  you'd expect them. As you would URLT_HTTPS, next to URLT_HTTP.

* Code review checks and fixes to prevent pspurious damage to url->parent structures:
  now the access to this element is critical-sectioned /everywhere/ using LOCK_URL(u); existed
  in 95% of the places already, now all code has been checked.

* Several fixes for multithreaded GTK GUI use. Most important thing which
  was missing: a call to gtk_threads_init().

* JavaScript: updated HTML tag/attribute tables to recognize all
  onXYZ=... JavaScript event attributes in HTML + added the full
  set of attributes to the url pattern class/object which is
  available in pavuk's own JavaScript extension.

For information on current development see here.