ftp.digital.com Mirror information                       Last Revised: 1/12/2007
================================================================================
The mirroring of the FTP archive on ftp.digital.com was done in multiple stages.
These were:
  1. Create download queues
  2. Process download queue for root directory
  3. Process other download queues
  4. Create a recursive directory listing using ftpls
  5. Get any more recent files in the ftp root directory
  
The mirroring operation was performed over about three weeks. Different parts
  were done around the dates shown below:
               Queues Created: 13/11/2007 -> 14/11/2007
                  Downloading: 20/11/2007 -> 23/11/2007 [first run, ADSL]
                               27/11/2007 -> 30/11/2007 [second run, ADSL]
                               30/11/2007 -> ?/12/2007 [final run, dialup]
  Recursive directory listing: 30/11/2007
  Final root directory update: 1/12/2007

Mirror Consistency
---------------------------------
The mirroring operation was performed while HP was perparing to move the Alpha
systems firmware from ftp://ftp.digital.com/pub/DEC/Alpha/firmware/ to somewhere
on their own FTP server. As a result at least a number of things are known to
have changed during the mirroring operation - it is possible that other changes
may have gone unseen. It is not believed that anything but perhaps the root
directory is in an inconsistent state.

The following changes are known to have happened:
  - /.in.Index-byname. appeared on the 30/11/2007
  - /.in.Index-bytime. appeared on the 29/11/2007
  - /ap0610203.adi.gz was removed during download leaving it unfinished at 1.3MB
  - /dave.test was removed
  - /du.out was removed
  - /pv was removed
  - All files were removed from /pub/games, subdirectories seem untouched
  - All files were removed from /pub/DEC/xcd-old - only the legal rules symlink
    remains.
Of the files that were removed, all but ap0610203.adi.gz were fully downloaded
before their removal. ap0610203.adi.gz has been barely downloaded and is not
of any real use. The files that appeared on the 30/11/2007 and 29/11/2007 will
not be in the index files as for some reason the index files stopped being
updated on 22/11/2007.

The files removed from /pub/games and /pub/DEC/xcd-old will be restored from a
previous mirroring attempt where they remain intact.

The latest version of the index files are available in the final-index
directory. These have not been merged with the mirror yet as it would likely
leave the mirror in a more inconsistent state - unless other things unknown
have changed the only difference between the two should be the removed files
in the root directory.

Recursive Listings
---------------------------------
A recursive directory listing of the server is available in this directory with
the name ls-lR.txt. The listing was created with the ftpls program:
  ftpls ftp://ftp.digital.com/ -R --raw --print-dir | tee ls-lR.txt
  
Hidden Files
---------------------------------
All known files in /hypertext have been mirrored. However, the du.out file once
present in the root directory suggests that there was much more in there than is
known to google, usenet or the internet archive. The empty directories under
/hypertext and /bin were created with the help of the du.out file.

Symbolic Links
---------------------------------
Two approaches have been taken with symbolic links in the mirroring process:
  - large linked directories have been ignored
  - small linked files and directories have been replaced with copies of the
    files/directories they originally pointed to. This means that all the
    various 00README-Legal-Rules-Regs are no longer symlinks but copies of the
    original file.
One should be able to construct a complete list of all symlinks with their
targets from the recursive listing of the server. Using this file one could
write a script to rebuild all symbolic links bringing the mirror back to its
original form.

Software
---------------------------------
The mirror was created with FileZilla 2.2.32 under Windows XP SP2. The version
of FileZilla was already quite out of date when the mirroring began - an upgrade
was not possible due to queue export incompatibilities between versions. 
The version used has a number of bugs that could possibly have prevented some
files from being downloaded. The use of Windows NT rather than UNIX could also
possibly cause minor problems.

At some point in the future the recursive directory listing will be used to
check the mirror for missing files and to produce a shell script for
restoring the symbolic links.
================================================================================