WASD Hypertext Services - Technical Overview

6 - Server Configuration

6.1 - "Securing" The Site
    6.1.1 - Directory Listings
    6.1.2 - Server Reports
    6.1.3 - Scripting
    6.1.4 - Server Side Includes
    6.1.5 - Authorization
    6.1.6 - Miscellaneous Issues
6.2 - Content-Type Configuration
    6.2.1 - Adding Content-Types
    6.2.2 - MIME.TYPES
    6.2.3 - Unknown Content-Types
    6.2.4 - Explicitly Specifying Content-Type
6.3 - Virtual Services
6.4 - Request Throttling
6.5 - Error Reporting
    6.5.1 - Basic and Detailed
    6.5.2 - Site Specific
6.6 - OPCOM Logging
6.7 - Access Logging
    6.7.1 - Log Format
    6.7.2 - Log Period
    6.7.3 - Log Per-Service
    6.7.4 - Log Naming
    6.7.5 - Access Tracking
6.8 - Include File Directive
[next] [previous] [contents] [full-page]

Server configuration basically concerns itself with the very fundamental behaviour of the server process.  Requirements such as buffer and cache sizes, timeout values, scripting limits, content-types, icons, number and type of services offerted, etc., are determined from configuration information. 

By default, the system-table logical name HTTPD$CONFIG locates a common configuration file, unless an individual configuration file is specified using a job-table logical name.  Simple editing of this file changes the configuration.  Comment lines may be included by prefixing them with the hash "#" character.  Configuration file directives are not case-sensitive.  Any changes to the configuration file can only be enabled by restarting the HTTPd process using the following command on the server system. 

  $ HTTPD /DO=RESTART

A server's currently loaded configuration can be interrogated.  See 15 - Server Administration for further information. 


6.1 - "Securing" The Site

This section does not pretend to be a complete guide to keeping the "bad guys" out.  It does provide a short guide to making a site more-or-less liberal in the way the server supplies information about the site and itself.  The reader is also strongly recommended to a number of hard and Web based resources on this topic. 

The WASD package had it's genesis in making the VMS operating system and associated resources, in a development environment, available via Web technology.  For this reason the default configuration is fairly liberal, providing information of use in a technical environment, but that may be superfluous or less-than-desirable in other, possibly commercial environments.  For instance, directory listings can contain VMS file system META information, error reports can be generated with similar references along with reporting source code module and line information.  This section refers to configuration directives discussed in more detail later in this chapter and in the next. 


6.1.1 - Directory Listings

Published guidelines for securing a Web site generally advise against automatic directory listing generation.  Where a home page is not available this may leak information on other directory contents, provide parent and child directory access, etc.  Compounding this is the WASD facility to force a listing by providing a directory URL with file wildcards (not to decry it's usefulness in some environments). 


6.1.2 - Server Reports

Reports are pages generated by the server, usually to indicate an error or other non-success condition, but sometimes to indicate success (e.g. after a successful file upload).  Reports provide either basic or detailed information about the situation.  Sometimes the detailed information includes VMS file system details, system status codes etc.  To limit this information to a minimum indication adjust the following directives. 


6.1.3 - Scripting

Scripting has been a notorious source of server compromise, particularly within Unix environments where script process shell command-line issues require special attention.  The WASD CGI scripting interface does not pass any arguments on the command line, and is careful not to allow substitution when constructing it's CGI environment.  Never-the-less, script behaviours cannot be guranteed and care should be exercised in their deployment. 


6.1.4 - Server Side Includes

SSI documents are pages containing special markup directives interpreted by the server and replaced with dynamic content.  This can include detail about the server, the file or files making up the document, and can even include DCL commands and procedure activation for supplying content into the page.  All this by anyone who can author on the site. 


6.1.5 - Authorization

Authorization issues imply controlling access to various resources and actions and therefore require careful planning and implementation if compromise is to be avoided.  WASD has a quite capable and versatile authorization and authentication environment, with a significant number of considerations.  The reader refered to the chapter on this topic, 12 - Authentication and Authorization

WASD authorization cannot be enabled without the administrator configuring at least three resources, and so therefore cannot easily be "accidentally" activated.  One of these is the addition of a startup qualifier controlling where authentication information may be sourced.  Another the server configuration file.  The third, mapping paths against authorization configuration. 

For sites that may be particularly sensitive about inadvertant access to some resources it is possible to use the authorization configuration file as a type of cross-check on the mapping configuration file.  The server /AUTHORIZATION=ALL startup qualifier forces all access to be authorized (even if some are marked "none").  This means that if something "escapes" via the mapping file it will very likely be "caught" by an absence in the authorization file. 


6.1.6 - Miscellaneous Issues

Although it is of limited usefulness because server identity may be deduced from behaviour and other indicators the exact server and version may be obscured by using the otherwise undocumented /SOFTWARE= qualifier to change the server identification string to (basically) whatever the administrator desires.  This identification is included as part of all HTTP response headers. 

Historically and by default server configuration and authorization sources are contained within the server package tree.  There is no reason why they cannot be located anywhere the site prefers.  Generally all that is required is a change to logical name definition and server startup. 


Package Tree

On many sites there may be little need to access the full, or any of the WASD package tree.  A combination of mapping and/or authorization rules can relatively simply block or control access to it.  These examples can be easily tailored to suit a site's specific requirements. 

This example shows blocking all access to the /ht_root/ tree, except for documentation, source code, examples and exercise (performance results) areas. 

  # HTTPD$MAP
  pass /ht_root/doc/*
  pass /ht_root/src/*
  pass /ht_root/example/*
  pass /ht_root/exercise/*
  fail /ht_root/*

The next example forbids all access to the package tree unless authorized (the authorization detail would vary according to the site).  It also allows modify access for the Admin Menu and to the /ht_root/local/ area. 

  # HTTPD$MAP
  pass /ht_root/*
 
  # HTTPD$AUTH
  [WASD_WEB_ADMIN=id]
  /httpd/-/admin/* r+w
  /ht_root/local/* r+w
  /ht_root/* r


6.2 - Content-Type Configuration

HTTP uses an implementation of the MIME (Multi-purpose Internet Mail Extensions) specification for identifying the type of data returned in a response.  A MIME content-type consists of a plain text string describing the data as a type and slash-separated subtype, as illustrated in the following examples:

  text/html
  text/plain
  image/gif
  image/jpeg
  application/octet-stream
The content-type is returned to the client as part of the HTTP response, the client then using this information to correctly process and present the data contained in that response. 


6.2.1 - Adding Content-Types

In common with most HTTP servers WASD uses a file's suffix (extension, type, e.g. ".HTML, ".TXT", ".GIF"" to identify the data type within the file.  The [AddType] directive is used during configuration to bind a file type to a MIME content-type.  To make the server recognise and return specific content-types these directives map file types to content-types. 

With the VMS file system there is no effective file characteristic or algorithm for identifying a file's content without an exhaustive examination of the data contained there-in ... a very expensive process (and probably still inconclusive in many cases), hence the reliance on the file type. 

NOTE

When adding a totally new content-type to the configuration be sure also to bind an icon to that type using the [AddIcon] directive (see below).  If this is not done the default icon specified by [AddDefaultIcon] is displayed.  If that is not defined then a directory listing shows "[?]" in place of an icon. 

Mappings using [AddType] look like these. 

  [AddType]
  .html  text/html   HyperText Markup Language
  .txt   text/plain  plain text
  .gif   image/gif   image (GIF)
  .hlb   text/x-script /Conan  VMS Help library
  .decw$book   text/x-script   /HyperReader    Bookreader book
  *  internal/x-unknown   application/octet-stream


6.2.2 - MIME.TYPES

To allow the server to share content-type definitions with other MIME-aware applications, and for WASD scripts to be able to perform their own mapping on a shared understanding of MIME content it is possible to move the file suffix to content-type mapping from a collection of [AddType]s in HTTPD$CONFIG to an external file.  This file is usually named MIME.TYPES and is specified in HTTPD$CONFIG using the [AddMimeTypesFile] directive. 

Mappings using MIME.TYPES look like these. 

  # MIME type			Extension
  application/msword            doc
  application/octet-stream      bin dms lha lzh exe class
  application/oda               oda
  application/pdf               pdf
  application/postscript        ai eps ps
  application/rtf               rtf

A leading content-type is mapped to single or multiple file suffixes.  A general MIME.TYPES file commonly has content-types listed with no corresponding file suffix.  These are ignored by WASD.  Where a file suffix is repeated during configuration the latter version completely supercedes the former (with the Admin Menu showing an italicised and struck-through content-type to help identify duplicates). 

To allow the configuration information used by the server to generate directory listings with additional detail, WASD-specific extensions to the standard MIME.TYPES format are provided.  These are "hidden" in comment structures so as not to interfere with non-WASD application use.  All begin with a hash then an exclamation character ("#!") then another reserved character indicating the purpose of the extension.  Existing comments are unaffected provided the second character is anything but an exclamation mark! 

These directives are placed following the MIME-type entry they apply to.  An example of the contents of a MIME.TYPES file with various WASD extensions. 

  # MIME type			Extension
  application/msword            doc
  #! MS Word document
  #![DOC] /httpd/-/doc.gif
  application/octet-stream      bin dms lha lzh exe class
  #! binary content
  #![BIN] /httpd/-/binary.gif
  application/oda               oda
  application/pdf               pdf
  application/postscript        ai eps ps
  #! Adobe PostScript
  #![PS.] /httpd/-/postscript.gif
  application/rtf               rtf
  #! Rich Text Format
  #![RTF] /httpd/-/rtf.gif
  application/x-script          bks decw$bookshelf
  #! DEC Bookshelf
  #!/cgi-bin/hypershelf
  application/x-script          bkb decw$book
  #![BKR] /httpd/-/script.gif
  #! DEC Book
  #!/cgi-bin/hyperreader

Other reserved characters have been specified for development purposes but are not (perhaps currently) employed by the HTTP server. 


6.2.3 - Unknown Content-Types

If a file type is not recognised (i.e. no [AddType] or [AddMimeTypesFile] mapping corresponding to the file type) then by default WASD identifies its data as application/octet-stream (i.e. essentially binary data).  Most browsers respond to this content-type with a download dialog, allowing the data to be saved as a file.  Most commonly these unknown types manifest themselves when authors use "interesting" file names to indicate their purpose.  Here are some examples the author has encountered:

  README.VMS
  README.1ST
  READ-ME.FIRST
  BUILD.INSTRUCTIONS
  MANUAL.PT1 (.PT2, ...)

If the site administrator would prefer another default content-type, perhaps "text/plain" so that any unidentified files default to plain text, then this may be configured by specifying that content-type as the description of the catch-all file type entry.  Examples (use one of):

  [AddType]
  *  internal/x-unknown
  *  internal/x-unknown  application/octet-stream
  *  internal/x-unknown  text/plain
  *  internal/x-unknown  something/else-entirely
It is the author's opinion that unidentified file types should remain as binary downloads, not "text" documents, which they are probably more often not, but it's there if it's wanted. 


6.2.4 - Explicitly Specifying Content-Type

When accessing files it is possible to explicitly specify the identifying content-type to be returned to the browser in the HTTP response header.  Of course this does not change the actual content of the file, just the header content-type!  This is primarily provided to allow access to plain-text documents that have obscure, non-"standard" or non-configured file extensions. 

It could also be used for other purposes, "forcing" the browser to accept a particular file as a particular content-type.  This can be useful if the extension is not configured (as mentioned above) or in the case where the file contains data of a known content-type but with an extension conflicting with an already configured extension specifying data of a different content-type. 

Enter the file path into the browser's URL specification field ("Location:", "Address:"). Then, for plain-text, append the following query string:

  ?httpd=content&type=text/plain

For another content-type substitute it appropriately.  For example, to retrieve a text file in binary (why I can't imagine :^) use

  ?httpd=content&type=application/octet-stream

This is an example:

  file.unknown

  file.unknown?httpd=content&type=text/plain

It is posssible to "force" the content-type for all files in a particular directory.  Enter the path to the directory and then add

  ?httpd=index&type=text/plain

(or what-ever type is desired).  Links to files in the listing will contain the appropriate "?httpd=content&type=..." appended as a query string. 

This is an example:

  *.*

  *.*?httpd=index&type=text/plain


6.3 - Virtual Services

The WASD server is capable of concurrently supporting the same host name on different port numbers and a number of different host names (DNS aliased or multi-homed) using the same port number.  This capability is generally known as a virtual server.  Virtual services offer versatile and powerful multi-site capabilities using the one system and server process.  Service determination is based on the contents of the request's "Host:" header field.  If none is present it defaults to base service for the interface's IP address and port. 

The same mechanism also effectively allows a single instance of the configuration files to support multiple server processes (using the /SERVICE qualifier), either on the one system or across multiple systems, as in a cluster (i.e. supports "virtual" and "real" servers.) See STARTUP_SERVER.COM for further information on startup support for these configurations. 

WASD provides server process run-time parameters via the HTTPD$CONFIG configuration file.  These provide settings for logging, scripting, timeouts, file content-type mappings, etc.  The HTTPD$MSG file provides configurable system messages. 


[Service]

Using the [Service] HTTPD$CONFIG configuration parameter or the /SERVICE qualifier the server creates an HTTP service for each specified.  If the host name is omitted it defaults to the local host name.  If the port is omitted it defaults to 80. The first port specified in the service list becomes the "administration" port of the server, using the local host name, appearing in administration reports, menus, etc.  This port is also that specified when sending control commands via the /DO= qualifier (see 5.3.2 - Server Command Line Control). 

This rather contrived example shows a server configured to provide four services over two host names. 

  [Service]
  alpha.wasd.dsto.defence.gov.au
  alpha.wasd.dsto.defence.gov.au:8080
  beta.wasd.dsto.defence.gov.au
  beta.wasd.dsto.defence.gov.au:8000

Note that both the HTTPD$SERVICE configuration file (see 8 - Service Configuration and the /SERVICE= command-line qualifier (see 5.3 - HTTPd Command Line) override this directive. 


HTTPD$SERVICE

If the logical name HTTPD$SERVICE is defined (and it is optional) the HTTPD$CONFIG [Service] directive is not used.  As mentioned, this configuration file is optional.  For simple sites, those containing one or two services, the use of a separate service configuration file is probably not warranted.  Once the number begins to grow this file offers an easier-to-manage interface for those services. 

See 8 - Service Configuration for further detail. 


[[virtual-server]]

The essential profile of a site is established by it's mapped resources and any authorization controls, the HTTPD$MAP and HTTPD$AUTH configuration files respectively, and these two files support directives that allow configuration rules to be applied to all virtual services (i.e. a default), to a host name (all ports), or to a single specified service (host name and specific port). 

To restrict rules to a specified server (virtual or real) add a line containing the server host name, and optionally a port number, between double-square brackets.  All following rules will be applied only to that service.  If a port number is not present it applies to all ports for that service name, otherwise only to the service using that port.  To resume applying rules to all services use a single asterisk instead of a host name.  In this way default (all service) and server-specific rules may be interleaved to build a composite environment, server-specific yet with defaults.  Note that service-specific and service-common rules may be mixed in any order allowing common rules to be shared.  This descriptive example shows a file with one rule per line. 

  # just an example
  this rule applies to all services
  so does this
  and this one
  [[alpha.wasd.dsto.defence.gov.au]]
  this one however applies only to ALPHA, but to all ports
  as indeed does this
  [[beta.wasd.dsto.defence.gov.au:8000]]
  now we switch to the BETA service, but only port 8000
  another one only applying to BETA
  and a third
  [[*]]
  now we have a couple default rules
  that again apply to all servers

Both the mapping and authorization modules report if rules are provided for services that are not configured for the particular server process (i.e. not in the server's [Service] or /SERVICE parameter list).  This provides feedback to the site administrator about any configuration problems that exist, but may also appear if a set of rules are shared between multiple processes on a system or cluster where processes deliver differing services.  In this latter case the reports can be considered informational, but should be checked initially and then occasionally for misconfiguration. 


Unknown Virtual Server

If a service is not configured for the particular host address and port of a request one of two actions will be taken. 

  1. If the configuration directive [ServiceNotFoundURL] is set the request will be redirected to the specified URL.  This should contain a specific host name, as well as message page.  For the default page use:
      [ServiceNotFoundURL]  //server.host.name/httpd/-/servicenotfound.html
    
  2. If the above directive is not set the request is mapped using the default rules (e.g. [[*]]). It is possible to specify a rule set containing a default rule for each virtual server.  The unmatched request is then handled by a fallback rule, as illustrated in the following. 
      pass /*/-/admin/*
      pass /*/-/* /ht_root/runtime/*/*
      exec /cgi-bin/* /cgi-bin/*
      [[virtual1.host.name]]
      /* /web/virtual1/*
      / /web/virtual1/
      [[virtual2.host.name]]
      /* /web/virtual2/*
      / /web/virtual2/
      [[virtual3.host.name]]
      /* /web/virtual3/*
      / /web/virtual3/
      [[*]]
      /* /web/servicenotfound.html
    

This applies to dotted-decimal addresses as well as alpha-numeric.  Therefore if there is a requirement to connect via a numeric IP address such a service must have been configured. 

Note also that the converse is possible.  That is, it's possible to configure a service that the server cannot ever possibly respond to because it does not have an interface using the IP address represented by the service host. 


6.4 - Request Throttling

Request "throttling" is a term adopted to describe controlling the number of requests that can be processing against any specified path at any one time.  Requests in excess of this value are First-In-First-Out (FIFO) queued, up to an optional limit, waiting for a currently processing request to conclude allowing the next queued request to resume processing.  This is primarily intended to limit concurrent resource-intensive script execution but could be applied to any resource path.  Here's one dictionary description. 

throttle n 1: a valve that regulates the supply of fuel to the engine [syn: accelerator, throttle valve] 2: a pedal that controls the throttle valve; "he stepped on the gas" [syn: accelerator, accelerator pedal, gas pedal, gas, gun] v 1: place limits on; "restrict the use of this parking lot" [syn: restrict, restrain, trammel, limit, bound, confine] 2: squeeze the throat of; "he tried to strangle his opponent" [syn: strangle, strangulate] 3: reduce the air supply; of carburetors [syn: choke]

This is applied to a path (or paths) using the HTTPD$MAP mapping SET THROTTLE= rule (also see 10.3.5 - SET Rule).  The general format is

  set path throttle=n1[,n2,n3,n4,t/o1,t/o2]
  set path throttle=from[,to,resume,busy,t/o-queue,t/o-busy]
where

One way to read a throttle rule is "begin to throttle (queue) requests from the n1 value up to the n2 value, after which the queue is FIFOed up to the n3 value when it resumes queuing-only, up until the busy n4 value". 

Each integer represents the number of concurrent requests against the throttle rule path.  Parameters not required may be specified as zero or omitted in a comma-separated list.  The schema of the rule requires that each successive parameter be larger than that preceding it.  This basic consistency check is performed when the rule is loaded. 

For any rule the possible maximum number of requests that can be processed at any one time may be simply calculated through the addition of the n1 value to the difference of the n3 and n2 values (i.e. max = n1 + (n3 - n2)). The maximum concurrently queued as the difference of the n4 and the maximum concurrently processed. 

A comprehensive throttle statistics report is available from the Admin Menu (see 15 - Server Administration). 


Examples

  1. throttle=10

    Requests up to 10 are concurrently processed.  When 10 is reached futher requests are queued to server capacity. 

  2. throttle=10,20

    Concurrent requests to 10 are processed immediately.  From 11 to 20 requests are queued.  After 20 all requests are queued but also result in a request FIFOing off the queue to be processed (queue length is static, number being processed increases to server capacity). 

  3. throttle=15,30,40

    Concurrent requests up to 15 are immediately processed.  Requests 16 through to 30 are queued, while 31 to 40 requests result in the new requests being queued and waiting requests being FIFOed into processing.  Concurrent requests from 41 onwards are again queued, in this scenario to server capacity. 

  4. throttle=10,20,30,40

    Concurrent requests up to 10 are immediately processed.  Requests 11 through to 20 will be queued.  Concurrent requests from 21 to 30 are queued too, but at the same time waiting requests are FIFOed from the queue (resulting in 10 (n1) + 10 (n3-n2) = 20 being processed).  From 31 onwards requests are just queued.  Up to 40 concurrent requests may be against the path before all new requests are immediately returned with a 503 "busy" status.  With this scenario no more than 20 can be concurrently processed with 20 concurrently queued. 

  5. throttle=10,,,30

    Concurrent requests up to 10 are processed.  When 10 is reached requests are queued up to request 30. When request 31 arrives it is immediately given a 503 "busy" status. 

  6. throttle=10,20,30,40,00:02:00

    This is basically the same as scenario 4) but with a resume-on-timeout of two minutes.  If there are currently 15 (or 22 or 28) requests (n1 exceeded, n3 still within limit) the queued requests will begin processing on timeout.  Should there be 32 processing (n3 has reached limit) the request will continue to sit in the queue.  The timeout would not be reset. 

  7. throttle=15,30,40,,,00:03:00

    This is basically the same as scenario 3) but with a busy-on-timeout of three minutes.  When the timeout expires the request is immediately dequeued with a 503 "busy" status. 


Mapping Reload

Throttling is applied using mapping rules.  The set of these rules may be changed within an executing server using map reload functionality.  This means the number of, and/or contents of, throttle rules may change during server execution.  The throttle functionality needs to be independent of the the mapping functionality (requests are processed independently of mapping rules once the rules have been applied).  After a mapping reload the contents of the throttle data structures may be at variance with the constraints currently executing requests began processing under. 

This should have little deleterious effect.  The worst case is mis-applied constraints on the execution limits of changed request paths, and slightly confusing data in the Throttle Report.  This quickly passes as requests being processed under the previous throttle constraints conclude and an entirely new collection of requests created using the constraints of the currently loaded rules are processed. 


6.5 - Error Reporting

By default the server provides it's own internal error reporting facility.  These reports may be configured as basic or detailed on a per-path basis, as well as determining the basic "look-and-feel".  For more demanding requirements the [ErrorReportPath] configuration directive allows a redirection path to be specified for error reporting, permitting the site administrator to tailor both the nature and format of the information provided.  A Server Side Include document, CGI script or even standard HTML file(s) may be specified.  Generally an SSI document would be recommended for it's simplicity yet versatility. 


6.5.1 - Basic and Detailed

Internally generated error reports are the most efficient.  These can be delivered with two levels of error information.  The default is more detailed. 

ERROR 404  -  The requested resource could not be found.
Document not found  ...  /ht_root/index.html
(document, bookmark, or reference requires revision)
Additional information:  1xx2xx3xx4xx5xxHelp

WASD/7.0.0 Server at wasd.dsto.defence.gov.au Port 80

There is also the more basic. 

ERROR 404  -  The requested resource could not be found.
Additional information:  1xx2xx3xx4xx5xxHelp

WASD/7.0.0 Server at wasd.dsto.defence.gov.au Port 80

These can be set per-server using the [ReportBasicOnly] configuration directive, or on a per-path basis in the HTTPD$MAP configuration file.  The basic report is intended for environments where traditionally a minimum of information might be provided to the user community, both to reduce site configuration information leakage but also where a general user population may only need or want the information that a document was either found or not found.  The detailed report often provides far more specific information as to the nature of the event and so may be more appropriate to a more technical group of users.  Either way it is relatively simple to provide one as the default and the other for specific audiences.  Note that the detailed report also includes in page <META> information the code module and line references for reported errors. 

To default to a basic report for all but selected resource paths introduce the following to the top of the HTTPD$MAP configuration file. 

  # default is basic reports
  set /* report=basic
  set /internal-documents/* report=detailed
  set /other/path/* report=detailed

To provide the converse, default to a detailed report for all but selected paths use the following. 

  # default is detailed reports
  set /web/* report=basic


Other Customization

The additional reference information included in the report may be disabled using the appropriate HTTPD$MSG [status] message item.  Emptying this message results in an error report similar to the following. 

ERROR 404  -  The requested resource could not be found.

WASD/7.0.0 Server at wasd.dsto.defence.gov.au Port 80

The server signature may be disabled using the HTTPD$CONFIG [ServerSignature] configuration directive.  This results in a minimal error report. 

ERROR 404  -  The requested resource could not be found.

A simple approach to providing a site-specific "look-and-feel" to server reports is to customize the [ServerReportBodyTag] HTTPD$CONFIG configuration directive.  Using this directive report page background colour, background image, text and link colours, etc., may be specified for all reports.  It is also possible to more significantly change the report format and contents (within some constraints), without resorting to the site-specific mechansims refered to below, by changing the contents of the appropriate HTTPD$MSG [status] item.  This should be undertaken with care. 


6.5.2 - Site Specific

Customized error reports can be generated for all or selected HTTP status status associated with errors reported by the server using the HTTPD$CONFIG [ErrorReportPath] and HTTPD$SERVER [ServiceErrorReportPath] configuration directives.  To explicitly handle all error reports specify the path to the error reporting mechanism (see description below) as in the following example. 

  [ErrorReportPath] /httpd/-/reporterror.shtml

To handle only selected error reports add the HTTP status codes following the report path.  In this example only 403 and 404 errors are explicitly handled, the rest remain server-generated.  This is particularly useful for static error documents. 

  [ErrorReportPath] /httpd/-/reporterror.shtml 403 404

Site-specific error reporting works by internal redirection.  When an error is reported the original request is concluded and the request reconstructed using the error report path before internally being reprocessed.  For SSI and CGI script handlers error information becomes available via a specially-built query string, and from that as CGI variables in the error report context.  One implication is the original request path and query string are no longer available.  All error information must be obtained from the error information in the new query string. 

It is suggested with any use of this facility the reporting document(s) be located somewhere local, probably HT_ROOT:[RUNTIME.HTTPD], and then enabled by placing the appropriate path into the [ErrorReportPath] configuration directive. 

  [ErrorReportPath] /httpd/-/reporterror.shtml

Note that virtual services can subsequently have this path mapped to other documents (or even scripts) so that some or all services may have custom error reports.  For instance the following arrangement provides each host (service) with an customized error report. 

  # HTTPD$CONFIG
  [ErrorReportPath] /errorreport.shtml
 
  # HTTPD$MAP
  [[alpha.wasd.dsto.gov.au]]
  pass /errorreport.shtml /httpd/-/alphareport.shtml
  [[beta.wasd.dsto.gov.au]]
  pass /errorreport.shtml /httpd/-/betareport.shtml
  [[gamma.wasd.dsto.gov.au]]
  pass /errorreport.shtml /httpd/-/gammareport.shtml


Using Static HTML Documents

Static HTML documents are a good choice for site-specific error messages.  They are very low overhead and are easily customizable.  One per possible response error status code is required.  When providing an error report path including a "!UL" introduces the response status code into the file path, providing a report path that includes a three digit number representing the HTTP status code.  A file for each possible or configured code must then be provided, in this example for 403 (authorization failure), 404 (resource not found) and 502 (bad gateway/script). 

  [ErrorReportPath] /httpd/-/reporterror!UL.html 403 404 502

This mapping will generate paths such as the following, and require the three specified to respond to those errors. 

  /httpd/-/reporterror403.html
  /httpd/-/reporterror404.html
  /httpd/-/reporterror502.html


Using an SSI Document

SSI documents provide the versatility of dynamic report generation for but they do take time and CPU for processing, and this may be a significant consideration on busy sites. 

Three example SSI error report documents are provided. 

  1. HT_ROOT:[EXAMPLE]REPORTERROR1.SHTML
    Provides a report identical with those internally generated in versions prior to v7.0.

  2. HT_ROOT:[EXAMPLE]REPORTERROR2.SHTML
    This is a minor variation, showing how the format may be easily customized.

  3. HT_ROOT:[EXAMPLE]REPORTERROR3.SHTML
    This version has a radically different format and content, with much less specific error information (which some administrator's may consider advantageous). When generated these reports look something like this.

  4. HT_ROOT:[EXAMPLE]REPORTERROR4.SHTML
    This example uses the report format provided with WASD v7.0 and later, and look something like this.

  5. HT_ROOT:[EXAMPLE]REPORTERROR5.SHTML
    This is another variation, showing how the format may be easily customized. When generated this report looks something like this.

The following SSI variables are available specifically for generating error reports.  The <!--#printenv --> statement near the top of the file may be uncommented to view all SSI and CGI variables available. 


Using a Script

It is also possible to report using a script.  This is discouraged dues to it's relative complexity, and should only be undertaken for verify specific reasons of limitation within the SSI implementation.  The source code HT_ROOT:[SRC.MISC]REPORTERROR.C provides such an implementation example. 


6.6 - OPCOM Logging

Significant server events may be optionally displayed via a selected operator's console and recorded in the operator log.  Various categories of these events may be selectively enabled via HTTPD$CONFIG directives (see 7 - Server Configuration Directives). 


6.7 - Access Logging

WASD provides a versatile access log, allowing data to be collected in Web-standard common and combined formats, as well as allowing customization of the log record format.  It is also possible to specify a log period.  If this is done log files are automatically changed according to the period specified.  Exclude requests from specified hosts using the [LogExcludeHosts] configuration parameter. 


6.7.1 - Log Format

The configuration parameter [LogFormat] and the server qualifier /FORMAT specifies one of three pre-defined formats, or a user-definable format.  Most log analysis tools can process the three pre-defined formats.  There is a small performance impost when using the user-defined format, as the log entry must be specially formatted for each request. 


User-Defined

The user-defined format allows customised log formats to be specified using a selection of commonly required data.  The specification must begin with a character that is used as a substitute when a particular field is empty (use "\0" for no substitute, as in the "windows log format" example below). 

Two different "escape" characters introduce the following parameters:


A "!" followed by

AR - authentication realm (if any)
AU - authenticated user name (if any)
BB - bytes in body (excludes response header)
BY - bytes in response (includes header)
CA - client address
CN - client host name (or address if DNS lookup disabled)
EM - request elapsed time in milliseconds
ES - request elapsed time in fractional seconds
ID - session track ID
ME - request method
PA - request path (not to be confused with "RQ")
QS - request query string (if any)
RF - referer (if any)
RQ - complete request string (see below)
RS - response status code
SN - server host name
SC - script name (if any)
SM - request scheme (http: or https:)
SP - server port
TC - request time (common log format)
TG - request time (GMT)
TV - request time (VMS format)
UA - user agent


A "\" followed by

0 - a null character (used to define the empty field character)
!  - insert an "!"
\ - insert a "\"
n - insert a newline
q - insert a quote (so that in DCL the quotes won't need escaping!)
t - insert a TAB


Any other character is directly inserted into the log entry. 


Note on "PA" and "RQ"

The "PA" and "RQ" have distinct roles.  In general the "RQ" (request) directive will always be used as this is the full request string; script component (if any), path string and query string component (if any).  The "PA" directive is merely the path string after any script and query string components have been removed. 


Examples

  1. The equivalent of the common log format is:
      -!CN - !AU [!TC] \q!RQ\q !RS !BY
    

  2. The combined log format could be specified as:
      -!CN - !AU [!TC] \q!RQ\q !RS !BY \q!RF\q \q!UA\q
    

  3. The O'Reilly WebSite "windows log format" would be created by:
      \0!TC\t!CA\t!SN\t!AR\t!AU\t!ME\t!PA\t!RQ\t!EM\t!UA\t!RS\t!BB\t
    

  4. The common log format with appended request duration in seconds could be provided using:
      -!CN - !AU [!TC] \q!RQ\q !RS !BY !ES
    


6.7.2 - Log Period

The access log file may have a period specified against it, producing an automatic generation of log file based on that period.  This allows logs to be systematically named, ordered and kept to a managable size.  The period specified can be one of

The log file changes on the first request after midnight of the new period.  When using a weekly period the new log file comes into effect on the first request following midnight on the specified day. 

When using a periodic log file, the file name specified by HTTPD$LOG or the configuration parameter [LogFile] is partially ignored, only partially because the directory component of it is used to located the generated file name.  The periodic log file name generated comprises

as in the following example
  HT_LOGS:WASD_80_19971013_ACCESS.LOG

For the daily period the date represents the request date.  For the weekly period it is the date of the previous (or current) day specified.  That is, if the request occurs on the Wednesday for a weekly period specified by Monday the log date show the last Monday's. For the monthly period it uses the first. 


6.7.3 - Log Per-Service

By default a single access log file is created for each HTTP server process.  Using the [LogPerService] configuration directive a log file for each service provided by the HTTPd is generated (see 6.3 - Virtual Services).  The [LogNaming] format can be any of "NAME" (default) which names the log file using the first period-delimited component of the IP host name, "HOST" which uses as much of the IP host name as can be accomodated within the maximum 39 character filename limitation (of ODS-2), or "ADDRESS" which uses the full IP host address in the name.  Both HOST and ADDRESS have hyphens substituted for periods in the string.  If these are specified then by default the service port follows the host name component.  This may be suppressed using the [LogPerServiceHostOnly] directive, allowing a minimum extra 3 characters in the name, and combining entries for all ports associated with the host name (for example, a standard HTTP service on port 80 and an SSL service on port 443 would have entries in the one file). 


6.7.4 - Log Naming

When per-period or per-service logging is enabled the access log file has a name generated.  Part of this name is the host's name or IP address.  By default the host name is used, however if the host IP address is specified the dot-numeric address is used, hyphens being substituted for the periods.  Accepted values for the [LogNaming] configuration directive are:

Examples of generated per-service (non-per-period) log names:

  HT_LOGS:131-185-250-202_80_ACCESS.LOG
  HT_LOGS:WASD-DSTO-DEFENCE-GOV-AU_80_ACCESS.LOG
  HT_LOGS:WASD_80_ACCESS.LOG

Examples of generated per-period (with/without per-service) log names:

  HT_LOGS:131-185-250-202_80_19971013_ACCESS.LOG
  HT_LOGS:WASD-DSTO-DEFENCE-GO_80_19971013_ACCESS.LOG
  HT_LOGS:WASD_80_19971013_ACCESS.LOG


6.7.5 - Access Tracking

The term access tracking describes the ability to follow a single user's accesses through a particular site or group of related sites.  This is accomplished by setting a unique cookie in a user's browser.  This cookie is then sent with all requests to that site.  The site detects the cookie's unique identifier, or token, and includes it the access log, allowing the user's route through the site or sites to be reviewed.  Note that a browser must have cookies enabled for this mechanism to operate. 

WASD access tracking is controlled using the [Track...] directives.  The tracking cookie uses an opaque, nineteen character string as the token (e.g. "ORoKJAOef8sAAAkuACc").  This token is spatially and temporally completely unique, generated the first time a user's browser accesses the site.  This token is by default added to the server access log in the common format "remote-ID" location.  It can also be placed into custom logs.  From this identifier in the logs a session's progress may be easily tracked.  Note that the token contains nothing related to the user's actual identity!  It is merely a unique identifier that tags a single browser's access trail through a site. 

The [Track] directive enables access tracking on a per-server basis.  By default all non-proxy services will then have tracking enabled.  Individual services may be then be disabled (or enabled in the case of proxy services) using the per-service ";notrack" and ";track" parameters. 

By default a session track token expires when the user closes the browser.  To encourage the browser to keep this token between uses enable multi-session tracking using the [TrackMultiSession] directive.  Note that browsers may dispose of any cookie at any time resources become scarce, and that users can also remove them. 

Session tracking can be extended from the default of the local server (virtual if applicable) to a group of servers within a local domain.  This means the same, initial identifier appears in the logs of all WASD servers in a related group of hosts.  Of course tracking must be enabled on all servers.  The host grouping is specified using the [TrackDomain] directive (this follows the general rules governing cookie domain behaviour - see RFC2109). Most host grouping require a minimum of three dots in the specification.  For example (note the leading dot)

  .site.org.domain

which would match the following servers, "curly.site.org.domain", "larry.site.org.domain", "moe.site.org.domain", etc.  Sites in top-level domains (e.g. "edu", "com", "org") need only specify a minimum of two periods. 


6.8 - Include File Directive

WASD uses multiple configuration files for a server and it's site, each one providing for a different functional aspect ... configuration, virtual services, path mapping, authorization, etc.  Generally these configuration files are "flat", with all required directives included in a single file.  This provides a simple and straight-forward approach suitable for most sites and allows for the provision of Admin Menu on-line configuration of several aspects. 

It is also possible to build site configurations by including the contents of referenced files.  This may provide a structure and flexibility not possible using the flat-file approach.  All WASD configuration files allow the use of an [IncludeFile] directive.  This takes a VMS file specification parameter.  The file's contents are then loaded and processed as if part of the parent configuration file.  These included files are allowed to be nested to a depth of two (i.e. the configuration file can include a file which may then include another file). 

The following is an example used to build up the mapping rules for four virtual services supported on the one server. 

  # HTTPD$MAP
 
  [[alpha.site.com]]
  [IncludeFile]  HT_ROOT:[LOCAL]MAP_ALPHA_80.CONF
  [[alpha.site.com:443]]
  [IncludeFile]  HT_ROOT:[LOCAL]MAP_ALPHA_443.CONF
 
  [[beta.site.com]]
  [IncludeFile]  HT_ROOT:[LOCAL]MAP_BETA_80.CONF
  [[beta.site.com:443]]
  [IncludeFile]  HT_ROOT:[LOCAL]MAP_BETA_443.CONF
 
  [[*]]
  [IncludeFile]  HT_ROOT:[LOCAL]MAP_COMMON.CONF
NOTE

Such configurations cannot be managed using Admin Menu interfaces.  Files containing [IncludeFile] directives are noted during server startup and if an Admin Menu configuration interface is accessed where this would be a problem an explanatory message and warning is provided.  A configuration can still be saved but the resulting configuration will be a flat-file representation of the server configuration, not the original hierarchical one. 


[next] [previous] [contents] [full-page]