More about Perl and CGIplus/RTE
 Version 1.2.1, 17th May 2003


Perl is a powerful language and should be used to greater advantage. Unfortunately in a CGI context the overheads of starting Perl (particularly under VMS) can severly impact request response time and system overhead.

WASD's persistant scripting (CGIplus) and persistant Run-Time Environment (RTE) both improve script response time and reduce system impact by allowing scripting resources to be loaded into a process just once before processing multiple requests. For general information on Perl and WASD, and for detailed information on CGIplus and RTE see WASD Scripting Environment. Just compare the latency difference between the CGI, CGIplus and RTE usage (after the first access) of exactly the same script in examples provided in the following two sections. Of course you require Perl to be installed and functioning from the command line before it can be used as a scripting tool!

Other general information on WASD and Perl may also be found in the above document.

Developed and tested against (and probably only suitable for) Perl v5.8.0 and v5.6.1.

This kit is designed for the WASD 8.1 (or later) environment and so requires that the example scripts be installed into the server's scripting file space before use. The procedure INSTALL.COM can assist with this.


CGI, CGIplus, RTE ... which to choose?

With multiple WASD scripting technologies available choosing which is best suited to a given application can somtimes seem daunting. This table attempts to summarize the issues.


TechnologyDescription
CGI Standard CGI is just that, the lowest common denominator between server environments. Nothing persists (except perhaps the process). The script is activated with each request, receiving request parameters via CGI environment variables. Processing occurs, with response output. The script runs down and the process becomes quiescent or exits. Perl engine activation, script source parsing (and any required modules), script activation, execution and rundown are all fairly expensive undertakings. Eliminating one or more of these improves latency and reduces system resource consumption.
CGIplus CGIplus allows the script itself to persist. This means Perl engine activation, script source parsing and script resource instantiation occurs only once for any number of successive requests. This is particularly significant with highly latent resources, such as databases, network connections, etc. CGIplus scripts must be designed and implemented to take persistence into account. Care must be taken to ensure that there is no leakage of data between requests and that resources used and no longer required are released - script rundown can no longer be used to perform these tasks! It is generally a simple undertaking to make a script able to behave in a CGI or CGIplus manner depending on how it is activated. The Perl module CGIplus.pm is available to perform the few extra steps required for CGIplus.
RTE A Run-Time Environment allows the scripting engine to persist. For interpreters such as Perl this can significantly reduce engine activation latency and resource consumption. The script itself does not persist and so all state is lost between activations. In the case of the Perl RTE engine the parsed code for that script is cached and will be used again when the script is next activated. Eliminating this step on subsequent activations provides a significant saving in latency and system resources. Generally any plain old CGI script can be used with the RTE, no specific changes are required for this environment. It can be a little fussy about namespace. All variables must have an explicit package reference or be "my"ed.

More information on these technologies is available in the WASD Scripting Overview.


Hints and Kinks

A few suggestions for when using Perl with WASD.


CGI Variables

The Perl (and PerlRTE) interpreter needs to be informed about the source of it's environment variables. In WASD's case the CGI variables are stored as DCL symbols. This requires the definition of a logical name, which can be on a system-wide basis (which of course affects all Perl scripting).

  $ DEFINE /SYSTEM PERL_ENV_TABLES CLISYM_GLOBAL,LNM$PROCESS

Alternatively, it may be applied only to the scripting environment through the use of a Perl "wrapper" procedure. The example PERL.COM shows how this can be done.

Note that the next comment only applies to standard Perl CGI. It does not apply when using the PerlRTE or the CGIplus.pm module described below. With these internal processing performs all such munging transparently.

Many, perhaps most, open-source Perl CGI scripts will be using CGI variable names without the common VMS (and WASD-default) prefix of "WWW_". These will not function correctly using $ENV{} to retrieve values with the standard Perl interpreter. To remove the "WWW_" prefix add an appropriate rule to HTTPD$MAP.

 set /cgi*-bin/*.pl CGIprefix=


POSTed Requests

CGI.pm and Perl v5.6.0 could not read a POSTed multipart stream satisfactorily producing the error "CGI.pm: Server closed socket during multipart read (client aborted?)". This is apparently a known problem fixed by migrating to Perl 5.6.1 (or later) and it's CGI.pm.

When not using the PerlRTE engine some redirection is necessary to get the correct stream to connect to Perl's <STDIN>. It controls the script process via SYS$INPUT and supplies any POSTed body via the separate stream HTTP$INPUT requiring redirection before Perl engine activation.

  $ define /user perl_env_tables clisym_global,lnm$process
  $ define /user sys$input http$input
  $ perl device:[directory]script.pl

This approach can be seen in use with PerlRTEexample4.pl and PerlRTEexample4.com below. Although PerlRTE does not actually require this because the two scripts can be used in both environments (to illustrate differences in latency) it is used in both.

There are also often issues in getting POSTed content to the script without VMS' record boundary munging interfering. This generally requires the <STDIN> stream being placed into binary mode, sometimes not trivial in standard VMS Perl. The PerlRTE environment can be used as if it was the standard Perl verb eminiating this requirement. See the comments in the PERLRTE.C prologue. This is done with the standard CGI example of PerlRTE_example5 (look at PerlRTE_example5.com).


Binary Streams

By default VMS I/O streams are record-oriented and RMS has a tendency to want to adjust carriage-control on these. This is often counter-productive when needing to return a binary (non-textual) response.

The VMS::stdio Perl extension allows I/O streams to be very finely controlled using low-level C-RTL and RMS functionality. See the appropriate Perl documentation page. The CGIplus.pm module described below uses this extensively.


Perl RTE

The Perl Run-Time Environment provides a persistent Perl engine, caching both the running Perl interpreter and any one or more Perl scripts activated and parsed by that interpreter.

The Perl Run-Time Environment should execute any standard Perl script using any collection of modules. This approach uses standard techniques and code described in the "perlembed" document to load and keep cached multiple script and module sources. How isolated are these scripts in reality? The document indicates effectively enough! Each is treated as an autonomous package and so storage restrictions etc. need to be observed. However apart from that it would seem as if any old (perhaps slightly tweaked) CGI script could be used within this environment.

The embedding code maintains the last modification time of each script cached and checks this against the last modification time of the script file before each activiation. If there is a difference in the two times (i.e. the file has changed in some way) the cache is overwritten with a fresh evaluation of the script. There is no need to explicitly flush this cache in any way.

The CGI environment variables are available via the ENV associative array. There are a number of ways to customize the startup of this environment. See the PerlRTE.c and PerlRTeng.c sources for detailed information.


Demonstrations

These examples and demonstrations rely on the following configuration.

  # HTTPD$CONFIG
  [DclScriptRunTime]
  .PL PERL
  # or perhaps (depending on the local system)
  # .PL $PERL_ROOT:[000000]PERL.EXE

  # HTTPD$MAP
  exec+ /plrte/* (CGI-BIN:[000000]PERLRTE.EXE)/cgi-bin/*

Note the differences in latency between standard CGI and the RTE!

  Script Sources:  PerlRTE_example1.pl
PerlRTE_example2.pl
PerlRTE_example3.pl (loads the CGI.pm module for extra latency)
PerlRTE_example4.pl (it's standard CGI POST wrapper PerlRTE_example4.com)
PerlRTE_example5.pl (it's standard CGI POST wrapper PerlRTE_example5.com)
Standard CGI:  /cgi-bin/PerlRTE_example1
/cgi-bin/PerlRTE_example2
/cgi-bin/PerlRTE_example3
one two three
using PerlRTEexample5
 (upload will not work with Perl 5.6.0)
If the CGI variables appear to be empty make adjustments in line with CGI Variables.
Persistent RTE:  /plrte/PerlRTE_example1   [restart RTE]
/plrte/PerlRTE_example2   [restart RTE]
/plrte/PerlRTE_example3   [restart RTE]
one two three
using PerlRTEexample5
 (upload will not work with Perl 5.6.0)


20x Performance

Benchmarking the PerlRTE_example3.pl script (which requires CGI.PM) using ApacheBench on the author's development system indicated 1.7 and 32 requests/second for the CGI and RTE usages respectively.  Expressed another way, the RTE usage responded 20 times faster!!  Remember this is the same script - no changes apart from invocation path.


Scripting Engine

The Perl RTE is suitable for use as the main Perl scripting engine. That is, it can activate standard CGI, CGIplus and RTE scripts. To enable this for the execution of all CGI Perl scripts by the server add or change the following HTTPD$MAP rules and reload.

  if (!script-name:/plrte/*) map /cgi-bin/*.pl* /plrte/*.pl*
  exec+ /plrte/* (CGI-BIN:[000000]PERLRTE.EXE)/cgi-bin/*

Also see mapping caution in CGIplus.pm.


Building PerlRTE

As there is some minor variation in socket support between Perl 5.8 and 5.6 the build procedure needs to be directed which environment to support. The default is 5.8.

Compile*+ Link for Perl 5.8

  $ SET DEFAULT HT_ROOT:[SRC.PERL]
  $ @BUILD_PERLRTE BUILD 5.8

Link-only for Perl 5.8

  $ SET DEFAULT HT_ROOT:[SRC.PERL]
  $ @BUILD_PERLRTE LINK 5.8

Compile*+ Link for Perl 5.6

  $ SET DEFAULT HT_ROOT:[SRC.PERL]
  $ @BUILD_PERLRTE BUILD 5.6

Link-only for Perl 5.6

  $ SET DEFAULT HT_ROOT:[SRC.PERL]
  $ @BUILD_PERLRTE LINK 5.6

* To compile the RTE sources the Perl distribution header files must available from PERL_ROOT:[000000].

As this release is designed for the post WASD 8.1 environment it is then necessary to install the Perl RTE interpreter into the server scripting file space.

  $ @INSTALL


CGIplus Perl

CGIplus Perl is an alternative, perhaps complimentary method to Perl RTE for increasing efficiency and reducing request latency, again using the idea of persistence. This time it is up to the script to maintain the state.

These examples also demonstrate the use of the VMS::DCLsym and VMS::Stdio extensions. The principles may be more generally applied to other scripts.

Script Source:  CGIplusPM_example1.pl
Standard CGI:  /cgi-bin/CGIplusPM_example1/ht_root/src/perl/?this+is+a+query+string
CGIplus:  /cgiplus-bin/CGIplusPM_example1/ht_root/src/perl/?this+is+a+query+string   [restart]


30x Performance

Benchmarking this using ApacheBench on the author's development system indicated 0.8 and 25 requests/second for the CGI and CGIplus usages respectively.  Expressed another way, the CGIplus usage responded approximately 30 times faster!  Remember this is the same script - no changes apart from invocation path.


CGIplus.pm

The CGIplus.pm Perl module is intended to assist authors write scripts that may be used transparently in both vanilla CGI environments (including non-WASD) as well as under WASD CGIplus. Perl implementation refinements courtesy of Dick Munroe (munroe@csworks.com).

A script using CGIplus.pm (or any other autonomous CGIplus behaviour) should never be activated using an RTE path; that is, one using the following mapping syntax.

  exec+ ($cgi-bin:[000000]perlrte.exe)/plrte/* /cgi-bin/* 

When an RTE becomes quiescent the server will give it another script. With the CGIplus.pm request processing loop is active servicing the one script. This unintended and probably incorrect script will become active. Always activate CGIplus.pm enabled scripts via a CGIplus path.

  exec+ /plplus/* /cgi-bin/* 

CGIplus.pm will detect such a mapping mistake and die!


Binary Responses

VMS' RMS complicates output streams under Perl. This is a particular issue with CGIplus end-of-file sentinals, which must be output as a single record. CGIplus.pm attempts to provide a simple mechanism for providing binary streams if necessary, while still ensuring it's own records are not interfered with. This uses Charles Bailey's VMS::Stdio extension module built into most versions of VMS Perl.

This script demonstrates how the module's stream functions can be directly used.

Script Source:  CGIplusPM_example2.pl
Standard CGI:  /cgi-bin/CGIplusPM_example2
CGIplus:  /cgiplus-bin/CGIplusPM_example2   [restart]

This script demonstrates how simply to return a binary file as a response.

Script Source:  CGIplusPM_example3.pl
Standard CGI:  /cgi-bin/CGIplusPM_example3
CGIplus:  /cgiplus-bin/CGIplusPM_example3   [restart]

Any accessable image location may be added to these scripts following the script part. Again, because these are being accessed via a CGIplus script, notice the difference in latency between the initial and subsequent requests.


Notes:

There have been a number of changes implemented in the CGIplus.pm supplied with PerlRTE 1.2. These should be backward compatible. The original module is supplied in the source directory if required.

  1. The module can be used with PerlRTE v1.2 as the scripting engine (the previous version could not).

  2. All CGI variables names have any leading "WWW_" automatically stripped. This is a more common convention for CGI variables names. This behaviour can be changed - check the module source.

  3. CGI variables are now also available from the more standard ENV associative array. Previously they were only accessable using CGIplus::var().


Useful Example Script(s)