Sun Microsystems
Products & Services
 
Support & Training
 
 

Previous Previous     Contents     Index     Next Next

Composing a Checkpointing Job Script

Shell scripts for kernel-level checkpointing are the same as regular shell scripts.

Shell scripts for user-level checkpointing jobs differ from regular batch scripts only in their ability to properly handle getting restarted. The environment variable RESTARTED is set for checkpointing jobs which are restarted. Use this variable to skip sections of the job script that need to be executed only during the initial invocation.

A transparently checkpointing job script might look like Example 4-3.

Example 4-3 Example of a Checkpointing Job Script

#!/bin/sh
#Force /bin/sh in Grid Engine
#$ -S /bin/sh

# Test if restarted/migrated
if [ $RESTARTED = 0 ]; then
     # 0 = not restarted
     # Parts to be executed only during the first
     # start go in here
     set_up_grid
fi

# Start the checkpointing executable
fem
#End of scriptfile

The job script restarts from the beginning if a user-level checkpointing job is migrated. The user is responsible for directing the program flow of the shell script to the location where the job was interrupted. Doing so skips those lines in the script that must be executed more than once.


Note - Kernel-level checkpointing jobs are interruptible at any point of time. The embracing shell script is restarted exactly from the point where the last checkpoint occurred. Therefore the RESTARTED environment variable is not relevant for kernel-level checkpointing jobs.


Submitting, Monitoring, or Deleting a Checkpointing Job From the Command Line

Type the following command with the appropriate options:

# qsub options arguments

The submission of a checkpointing job works in the same way as for regular batch scripts, except for the qsub -ckpt and qsub -c commands. These commands request a checkpointing mechanism. The commands also define the occasions at which checkpoints must be generated for the job.

The -ckpt option takes one argument, which is the name of the checkpointing environment to use. See "Configuring Checkpointing Environments" in N1 Grid Engine 6 Administration Guide.

The -c option is not required. -c also takes one argument. Use the -c option to override the definitions of the when parameter in the checkpointing environment configuration. See the checkpoint(5) man page for details.

The argument to the -c option can be one of the following one-letter selection, or any combination thereof. The argument can also be a time value.

  • n - No checkpoint is performed. n has the highest precedence.

  • s - A checkpoint is generated only if the sge_execd on the jobs host is shut down.

  • m - Generate checkpoint at minimum CPU interval defined in the corresponding queue configuration. See the min_cpu_interval parameter in the queue_conf(5) man page.

  • x - A checkpoint is generated if the job is suspended.

  • interval - Generate checkpoint in the given interval but not more frequently than defined by min_cpu_interval. The time value must be specified as hh:mm:ss. This format specifies two digit hours, minutes, and seconds, separated by colons.

The monitoring of checkpointing jobs differs from monitoring regular jobs. Checkpointing jobs can migrate from time to time. Checkpointing jobs are therefore not bound to a single queue. However, the unique job identification number and the job name stay the same.

The deletion of checkpointing jobs works in the same way as described in Monitoring and Controlling Jobs From the Command Line.

Submitting a Checkpointing Job With QMON

Follow the instructions in Submitting Advanced Jobs With QMON, taking note of the following additional information.

The submission of checkpointing jobs with QMON is identical to submitting regular batch jobs, with the addition of specifying an appropriate checkpointing environment. As explained in Submitting Advanced Jobs With QMON, the Submit Job dialog box provides a field for the checkpointing environment that is associated with a job. Next to the field is a button that opens the following Selection dialog box.

Dialog box titled Select an Item. Shows available checkpoint
objects and a field for selecting one. Shows OK, Cancel, and Help buttons.

Here you can select a suitable checkpoint environment from the list of available checkpoint objects. Ask your system administrator for information about the properties of the checkpointing environments that are installed at your site. Or see "Configuring Checkpointing Environments" in N1 Grid Engine 6 Administration Guide.

File System Requirements for Checkpointing

When a user-level checkpoint or a kernel-level checkpoint that is based on a checkpointing library is written, a complete image of the virtual memory covered by the process or job to be checkpointed must be dumped. Sufficient disk space must be available for this purpose. If the checkpointing environment configuration parameter ckpt_dir is set, the checkpoint information is dumped to a job private location under ckpt_dir. If ckpt_dir is set to NONE, the directory where the checkpointing job started is used. See the checkpoint(5) man page for detailed information about the checkpointing environment configuration.


Note - You should start a checkpointing job with the qsub -cwd script if ckpt_dir is set to NONE.


Checkpointing files and restart files must be visible on all machines in order to successfully migrate and restart jobs. File visibility is an additional requirement for the way file systems must be organized. Thus NFS or a similar file system is required. Ask your cluster administration if your site meets this requirement.

If your site does not run NFS, you can transfer the restart files explicitly at the beginning of your shell script. For example, you can use rcp or ftp, in the case of user-level checkpointing jobs.

Previous Previous     Contents     Index     Next Next