3    Editing Files with the sed Editor

The sed stream editor is a program that works much like the interactive ed program, but you do not need to know how to use the ed line editor to use the material presented here. Unlike ed, sed edits files by using a prepared list of commands, called a script, instead of interacting with the user. This method of operation makes sed particularly well suited for tasks like the following:

• Editing large files

• Performing complex editing operations many times without extensive retyping and cursor positioning

• Performing global changes in one pass through the input

3.1    Overview of the sed Editor

The sed stream editor receives its input from standard input or from a named file, changes that input as directed by commands in a command file or on the command line, and writes the resulting stream to standard output. If you specify more than one input file, sed processes each file in sequence and concatenates the results to standard output. If you do not provide a command file and do not use any with the sed command flags, sed copies standard input to standard output without change. The editor keeps only a few lines of the file being edited in memory at one time and does not use temporary files. Therefore, the size of the file to be edited is limited only by the available disk space.

The command script for sed can be a file that you create before running the editor, a series of commands you enter as a command flag, or both. The editor cannot process more than 99 commands in a single invocation; for this reason or to accomplish certain extremely complex editing tasks, you might need to pipe the output from sed into another instance of sed.

3.2    Running the sed Editor

The syntax for the sed command is as follows:

sed [ -n ] [ -e script ] [ -f script_file ] [ source_file1 [ source-file2... ] ]

Table 3-1 describes the flags for the sed command.

Table 3-1: Flags for the sed Command

The order of presentation of the -e and -f options is important. Usually, you create a command file containing the desired editing commands before running sed. The sed editor's command set is powerful and requires little typing. Each command in the command file can be on a separate line, or you can place multiple commands on one line by separating them with semicolons (;). For example, either of the following two scripts will delete all lines beginning with .ne, .RE, or .RS:

Script 1:

/^\.ne/d
/^\.R[ES]/d

Script 2:

/^\.ne/d;/^\.R[ES]/d

Once you create the command file (cmdfile in the following example), enter the sed command as in this example:

$  sed -f cmdfile infile > outfile

This command edits infile using the commands contained in cmdfile, and writes the output to outfile. The input file is not changed.

With a short editing script, you can accomplish the same job by entering the editing commands as an argument to the -e flag on the command line:

$  sed -e '/^\.ne/d;/^\.R[ES]/d' infile > outfile

If you use the -e and -f flags together on a command line, sed applies all the commands specified by both flags, in the order in which the flags appear. For example:

$   echo "s/line/foo/" > sedx
$   echo "Test line" | sed -f sedx -e 's/line/bar/'

Test foo

Test bar

You can use the -e and -f flags more than once with a given sed command. For example:

$  sed -f script1 -e 's/foo/bar/' -f script2 msgs > msgs2

When you start sed, the editor reads and compiles the command script, checking for syntax and organizing the commands for efficiency. It then reads the input file one line at a time into an area of memory called the pattern space. The editor then tries to match the addresses specified by the commands in the script, one after another, to the lines in the pattern space. Whenever a command's address matches any line or lines in the pattern space, sed applies that editing command to the matched text.

Commands are applied in sequence to the text, and the results of each command are used as the input for subsequent commands. When no more commands match a given line in the pattern space, sed writes that line to the output, reads more input, and repeats the process. Figure 3-1 is a flowchart of this sequence. Compare the operation of sed with the very similar operation of the awk program, shown in Figure 2-1.

Figure 3-1: Sequence of sed Processing

Some editing commands change the way the editing process operates by causing the editor to bypass other script commands, by inhibiting the writing of certain lines (by deleting them), or by ending the process prematurely.

3.3    Selecting Lines for Editing

The sed editor identifies lines to be edited by matching addresses. An address can be either a line number or a context address:

• Line numbers

  The first line in the input stream is line 1, and each successive line increments the line counter by one. The dollar sign ($) is a shorthand way to specify the last line of the input stream.

  If you edit more than one file in a single invocation of sed, the line counter is cumulative across all the files edited; for example, if the first file contains 100 lines, the first line of the second file is line 101.

• Context addresses

  A context address is a regular expression enclosed in pattern delimiters (usually slashes); for example, /^\.R[ES]/ matches any line beginning with either .RE or .RS.

You can specify any character as a pattern delimiter for a given command by preceding the first use of the character with a backslash. For example, the following two patterns are interpreted identically:

/abc/
\xabcx

In the second pattern, the letter x is used as the pattern delimiter. If you use an alternative pattern delimiter in this way, you can match a literal occurrence of that character by preceding it with a backslash; the pattern \x\xyzx matches the string "xyz".

The sed editor recognizes the standard set of basic regular expressions described in Chapter 1. In addition to these expressions, sed recognizes the special expressions shown in Table 3-2.

Table 3-2: Special Regular Expressions Recognized by sed

Some sed commands do not accept addresses. Commands that accept addresses behave differently depending on the number of addresses, as follows:

• If no address is specified, the command is applied to every line in the input stream.

• If one address is specified, the command is applied to each line that matches the address.

• If two addresses are specified, the command is applied to a group of lines starting with a line that matches the first address and ending with the first subsequent line that matches the second address. The editor then tries to match the first address again to find another range.

Note

If two addresses are specified but sed cannot find a line matching the ending address, sed operates on every line from the first address to the end of the file.

3.4    Summary of sed Commands

Each sed command consists of a single letter with optional addresses. Some commands require arguments and accept qualifiers that alter their behavior. Do not include any space between the addresses and the letter. If you use two addresses with a command, separate them with a comma. The r and w commands and the w flag for the s command require a single space between the letter and the argument; otherwise, do not include any space between the letter and the argument.

Table 3-3, Table 3-4, and Table 3-5 describe the individual sed commands, showing the syntax of each. In these tables, the following conventions apply:

• The term "range of lines" can mean a single line, a group of lines, or all lines, as specified by the number of addresses given to the command.

• Brackets [ ] enclose optional elements. Nested brackets indicate that the nested element can be used only if the enclosing element is present.

• Italic (slanted) type indicates a general name for an object that you specify; for example, file represents a command argument that must be the name of a file.

The following example illustrates a correctly formed s command with all optional elements:

1,/^$/s/vizier//g

This example processes the header of a mail message (line 1 to the first completely blank line), replacing the string vizier with nothing wherever the string occurs on any line in the specified range.

Table 3-3: Text Editing and Movement Commands

Table Notes:

1. If the text to be written consists of multiple lines, each line except the last must have a backslash (\) before the terminal newline character. The text is always written regardless of anything subsequent commands do to the line that caused it to be written, including deletion of that line. It is neither scanned for address matches nor affected by subsequent editing commands, and it has no effect on the editor's line counter.

2. If no addresses are given, the d command deletes all lines in the pattern space; unless constrained by a range controlling a group of commands in braces, the command deletes the entire contents of the file.

3. Include exactly one space between the r command and the file name. If file cannot be accessed, sed behaves as if it had read an empty file and gives no abnormal indication. A combined maximum of 10 files can be named for reading or writing in any given editing process.

4. See Section 3.5 for descriptions of the s command's optional flags.

5. Include exactly one space between the w command and the file name. If file exists, it is overwritten; if not, it is created. A combined maximum of 10 files can be named for reading or writing in any given editing process.

Table 3-4: Buffer Manipulation Commands

Table 3-5: Flow-of-Control Commands

3.5    String Replacement

The s command performs string replacement on the indicated lines in the input file. If the editor finds a string of characters in the input file that satisfies the pattern expression expr, it replaces that string with the set of characters specified in string. The string argument is not a regular expression, and it is not scanned or otherwise interpreted except as follows:

• Any backslash characters (\) appearing in string must be escaped. See Table 3-3 for an explanation of how to handle slash characters (/) in string.

• The following two special symbols can be used in string:

  • Ampersand (&)

    This symbol in string is replaced by the exact string of characters in the input lines that matched expr. For example, apply the command s/[Bb]oy/&s/ to the following line:
    

    The boy watched the game.
    

    This command tells sed to find either Boy or boy in the input line and copy whichever pattern it finds to the output with an appended "s". Because the command finds boy, it transfers that string to the output with the modification, and the result is as follows:
    

    The boys watched the game.
    

  • Back-reference expression (\n)

    The number n is a single digit. This symbol in string is replaced by the string in the input line that matches the nth subexpression in expr. Subexpressions in basic regular expressions are delimited by backslash-parentheses sets, \( and \). For example, apply the command s/\(stu\)\(dy\)/\1r\2/ to the following line:
    

    The study chair.
    

    This command tells sed to find study in the input line and copy that pattern to the output with an "r" inserted in the middle. The result is as follows:
    

    The sturdy chair.
    

You can modify the behavior of the s command with flags, as follows:

• Usually, only the first matching string in each line of the range is replaced. The g (global) flag causes sed to make the substitution for all matching strings anywhere on any line in the range. Note that the matching strings do not have to be identical; the expression expr is evaluated again for each potential match.

• The p (print) flag instructs sed to write the indicated lines explicitly after making any substitutions; this writing action is in addition to sed's normal operation.

• The w file (write) flag instructs sed to write the indicated lines to the named file after making any substitutions. Include exactly one space between the w flag and the file name.

Any or all of these flags can be used with a given s command; in combinations, the w flag must be the last flag specified.