The sed command in UNIX is one of the most useful tools available in your arsenal. sed stands for stream editor and it can be used to perform functions on files or standard input text streams such as searching, find and replace, insertion, or deletion. In this tutorial, we will cover some of the basic commands of sed.
Basic Usage
$ sed [options] command [input-file]
sed will operate on a stream of text that it either reads from an input file or from standard input (STDIN). By default sed will send its results to standard output (STDOUT). Executing both of the following commands will print the contents of the /etc/services file to the screen:
$ sed '' /etc/services
$ cat /etc/services | sed ''
Output:
# Network services, Internet style
#
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
#
# Updated from http://www.iana.org/assignments/port-numbers and other
# sources like http://www.freebsd.org/cgi/cvsweb.cgi/src/etc/services .
# New ports will be added on request if they have been officially assigned
# by IANA and used in the real-world or are needed by a debian package.
# If you need a huge list of used numbers please install the nmap package.
...
...
Printing
The commands above demonstrate that without any operations sed would print the output of the file to STDOUT. This isn’t entirely useful as we already have other commands that will print the contents of a file to STDOUT, so let’s explore the sed’s print command 'p'.
We’ll modify the command above to use the print command:
$ sed 'p' /etc/services
Output:
# Network services, Internet style
# Network services, Internet style
#
#
# Note that it is presently the policy of IANA to assign a single well-known
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
# even if the protocol doesn't support UDP operations.
...
...
Notice how each line is printed twice? sed will put each line of text that is to be processed into what is known as the “Pattern” space. Before processing a line sed will place the line into the pattern space and then apply all specified commands to the pattern space. By default sed prints the resulted text from the pattern space to STDOUT. The command above is explicitly telling sed to print the contents of the pattern space.
sed can be instructed to not print the pattern space by default by using the -n flag:
$ sed -n 'p' /etc/services
Output:
# Network services, Internet style
#
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
...
...
Operations by Address Ranges
Using addressing allows you to target specific parts of a text stream to operate on. For instance, we can have sed print out a specific line by providing the line number to the print command.
$ sed -n '1p' /etc/services
Output:
# Network services, Internet style
Placing the number 1 in from of the print command informed sed to only perform the print operation on line number 1. We can extend this and make sed only perform a specific operation on a range of lines. For instance the following will print the first 4 lines:
$ sed -n '1,4p' /etc/services
Output:
# Network services, Internet style
#
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
The above address range has instructed sed to only perform the print operation on all lines between line 1 and line 4. An alternative is to specify an offset from the initial line. This can be done by adding using the + character followed by the offset amount, like so:
$ sed -n '1,+3p' /etc/services
Output:
# Network services, Internet style
#
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
sed can also be instructed to perform operations on every line for a given interval by using the ~ character. For instance, the following will perform the print operation on every 2nd line starting at line 1.
$ sed -n '1~2p' /etc/services
Output:
# Network services, Internet style
# Note that it is presently the policy of IANA to assign a single well-known
# even if the protocol doesn't support UDP operations.
# Updated from http://www.iana.org/assignments/port-numbers and other
# New ports will be added on request if they have been officially assigned
# If you need a huge list of used numbers please install the nmap package.
tcpmux 1/tcp # TCP port service multiplexer
...
...
Operations by Pattern
sed can not only perform operations based on address ranges but it can also perform operations on based on patterns defined by a regular expression. The regular expression must be enclosed by slashes /. For instance, we can print all line that beginning with a certain number:
$ sed -n '/443/p' /etc/services
Output:
# Updated from https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml .
http 80/tcp www # WorldWideWeb HTTP
https 443/tcp # http protocol over TLS/SSL
http-alt 8080/tcp webcache # WWW caching service
or we can use more complex regular expressions for instance to find any line that begins with a # character:
$ sed -n '^#/p' /etc/services
# Network services, Internet style
#
# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
#
# Updated from https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml .
#
# New ports will be added on request if they have been officially assigned
# by IANA and used in the real-world or are needed by a debian package.
...
...
Inverse Operation
The above show how we can make sed perform operations on specific lines but what about if we wanted to perform operations on all lines but the ones matching the address specification?
To operate on all lines that do not match the pattern we need to append the ! character to the end of the address specification.
$ sed -n '/^#/!p' /etc/services
Output
tcpmux 1/tcp # TCP port service multiplexer
echo 7/tcp
echo 7/udp
discard 9/tcp sink null
discard 9/udp sink null
systat 11/tcp users
daytime 13/tcp
daytime 13/udp
netstat 15/tcp
...
...
Deletion
Above we have demonstrated the power of sed using the print command, however, we can replace the print command with other sed commands. For instance, we could perform text deletion instead of printing by replacing the p command with the d command.
Since we don’t want to be deleting lines from the /etc/services file we’ll use the following text file:
This is a test file.
#
It'll be used to demonstrate sed's text deletion command.
#
The file is very simple but
#
it will demonstrate how we can remove a range of lines.
As mentioned we can instruct sed to remove all lines between a given range for such as removing all lines between line 2 and 6 by running the following command:
$ sed '2,6d' testfile.txt
Output:
This is a test file.
it will demonstrate how we can remove a range of lines.
Note: In this case we don’t need -n as sed will print every line that is not deleted.
We can also use the regular expressions as we did with the print command to remove any line that matches the regular expression. The command below will remove all lines beginning with the # character:
$ sed '/^#/d' testfile.txt
Output:
This is a test file.
It'll be used to demonstrate sed's text deletion command.
The file is very simple but
it will demonstrate how we can remove a range of lines.
It is important to note that the deletion command above has not made changes to the source file. The output of the edits are only being sent to STDOUT.
If we want the edits to persist then we need to redirect the output to another file:
$ sed '2,6d' testfile.txt > editted-testfile.txt
$ cat editted-testfile.txt
Output:
This is a test file.
it will demonstrate how we can remove a range of lines.
Alternatively, we can specify that sed should perform in-place editing with the -i option. This option will have sed overwrite the source file with the edits. Care should be taken when using the -i option and therefore it is recommended to test your edits without the -i option first.
sed -i '2,6d' testfile.txt
Thankfully though, sed can create a backup file of the source file before editing the file. To create a backup all you need to do it add a backup extension after the -i option:
sed -i.bak '2,6d' testfile.txt
This will create a backup of the source file called testfile.txt.bak and then perform the edits on the original file.