sed (Stream Editor) In Bash Scripting

Another powerful filter in GNU/Linux is sed (stream editor), which is a program for performing basic editing tasks on the output of another program or on a file.

sed or stream editor can be used to perform basic text modification on an input stream (a file or input from a pipeline). sed' works by making only one pass over the input(s), and is consequently more efficient than other filter programs.

When used as is, the output is only produced on the screen, the actual file is not changed at all, so, if you want to change the file, then you can use a redirection into another file or by specifying the edit option, which would take a look at below

Its syntax is as follows:

sed action [files]

You specify an action that you want to perform, and on what file or files you want to specify that action on. If you don't specify a file, then it performs the action on the input from either the standard input or a piped information from another program.

sed can perform several actions at a time, e.g

• sed -e action1 -e action2 [files]
• sed -f scriptfile [files] - You take all the action you want to perform and put it in a script file, the f option would open the script file, examine all the actions in it, and then it performs all those actions on the given input.

Actions specified on the command line are almost always enclosed in single quotes to prevent shell interpretation of special characters, so, when using the sed program, you need to take note of this.

Let's take a look at some of the actions you can do with sed.

Substitution In sed

s/foo/bar/- This is a text substitution, and it changes the first occurrence of foo on each line to bar

This is how you can do it practically:

sed 's/Hello/World/' textfile.txt

The above command would replace all instances of "Hello" with "World" in "testfile.txt'

'sed' writes output to standard output (screen) by default, so, whatever you do it won't replace the final process. Use '-i' to edit files directly instead of printing to standard output (screen) only.

The following command modifies 'testfile.txt' and does not produce any output:

sed -i 's/Hello/World' textfile.txt

By default 'sed' prints all processed input ( with the exception of input that has been modified or deleted by commands such as 'd'). You can use '-n' to suppress output, and the 'p' command to print specific lines.

The following example prints only line 20 of the input file:

sed -n '20p' textfile.txt

The following example would print line 1 to 3:

user@blog:~$ sed -n '1,3 p ' /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin

To perform several actions, you can do the following:

sed -e 's/Hello/World' -e 's/foo/bar/' textfile.txt

To change the shell of a a pi user from /bin/bash to /usr/bin/sh, we do:

user@blog:~$ sed -n ' /^pi/ s/\/bin\/bash/\/usr\/bin\/sh/p ' /etc/passwd
user@blog:~$ pi:x:0:0:pi:/pi:/usr/bin/sh

The issue was that the default delimiter is a forward slash so I escaped each
forward slash, so, you just need to add a backward slash for the escape to come through, I have highlighted it with red.

This is one option, another way to do this is to know that the first delimiter we use defines the delimiters. Meaning, you can use any character as a delimiter. We can use a dash (-) to specify the delimiter:

user@blog:~$ sed -n ' /^root/ s-/bin/bash-/usr/bin/sh-p ' /etc/passwd
user@blog:~$ pi:x:0:0:pi:/pi:/usr/bin/sh

Alternatively, you can also use an @ symbol, so, its your call:

sed -n ' /^pi/ s@/bin/bash@/usr/bin/sh@p ' /etc/passwd

Don't forget, this would only print to the standard out, so, if you want to edit the file directly, then you use the -i option:

sed -i ' /^pi/ s@/bin/bash@/usr/bin/sh@p ' /etc/passwd

Working on a specific range

It is possible to specify the range of line number, this way, you can restrict the level of what the process would do:

For example, to print a line that only has the word root, you can do:

user@blog:~$ sed -n '/^root/ p' /etc/passwd
root:x:0:0:root:/root:/bin/bash

The above can also be done in the grep command

If you want to perform the action on line 1 - 2, you do:

user@blog:~/bin$ sed 1,2's/foo/bar/' text
bar
bar
foo
foo

As you can see, it only restricts the process to line 1 and 2.

What if you want to perform an action on a specific line all the way to the last line, then you can do:

user@blog:~/bin$ sed '3,$''s/foo/bar/' text
foo
foo
bar
bar

The $ refers to the last line, so, I am telling sed to work from line 3 all the way to the last line. Notice that I am using a single quote, this is to prevent shell interpretation of special characters.

Deleting Lines In sed

You can also delete a specific line in sed, so, to delete the line 5 to 10, you do:

sed -i '5,20d' textfile.txt

or if you want to delete all lines with a specific word in it, you do:

sed -i '/foo/d' textfile.txt

It is also possible to delete all lines except for lines that match certain criteria:

So, for example, if you don't want to delete from line 10 to the end of the line in the file, you do:

sed -i '5,$!d' textfile.txt

This would delete all the lines that do not fall in that range, so, in our case, it would only delete lines 1 to 4.

Or you can also specify that you don't want a line that contains a certain word to be deleted, e.g

sed -i '/foo/!d' textfile.txt

We are saying delete every line, except the lines that have the foo word included.

To conclude this guide, let me show you an interesting thing you can do with sed, look at the following script:

#!/bin/sed -f

s/myusername/newusername/

/bar/d

This is very similar to a shell script, in short, it is a shell script, just with a sed interpreter, when you run this program, it will be an executable script that you can run like any other executable script, but instead of being interpreted by the shell program, it will be interpreted by the sed program.

As you can see the lines of code in the script are sed actions rather than shell script command, so what are we actually doing?

We are basically substituting "myusername" with "newusername", and we are also deleting any lines that we find in the input that contains "bar"

I'll save the script as sedscript, so, let me show you an example of how we can utilize it:

lastlog | sedscript

I am piping the output of lastlog to the sedscript, and it would perform the action and processes anything that it matches.