What Two Linux Commands Can Be Used For Pattern Matching And Filtering Of Output?

In this article, nosotros will review a number of control line tools that human activity as filters in Linux. A filter is a program that reads standard input, performs an operation upon it and writes the results to standard output.

For this reason, it tin be used to process information in powerful means such equally restructuring output to generate useful reports, modifying text in files and many other arrangement administration tasks.

With that said, below are some of the useful file or text filters in Linux.

1. Awk Command

Awk is a remarkable pattern scanning and processing language, it tin be used to build useful filters in Linux. You lot can starting time using it by reading through our Awk series Office one to Part thirteen.

Additionally, also read through the awk human page for more info and usage options:

$ man awk

2. Sed Command

sed is a powerful stream editor for filtering and transforming text. We've already written a two useful manufactures on sed, that you can get through information technology here:

How to use GNU 'sed' Command to Create, Edit, and Manipulate files in Linux
xv Useful 'sed' Command Tips and Tricks for Daily Linux System Assistants Tasks

The sed man folio has added control options and instructions:

$ man sed

3. Grep, Egrep, Fgrep, Rgrep Commands

These filters output lines matching a given pattern. They read lines from a file or standard input, and print all matching lines by default to standard output.

Note: The master program is grep, the variations are but the same equally using specific grep options as beneath (and they are yet being used for backward compatibility):

$ egrep = grep -E $ fgrep = grep -F $ rgrep = grep -r

Beneath are some basic grep commands:

          [email protected] ~ $          grep "aaronkilik" /etc/passwd          aaronkilik:x:1001:1001::/home/aaronkilik:          [email protected] ~ $          cat /etc/passwd | grep "aronkilik"          aaronkilik:x:1001:1001::/domicile/aaronkilik:

You tin can read more about What's Difference Between Grep, Egrep and Fgrep in Linux?.

iv. head Command

caput is used to brandish the get-go parts of a file, it outputs the first 10 lines past default. You tin can use the -northward num flag to specify the number of lines to be displayed:

          [email protected] ~ $          head /var/log/auth.log          January  2 ten:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (uid=0) Jan  two 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session airtight for user root Jan  two x:51:34 TecMint sudo:  tecmint : TTY=unknown ; PWD=/dwelling/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py January  two 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (uid=0) January  two ten:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root Jan  two 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root past (uid=0) January  2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root January  ii 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root past (uid=0) Jan  2 eleven:05:01 TecMint CRON[4138]: pam_unix(cron:session): session airtight for user root Jan  2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (uid=0)          [email protected] ~ $          head  -north v /var/log/auth.log          Jan  ii ten:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (uid=0) Jan  2 x:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root Jan  2 10:51:34 TecMint sudo:  tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py January  2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Jan  two 10:51:39 TecMint sudo: pam_unix(sudo:session): session airtight for user root

Learn how to utilize head command with tail and cat commands for constructive usage in Linux.

5. tail Command

tail outputs the last parts (ten lines by default) of a file. Utilize the -north num switch to specify the number of lines to be displayed.

The command below will output the terminal five lines of the specified file:

          [email protected] ~ $          tail -due north 5 /var/log/auth.log          January  6 thirteen:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22. January  half-dozen xiii:01:27 TecMint sshd[1269]: Server listening on :: port 22. Jan  half dozen thirteen:01:27 TecMint sshd[1269]: Received SIGHUP; restarting. January  6 xiii:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22. Jan  6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.

Additionally, tail has a special option -f for watching changes in a file in real-time (especially log files).

The following control will enable you monitor changes in the specified file:

          [email protected] ~ $          tail -f /var/log/auth.log          Jan  six 12:58:01 TecMint sshd[1269]: Server listening on :: port 22. January  6 12:58:xi TecMint sshd[1269]: Received SIGHUP; restarting. January  6 12:58:12 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22. January  six 12:58:12 TecMint sshd[1269]: Server listening on :: port 22. Jan  6 thirteen:01:27 TecMint sshd[1269]: Received SIGHUP; restarting. Jan  6 13:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22. January  vi xiii:01:27 TecMint sshd[1269]: Server listening on :: port 22. Jan  half dozen 13:01:27 TecMint sshd[1269]: Received SIGHUP; restarting. Jan  6 xiii:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22. January  6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.

Read through the tail man page for a complete listing of usage options and instructions:

$ homo tail

half-dozen. sort Command

sort is used to sort lines of a text file or from standard input.

Below is the content of a file named domains.list:

          [electronic mail protected] ~ $          cat domains.list          tecmint.com tecmint.com news.tecmint.com news.tecmint.com linuxsay.com linuxsay.com windowsmint.com windowsmint.com

You lot can run a unproblematic sort command to sort the file content like so:

          [email protected] ~ $          sort domains.list          linuxsay.com linuxsay.com news.tecmint.com news.tecmint.com tecmint.com tecmint.com windowsmint.com windowsmint.com

Y'all tin can utilise sort command in many ways, go through some of the useful articles on sort command every bit follows:

fourteen Useful Examples of Linux 'sort' Command – Office i
vii Interesting Linux 'sort' Command Examples – Function 2
How to Detect and Sort Files Based on Modification Appointment and Fourth dimension
How to Sort Output of 'ls' Command Past Concluding Modified Date and Fourth dimension

7. uniq Command

uniq command is used to report or omit repeated lines, it filters lines from standard input and writes the outcome to standard output.

Later running sort on an input stream, yous can remove repeated lines with uniq equally in the instance beneath.

To indicate the number of occurrences of a line, utilise the -c option and ignore differences in instance while comparing by including the -i selection:

          [email protected] ~ $          cat domains.list          tecmint.com tecmint.com news.tecmint.com news.tecmint.com linuxsay.com linuxsay.com windowsmint.com          [electronic mail protected] ~ $          sort domains.list | uniq -c          2 linuxsay.com ii news.tecmint.com 2 tecmint.com one windowsmint.com

Read through the uniq man page for further usage info and flags:

$ man uniq

eight. fmt Command

fmt simple optimal text formatter, it reformats paragraphs in specified file and prints results to the standard output.

The post-obit is the content extracted from the file domain-list.txt:

1.tecmint.com two.news.tecmint.com 3.linuxsay.com four.windowsmint.com

To reformat the higher up content to a standard list, run the post-obit control with -w switch is used to define the maximum line width:

          [email protected] ~ $          cat domain-listing.txt          1.tecmint.com two.news.tecmint.com 3.linuxsay.com 4.windowsmint.com          [e-mail protected] ~ $          fmt -west ane domain-list.txt          i.tecmint.com  2.news.tecmint.com  3.linuxsay.com  4.windowsmint.com

9. pr Command

pr command converts text files or standard input for printing. For example on Debian systems, you can list all installed packages as follows:

$ dpkg -fifty

To organize the list in pages and columns fix for printing, issue the following command.

          [email protected] ~ $          dpkg -50 | pr --columns 3 -l twenty          2017-01-06 xiii:19                                                  Folio 1   Desired=Unknown/Install ii  adduser		ii  apg | Status=Not/Inst/Conf- ii  adwaita-icon-theme	ii  app-install-data |/ Err?=(none)/Reinst-r ii  adwaita-icon-theme- two  apparmor ||/ Name		two  alsa-base		ii  apt +++-=================== two  alsa-utils		2  apt-clone ii  accountsservice	ii  anacron		2  apt-transport-https two  acl			2  apache2		ii  apt-utils ii  acpi-support	ii  apache2-bin		ii  apt-xapian-alphabetize ii  acpid		ii  apache2-data	ii  aptdaemon ii  add-apt-key		ii  apache2-utils	ii  aptdaemon-data   2017-01-06 13:xix                                                  Page 2   ii  aptitude		ii  avahi-daemon	ii  bind9-host ii  bent-common	2  avahi-utils		ii  binfmt-support ii  apturl		ii  aview		ii  binutils 2  apturl-common	ii  banshee		ii  bison ii  archdetect-deb	ii  baobab		2  blt ii  aspell		ii  base-files		ii  blueberry ii  aspell-en		ii  base-passwd		ii  bluetooth ii  at-spi2-core	two  bash		two  bluez ii  attr		2  fustigate-completion	two  bluez-cups ii  avahi-autoipd	ii  bc			ii  bluez-obexd  .....

The flags used here are:

--column defines number of columns created in the output.
-l specifies folio length (default is 66 lines).

10. tr Command

This tool translates or deletes characters from standard input and writes results to standard output.

The syntax for using tr is as follows:

$ tr options set1 set2

Take a expect at the examples below, in the kickoff command, set1( [:upper:] ) represents the case of input characters (all upper case).

And then set2([:lower:]) represents the case in which the resultant characters will be. Information technology's same thing in the 2nd example and the escape sequence \n means print output on a new line:

          [email protected] ~ $          repeat "World wide web.TECMINT.COM" | tr [:upper:] [:lower:]          www.tecmint.com          [e-mail protected] ~ $          echo "news.tecmint.com" | tr [:lower:] [:upper:]          NEWS.TECMINT.COM

11. more Command

more command is a useful file perusal filter created basically for certificate viewing. It shows file content in a page like format, where users can printing [Enter] to view more information.

You can use it to view big files similar so:

          [email protected] ~ $          dmesg | more than          [    0.000000] Initializing cgroup subsys cpuset [    0.000000] Initializing cgroup subsys cpu [    0.000000] Initializing cgroup subsys cpuacct [    0.000000] Linux version 4.4.0-21-generic ([email protected]) (gcc version v.3.1 20160413 (Ubuntu v.iii.1-14ubuntu2) ) #37-Ubuntu SMP Mon April 18 xviii:33:37 UTC 2016 (Ubuntu iv.4.0-21.37-generic  4.4.6) [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.four.0-21-generic root=UUID=bb29dda3-bdaa-4b39-86cf-4a6dc9634a1b ro tranquillity splash vt.handoff=7 [    0.000000] KERNEL supported cpus: [    0.000000]   Intel GenuineIntel [    0.000000]   AMD AuthenticAMD [    0.000000]   Centaur CentaurHauls [    0.000000] x86/fpu: xstate_offset[ii]:  576, xstate_sizes[2]:  256 [    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating indicate registers' [    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [    0.000000] x86/fpu: Supporting XSAVE characteristic 0x04: 'AVX registers' [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [    0.000000] x86/fpu: Using 'eager' FPU context switches. [    0.000000] e820: BIOS-provided physical RAM map: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable [    0.000000] BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a56affff] usable [    0.000000] BIOS-e820: [mem 0x00000000a56b0000-0x00000000a5eaffff] reserved [    0.000000] BIOS-e820: [mem 0x00000000a5eb0000-0x00000000aaabefff] usable          --More--

12. less Control

less is the opposite of more command above simply information technology offers extra features and it'due south a little faster with large files.

Use information technology in the same fashion as more:

          [e-mail protected] ~ $          dmesg | less          [    0.000000] Initializing cgroup subsys cpuset [    0.000000] Initializing cgroup subsys cpu [    0.000000] Initializing cgroup subsys cpuacct [    0.000000] Linux version iv.iv.0-21-generic ([email protected]) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon April eighteen 18:33:37 UTC 2016 (Ubuntu 4.four.0-21.37-generic  4.4.6) [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=bb29dda3-bdaa-4b39-86cf-4a6dc9634a1b ro quiet splash vt.handoff=7 [    0.000000] KERNEL supported cpus: [    0.000000]   Intel GenuineIntel [    0.000000]   AMD AuthenticAMD [    0.000000]   Centaur CentaurHauls [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[ii]:  256 [    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating signal registers' [    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [    0.000000] x86/fpu: Supporting XSAVE characteristic 0x04: 'AVX registers' [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [    0.000000] x86/fpu: Using 'eager' FPU context switches. [    0.000000] e820: BIOS-provided physical RAM map: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable [    0.000000] BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a56affff] usable [    0.000000] BIOS-e820: [mem 0x00000000a56b0000-0x00000000a5eaffff] reserved [    0.000000] BIOS-e820: [mem 0x00000000a5eb0000-0x00000000aaabefff] usable :

Larn Why 'less' is Faster Than 'more' Command for constructive file navigation in Linux.

That'south all for now, do let usa know of whatever useful control line tools not mentioned here, that act every bit a text filters in Linux via the comment section below.

If You Appreciate What We Practice Here On TecMint, Y'all Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what y'all are reading, please consider buying u.s. a java ( or ii ) as a token of appreciation.

Support Us