Understanding the `uniq` Command in Linux

The uniq command in Linux is a command-line utility that is used to report or filter out repeated lines in a file. It is most effective when used on sorted files: lines are compared with the immediately preceding line, so identical lines must be consecutive to be recognized as duplicates.

Syntax of uniq Command`

The basic syntax of theuniq` command is as follows:

uniq [OPTION]... [INPUT [OUTPUT]]

• INPUT is the name of the input file. If no input file is provided or if it is -, uniq reads from standard input.
• OUTPUT is the name of the output file. If no output file is provided, uniq writes to standard output.
• OPTION are the command-line options that can be provided to uniq to alter its behavior.

Options for uniq Command

Here is a table summarizing the most common options available for the uniq command:

Option	Shorthand	Description
--count	-c	Prefixes lines by the number of occurrences
--repeated	-d	Only prints duplicate lines, one for each group
--unique	-u	Only prints lines that are not followed by a duplicate
--all-repeated	-D	Prints all duplicate lines
--ignore-case	-i	Ignores differences in case when comparing
--skip-fields=N	-f N	Skips the first N fields
--skip-chars=N	-s N	Skips the first N characters
--check-chars=N	-w N	Compares no more than N characters in lines
--group	-g	Groups duplicate lines together with an empty line between groups
--zero-terminated	-z	End lines with 0 byte, not newline (useful for files with newline in their names)
--help		Displays a help message and exits
--version		Outputs version information and exits

Creating a Sample Text File Using vim

Before we dive into examples, let’s create a sample text file named list.txt using vim:

vim list.txt

Once you're in vim, press i to switch to insert mode and then type or paste the following lines:

apple
banana
apple
Apple
banana
cherry
cherry

To save the file and exit vim, press Esc, type :wq, and then press Enter.

Examples of Using uniq

Example 1: Basic Usage

To simply filter out repeated lines:

sort list.txt | uniq

This will output:

apple
Apple
banana
cherry

Example 2: Counting Occurrences

To count the number of occurrences of each line:

sort list.txt | uniq -c

Output:

      2 apple
      1 Apple
      2 banana
      2 cherry

Example 3: Display Only Duplicate Lines

To display only the lines that are repeated:

sort list.txt | uniq -d

Output:

apple
banana
cherry

Example 4: Display Unique Lines

To display only the lines that are not followed by a duplicate:

sort list.txt | uniq -u

Output:

Apple

Example 5: Ignoring Case

To ignore the case when comparing lines:

sort list.txt | uniq -i -d

Output:

apple
banana
cherry

Example 6: Skip Fields

To skip the first field when comparing lines (assuming some delimiter like a space separates fields):

sort list_with_fields.txt | uniq -f 1

This would only compare lines ignoring the first field.

Example 7: Check Specific Characters

To only compare the first N characters of each line:

sort list.txt | uniq -w N

Replace N with the number of characters you want to compare.

Example 8: Grouping Duplicates

To group duplicate lines together:

sort list.txt | uniq -g

Output:

apple

apple

Apple
banana

banana
cherry

cherry

Remember that uniq is most effective on sorted input. Without sorting, uniq would not properly identify duplicates unless they are adjacent in the file.

By combining uniq with other text processing tools like

What Can You Do Next 🙏😊

If you liked the article, consider subscribing to Cloudaffle, my YouTube Channel, where I keep posting in-depth tutorials and all edutainment stuff for software developers.