Mastering the uniq Command in Linux

The uniq command in Linux is a utility that filters out adjacent duplicate lines from input text. Often used with the sort command, uniq is a vital tool for text processing tasks. This article will delve into the syntax, options, and practical examples of using uniq.

Syntax of uniq

The basic syntax for the uniq command is:

uniq [OPTION] [INPUT [OUTPUT]]

• OPTION: Flags that modify how uniq operates.
• INPUT: The input file. If not specified, uniq reads from standard input.
• OUTPUT: The output file. If not specified, uniq writes to standard output.

Options for uniq

Here is a table outlining some commonly used uniq options, their shorthand, and descriptions:

Option	Shorthand	Description
--count	-c	Prefix lines by the number of occurrences
--repeated	-d	Only print duplicate lines
--all-repeated	-D	Print all duplicate lines
--ignore-case	-i	Ignore differences in case when comparing
--unique	-u	Only print unique lines
--skip-fields=N	-f N	Skip N fields on each line before checking for uniqueness
--skip-chars=N	-s N	Skip N characters before checking for uniqueness
--check-chars=N	-w N	Compare no more than N characters in lines

Practical Examples

Basic Usage

Consider a file called numbers.txt with the following contents:

Create numbers.txt file

1
2
2
3
3
3
4
4
5

Run uniq to remove adjacent duplicate lines:

uniq numbers.txt

Output:

1
2
3
4
5

Counting Occurrences

To count the occurrences of each line, use the -c option:

uniq -c numbers.txt

Output:

1 1
1 2
2 3
2 4
1 5

Display Only Duplicate Lines

To display only the duplicate lines, use the -d option:

uniq -d numbers.txt

Output:

2
3
4

Display Only Unique Lines

To display only the unique lines, use the -u option:

uniq -u numbers.txt

Output:

1
5

Skipping Fields

To skip the first field before comparing, use the -f option:

apple 1
orange 1
banana 2

Run:

uniq -f 1 numbers.txt

Output:

apple 1
banana 2

Combining Multiple Options

You can combine multiple options to execute more complex queries. For instance, to display only the unique lines while ignoring case:

uniq -ui numbers.txt

Conclusion

The uniq command is extremely useful for text manipulation tasks in Linux. Its real power is unleashed when used in combination with other text processing utilities like sort, awk, and sed. As seen from the examples, uniq offers various options to skip fields, count occurrences, and display only unique or duplicate lines, making it a versatile command you should get familiar with.

What Can You Do Next 🙏😊

If you liked the article, consider subscribing to Cloudaffle, my YouTube Channel, where I keep posting in-depth tutorials and all edutainment stuff for software developers.