uniq command in Linux is a command-line utility that is used to report or
filter out repeated lines in a file. It is most effective when used on sorted
files: lines are compared with the immediately preceding line, so identical
lines must be consecutive to be recognized as duplicates.
The basic syntax of theuniq` command is as follows:
uniq [OPTION]... [INPUT [OUTPUT]]
INPUTis the name of the input file. If no input file is provided or if it is
uniqreads from standard input.
OUTPUTis the name of the output file. If no output file is provided,
uniqwrites to standard output.
OPTIONare the command-line options that can be provided to
uniqto alter its behavior.
Here is a table summarizing the most common options available for the
|Prefixes lines by the number of occurrences
|Only prints duplicate lines, one for each group
|Only prints lines that are not followed by a duplicate
|Prints all duplicate lines
|Ignores differences in case when comparing
|Skips the first N fields
|Skips the first N characters
|Compares no more than N characters in lines
|Groups duplicate lines together with an empty line between groups
|End lines with 0 byte, not newline (useful for files with newline in their names)
|Displays a help message and exits
|Outputs version information and exits
Creating a Sample Text File Using
Before we dive into examples, let’s create a sample text file named
Once you're in
i to switch to insert mode and then type or paste
the following lines:
To save the file and exit
:wq, and then
Examples of Using
Example 1: Basic Usage
To simply filter out repeated lines:
sort list.txt | uniq
This will output:
Example 2: Counting Occurrences
To count the number of occurrences of each line:
sort list.txt | uniq -c
Example 3: Display Only Duplicate Lines
To display only the lines that are repeated:
sort list.txt | uniq -d
Example 4: Display Unique Lines
To display only the lines that are not followed by a duplicate:
sort list.txt | uniq -u
Example 5: Ignoring Case
To ignore the case when comparing lines:
sort list.txt | uniq -i -d
Example 6: Skip Fields
To skip the first field when comparing lines (assuming some delimiter like a space separates fields):
sort list_with_fields.txt | uniq -f 1
This would only compare lines ignoring the first field.
Example 7: Check Specific Characters
To only compare the first N characters of each line:
sort list.txt | uniq -w N
N with the number of characters you want to compare.
Example 8: Grouping Duplicates
To group duplicate lines together:
sort list.txt | uniq -g
uniq is most effective on sorted input. Without sorting,
would not properly identify duplicates unless they are adjacent in the file.
uniq with other text processing tools like
What Can You Do Next 🙏😊
If you liked the article, consider subscribing to Cloudaffle, my YouTube Channel, where I keep posting in-depth tutorials and all edutainment stuff for software developers.