Remove Duplicate Lines

Remove Duplicate Lines and Keep the Original Order

This tool helps you remove duplicate lines from email lists, keyword lists, log extracts or any text with one item per line. It keeps the first occurrence of every line, drops later copies and never re-orders what is left.

Example

Input        Default          Case sensitive off
banana       banana           banana
Apple        Apple            Apple
cherry       cherry           cherry
apple        apple            (blank)
(blank)      (blank)          date
Banana       Banana
date         date
cherry

With the default settings only the second “cherry” is removed. When case sensitivity is off, “apple” and “Banana” are also treated as repeats of earlier lines, so three lines go.

Options explained

• Case sensitive (on by default): lines must match letter for letter, including capitals. Turn it off to treat “Apple” and “apple” as the same entry.
• Ignore surrounding spaces (on by default): leading and trailing spaces are ignored when comparing, so “ apple ” matches “apple”. The first copy is kept exactly as you typed it, spaces included.
• Remove empty lines (off by default): deletes blank and whitespace-only lines. When it is off, blank lines are de-duplicated like any other line, so only the first blank line survives.

Why order matters

The classic Unix uniq command only removes duplicates that sit next to each other, so it is usually combined with sort. That changes the order of your data. This tool behaves like awk '!seen[$0]++': it remembers every line it has seen and keeps the list in its original sequence. Ranked keyword lists or step-by-step lists stay intact.

If you do want the result sorted as well, run it through Sort Text Lines, which also has its own duplicate filter.

Common pitfalls

• Look-alike lines. Lines that differ only by an invisible character, such as a zero-width space or a non-breaking space in the middle, are not identical. Clean them first with the Text Cleaner.
• Different spacing inside a line. Trimming only affects the ends of a line. “New York” with two spaces and “New York” with one are still different.
• Comma-separated data. The tool compares whole lines. To compare two versions of a list instead of cleaning one, use the Diff Checker.

The same job in a spreadsheet or on the command line

If your list already lives somewhere else, these are the closest equivalents and how they differ:

• Excel: Data → Remove Duplicates keeps the first copy in place, but compares without regard to case and changes the sheet directly.
• Google Sheets: =UNIQUE(A:A) returns the distinct values in their original order in a new range.
• Command line: sort -u removes duplicates but sorts the list; awk '!seen[$0]++' list.txt keeps the order, like this tool.

The tool is handy when you need both order preservation and control over case and surrounding spaces, or when the list comes from an email or chat rather than a file.

Privacy

De-duplication runs entirely in your browser, which makes it safe for customer lists and other private data.