What this tool does
Lists collect repeats. An email list exported from two sources, a keyword list merged from several tools, a column of names copied from a spreadsheet or a log file with the same error many times all contain duplicate lines. This tool reads your text one line at a time and keeps only the first copy of each, in the order they first appeared, so the cleaned list is as short as it can be without losing anything.
Matching is exact by default, with options to ignore capital letters, trim spaces at the beginning and end of each line, drop empty lines and sort the output. Statistics show how many lines you started with, how many are unique and how many were removed.
How to remove duplicate lines
- Paste your list into the box, with one item on each line.
- Choose your options: case sensitive, trim spaces, remove empty lines and sort A-Z.
- Press Remove duplicates.
- Check the statistics and the result.
- Copy the cleaned list, or download it as a text file.
The options explained
- Case sensitive: when off, Apple and apple count as the same line and only the first is kept. Turn it on if capitals matter, as in passwords or case-sensitive file names.
- Trim spaces: removes spaces before and after each line, so that apple and apple with a trailing space are treated as duplicates. This is the most common reason duplicates survive.
- Remove empty lines: deletes blank lines so that the result has no gaps.
- Sort A-Z: puts the unique lines in alphabetical order, with numbers sorted naturally.
Where duplicate removal helps
- Email marketing: merge subscriber lists without sending twice to the same address.
- SEO and advertising: combine keyword lists from several tools into one clean list.
- Data cleaning: remove repeated IDs, URLs, product codes or names before importing a file.
- Development and IT: shrink log files and lists of hostnames to the unique entries.
- Writing and research: clean up reading lists and reference lists.
Limits and good practice
The tool compares whole lines, so a list where items are separated by commas on one line needs to be split into separate lines first. For spreadsheet data with several columns, deduplicate on the one column that identifies a record. Very large lists with hundreds of thousands of lines work best on a fast computer. Always keep your original list until you have checked the result.
Example: cleaning a merged email list
Suppose you exported subscribers from two systems and joined them into one list. Paste the combined list, keep Trim spaces and Remove empty lines on, and turn off Case sensitive, because email addresses are normally treated without regard to case. Press Remove duplicates, and the result is one entry per address. The statistics tell you how many repeats were removed, which is a quick way to see how much the two lists overlapped.
Example: removing duplicates
| Input lines | Result |
|---|---|
| apple, Apple, banana, apple, (empty), cherry, banana | apple, banana, cherry (case-insensitive, empty lines removed) |
| 10, 2, 10, 33, 2 | 10, 2, 33 |
| a.com, a.com , b.com | a.com, b.com (with Trim spaces on) |
Frequently Asked Questions
How do I remove duplicate lines?
Paste your list, one item per line, and press Remove duplicates. The unique lines appear in the result box.
Which copy of a duplicate is kept?
The first one. The order of the remaining lines is unchanged unless you turn on sorting.
Why were some duplicates not removed?
Usually because of hidden differences such as trailing spaces or capital letters. Turn on Trim spaces and turn off Case sensitive.
Can I remove duplicates from a CSV or Excel column?
Yes. Copy the column, paste it here, clean it and paste the result back.
Is there a limit on the number of lines?
There is no fixed limit, but very large lists can slow down your browser.
Can I get the duplicates instead of the unique lines?
This tool keeps the unique lines. To see what changed, use the Text Diff Checker on the original and the cleaned list.
Is my list uploaded anywhere?
No. The tool runs in your browser and your data stays on your device.