LINES -> UNIQUE · Local tool
Remove Duplicate Lines
Remove repeated lines, or show only the ones that repeat.
Last updated
How do I remove duplicate lines from a list?
Paste the lines into filetity and the repeats are removed, keeping the first appearance of each so the order the list was in survives. It also counts what went, and can do the opposite and show only the lines that appear more than once, which is what you want when the duplicates are the thing you are investigating. Two options do the real work on messy lists: ignoring case, for names and email addresses, which keeps the first spelling rather than a lowercased one, and trimming each line, which is on by default because an invisible trailing space is the most common reason a deduplicated list still looks like it has duplicates in it. Everything runs in the browser tab with nothing uploaded.
01 / Your device
the file stays here
02 / Browser memory
the work runs locally
03 / Back to you
saved on this device
When you would use this
Situations, not settings.
- A mailing list assembled from several sources
- Ignore case so Bob and bob count as one, and the first spelling is the one kept rather than a lowercased version.
- Finding out what is duplicated rather than clearing it
- Show only repeats lists each line that appears more than once, once, which is what you want when the duplicates are the problem.
- A list that still looks duplicated after deduplicating
- Almost always an invisible trailing space, which is impossible to see in a text box. Trimming is on by default for that reason.
How it works / no cloud
A conversion engine, not an upload form.
The first appearance of each line is kept, so the order somebody chose survives, which matters when the list is ranked or chronological. Case and surrounding whitespace can each be ignored, which is what a list of emails or names actually needs and is the thing a plain set cannot do.
- Offline ready
- Installable app
- No file limit queue
Straight answers
No accordion. Nothing hidden.
- Is it free?
- Yes, with no watermark and no page limit.
- Does it need an account?
- No. There is no account system.
- Is my file processed on my device?
- Yes. The file is read by your browser and never uploaded. There is no server to send it to.
- What formats are supported?
- Any lines in. Either the unique lines, keeping the first of each, or only the lines that appear more than once.
- What are the limits?
- Lines are compared whole. Two lines differing anywhere at all are different lines, unless case or edge whitespace is being ignored.
- Does it work on mobile?
- Yes, on any modern mobile browser. Large files are limited by the memory the phone gives the browser.
- Does it work offline?
- Yes, once the page has loaded. The tool keeps working with no network connection.
- Which copy of a duplicate is kept?
- The first. That preserves whatever order the list was in, which matters when it is ranked, chronological or simply something a person arranged. Keeping the last would silently reorder the result, which is rarely what anyone wants.
- Are Bob and bob the same line?
- Not by default, and yes if you turn on ignore case. Names and email addresses are the usual reason to want it. When it is on, the first spelling is kept rather than a lowercased version, so the list keeps the capitalisation somebody typed.
- How do I find the duplicates instead of removing them?
- Switch to showing only repeats, which lists each line that appears more than once, once. That is the option you want when the duplicates are the problem you are investigating rather than noise you are clearing away.
- What about lines that differ only by a trailing space?
- They are different lines unless trimming is on, which it is by default. An invisible trailing space is the most common reason a deduplicated list still looks like it has duplicates in it, and it is impossible to see in a text box.
- Does it change the order of what is left?
- No. The lines that survive come back in exactly the order they first appeared. If you want them sorted as well, run the result through the line sorter, which keeps the two operations separate and predictable.
Keep working locally
3 available · more coming