Remove Duplicate Lines
Deduplicate any list with explicit case, trim and Unicode options — order preserved, removed lines auditable, entirely on this device.
Order is preserved — the first occurrence of each line is kept.
Press ? for shortcuts
Do it offline:$ awk '!seen[$0]++' file.txt
How to use it
- Paste your lines, or drop a .txt or .csv file up to 50 MB — small lists dedupe as you type; past 2 MB a button states the size first.
- Pick what makes two lines equal — case-insensitive, trim before comparing, Unicode-normalize (NFC), remove blank lines — and keep first or keep last.
- Read the summary, then open the Removed tab to audit which lines were dropped.
- Copy or download the unique lines, or share a link carrying exactly the state you chose.
Paste a list and repeated lines vanish while the survivors hold their positions — first occurrence wins by default, last if you prefer. Checkboxes decide what counts as identical: letter case, surrounding whitespace, Unicode spelling, blank lines. A summary reports the counts, a Removed tab shows every dropped line, and the whole job runs on this device.
How it works
Lines split on CRLF, LF and CR; the terminator stays out of the comparison, so Windows and Unix spellings of one line are one duplicate. Kept lines keep their own endings, and a final newline never becomes a phantom empty line.
Ticked options build the comparison key in a declared order — trim, then NFC, then a locale-free lowercase. Only the key is transformed; the emitted line is byte-for-byte the one you pasted.
Keep first walks the list once with a set of seen keys — first-occurrence order. Keep last pre-computes each key's final position and emits the lines standing there — last-occurrence order.
Working state is one pass, index lists and a key set — no per-line copies — so multi-million-line files fit under the 50 MB cap.
Examples
Ana@example.com
ana@example.com
BOB@example.com
bob@example.com
carol@example.com
Ana@example.com
BOB@example.com
carol@example.com
Two addresses differ only by capitalization, so the summary reads 5 lines → 3 unique; 2 duplicates removed and each survivor keeps its first spelling. The dropped pair waits in the Removed tab.
ERROR TimeoutError: upstream timed out
at fetchRetry (queue.js:71)
at run (queue.js:12)
ERROR TimeoutError: upstream timed out
at fetchRetry (queue.js:71)
at run (queue.js:12)
WARN retrying in 2s
ERROR TimeoutError: upstream timed out
at fetchRetry (queue.js:71)
at run (queue.js:12)
WARN retrying in 2s
A retried request pasted the same three stack-trace lines twice. With every option off, each first appearance stays put and the repeat block moves whole into the audit view.
café
café
latte
café
latte
The two cafés are different byte sequences — é composed versus e plus a combining accent. With NFC on they merge and the first spelling survives; with it off both stay and a hint chip reads: 1 line differs only by Unicode normalization — enable Unicode-normalize (NFC)?
Every example on this page is executed against the same engine that powers the tool, on every build.
Gotchas worth knowing
CRLF and LF spellings are one duplicate
Mixed endings raise a chip: compared as equal, kept as-is. Each keeper carries its own terminator, so a half-Windows, half-Unix file leaves in that mixture — this page is not a line-ending converter.
A trailing newline is not an extra line
Text ending in a newline has no ghost empty line after it — the counts say so, and the final terminator survives. An unterminated last line stays unterminated.
Blank lines collapse — or disappear entirely
With nothing ticked, later blanks are duplicates of the first — paragraph gaps collapse to one, and a chip says why. Remove blank lines deletes every whitespace-only line, the first included.
Keep last changes the order on purpose
Keep last lists each survivor where its final occurrence stood — natural for logs where the newest repeat matters. The Removed tab stays in input order.
sort -u is not the same operation
sort -u reorders your list as a side effect, and plain uniq only drops adjacent repeats. The awk one-liner below is the faithful CLI twin: first occurrence kept, everything else in place.
Turkish İ will not fold to i
Case-insensitive mode lowercases without a locale, so dotted İ becomes i plus a combining dot and stays distinct from plain i. For locale-aware casing itself, see the Case Converter page.
One enormous line is still one line
A minified bundle pasted as one huge line is simply a list of length one. Past 256 KB the work leaves the main thread; past 2 MB an explicit press runs it.
Common errors
This tool handles up to 50 MB; beyond that, try the CLI: awk '!seen[$0]++' file.txt- Past the cap the page declines the whole input rather than deduplicating a slice, and your paste stays put. Underneath, the engine answers with “Input is larger than the 50 MB limit for this tool.” — typed data, never a crash.
This file doesn't look like text (NUL bytes at offset 8) — did you mean a hex viewer?- A NUL byte in the first 8,192 bytes marks a dropped file as binary — a PNG header trips it at its ninth byte — and the input pane is left untouched. UTF-16 text sniffs as binary too; re-save as UTF-8.
FAQ
Does removing duplicates change the order of my lines?
Not under the default: keep first leaves every survivor where it first appeared, and a caption states the active rule. Keep last switches to last-occurrence order; only the separate sort checkbox alphabetizes.
Why were two lines that look the same both kept?
They differ invisibly — a trailing space, a tab, capitalization, or an accent composed two ways. When over 5% of unique lines are near-misses of one kind, a chip names the difference and one click fixes it.
Is my list uploaded to a server?
No. Deduplication runs in your browser, dropped files are read locally without a network request, and nothing leaves unless you create a share link — it encodes only the state you see.
Keyboard shortcuts
- ⌘/Ctrl + Enter
- Re-run the dedupe — required past 2 MB.
- ⌘/Ctrl + S
- Download the result as .txt instead of saving the page.
- ⌘/Ctrl + C
- Copies the whole result while nothing is selected; a real selection copies itself.
- ⌘/Ctrl + Shift + C
- Force-copy the whole result even mid-selection.
- t
- Flip the theme — idle while typing.
- ? or ⌘/Ctrl + /
- Bring up the shortcut reference.
- Esc
- Close the topmost dialog without touching your input.