Remove Duplicate Rows (CSV)
Drop repeated CSV rows by whole row or chosen columns, then sort and filter what is left — header kept, counts shown, all on this device.
With no header row the columns are named col1, col2, col3 in order. Detected from the input — click to choose a different delimiter. Repeated header names are numbered (email, email_2) and picked by position, so the right column is never a guess.
Order: filter → remove duplicates → sort.
Dropped files are read in this tab. Nothing is uploaded.
Press ? for shortcuts
Do it offline:$ awk 'NR==1 || !seen[$0]++' data.csv
How to use it
- Paste the table, or drop a .csv or .tsv file — small tables answer while you type.
- Choose the duplicate key: the whole row, or add one key column at a time — repeated headings arrive numbered (email, email_2), so the right column is never a guess.
- Add sort keys and conditions if you want them — each key has its own direction and comparator; conditions join with AND or OR under one invert switch.
- Check the census, read the removed rows, then copy or download.
An export that repeats itself is not a text problem: the repeat is a row, and whether two rows count as one depends on which columns you compare. Point the duplicate key at the whole row or at chosen columns, keep the earliest or latest occurrence, then order and narrow what survives — counts and dropped rows included, on this device.
How it works
Three stages run in one fixed order, printed beside the controls: conditions, duplicates, ordering. Conditions go first so the duplicate count describes the rows you keep; duplicates go next so first and last mean as pasted.
Identity is exact text from the decoded cell: two rows are one row when every key cell matches character for character, so 00123 is not 123 and Ana is not ana. Ignore case and trim edges compare a changed copy, never the cells that come out.
Ordering is stable, which is what makes a sorted export diffable. Rows tied on every key leave in the order they arrived, so re-running these settings over an edited file moves what changed and nothing else. Reversing a key reverses values, not that tie-break.
Each key picks how its cells are read. Text compares code unit by code unit, no locale involved. Number reads a whole cell as a number — sign, digits, one dot, optional exponent — so 9 lands below 12, and Version-style chunks digits and letters: item2 before item10.
Examples
id,email,plan
1,Ana@Example.com,pro
2,ana@example.com ,free
3,bob@example.com,pro
4,BOB@example.com,teamid,email,plan
1,Ana@Example.com,pro
3,bob@example.com,proFour rows, two people. Keyed on the email column with ignore case and trim edges on, row 1's capitalised address and row 2's space-padded one are one person: 4 rows → 2 kept · 0 filtered out · 2 duplicates removed.
region,city,orders
EMEA,Berlin,7
APAC,Osaka,12
EMEA,Lisbon,120
APAC,Perth,9
EMEA,Dublin,region,city,orders
APAC,Osaka,12
APAC,Perth,9
EMEA,Lisbon,120
EMEA,Berlin,7
EMEA,Dublin,Region ascending as text, orders descending as numbers, so each region's busiest city comes first — 120 above 7. Set that key to Text and you get 7 above 120 and 9 above 12: 9 outranks 1 one character in. Dublin's missing figure stays last either way.
ticket,assignee,status
4102,ana@example.com,open
4103,bot+ci@example.com,open
4104,ana@example.com,open
4105,bot+nightly@example.com,closed
4106,ana@example.com,openticket,assignee,status
4106,ana@example.com,openOne inverted condition — matches regex ^bot\+ on assignee — drops both robot tickets. Keyed then on assignee plus status, keep last, only the newest of three open tickets remains: 5 rows → 1 kept · 2 filtered out · 2 duplicates removed.
Every example on this page is executed against the same engine that powers the tool, on every build.
Gotchas worth knowing
Leading zeros are not a number here
Part code 00123 and quantity 123 are different text and stay two rows. A sort key set to Number reads them as equal, so the tie falls back to position. Identity never switches to Number.
Empty cells and unreadable numbers keep their end
Blank key cells collect at the bottom. Under Number, anything not a plain number — a thousands separator, a currency mark, the word pending — sits after every figure in arrival order, with a note naming the column.
A cell is one cell, whatever it contains
A cell carrying a separator or a line break inside it is still one value, and two such cells match when their contents do. Short rows are filled out before comparison.
Pattern matching has a work budget
The matches regex operator is metered: it stops with a typed refusal naming the row once a run has examined about five million characters of that column. Contains, starts with and ends with have no ceiling.
How the awk one-liner differs
The command below keeps the first copy of each identical line and pins line 1 as the heading. It cannot key on chosen columns, ignore case, or keep the last occurrence.
Common errors
Input is larger than the 100 MB limit for this tool. For bigger files use the CLI: awk 'NR==1 || !seen[$0]++' data.csv- No slice is attempted: the input is declined and left where it is. That first sentence is the intake engine's own typed refusal, printed as it arrives; the banner only adds the command to reach for instead.
Condition 1, column "name": that pattern is not a valid regular expression — Invalid regular expression: /a(/: Unterminated group; fix the pattern, or switch the operator to Contains.- The middle clause is the JavaScript engine's own complaint, passed through untouched, so an unclosed group says so. Nothing runs while a condition is unreadable.
Row 22,785: the pattern passed its 5,000,000-step budget for this file and the filter stopped there — simplify the pattern, or use Contains, Starts with or Ends with.- A step is one test plus one per character examined, re-checked every 256 rows, so the row named is where the run stopped rather than an estimate.
FAQ
What exactly makes two CSV rows duplicates?
Matching text in every key cell — the whole row, or the columns you added as keys. Capitalisation, edge whitespace and leading zeros all differ until ignore case or trim edges is on.
Do rows that tie on my sort keys get shuffled?
No — tied rows leave in the order they arrived, in both directions, because position is the final tie-break. Re-run after an edit and only what truly moved will have moved.
Does my CSV go to a server?
It does not. The table is read, keyed and reordered in this tab, and a dropped file never becomes a request. A permalink carries only what you can already see.
Keyboard shortcuts
- ⌘/Ctrl + Enter
- Re-run the three stages; a large table waits for this.
- ⌘/Ctrl + S
- Save the resulting rows as .csv.
- ⌘/Ctrl + C
- Takes the whole result unless text is selected.
- ⌘/Ctrl + Shift + C
- Take the result despite a selection.
- t
- Switch palettes; idle mid-typing.
- ? or ⌘/Ctrl + /
- Open this list in the tool.
- Esc
- Dismiss the top panel, table untouched.