Data cleaning 101: a few commands to clean csv or tsv files

Print duplicates: $ sort file.csv | uniq -d Create a new file without duplicated: $ awk !seen[$0]++ file.csv > no_dup.csv Remove rows without email addresses: $ grep @ file.csv > newfile.csv Remove ^M from file.csv: $ vim file.csv :%s/^M//g NB: CTRL + V then CTRL + M to print ^M Visidata Open file with Visidata: $ vd file.csv To edit a cell, press e, edit, then press Enter. To delete a row, press d. To expand a column so you see all the text, press _ to toggle between the 2 width. To save a file, press CTRL + S, then press Enter.