The files from this session can be found here: https://github.com/ireapps/nicar-2024-data-wrangling-with-code

CSVKit in terminal

Everywhere it says [your csv], replace it with the name of your CSV file including the .csv

• Install CSVKit
  • sudo pip install csvkit
• Print out the first 5 rows of the CSV:
  • cat [your csv] | head
• Check the headers:
  • csvcut -n [your csv]
• Remove the first row
  • csvcut [your csv] -K 1 > [new csv name]
• Extract a specific value
  • Gives most common values in the column given by the column id
    • csvcut -c [column id] [your csv] | csvstat
  • Gives all unique values in a column
    • csvcut -c [column id] [your csv] | sort | uniq
  • Pick the [value] in the column you want
    • csvgrep -c [column id] -m [value] [your csv]

Data Cleaning Tips

Think about reproducibility

What will be helpful for future years of data?

Look for patterns in the data

How can you tell when there is a new entry?

Are certain pieces of information formatted in a way you can check for?

Use trial and error