csv-pipeline

The value here is that it keeps CSV and JSON handling in one mental model, so a transform chain does not need a different tool per hop. The pipeline framing (read, normalize, transform, aggregate, report) is a useful checklist because the normalize step is where most CSV work actually goes wrong: mixed encodings, delimiter sniffing, embedded newlines in quoted fields, and header whitespace. Trade-off to be aware of: convenience pipelines tend to assume a single delimiter and UTF-8, so on files exported from regional Excel configs you should inspect the first few lines and the byte-order mark before trusting the parse. For heavy data volumes I would rather push the join down to a real database than hold the frame in the agent's context, since an agent summarizing a large frame is a memory and token trap. Keep raw input immutable and write every stage to a new file so a bad transform is one rollback away.

Use case: Turning an exported CSV into a normalized dataset plus a short aggregate report without losing the ability to trace each number back to a row.

# Inspect before you transform The expensive part of a tabular job is never the transform. It is discovering, three steps later, that the column you aggregated was a string, or that the file had a byte-order mark in front of the first header. The csv-pipeline skill ships the inspection commands first for exactly this reason. ## Inspect in four cheap moves Start every job with the same four commands: preview the first rows, count the data rows, print the header line, and count distinct values in the column you plan to group by. ```bash head -5 data.csv tail -n +2 data.csv | wc -l head -1 data.csv tail -n +2 data.csv | cut -d',' -f3 | sort -u | wc -l ``` The last one is the one people skip and the one that matters. If the distinct count in your grouping column equals the row count, you are not grouping anything; if it is two, your filter is pointless. ## Then filter with awk, keeping the header A filter that drops the header makes every downstream step guess at column meanings. Keep the first record unconditionally: ```bash awk -F',' 'NR==1 || $3 > 100' data.csv > filtered.csv ``` ## Escalate to Python when the shape breaks Plain shell tooling assumes one record per line and a delimiter that never appears inside a field. Quoted commas, embedded newlines and escaped quotes all violate that assumption, and awk will split them wrong without complaining. When the inspection step shows any of those, stop reaching for cut and sort and use a real parser instead of a more elaborate pipeline. ## Write the report from the transformed file Report generation should read the file you wrote, not re-derive the data. A report that recomputes its own aggregates cannot be re-run against a fixed input, so you can never reproduce last week's numbers.