Competencies: CLI & Terminal > Data Processing Tools
Data Processing Tools
Body of Knowledge
| Topic | Description | Relevance | Career Tracks |
|---|---|---|---|
jq Fundamentals |
JSON querying, dot notation, array indexing, select filters, pipes within jq, identity operator. |
Critical |
DevOps, API Developer, Data Engineer |
jq Advanced |
Map/reduce, conditionals, string interpolation, custom functions, recursive descent, @base64/@uri. |
High |
DevOps, API Developer, Data Engineer |
jq Streaming |
Large file processing (--stream), streaming patterns, memory-efficient JSON processing. |
Medium |
Data Engineer, SRE |
yq |
YAML/XML/TOML processing, jq-like syntax for YAML, Kubernetes manifest manipulation, format conversion. |
High |
DevOps, Platform Engineer, SRE |
Miller (mlr) |
CSV/TSV/JSON processing, SQL-like operations, format conversion, statistical operations, streaming. |
Medium |
Data Engineer, Analytics |
csvkit |
CSV toolkit, csvstat, csvcut, csvsql, csvjoin, data inspection and transformation for tabular data. |
Medium |
Data Engineer, Analytics |
datamash |
GNU tool for statistical operations, grouping, transpose, crosstabs, summary statistics on tabular data. |
Low |
Data Engineer, Analytics |
xsv |
Fast CSV toolkit in Rust, indexing, searching, joining, statistics, slicing, format conversion. |
Medium |
Data Engineer, SRE |
fx |
Interactive JSON viewer, keyboard navigation, jq integration, color output, large file support. |
Low |
Developer, DevOps |
gron |
Make JSON greppable, flatten JSON to path=value, unflatten back to JSON, pipeline-friendly. |
Low |
DevOps, Developer |
Personal Status
| Topic | Level | Evidence | Active Projects | Gaps |
|---|---|---|---|---|
jq |
Advanced |
JSON transformation for API responses — select, map, reduce, string interpolation, conditional logic, custom functions; used extensively with domus-api and ISE ERS API |
No jq streaming (--stream) for large files, no jqplay-style debugging |