Data Transforms

Convert between data formats without information loss — JSON, YAML, CSV, XML, TOML.

JSON to YAML

Convert JSON to YAML — yq with pretty-print

yq -P config.json

Convert JSON to YAML file — redirect output

yq -P config.json > config.yml

JSON stdin to YAML — useful in pipelines

curl -s https://api.example.com/data | yq -P

Convert JSON to YAML preserving comments — add comments post-conversion

jq '.' config.json | yq -P '... comments = ""'

YAML to JSON

Convert YAML to JSON — yq output mode

yq -o=json config.yml

YAML to compact JSON — single line, no whitespace

yq -o=json -I=0 config.yml

YAML to JSON then process with jq — the standard pipeline

yq -o=json config.yml | jq '.devices[] | select(.role == "core")'

Multiple YAML documents to JSON array — handle multi-doc YAML

yq -o=json -s '.' multi-doc.yml

JSON to CSV

jq @csv — structured JSON to CSV with header

jq -r '["hostname","ip","vlan"], (.[] | [.hostname, .ip, .vlan]) | @csv' devices.json

Nested JSON to flat CSV — use jq to flatten first

jq -r '.devices[] | [.hostname, .network.ip, .network.vlan] | @csv' inventory.json

JSON to TSV — tab-separated for Unix tool consumption

jq -r '.[] | [.hostname, .ip, .vlan] | @tsv' devices.json

JSON API response to CSV — extract and transform in one pipeline

curl -s https://api.example.com/devices \
  | jq -r '["name","status","ip"], (.results[] | [.name, .status, .ip]) | @csv' \
  > devices.csv

CSV to JSON

csvjson — CSV to JSON array of objects

csvjson inventory.csv | jq '.'

miller — CSV to JSON with type inference

mlr --icsv --ojson cat inventory.csv

awk CSV to JSON — no dependencies, handles simple CSV

awk -F, 'NR==1{split($0,h); next} {printf "{"; for(i=1;i<=NF;i++){printf "\"%s\":\"%s\"", h[i], $i; if(i<NF) printf ","} print "}"}' inventory.csv | jq -s '.'

jq from TSV — read tab-separated input into JSON

jq -Rn '[inputs | split("\t") | {hostname: .[0], ip: .[1], vlan: (.[2] | tonumber)}]' devices.tsv

XML to JSON

xmlstarlet to extract, jq to structure — two-stage pipeline

xmlstarlet sel -t -m '//device' \
  -v 'concat(hostname, "\t", ip, "\t", @role)' -n inventory.xml \
  | jq -Rn '[inputs | split("\t") | {hostname: .[0], ip: .[1], role: .[2]}]'

Python one-liner — xmltodict for direct XML to JSON

python3 -c "
import xmltodict, json, sys
with open('inventory.xml') as f:
    print(json.dumps(xmltodict.parse(f.read()), indent=2))
"

yq with XML input — yq v4 supports XML

yq -p=xml -o=json inventory.xml

TOML Conversions

TOML to JSON — via yq

yq -p=toml -o=json config.toml

TOML to YAML — chain input/output parsers

yq -p=toml -o=yaml config.toml

JSON to TOML — reverse direction (yq v4.30+)

yq -p=json -o=toml config.json

Template Rendering

envsubst — substitute environment variables in templates

export HOSTNAME="modestus-aw" DOMAIN="inside.domusdigitalis.dev" VLAN=10
envsubst < config.template.json > config.json

envsubst with restricted variable list — only expand named variables

envsubst '$HOSTNAME $DOMAIN' < config.template.yml > config.yml

Heredoc + jq — construct JSON from shell variables without templates

HOSTNAME="modestus-aw"
IP="10.50.1.100"
VLAN=10

jq -n \
  --arg hostname "$HOSTNAME" \
  --arg ip "$IP" \
  --argjson vlan "$VLAN" \
  '{hostname: $hostname, ip: $ip, vlan: $vlan}'

jq with --argjson for mixed types — strings stay strings, numbers stay numbers

jq -n \
  --arg name "sw-core-01" \
  --argjson enabled true \
  --argjson vlans '[10, 20, 30]' \
  '{name: $name, enabled: $enabled, vlans: $vlans}'

yq template rendering — merge a values file into a template

yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' template.yml values.yml

Python One-liners for Conversion

YAML to JSON — Python stdlib + PyYAML

python3 -c "import yaml, json, sys; print(json.dumps(yaml.safe_load(sys.stdin), indent=2))" < config.yml

JSON to YAML — reverse direction

python3 -c "import yaml, json, sys; print(yaml.dump(json.load(sys.stdin), default_flow_style=False))" < config.json

INI to JSON — Python configparser

python3 -c "
import configparser, json, sys
c = configparser.ConfigParser()
c.read('config.ini')
print(json.dumps({s: dict(c[s]) for s in c.sections()}, indent=2))
"

Environment file to JSON — parse KEY=VALUE into JSON object

awk -F= '/^[^#]/ {gsub(/"/, "", $2); printf "\"%s\": \"%s\",\n", $1, $2}' .env \
  | sed '$ s/,$//' | { echo '{'; cat; echo '}'; } | jq '.'

Round-trip Verification

Verify lossless JSON to YAML round-trip — diff the canonical forms

diff <(jq -S . original.json) <(yq -P original.json | yq -o=json | jq -S .)

Verify YAML to JSON round-trip

diff <(yq -S '.' original.yml) <(yq -o=json original.yml | yq -P | yq -S '.')

Compare two files in different formats — normalize to JSON then diff

diff <(yq -o=json -S file1.yml) <(jq -S . file2.json)