jq & yq Mastery

jq is the essential tool for JSON processing in pipelines, APIs, and automation. Master jq and yq to manipulate configuration data, parse API responses, and build powerful data transformation pipelines.

Why jq/yq Matters

Domain Applications

API Automation

Parse REST responses, extract data, transform payloads

Configuration Management

Manipulate JSON/YAML configs, generate templates

Log Processing

Parse JSON logs, filter events, aggregate metrics

CI/CD Pipelines

Transform manifests, inject variables, validate schemas

Monad Transforms

Build routing logic for SIEM pipelines

Infrastructure as Code

Process Terraform state, Kubernetes manifests

Curriculum

Module Description Level

Fundamentals

Basic filters, identity, field access, arrays

Core

Operators

Comparison, arithmetic, logical, string operations

Core

Transformations

Object construction, array manipulation, restructuring

Intermediate

Conditionals

if-then-else, select, alternative operator

Intermediate

Functions

Built-in functions, map, reduce, group_by, custom defs

Advanced

Infrastructure Patterns

API parsing, config generation, log processing

Reference

yq for YAML

YAML processing, Kubernetes manifests, Ansible

Reference

Quick Reference

Basic Syntax

# Pretty print
cat file.json | jq .

# Extract field
jq '.name' file.json

# Array element
jq '.[0]' file.json

# Nested access
jq '.user.email' file.json

# Multiple fields
jq '{name: .name, id: .id}' file.json

Essential Operators

# Pipe (chain operations)
jq '.users | .[0] | .name'

# Array iteration
jq '.users[]'

# Filter
jq '.users[] | select(.active == true)'

# Map
jq '.users | map(.name)'

# Conditional
jq 'if .status == "ok" then .data else empty end'

Common Patterns

# Extract keys
jq 'keys' file.json

# Count array elements
jq '.users | length'

# Group by field
jq '.logs | group_by(.level)'

# Sort by field
jq '.items | sort_by(.date)'

# Unique values
jq '.tags | unique'

jq vs yq

Feature jq yq

Input

JSON

YAML, JSON, XML, CSV

Output

JSON

YAML, JSON, XML, CSV

Use Case

API responses, JSON logs

Kubernetes, Ansible, configs

Syntax

jq native

jq-like (Mike Farah version)

Tool Installation

# Arch Linux
pacman -S jq yq

# Ubuntu/Debian
apt install jq
# yq (Go version by Mike Farah)
wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq
chmod +x /usr/local/bin/yq

# macOS
brew install jq yq

Related Resources