jq Guide: Live Data

The jq sessions use synthetic test data. This guide uses live data from your actual repos — gh api, git log, and awk-generated JSON. Different skill, different muscle.

Prerequisites

  • Completed at least jq Sessions 01-03

  • gh CLI authenticated

  • Your domus repos cloned locally

Pattern 1: GitHub API — Repo Inventory

# List your repos (JSON) — quote the URL for zsh
gh api 'users/EvanusModestus/repos?per_page=100' | jq '.[] | {name, language, stars: .stargazers_count}'

Sort by most recently updated:

gh api 'users/EvanusModestus/repos?per_page=100' |
  jq -r 'sort_by(.updated_at) | reverse | .[] | "\(.updated_at | split("T")[0])\t\(.name)"'

Pattern 2: GitHub API — File Tree

# Count .adoc files in a repo
gh api 'repos/EvanusModestus/domus-captures/git/trees/main?recursive=1' |
  jq '[.tree[] | select(.path | endswith(".adoc"))] | length'

Group files by extension:

gh api 'repos/EvanusModestus/domus-captures/git/trees/main?recursive=1' |
  jq '.tree | map(.path | split(".") | last) | group_by(.) | map({ext: .[0], count: length}) | sort_by(.count) | reverse'

Pattern 3: GitHub API — Code Search

# Search across repos — note full_name (snake_case, not camelCase)
gh search code "vault seal" --owner EvanusModestus --json repository,path,textMatches |
  jq '.[] | {repo: .repository.full_name, file: .path, match: .textMatches[].fragment}'

Unique repos from search:

gh search code "vault seal" --owner EvanusModestus --json repository |
  jq '[.[].repository.full_name] | unique'

Pattern 4: Git Log as JSON

# Structured commit log
git -C ~/atelier/_bibliotheca/domus-captures log --pretty=format:'{"hash":"%h","date":"%aI","subject":"%s"}' -20 |
  jq -s 'sort_by(.date) | reverse | .[].subject'

Commits per month:

git -C ~/atelier/_bibliotheca/domus-captures log --pretty=format:'{"date":"%aI","subject":"%s"}' -100 |
  jq -s 'map(.date | split("T")[0] | split("-")[0:2] | join("-")) | group_by(.) | map({month: .[0], count: length}) | sort_by(.month)'

Filter by subject pattern:

git -C ~/atelier/_bibliotheca/domus-captures log --pretty=format:'{"hash":"%h","subject":"%s"}' -100 |
  jq -s '.[] | select(.subject | test("STD-"))'

Pattern 5: AsciiDoc Headers as JSON

Use awk to extract document structure, then query with jq:

awk '/^=+ / {
  level = 0
  for (i=1; i<=length($0); i++) {
    if (substr($0,i,1) == "=") level++; else break
  }
  title = substr($0, level+2)
  gsub(/"/, "\\\"", title)
  printf "{\"level\":%d,\"title\":\"%s\",\"file\":\"%s\"}\n", level, title, FILENAME
}' ~/atelier/_bibliotheca/domus-captures/docs/asciidoc/modules/ROOT/pages/*.adoc |
  jq -s 'group_by(.level) | .[] | {level: .[0].level, count: length}'

Count headers per file, sorted:

# Pipe the same awk output into:
jq -s 'group_by(.file) | map({file: .[0].file | split("/") | last, sections: length}) | sort_by(.sections) | reverse | .[:10]'

Pattern 6: Cross-Repo Activity Dashboard

for repo in domus-captures domus-infra-ops domus-ise-linux domus-netapi-docs domus-secrets-ops; do
  git -C ~/atelier/_bibliotheca/$repo log --pretty=format:"{\"repo\":\"$repo\",\"date\":\"%aI\",\"subject\":\"%s\"}" -5 2>/dev/null
done | jq -s 'sort_by(.date) | reverse | .[:15] | .[] | "\(.date | split("T")[0]) [\(.repo)] \(.subject)"' -r

Common Gotchas with Live Data

  1. Quote URLs in zsh — ? is a glob character: gh api 'url?param=val'

  2. GitHub uses snake_case — full_name, not fullName; stargazers_count, not stars

  3. -s (slurp) for git log — each line is a separate JSON object; -s collects into array

  4. Escape quotes in awk output — gsub(/"/, "\\\"", title) prevents broken JSON

  5. Rate limits — gh api respects GitHub’s rate limit. If you hit it: gh api rate_limit | jq '.rate'

Exercises

  1. [ ] Find your 5 most recently pushed repos with gh api + jq

  2. [ ] Count commits per author across 3 repos

  3. [ ] Build a JSON inventory of all AsciiDoc files across your domus repos

  4. [ ] Create a "standards mentioned in commits" report using git log + jq + test("STD-")