3. JSON and YAML — `jq`, `yq`

🎯 What you'll learn in this chapter:
jq basics — identity, field access, array operations
Filter and transform — select, map, group_by, sort_by
Creating objects — {name: .a, age: .b}
String and number operations
curl + jq — the most powerful pipeline (a daily tool for every DevOps engineer)
yq — the same syntax for YAML
Real examples — GitHub API, AWS, Kubernetes
⏱ Time: ~35 minutes 🧪 Exercises: bashlings watch — 6 interactive exercises ready (exercises/13_jq/)

3.1. Why do you need `jq`?

60-70% of a DevOps and SRE workday is JSON. API responses, Kubernetes manifests, AWS CLI output, Docker inspect, GitHub Actions output. It's all JSON.

Parsing JSON in Bash with grep or awk is dangerous and error-prone:

bash

# ❌ DANGEROUS — breaks if line order or escaping changes
curl -s api.com/user | grep '"name"' | cut -d'"' -f4

jq is a tool built for JSON that understands JSON. It understands structure and doesn't rely on regex.

bash

# ✓ CORRECT — structural
curl -s api.com/user | jq -r '.name'

Core idea

If you work with JSON, use jq. Never parse JSON with grep/awk.

3.2. Installation

bash

# macOS
brew install jq

# Ubuntu / Debian
sudo apt install jq

# Alpine
apk add jq

# Static binary (anywhere)
curl -L https://github.com/jqlang/jq/releases/latest/download/jq-linux-amd64 -o /usr/local/bin/jq
chmod +x /usr/local/bin/jq

# Check version
jq --version
# jq-1.7.1

3.3. Identity filter and pretty-print

The simplest one — the identity filter .:

bash

echo '{"name":"Ali","age":25}' | jq '.'

Result (pretty-formatted, colorized):

json

{
  "name": "Ali",
  "age": 25
}

Basic flags

Flag	Meaning
`-r`	Raw — strip quotes from strings
`-c`	Compact — on a single line
`-s`	Slurp — read all input as one array
`-n`	Null input (jq generates the output)
`-e`	Exit code (if output is `null` or `false` → 1)
`-S`	Sort keys
`-a`	ASCII-only output (escape unicode)
`--tab`	Indent with tabs

bash

# Raw (no quotes)
echo '"hello"' | jq '.'    # "hello"
echo '"hello"' | jq -r '.' # hello

# Compact
echo '{"a":1,"b":2}' | jq -c '.'
# {"a":1,"b":2}

Always use -r in the shell

If you write the result of jq into a shell variable, -r is mandatory. Otherwise it stays inside "quotes".

bash

name=$(echo '{"name":"Ali"}' | jq '.name')
echo "$name"            # "Ali"  ← quotes remain
name=$(echo '{"name":"Ali"}' | jq -r '.name')
echo "$name"            # Ali     ← clean

3.4. Field access

Simple field

bash

echo '{"name":"Ali","age":25,"city":"Toshkent"}' | jq -r '.name'
# Ali

echo '{"name":"Ali","age":25}' | jq '.age'
# 25

Nested field

bash

echo '{"user":{"email":"ali@example.com","verified":true}}' | jq -r '.user.email'
# ali@example.com

Bracket notation (for special characters)

bash

# When the key contains a space, digits, or a hyphen
echo '{"user-id":42}' | jq '.["user-id"]'
# 42

Optional access — `?`

bash

# If the field is missing — null (not an error)
echo '{"name":"Ali"}' | jq '.email?'
# null

# Access without the question mark:
echo '{"name":"Ali"}' | jq '.email'
# null

# But if the input is a string or array, .field raises an error —
# `.field?` does a "silent fail":
echo '"just a string"' | jq '.email?'    # null (not an error)
echo '"just a string"' | jq '.email'     # ERROR

3.5. Array operations

bash

# Example input:
data='[10, 20, 30, 40, 50]'

By index

bash

echo "$data" | jq '.[0]'    # 10
echo "$data" | jq '.[-1]'   # 50 (last)
echo "$data" | jq '.[2]'    # 30

Slice — `.[start:end]`

bash

echo "$data" | jq '.[1:3]'  # [20, 30] (indices 1..2)
echo "$data" | jq '.[:2]'   # [10, 20] (first 2)
echo "$data" | jq '.[-2:]'  # [40, 50] (last 2)

Iteration — `.[]`

bash

echo "$data" | jq '.[]'
# 10
# 20
# 30
# 40
# 50

.[] — each element separately (multi-output).

Size — `length`

bash

echo "$data" | jq 'length'                    # 5
echo '{"a":1,"b":2,"c":3}' | jq 'length'      # 3 (number of object keys)
echo '"hello"' | jq 'length'                  # 5 (number of characters)

Array of objects

bash

users='[
  {"name":"Ali","age":25},
  {"name":"Vali","age":30},
  {"name":"Gulnora","age":28}
]'

# Print all
echo "$users" | jq '.[]'

# Names only
echo "$users" | jq -r '.[].name'
# Ali
# Vali
# Gulnora

# Name of just the 2nd element
echo "$users" | jq -r '.[1].name'   # Vali

3.6. Chaining inside a pipe `|`

Inside jq, | is similar to Bash's |, but it's a chain of filters.

bash

echo "$users" | jq '.[] | .name'
# Ali
# Vali
# Gulnora

echo "$users" | jq '.[] | .name | ascii_upcase'
# ALI
# VALI
# GULNORA

| passes the left result to the right side — just like a shell pipe.

3.7. Filter — `select(...)`

select(condition) — keeps the ones that match the condition.

bash

echo "$users" | jq '.[] | select(.age >= 28)'
# {"name":"Vali","age":30}
# {"name":"Gulnora","age":28}

echo "$users" | jq -r '.[] | select(.age >= 28) | .name'
# Vali
# Gulnora

Operators

Operator	Example
`==`	`.age == 25`
`!=`	`.status != "deleted"`
`>`, `<`	`.score > 80`
`>=`, `<=`	`.age >= 18`
`and`	`.age > 18 and .verified`
`or`	`.role == "admin" or .role == "owner"`
`not`	`.deleted

Checking existence

bash

# Does the field exist?
echo '{"a":1}' | jq 'has("a")'    # true
echo '{"a":1}' | jq 'has("b")'    # false

# String or array element
echo '"hello world"' | jq 'contains("world")'   # true
echo '[1,2,3]' | jq 'contains([2])'             # true

3.8. `map(...)` — transform

map(f) — a shorthand for [ .[] | f ].

bash

# Add 1 to each age
echo "$users" | jq 'map(.age + 1)'
# [26, 31, 29]

# Transform the objects
echo "$users" | jq 'map({ism: .name, yosh: .age})'
# [
#   {"ism":"Ali","yosh":25},
#   ...
# ]

3.9. Creating objects

bash

# Build a new object
echo '{"firstName":"Ali","lastName":"Karim"}' | jq '{
  fullName: (.firstName + " " + .lastName),
  initials: (.firstName[0:1] + .lastName[0:1])
}'
# {
#   "fullName": "Ali Karim",
#   "initials": "AK"
# }

`to_entries` and `from_entries`

Convert an object into an array of key-value pairs:

bash

echo '{"name":"Ali","age":25}' | jq 'to_entries'
# [
#   {"key":"name","value":"Ali"},
#   {"key":"age","value":25}
# ]

# The reverse:
echo '[{"key":"a","value":1},{"key":"b","value":2}]' | jq 'from_entries'
# {"a":1,"b":2}

A useful pattern — transform each value:

bash

# Convert each value to a string
echo '{"a":1,"b":2}' | jq 'with_entries(.value |= tostring)'
# {"a":"1","b":"2"}

3.10. `group_by`, `sort_by`, `unique_by`

bash

people='[
  {"name":"Ali","dept":"IT","age":25},
  {"name":"Vali","dept":"HR","age":30},
  {"name":"Gulnora","dept":"IT","age":28},
  {"name":"Bobur","dept":"HR","age":35}
]'

# Group by department
echo "$people" | jq 'group_by(.dept)'

# Sort by age
echo "$people" | jq 'sort_by(.age) | .[].name'
# "Ali"
# "Gulnora"
# "Vali"
# "Bobur"

# Reverse
echo "$people" | jq 'sort_by(.age) | reverse | .[].name'

# Unique
echo "$people" | jq 'unique_by(.dept) | .[].dept'
# "HR"
# "IT"

Computing per department

bash

echo "$people" | jq 'group_by(.dept) | map({
  dept: .[0].dept,
  count: length,
  avg_age: (map(.age) | add / length)
})'
# [
#   {"dept":"HR","count":2,"avg_age":32.5},
#   {"dept":"IT","count":2,"avg_age":26.5}
# ]

3.11. String and number operations

Strings

Function	Example
`ascii_upcase`	`"hello"
`ascii_downcase`	`"HELLO"
`length`	`"hello"
`split(",")`	`"a,b,c"
`join("-")`	`["a","b"]
`gsub("re"; "x")`	Global regex replace
`sub("re"; "x")`	Match the first regex
`ltrimstr("pre")`	Strip a prefix
`rtrimstr(".log")`	Strip a suffix
`tostring`	Number → string
`tonumber`	String → number

bash

echo '"hello world"' | jq -r 'ascii_upcase'
# HELLO WORLD

echo '["a","b","c"]' | jq -r 'join(",")'
# a,b,c

echo '"app-prod-v2.tar.gz"' | jq -r 'ltrimstr("app-") | rtrimstr(".tar.gz")'
# prod-v2

Numbers

Function	Meaning
`add`	Sum of an array
`min`, `max`	Smallest / largest
`floor`, `ceil`	Rounding
`fabs`	Absolute value

bash

echo '[10,20,30,40]' | jq 'add'           # 100
echo '[10,20,30,40]' | jq 'min'           # 10
echo '[10,20,30,40]' | jq 'max'           # 40
echo '[1.4,2.6]' | jq 'map(floor)'        # [1, 2]

3.12. `curl + jq` — the most powerful pipeline

Top 5 GitHub repositories (by stars)

bash

curl -s "https://api.github.com/search/repositories?q=language:rust&sort=stars" \
  | jq -r '.items[:5] | .[] | "\(.stargazers_count) \(.full_name)"'
# 96234 rust-lang/rust
# 78912 denoland/deno
# ...

Information about a GitHub user

bash

curl -s https://api.github.com/users/torvalds | jq '{
  name: .name,
  bio: .bio,
  repos: .public_repos,
  followers: .followers
}'
# {
#   "name": "Linus Torvalds",
#   "bio": null,
#   "repos": 6,
#   "followers": 235000
# }

A single field into a shell variable

bash

STARS=$(curl -s https://api.github.com/repos/rust-lang/rust | jq '.stargazers_count')
echo "Rust starlari: $STARS"

Filter + transform

bash

# Only open issues, the most recent 5
curl -s "https://api.github.com/repos/cli/cli/issues?state=open" \
  | jq -r '.[:5] | .[] | "[\(.number)] \(.title) — \(.user.login)"'

Multi-source — `jq -s` (slurp)

Read several JSON inputs as a single array:

bash

curl -s api1.com > a.json
curl -s api2.com > b.json

jq -s '.[0] + .[1]' a.json b.json
# Merge the two

3.13. Conditionals — `if-then-else-end`

bash

echo '{"age":17}' | jq 'if .age >= 18 then "kattalar" else "yoshlar" end'
# "yoshlar"

echo '{"age":25}' | jq 'if .age >= 18 then "kattalar" else "yoshlar" end'
# "kattalar"

There's also elif:

bash

echo '{"score":85}' | jq '
  if .score >= 90 then "A"
  elif .score >= 70 then "B"
  elif .score >= 50 then "C"
  else "F"
  end
'
# "B"

3.14. `yq` — for YAML

yq (Mike Farah's version, written in Go) — the same syntax as jq, but for YAML.

bash

# Install
brew install yq                # macOS
sudo snap install yq           # Ubuntu

# Version
yq --version

Reading a Kubernetes manifest

bash

cat deployment.yaml | yq '.spec.replicas'
# 3

# View the container image
yq '.spec.template.spec.containers[0].image' deployment.yaml
# nginx:1.21

Changing a value (in-place)

bash

# Set replicas to 5
yq -i '.spec.replicas = 5' deployment.yaml

# Update the tag
yq -i '.spec.template.spec.containers[0].image = "nginx:1.25"' deployment.yaml

YAML ↔ JSON conversion

bash

# YAML → JSON
yq -o=json deployment.yaml > deployment.json

# JSON → YAML
yq -p=json -o=yaml deployment.json

yq versions differ

There are two popular yq tools:

Mike Farah (Go) — jq syntax, the one shown here
Python (kislyuk/yq) — a jq wrapper, with slightly different syntax

Check which version you have with yq --version.

3.15. Real production examples

Example 1 — Filtering AWS EC2 instances

bash

aws ec2 describe-instances \
  | jq -r '.Reservations[].Instances[]
           | select(.State.Name == "running")
           | "\(.InstanceId) \(.InstanceType) \(.PublicIpAddress)"'
# i-0abc123 t3.medium 54.232.x.x
# i-0def456 t3.large  18.231.x.x

Example 2 — Docker container cleanup

bash

# `Exited` containers running for more than 1 hour
docker ps -a --format '{{json .}}' \
  | jq -r 'select(.State == "exited" and (.RunningFor | test("hour"))) | .ID' \
  | xargs -r docker rm

Example 3 — Kubernetes Pod statuses

bash

kubectl get pods -o json \
  | jq -r '.items[]
           | "\(.metadata.namespace)/\(.metadata.name) — \(.status.phase)"' \
  | sort -k2
# default/web-abc — Running
# default/web-def — Pending
# kube-system/coredns — Running

Example 4 — Log file analysis (JSON log format)

bash

# Apache structured logs
cat access.log | jq -r 'select(.status >= 500) | "\(.timestamp) \(.path) \(.status)"' \
  | head -20

Example 5 — Configuration backup audit

bash

# Kubernetes deployment image versions
for f in deployments/*.yaml; do
    image=$(yq '.spec.template.spec.containers[0].image' "$f")
    printf '%-30s %s\n' "$(basename "$f")" "$image"
done

3.16. Common mistakes

Classic pitfalls

Forgetting the -r flag.

bash

user=$(... | jq '.name')      # "Ali" — quotes remain!
user=$(... | jq -r '.name')   # Ali — clean

Not handling null values.

bash

email=$(... | jq -r '.email')   # if missing, returns "null" (a string!)
email=$(... | jq -r '.email // empty')   # empty string

Error when an object access is absent.

bash

... | jq '.user.email'        # if user is missing — ERROR
... | jq '.user.email?'       # returns null

Using .[] inside map.

bash

echo '[1,2,3]' | jq 'map(.[])'   # ERROR
echo '[1,2,3]' | jq 'map(. * 2)' # ✓

Forgetting @csv for CSV export.

bash

... | jq -r '[.name, .age] | @csv'
# "Ali",25

jq arithmetic precision. Large numbers are converted to IEEE 754 doubles — precision is lost after 16-17 digits.

Numeric vs string comparison.

bash

echo '{"v":"5"}' | jq '.v > 3'   # ERROR (string vs number)
echo '{"v":"5"}' | jq '(.v | tonumber) > 3'   # ✓

Trouble with jq and newlines. You may get a multiline string — use -c (compact) to turn it into a single line.

3.17. Exercises

🧪 Bashlings — interactive exercises

This chapter has 6 exercises with auto-checking via the bashlings CLI (prerequisite: jq must be installed):

bash

bashlings watch              # start from the first pending exercise
bashlings run jq1            # check a single exercise
bashlings hint jq1           # step-by-step hint

Source: exercises/13_jq/

Try these additional conceptual exercises as well:

User name — from the JSON {"user":{"name":"Ali","email":"ali@example.com"}}, extract just the email with jq -r.
Filter — from [{"age":15},{"age":20},{"age":17},{"age":30}], output the count of adults (>=18).
GitHub stars — get the number of stars for GitHub CLI via curl -s https://api.github.com/repos/cli/cli | jq -r '.stargazers_count'.
CSV transform — output [{"name":"Ali","age":25},{"name":"Vali","age":30}] in CSV form (via @csv).
YAML edit — in any deployment.yaml file, change replicas from 3 to 5 in-place (-i).

3.18. Summary

Concept	Key point
`jq '.'`	Identity, pretty-print
`jq '.field'`	Simple field access
`jq '.user.email'`	Nested
`jq '.field?'`	Optional access (null-safe)
`jq '.[0]'` / `jq '.[]'`	Array index / iteration
`jq 'length'`	Size
`jq '...	select(...)'`
`jq 'map(...)'`	Transform
`jq 'group_by(.x)'`	Grouping
`jq '{a: .b}'`	Object creation
`jq 'if X then Y else Z end'`	Condition
`jq -r`	Raw — mandatory in every shell variable
`jq -c`	Compact single line
`jq -s`	Slurp — read all input as an array
`yq '...'`	The same syntax for YAML

5 core ideas

jq for every JSON task — don't parse JSON with grep/awk.
-r for every shell variable — to strip the quotes.
Beware of null — use ? (optional) or a // empty default.
The curl + jq pipeline — DevOps's most-used chain.
yq (Mike Farah) — jq syntax for YAML. Needed for Kubernetes and Docker Compose.

🎉 Now you have all the tools for API integration: curl + jq + yq. In the next chapter — cron and systemd timers — scheduling tasks.

Next page: 4. Cron and task scheduling →

3. JSON and YAML — jq, yq ​

3.1. Why do you need jq? ​

3.2. Installation ​

3.3. Identity filter and pretty-print ​

Basic flags ​

3.4. Field access ​

Simple field ​

Nested field ​

Bracket notation (for special characters) ​

Optional access — ? ​

3.5. Array operations ​

By index ​

Slice — .[start:end] ​

Iteration — .[] ​

Size — length ​

Array of objects ​

3.6. Chaining inside a pipe | ​

3.7. Filter — select(...) ​

Operators ​

Checking existence ​

3.8. map(...) — transform ​

3.9. Creating objects ​

to_entries and from_entries ​

3.10. group_by, sort_by, unique_by ​

Computing per department ​

3.11. String and number operations ​

Strings ​

Numbers ​

3.12. curl + jq — the most powerful pipeline ​

Top 5 GitHub repositories (by stars) ​

Information about a GitHub user ​

A single field into a shell variable ​

Filter + transform ​

Multi-source — jq -s (slurp) ​

3.13. Conditionals — if-then-else-end ​

3.14. yq — for YAML ​

Reading a Kubernetes manifest ​

Changing a value (in-place) ​

YAML ↔ JSON conversion ​

3.15. Real production examples ​

Example 1 — Filtering AWS EC2 instances ​

Example 2 — Docker container cleanup ​

Example 3 — Kubernetes Pod statuses ​

Example 4 — Log file analysis (JSON log format) ​

Example 5 — Configuration backup audit ​

3.16. Common mistakes ​

3.17. Exercises ​

3.18. Summary ​

5 core ideas ​

3. JSON and YAML — `jq`, `yq`

3.1. Why do you need `jq`?

3.2. Installation

3.3. Identity filter and pretty-print

Basic flags

3.4. Field access

Simple field

Nested field

Bracket notation (for special characters)

Optional access — `?`

3.5. Array operations

By index

Slice — `.[start:end]`

Iteration — `.[]`

Size — `length`

Array of objects

3.6. Chaining inside a pipe `|`

3.7. Filter — `select(...)`

Operators

Checking existence

3.8. `map(...)` — transform

3.9. Creating objects

`to_entries` and `from_entries`

3.10. `group_by`, `sort_by`, `unique_by`

Computing per department

3.11. String and number operations

Strings

Numbers

3.12. `curl + jq` — the most powerful pipeline

Top 5 GitHub repositories (by stars)

Information about a GitHub user

A single field into a shell variable

Filter + transform

Multi-source — `jq -s` (slurp)

3.13. Conditionals — `if-then-else-end`

3.14. `yq` — for YAML

Reading a Kubernetes manifest

Changing a value (in-place)

YAML ↔ JSON conversion

3.15. Real production examples

Example 1 — Filtering AWS EC2 instances

Example 2 — Docker container cleanup

Example 3 — Kubernetes Pod statuses

Example 4 — Log file analysis (JSON log format)

Example 5 — Configuration backup audit

3.16. Common mistakes

3.17. Exercises

3.18. Summary

5 core ideas