Data Processing

Data Processing

Query, filter, and transform structured data (JSON, YAML, TOML) efficiently from the command line.

Tools

| Tool | Command | Use For |

|------|---------|---------|

| jq | jq '.key' file.json | JSON processing |

| yq | yq '.key' file.yaml | YAML/TOML processing |

jq Essentials

# Extract single field
jq '.name' package.json

# Extract nested field
jq '.scripts.build' package.json

# Extract from array
jq '.dependencies[0]' package.json

# Extract multiple fields
jq '{name, version}' package.json

# Navigate deeply nested
jq '.data.users[0].profile.email' response.json

# Filter by condition
jq '.users[] | select(.active == true)' data.json

# Transform each element
jq '.users | map({id, name})' data.json

# Count elements
jq '.users | length' data.json

# Raw string output
jq -r '.name' package.json

yq Essentials

# Extract field
yq '.name' config.yaml

# Extract nested
yq '.services.web.image' docker-compose.yml

# List all keys
yq 'keys' config.yaml

# List all service names (Docker Compose)
yq '.services | keys' docker-compose.yml

# Get container images (K8s)
yq '.spec.template.spec.containers[].image' deployment.yaml

# Update value (in-place)
yq -i '.version = "2.0.0"' config.yaml

# TOML to JSON
yq -p toml -o json '.' config.toml

Quick Reference

| Task | jq | yq |

|------|----|----|

| Get field | jq '.key' | yq '.key' |

| Array element | jq '.[0]' | yq '.[0]' |

| Filter array | jq '.[] \| select(.x)' | yq '.[] \| select(.x)' |

| Transform | jq 'map(.x)' | yq 'map(.x)' |

| Count | jq 'length' | yq 'length' |

| Keys | jq 'keys' | yq 'keys' |

| Pretty print | jq '.' | yq '.' |

| Compact | jq -c | yq -o json -I0 |

| Raw output | jq -r | yq -r |

| In-place edit | - | yq -i |

When to Use

• Reading package.json dependencies
• Parsing Docker Compose configurations
• Analyzing Kubernetes manifests
• Processing GitHub Actions workflows
• Extracting data from API responses
• Filtering large JSON datasets
• Config file manipulation
• Data format conversion

Additional Resources

For complete pattern libraries, load:

• ./references/jq-patterns.md - Arrays, filtering, transformation, aggregation, output formatting
• ./references/yq-patterns.md - Docker Compose, K8s, GitHub Actions, TOML, YAML modification
• ./references/config-files.md - package.json, tsconfig, eslint/prettier patterns