Shell Scripting

Shell Scripting

Purpose

Provides patterns and best practices for writing maintainable shell scripts with error handling, argument parsing, and portability considerations. Covers POSIX sh vs Bash decision-making, parameter expansion, integration with common utilities (jq, yq, awk), and testing with ShellCheck and Bats.

When to Use This Skill

Use shell scripting when:

• Orchestrating existing command-line tools and system utilities
• Writing CI/CD pipeline scripts (GitHub Actions, GitLab CI)
• Creating container entrypoints and initialization scripts
• Automating system administration tasks (backups, log rotation)
• Building development tooling (build scripts, test runners)

Consider Python/Go instead when:

• Complex business logic or data structures required
• Cross-platform GUI needed
• Heavy API integration (REST, gRPC)
• Script exceeds 200 lines with significant logic complexity

POSIX sh vs Bash

Use POSIX sh (#!/bin/sh) when:

• Maximum portability required (Linux, macOS, BSD, Alpine)
• Minimal container images needed
• Embedded systems or unknown target environments

Use Bash (#!/bin/bash) when:

• Controlled environment (specific OS, container)
• Arrays or associative arrays needed
• Advanced parameter expansion beneficial
• Process substitution <(cmd) useful

For detailed comparison and testing strategies, see references/portability-guide.md.

Essential Error Handling

Fail-Fast Pattern

#!/bin/bash
set -euo pipefail

# -e: Exit on error
# -u: Exit on undefined variable
# -o pipefail: Pipeline fails if any command fails

Use for production automation, CI/CD scripts, and critical operations.

Explicit Exit Code Checking

#!/bin/bash

if ! command_that_might_fail; then
    echo "Error: Command failed" >&2
    exit 1
fi

Use for custom error messages and interactive scripts.

Trap Handlers for Cleanup

#!/bin/bash
set -euo pipefail

TEMP_FILE=$(mktemp)

cleanup() {
    rm -f "$TEMP_FILE"
}

trap cleanup EXIT

Use for guaranteed cleanup of temporary files, locks, and resources.

For comprehensive error patterns, see references/error-handling.md.

Argument Parsing

Short Options with getopts (POSIX)

#!/bin/bash

while getopts "hvf:o:" opt; do
    case "$opt" in
        h) usage ;;
        v) VERBOSE=true ;;
        f) INPUT_FILE="$OPTARG" ;;
        o) OUTPUT_FILE="$OPTARG" ;;
        *) usage ;;
    esac
done

shift $((OPTIND - 1))

Long Options (Manual Parsing)

#!/bin/bash

while [[ $# -gt 0 ]]; do
    case "$1" in
        --help) usage ;;
        --verbose) VERBOSE=true; shift ;;
        --file) INPUT_FILE="$2"; shift 2 ;;
        --file=*) INPUT_FILE="${1#*=}"; shift ;;
        *) break ;;
    esac
done

For hybrid approaches and validation patterns, see references/argument-parsing.md.

Parameter Expansion Quick Reference

# Default values
${var:-default}              # Use default if unset
${var:=default}              # Assign default if unset
: "${API_KEY:?Error: required}"  # Error if unset

# String manipulation
${#var}                      # String length
${var:offset:length}         # Substring
${var%.txt}                  # Remove suffix
${var##*/}                   # Basename
${var/old/new}               # Replace first
${var//old/new}              # Replace all

# Case conversion (Bash 4+)
${var^^}                     # Uppercase
${var,,}                     # Lowercase

For complete expansion patterns and array handling, see references/parameter-expansion.md.

Common Utilities Integration

JSON with jq

# Extract field
name=$(curl -sSL https://api.example.com/user | jq -r '.name')

# Filter array
active=$(jq '.users[] | select(.active) | .name' data.json)

# Check existence
if ! echo "$json" | jq -e '.field' >/dev/null; then
    echo "Error: Field missing" >&2
fi

YAML with yq

# Read value (yq v4)
host=$(yq eval '.database.host' config.yaml)

# Update in-place
yq eval '.port = 5432' -i config.yaml

# Convert to JSON
yq eval -o=json config.yaml

Text Processing

# awk: Extract columns
awk -F',' '{print $1, $3}' data.csv

# sed: Replace text
sed 's/old/new/g' file.txt

# grep: Pattern match
grep -E "ERROR|WARN" logfile.txt

For detailed examples and best practices, see references/common-utilities.md.

Testing and Validation

ShellCheck: Static Analysis

# Check script
shellcheck script.sh

# POSIX compliance
shellcheck --shell=sh script.sh

# Exclude warnings
shellcheck --exclude=SC2086 script.sh

Bats: Automated Testing

#!/usr/bin/env bats

@test "script runs successfully" {
    run ./script.sh --help
    [ "$status" -eq 0 ]
    [ "${lines[0]}" = "Usage: script.sh [OPTIONS]" ]
}

@test "handles missing argument" {
    run ./script.sh
    [ "$status" -eq 1 ]
    [[ "$output" =~ "Error" ]]
}

Run tests:

bats test/

For CI/CD integration and debugging techniques, see references/testing-guide.md.

Defensive Programming Checklist

#!/bin/bash
set -euo pipefail

# Check required commands
command -v jq >/dev/null 2>&1 || {
    echo "Error: jq required" >&2
    exit 1
}

# Check environment variables
: "${API_KEY:?Error: API_KEY required}"

# Check files
[ -f "$CONFIG_FILE" ] || {
    echo "Error: Config not found: $CONFIG_FILE" >&2
    exit 1
}

# Quote all variables
echo "Processing: $file"        # ❌ Unquoted
echo "Processing: \"$file\""    # ✅ Quoted

Platform Considerations

macOS vs Linux Differences

# sed in-place
sed -i '' 's/old/new/g' file.txt    # macOS
sed -i 's/old/new/g' file.txt       # Linux

# Portable: Use temp file
sed 's/old/new/g' file.txt > file.txt.tmp
mv file.txt.tmp file.txt

# readlink
readlink -f /path                    # Linux only
cd "$(dirname "$0")" && pwd         # Portable

For complete platform differences, see references/portability-guide.md.

Script Categories

System Administration: Cron jobs, log rotation, backup automation

Build/Deployment: CI/CD pipelines, Docker builds, deployments

Development Tooling: Project setup, test runners, code generators

Container Entrypoints: Initialization, signal handling, configuration

Production Script Template

#!/bin/bash
set -euo pipefail

readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"

TEMP_DIR=""

cleanup() {
    local exit_code=$?
    rm -rf "$TEMP_DIR"
    exit "$exit_code"
}

trap cleanup EXIT

log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" >&2
}

main() {
    # Check dependencies
    command -v jq >/dev/null 2>&1 || exit 1

    # Parse arguments
    # Validate input
    # Process
    # Report results

    log "Completed successfully"
}

main "$@"

For complete production template, see examples/production-template.sh.

Tool Recommendations

Core Tools:

• jq: JSON parsing and transformation
• yq: YAML parsing (v4 recommended)
• ShellCheck: Static analysis and linting
• Bats: Automated testing framework

Installation:

# macOS
brew install jq yq shellcheck bats-core

# Ubuntu/Debian
apt-get install jq shellcheck

Related Skills

• linux-administration: System commands and administration
• building-ci-pipelines: Using scripts in CI/CD
• infrastructure-as-code: Terraform/Pulumi wrappers
• kubernetes-operations: kubectl scripts, Helm hooks
• writing-dockerfiles: Container entrypoints

Additional Resources

Reference Files:

• references/error-handling.md - Comprehensive error patterns
• references/argument-parsing.md - Advanced parsing techniques
• references/parameter-expansion.md - Complete expansion reference
• references/portability-guide.md - POSIX vs Bash differences
• references/testing-guide.md - ShellCheck and Bats guide
• references/common-utilities.md - jq, yq, awk, sed usage

Example Scripts:

• examples/production-template.sh - Production-ready template
• examples/getopts-basic.sh - Simple getopts usage
• examples/getopts-advanced.sh - Complex option handling
• examples/long-options.sh - Manual long option parsing
• examples/error-handling.sh - Error handling patterns
• examples/json-yaml-processing.sh - jq/yq examples

Utility Scripts:

• scripts/lint-script.sh - ShellCheck wrapper for CI
• scripts/test-script.sh - Bats wrapper for CI