Performance Tuning Agent Skill
Optimize Linux system performance. Configure kernel parameters, analyze bottlenecks, and tune resources. Use when improving system performance.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/BagelHole/DevOps-Security-Agent-Skills --skill performance-tuningThe instruction itself
20 sections, as written by the authorPerformance Tuning
Optimize Linux system performance through kernel parameter tuning, I/O scheduler selection, memory management, CPU governor configuration, and benchmarking. Covers methodology, real sysctl settings, and tool-based validation.
When to Use
- Server experiencing high latency, throughput bottlenecks, or resource exhaustion
- Preparing infrastructure for high-traffic events or load tests
- Tuning a database server, web server, or application host for production
- Diagnosing whether a bottleneck is CPU, memory, disk I/O, or network
- Establishing baseline performance metrics before and after changes
- Configuring kernel parameters for containers, VMs, or bare-metal hosts
Prerequisites
- Root or sudo access on the target system
sysstatpackage installed (providessar,iostat,mpstat)linux-toolsorperfpackage for CPU profiling- Benchmarking tools:
fio(disk),sysbench(CPU/memory),iperf3(network) - Baseline metrics collected before making any changes
Performance Analysis Methodology
Always follow this order:
- Collect baseline -- measure current performance with tools
- Identify bottleneck -- determine if CPU, memory, I/O, or network
- Change one parameter -- apply a single tuning change
- Measure impact -- re-run the same benchmark
- Document -- record the change and its effect
- Iterate or revert -- keep the change if beneficial, revert if not
System Monitoring Tools
# CPU and process monitoring
top # Interactive process viewer
htop # Enhanced interactive viewer
mpstat -P ALL 2 # Per-CPU utilization every 2 seconds
pidstat -u 2 # Per-process CPU usage
# Memory monitoring
free -h # Memory summary
vmstat 2 # Virtual memory stats every 2 seconds
# Columns: r=runnable, b=blocked, si/so=swap in/out, bi/bo=block I/O
# Disk I/O monitoring
iostat -xz 2 # Extended disk stats every 2 seconds
# Key columns: %util, await (latency), r/s, w/s
iotop -oP # Show processes doing I/O
# Network monitoring
sar -n DEV 2 # Network interface stats
ss -s # Socket summary
nstat # Network counters
# CPU profiling (requires perf)
perf top # Real-time function-level CPU profiling
perf stat -a sleep 10 # System-wide counters for 10 seconds
perf record -g -a sleep 30 # Record 30 seconds of call stacks
perf report # Analyze recorded data
# One-liner: check all major resources
echo "=== CPU ===" && mpstat 1 1 && echo "=== MEM ===" && free -h && echo "=== DISK ===" && iostat -x 1 1 && echo "=== NET ===" && ss -s
Sysctl Kernel Parameter Tuning
Network Tuning
# /etc/sysctl.d/60-network-performance.conf
# Increase the maximum socket receive/send buffer sizes
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
# TCP buffer auto-tuning (min, default, max in bytes)
net.ipv4.tcp_rmem = 4096 1048576 134217728
net.ipv4.tcp_wmem = 4096 1048576 134217728
# Increase connection backlog for high-traffic servers
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.core.netdev_max_backlog = 65535
# Enable TCP fast open (client and server)
net.ipv4.tcp_fastopen = 3
# Reuse TIME_WAIT sockets for new connections
net.ipv4.tcp_tw_reuse = 1
# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535
# TCP keepalive tuning (detect dead connections faster)
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
# Disable slow start after idle (keeps congestion window open)
net.ipv4.tcp_slow_start_after_idle = 0
# Enable BBR congestion control (requires kernel 4.9+)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
Memory Tuning
# /etc/sysctl.d/60-memory-performance.conf
# Reduce swappiness (0-100, lower = less swap usage)
# 10 for general servers, 1 for database servers
vm.swappiness = 10
# Dirty page ratios (controls when dirty data is flushed to disk)
# Lower values = more frequent, smaller writes (better for SSDs)
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
# For large-memory systems writing to fast storage
# vm.dirty_ratio = 40
# vm.dirty_background_ratio = 10
# Increase inotify limits (for apps watching many files)
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024
# Maximum number of open file descriptors system-wide
fs.file-max = 2097152
# Virtual memory overcommit
# 0 = heuristic (default), 1 = always overcommit, 2 = never overcommit
vm.overcommit_memory = 0
# For Redis or similar in-memory stores, use:
# vm.overcommit_memory = 1
# Disable Transparent Huge Pages if it causes latency spikes (common with databases)
# Done via boot parameter or runtime:
# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
Applying Sysctl Changes
# Apply all sysctl files
sysctl --system
# Apply a specific file
sysctl -p /etc/sysctl.d/60-network-performance.conf
# Set a parameter temporarily (lost on reboot)
sysctl -w vm.swappiness=10
# Verify a parameter
sysctl vm.swappiness
sysctl net.ipv4.tcp_congestion_control
I/O Scheduler Configuration
# Check the current scheduler for a device
cat /sys/block/sda/queue/scheduler
# Output example: [mq-deadline] none kyber bfq
# Set the scheduler temporarily
echo mq-deadline > /sys/block/sda/queue/scheduler # Good for databases
echo none > /sys/block/nvme0n1/queue/scheduler # Best for NVMe SSDs
echo bfq > /sys/block/sda/queue/scheduler # Good for interactive desktop
# Scheduler recommendations:
# NVMe SSD: none (noop) -- minimal overhead, hardware handles scheduling
# SATA SSD: mq-deadline -- provides fairness with low latency
# HDD: mq-deadline -- prevents starvation, good for databases
# Desktop: bfq -- prioritizes interactive I/O
# Make persistent via udev rule
cat <<'EOF' > /etc/udev/rules.d/60-io-scheduler.rules
# Set mq-deadline for rotational (HDD) devices
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"
# Set none for non-rotational (SSD/NVMe) devices
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
EOF
udevadm control --reload-rules
# Tune read-ahead for sequential workloads (database sequential scans)
blockdev --setrahead 4096 /dev/sda # 4096 sectors = 2 MB
# Enable TRIM for SSDs (weekly via systemd timer)
systemctl enable --now fstrim.timer
fstrim -av # Manual run
CPU Governor Configuration
# Check available governors
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
# Output: performance powersave schedutil
# Check current governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# Set all CPUs to performance mode (maximum frequency)
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
echo performance > "$cpu"
done
# Set using cpupower (if installed)
cpupower frequency-set -g performance
# Governor recommendations:
# Server (production): performance -- max frequency, lowest latency
# Server (general): schedutil -- kernel-driven dynamic scaling
# Laptop / idle server: powersave -- minimize power consumption
# Make persistent via systemd service
cat <<'EOF' > /etc/systemd/system/cpu-governor.service
[Unit]
Description=Set CPU governor to performance
[Service]
Type=oneshot
ExecStart=/usr/bin/cpupower frequency-set -g performance
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now cpu-governor
# Disable CPU boost (turbo) if consistent latency is needed
echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
Benchmarking Tools
fio -- Disk I/O Benchmarking
# Install fio
apt install -y fio # Debian/Ubuntu
dnf install -y fio # RHEL/CentOS
# Sequential read test (simulates backup reads)
fio --name=seq-read --ioengine=libaio --direct=1 --rw=read \
--bs=1M --numjobs=4 --size=1G --runtime=60 --time_based --group_reporting
# Sequential write test
fio --name=seq-write --ioengine=libaio --direct=1 --rw=write \
--bs=1M --numjobs=4 --size=1G --runtime=60 --time_based --group_reporting
# Random read (4K blocks -- simulates database IOPS)
fio --name=rand-read --ioengine=libaio --direct=1 --rw=randread \
--bs=4k --numjobs=16 --iodepth=64 --size=1G --runtime=60 --time_based --group_reporting
# Random write (4K blocks)
fio --name=rand-write --ioengine=libaio --direct=1 --rw=randwrite \
--bs=4k --numjobs=16 --iodepth=64 --size=1G --runtime=60 --time_based --group_reporting
# Mixed random read/write (70/30 -- typical database workload)
fio --name=mixed --ioengine=libaio --direct=1 --rw=randrw --rwmixread=70 \
--bs=4k --numjobs=8 --iodepth=32 --size=1G --runtime=60 --time_based --group_reporting
sysbench -- CPU and Memory Benchmarking
# Install: apt install -y sysbench (Debian) / dnf install -y sysbench (RHEL)
# CPU benchmark
sysbench cpu --threads=4 --time=30 run
# Memory benchmark
sysbench memory --threads=4 --time=30 --memory-block-size=1K --memory-total-size=100G run
iperf3 -- Network Benchmarking
# Install iperf3
apt install -y iperf3
# Start server on one host
iperf3 -s
# Run client test from another host
iperf3 -c <server-ip> -t 30 -P 4 # 30 seconds, 4 parallel streams
# Test with UDP (measure packet loss)
iperf3 -c <server-ip> -u -b 1G -t 30
# Reverse mode (server sends to client)
iperf3 -c <server-ip> -R -t 30
Quick-Reference Tuning Profiles
Web Server (nginx/Apache)
# /etc/sysctl.d/60-webserver.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fastopen = 3
net.ipv4.ip_local_port_range = 1024 65535
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
fs.file-max = 2097152
vm.swappiness = 10
Database Server (PostgreSQL/MySQL)
# /etc/sysctl.d/60-database.conf
vm.swappiness = 1
vm.dirty_ratio = 15
vm.dirty_background_ratio = 3
vm.overcommit_memory = 2
vm.overcommit_ratio = 80
net.core.somaxconn = 4096
fs.file-max = 2097152
# Disable THP for databases
# echo never > /sys/kernel/mm/transparent_hugepage/enabled
Troubleshooting
| Symptom | Diagnostic Command | Common Fix |
|---|---|---|
| High CPU, no obvious process | perf top, mpstat -P ALL 2 | Check for kernel-level issues: softirqs, interrupts |
| High load avg, low CPU usage | vmstat 2 (check b column) | I/O bottleneck: tune scheduler, check disk health |
| System swapping heavily | free -h, vmstat 2 (check si/so) | Reduce vm.swappiness, add RAM, find memory leak |
| Disk latency spikes | iostat -x 2 (check await) | Switch I/O scheduler, reduce dirty ratio, add SSD |
| "Too many open files" error | cat /proc/sys/fs/file-nr | Increase fs.file-max and LimitNOFILE |
| Network throughput low | iperf3 -c <server>, ethtool | Increase buffer sizes, enable BBR, check MTU |
| Application timeout under load | ss -s, sysctl net.core.somaxconn | Increase somaxconn and tcp_max_syn_backlog |
| Inconsistent latency | Check CPU governor | Set governor to performance, disable turbo boost |
Related Skills
linux-administration-- General system monitoring and managementsystemd-services-- Resource limits via cgroups in unit filesblock-storage-- Storage-level performance (LVM, RAID, filesystems)nfs-storage-- NFS-specific performance tuning
Other skills for the same job
different authors, same section of the catalogue
Integration with protocols.io API for managing scientific protocols. This skill should be used when working with protocols.io to search, create, update, or publish protocols; manage protocol steps and materials; handle discussions and comments; organize workspaces; upload and manage files; or integrate protocols.io functionality into workflows. Applicable for protocol discovery, collaborative protocol development, experiment tracking, lab protocol management, and scientific documentation.
Analyzes job descriptions and generates tailored resumes that highlight relevant experience, skills, and achievements to maximize interview chances
Generate Excalidraw diagrams from natural language descriptions. Use when asked to "create a diagram", "make a flowchart", "visualize a process", "draw a system architecture", "create a mind map", or "generate an Excalidraw file". Supports flowcharts, relationship diagrams, mind maps, and system architecture diagrams. Outputs .excalidraw JSON files that can be opened directly in Excalidraw.
Build and distribute Expo development clients locally or via TestFlight
Use when you have a written implementation plan to execute in a separate session with review checkpoints
Data structure for annotated matrices in single-cell analysis. Use when working with .h5ad files or integrating with the scverse ecosystem. This is the data format skill—for analysis workflows use scanpy; for probabilistic models use scvi-tools; for population-scale queries use cellxgene-census.
Benchling R&D platform integration. Access registry (DNA, proteins), inventory, ELN entries, workflows via API, build Benchling Apps, query Data Warehouse, for lab data management automation.
Comprehensive molecular biology toolkit. Use for sequence manipulation, file parsing (FASTA/GenBank/PDB), phylogenetics, and programmatic NCBI/PubMed access (Bio.Entrez). Best for batch processing, custom bioinformatics pipelines, BLAST automation. For quick lookups use gget; for multi-service integration use bioservices.
How to use it
Take bagelhole/performance-tuning from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.