Essential Linux System Administration & Performance Tuning Guide

Overview of Linux System Performance

Modern Linux servers require meticulous monitoring and tuning to handle high-throughput workloads. In this comprehensive reference guide, we examine kernel parameters, storage I/O, memory subsystem management, and real-time troubleshooting utilities.

1. CPU & Process Monitoring Tools

When an outage or slowdown occurs, these are the primary diagnostic commands to execute in sequence:

# Check overall load averages and per-core utilization
top -b -n 1 | head -n 20
htop

# High-resolution interactive process viewer
uptime
vmstat 1 5
mpstat -P ALL 1 3

Pro Tip: An average load higher than the number of physical CPU cores indicates processes are queueing up for CPU cycles or waiting on disk I/O.

2. Memory & Virtual Memory Management

Inspecting RAM usage and understanding buffer/cache reclamation:

# View human readable memory metrics
free -h -w

# Check for Out-Of-Memory (OOM) killer events in kernel ring buffer
dmesg -T | grep -i -E "oom|killed process"
journalctl -k --grep="Out of memory"

3. Tuning /etc/sysctl.conf for High Concurrency

For high-traffic web servers and reverse proxies (NGINX/HAProxy), apply the following network and file descriptor tuning parameters:

# /etc/sysctl.d/99-sysadmin-tuning.conf
fs.file-max = 2097152
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
vm.swappiness = 10

Reload the sysctl configuration immediately with:

sudo sysctl --system

4. Essential systemd Service Troubleshooting

# Check detailed service status and recent journal logs
sudo systemctl status nginx.service --no-pager -l

# View live stream of system logs for a specific unit
sudo journalctl -u nginx.service -f -o cat