Linux System Monitoring Guide

System monitoring is essential for maintaining healthy Linux systems. This guide covers tools and techniques for tracking system resources, performance, and diagnosing issues.

Overview: Monitoring Tools

Tool	Purpose	Best For
`top`	Interactive process viewer	Real-time system overview
`htop`	Enhanced top	User-friendly process management
`ps`	Process snapshot	Detailed process information
`free`	Memory usage	RAM and swap monitoring
`vmstat`	Virtual memory stats	System performance overview
`iostat`	I/O statistics	Disk performance monitoring
`df`	Disk space	Filesystem usage
`du`	Disk usage	Directory size analysis
`uptime`	System uptime	Load average
`mpstat`	CPU statistics	Per-CPU performance

Monitoring Processes with top

Basic top Usage

# Launch top
top

# Key commands in top:
# q - Quit
# k - Kill process
# r - Renice process
# P - Sort by CPU
# M - Sort by memory
# 1 - Show individual CPUs
# c - Show full command
# h - Help

top Output Explained

top - 10:25:33 up 5 days, 12:34,  2 users,  load average: 0.52, 0.58, 0.59
Tasks: 247 total,   1 running, 246 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.2 us,  2.1 sy,  0.0 ni, 92.3 id,  0.3 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  15892.3 total,   1234.5 free,   8234.2 used,   6423.6 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   6542.1 avail Mem

Load Average: 1min, 5min, 15min averages
CPU states: us=user, sy=system, ni=nice, id=idle, wa=I/O wait
Memory: Total, free, used, buffers/cache

Advanced top Options

# Update every 2 seconds
top -d 2

# Show specific user's processes
top -u username

# Batch mode (for scripting)
top -b -n 1 > top-snapshot.txt

# Show specific number of processes
top -n 1 -o %CPU | head -20

# Monitor specific PID
top -p 1234,5678

Enhanced Process Monitoring with htop

htop Advantages

# Install htop
sudo apt install htop  # Debian/Ubuntu
sudo yum install htop  # RHEL/CentOS

# Launch htop
htop

htop Key Features

F1-Help  F2-Setup  F3-Search  F4-Filter  F5-Tree  F6-Sort  F9-Kill  F10-Quit

# Interactive features:
- Mouse support
- Color-coded meters
- Tree view of processes
- Easy sorting and filtering
- Vertical and horizontal scrolling

htop Keyboard Shortcuts

# Navigation
Arrow keys - Move selection
Space - Tag process
U - Show user's processes
t - Tree view
K - Show/hide kernel threads

# Sorting
F6 or > - Select sorting
P - Sort by CPU
M - Sort by memory
T - Sort by time

# Actions
F9 or k - Kill process
F7/F8 - Nice/renice
F5 - Tree view toggle
/ - Search

Process Information with ps

Basic ps Usage

# Show all processes
ps aux

# Show process tree
ps auxf

# Show processes for current user
ps ux

# BSD style (most common)
ps aux

# Unix style with headers
ps -ef

ps Output Columns

# Common columns:
# USER - Process owner
# PID - Process ID
# %CPU - CPU usage percentage
# %MEM - Memory usage percentage
# VSZ - Virtual memory size
# RSS - Resident set size (physical memory)
# TTY - Terminal
# STAT - Process state
# START - Start time
# TIME - CPU time
# COMMAND - Command name

Custom ps Output

# Custom columns
ps -eo pid,user,%cpu,%mem,cmd

# Sort by memory
ps aux --sort=-%mem | head -10

# Sort by CPU
ps aux --sort=-%cpu | head -10

# Show threads
ps -eLf

# Show process tree
ps axjf

Finding Specific Processes

# Find by name
ps aux | grep nginx

# Find by user
ps -u username

# Find by PID
ps -p 1234

# Find parent-child relationships
ps -o pid,ppid,cmd

# Show full command line
ps aux | grep -v grep | grep processname

Memory Monitoring with free

Basic free Usage

# Show memory usage
free

# Human-readable format
free -h

# Show in megabytes
free -m

# Show in gigabytes
free -g

# Continuous monitoring (every 2 seconds)
free -h -s 2

Understanding free Output

free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       7.8Gi       1.2Gi       234Mi       6.4Gi       7.1Gi
Swap:         2.0Gi          0B       2.0Gi

Key metrics: - total - Total installed RAM - used - Memory in use - free - Completely unused memory - shared - Shared memory (tmpfs) - buff/cache - Kernel buffers and page cache - available - Estimate of memory available for applications

Memory Usage Analysis

# Show total memory
free -h | awk '/^Mem:/ {print $2}'

# Show used memory
free -h | awk '/^Mem:/ {print $3}'

# Calculate usage percentage
free | awk '/^Mem:/ {printf "%.2f%%\n", $3/$2 * 100}'

# Monitor swap usage
free -h | grep Swap

System Statistics with vmstat

Basic vmstat Usage

# Single snapshot
vmstat

# Update every 2 seconds, 10 times
vmstat 2 10

# Show in megabytes
vmstat -S M 2

# Detailed memory stats
vmstat -s

# Disk statistics
vmstat -d

Understanding vmstat Output

vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1234568  89234 6234234  0    0     2    15   45  321  5  2 92  1  0

Columns explained: - r - Processes waiting for CPU - b - Processes in uninterruptible sleep - swpd - Virtual memory used - free - Idle memory - si/so - Swap in/out per second - bi/bo - Blocks in/out per second - in - Interrupts per second - cs - Context switches per second - us/sy/id/wa - CPU time percentages

Disk Monitoring

Disk Space with df

# Show disk usage
df -h

# Show inode usage
df -i

# Show specific filesystem type
df -t ext4

# Exclude type
df -x tmpfs

# Show all filesystems
df -a

# Display totals
df -h --total

Directory Size with du

# Directory size
du -sh /var/log

# Top-level directories
du -h --max-depth=1 /var

# Sort by size
du -h /var | sort -rh | head -10

# Exclude patterns
du -h --exclude="*.log" /var/log

# Show total
du -ch /var/log/*.log | tail -1

Finding Large Files

# Find files larger than 100MB
find / -type f -size +100M -exec ls -lh {} \; | sort -k5 -hr

# Top 10 largest files
find /var -type f -exec du -h {} + | sort -rh | head -10

# Disk usage by type
find /var/www -name "*.jpg" -exec du -ch {} + | tail -1

I/O Monitoring with iostat

Basic iostat Usage

# Install sysstat package
sudo apt install sysstat

# Show I/O statistics
iostat

# Update every 2 seconds
iostat 2

# Extended statistics
iostat -x

# Show specific device
iostat -x sda 2

# CPU and device stats
iostat -xc 2

Understanding iostat Output

iostat -x 2
Device    r/s   w/s   rkB/s   wkB/s  %util
sda      45.2  12.3   1234.5   567.8  12.5

Key metrics: - r/s, w/s - Reads/writes per second - rkB/s, wkB/s - KB read/written per second - %util - Device utilization percentage

CPU Monitoring

CPU Info

# View CPU information
lscpu

# Count CPUs
nproc

# CPU details
cat /proc/cpuinfo

# Per-CPU usage with mpstat
mpstat -P ALL 2

Load Average

# Show load average
uptime

# Watch load average
watch -n 1 uptime

# Detailed load info
cat /proc/loadavg

Load average interpretation: - Less than CPU count: System is handling load - Equal to CPU count: Fully utilized - Greater than CPU count: Processes are waiting

Network Monitoring

Network Connections

# Active connections
netstat -tunlp

# Or with ss (modern alternative)
ss -tunlp

# Connection statistics
netstat -s

# Interface statistics
ifconfig
ip -s link

# Bandwidth usage
iftop  # Requires installation
nload  # Requires installation

Network Traffic

# Monitor bandwidth in real-time
iftop -i eth0

# Simple bandwidth monitor
nload eth0

# Network statistics
ip -s link show eth0

# Packet statistics
watch -n 1 'cat /proc/net/dev'

System Uptime and Load

# System uptime and load
uptime

# Who is logged in
w

# Last logins
last

# Last reboots
last reboot

# System boot time
who -b

Real-Time Monitoring Scripts

CPU Usage Alert

#!/bin/bash
# Alert if CPU usage > 80%
while true; do
    CPU=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
    if (( $(echo "$CPU > 80" | bc -l) )); then
        echo "High CPU usage: $CPU%"
    fi
    sleep 10
done

Memory Monitor

#!/bin/bash
# Monitor memory usage
while true; do
    MEM=$(free | awk '/Mem:/ {printf "%.2f", $3/$2 * 100}')
    echo "$(date '+%Y-%m-%d %H:%M:%S') Memory: ${MEM}%"
    sleep 60
done

Disk Space Alert

#!/bin/bash
# Alert if disk usage > 90%
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{print $5 " " $1}' | while read output; do
    usage=$(echo $output | awk '{print $1}' | cut -d'%' -f1)
    partition=$(echo $output | awk '{print $2}')
    if [ $usage -ge 90 ]; then
        echo "Warning: $partition at ${usage}%"
    fi
done

Performance Analysis

System Performance Overview

# All-in-one performance check
#!/bin/bash
echo "=== CPU ==="
mpstat 1 1

echo -e "\n=== Memory ==="
free -h

echo -e "\n=== Disk ==="
df -h

echo -e "\n=== Load ==="
uptime

echo -e "\n=== Top Processes ==="
ps aux --sort=-%cpu | head -6

Identify Performance Bottlenecks

# High CPU processes
ps aux --sort=-%cpu | head -10

# High memory processes
ps aux --sort=-%mem | head -10

# Disk I/O wait
iostat -x 2 5

# Network connections
ss -s

Quick Reference

Essential Commands

# Process monitoring
top                    # Interactive process viewer
htop                   # Enhanced top
ps aux                 # All processes snapshot

# Memory monitoring
free -h                # Memory usage
vmstat 2               # Virtual memory stats

# Disk monitoring
df -h                  # Disk space
du -sh /path          # Directory size
iostat -x             # I/O statistics

# System info
uptime                # Load average
lscpu                 # CPU info
uname -a              # System info

# Network
ss -tunlp             # Network connections
netstat -tunlp        # Network connections (older)