Linux Static Performance Tools Cheatsheet

Table of Contents

1. System Architecture Overview
2. Performance Analysis Tools by Layer
3. Hardware Layer Tools
4. Operating System Layer Tools
5. Application Layer Tools
6. Detailed Tool Reference

System Architecture Overview

The Linux system architecture consists of multiple layers, each with specific performance monitoring and analysis tools. Here's a detailed breakdown of each layer and its components:

Hardware Layer

• CPUs
• DRAM
• I/O Controllers
• Network Controllers
• Storage Devices

Operating System Layer Components

1. Device Drivers
2. Block Device
3. Network Device
4. Virtual Memory
5. File Systems
6. TCP/UDP
7. IP Stack
8. System Call Interface
9. Scheduler
10. Clock Source

Application Layer

• Applications
• System Libraries
• VFS (Virtual File System)
• Sockets

Performance Analysis Tools by Layer

Hardware Level Tools

CPU Analysis Tools

# CPU information and statistics
/proc/cpuinfo    # Detailed CPU information
cpuid            # CPU identification and features
lscpu            # CPU architecture information
cpu-x            # CPU identification tool

# CPU Performance Monitoring
numact1          # NUMA topology and statistics
lstopo           # Hardware topology viewer
dmidecode        # DMI table decoder

Memory Analysis

# Memory statistics and information
free -m          # Memory usage overview
vmstat           # Virtual memory statistics
dmidecode        # Memory hardware information

# Memory Performance Tools
lstopo           # Memory topology visualization
numact1          # NUMA memory statistics

Operating System Level Tools

File System Tools

# Disk and Filesystem Analysis
df -h            # Filesystem usage
lsblk            # Block device information
blockdev         # Block device operations
smartctl         # SMART disk monitoring
fdisk -l         # Disk partition information
lsusb            # USB device information

# Advanced Storage Analysis
mdadm            # RAID management and monitoring
dmesg            # Storage device messages

Network Analysis Tools

# Basic Network Tools
ip route         # IP routing table
tc              # Traffic control
ethtool         # Network interface settings
iwconfig        # Wireless interface configuration
lldptool        # Link Layer Discovery Protocol tool

# Network Statistics
netstat         # Network statistics
ss              # Socket statistics
ip              # IP configuration and statistics

System Call Interface Tools

# System Call Monitoring
strace          # System call tracer
ltrace          # Library call tracer
syscall         # System call information
sysctl          # Kernel parameter configuration

Scheduler Analysis

# Scheduler Tools
schedtool       # Process scheduler configuration
chrt            # Real-time scheduler priorities
taskset         # CPU affinity configuration

Application Level Tools

Process Analysis

# Process Monitoring
top             # Process activity monitoring
htop            # Interactive process viewer
ps              # Process status
pstree          # Process hierarchy view

System Library Analysis

# Library Tools
ldd             # Shared library dependencies
ldconfig        # Library path configuration
ld              # Link editor for object files

Detailed Tool Reference

System Information Tools

sysctl

# View all system parameters
sysctl -a

# View specific parameter
sysctl kernel.hostname

# Modify parameter
sysctl -w kernel.hostname=newname

# Load settings from file
sysctl -p /etc/sysctl.conf

dmesg

# View kernel messages
dmesg

# Clear kernel ring buffer
dmesg -c

# Show human-readable timestamps
dmesg -T

# Follow new messages
dmesg -w

Storage Performance Tools

smartctl

# Check disk health
smartctl -a /dev/sda

# Run short self-test
smartctl -t short /dev/sda

# Show test results
smartctl -l selftest /dev/sda

# Enable SMART
smartctl -s on /dev/sda

iostat

# Basic I/O statistics
iostat

# Extended statistics
iostat -x

# Device utilization
iostat -d

# Updates every 2 seconds
iostat 2

Network Performance Tools

ethtool

# Show interface settings
ethtool eth0

# Show interface statistics
ethtool -S eth0

# Set interface speed
ethtool -s eth0 speed 1000

# Check link status
ethtool -i eth0

tc (Traffic Control)

# Show queueing discipline
tc qdisc show

# Add rate limiting
tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms

# Show filter rules
tc filter show dev eth0

# Delete all rules
tc qdisc del dev eth0 root

Process Management Tools

schedtool

# Show process scheduler info
schedtool -p 1234

# Set SCHED_FIFO policy
schedtool -F -p 50 1234

# Set CPU affinity
schedtool -a 0-3 1234

# Show all scheduler policies
schedtool -h

numact1

# Show NUMA topology
numactl --hardware

# Run command on specific node
numactl --membind=0 command

# Show NUMA statistics
numactl --show

# Set memory policy
numactl --preferred=0 command

Virtual Memory Analysis

vmstat

# Memory statistics
vmstat

# Detailed memory info
vmstat -s

# Active/inactive memory
vmstat -a

# Disk statistics
vmstat -d

Advanced Analysis Tools

SystemTap

# Create simple probe
stap -e 'probe syscall.open { println(execname()) }'

# Monitor file operations
stap -e 'probe vfs.read { printf("%s read %d bytes\n", execname(), count) }'

# Track process creation
stap -e 'probe process.begin { printf("%s started\n", execname()) }'

perf

# CPU performance counters
perf stat command

# System-wide sampling
perf record -a sleep 10

# Show recorded data
perf report

# Live performance monitoring
perf top

Hardware Management

dmidecode

# Show all hardware info
dmidecode

# Show memory info
dmidecode -t memory

# Show CPU info
dmidecode -t processor

# Show system info
dmidecode -t system

Volume Management Tools

MegaCli

# Show RAID configuration
MegaCli -CfgDsply -aALL

# Check battery status
MegaCli -AdpBbuCmd -aALL

# Show drive information
MegaCli -PDList -aALL

# Check rebuild status
MegaCli -PDRbld -ShowProg -physdrv[252:2] -aALL

Performance Monitoring Best Practices

1. Regular Monitoring
2. Set up baseline measurements
3. Monitor trends over time

4. Create alerts for anomalies

5. Resource Usage

6. Track CPU utilization
7. Monitor memory consumption
8. Watch I/O patterns

9. Analyze network traffic

10. System Health

11. Check system logs
12. Monitor hardware status
13. Track error rates

14. Verify service status

15. Performance Optimization

16. Identify bottlenecks
17. Tune system parameters
18. Optimize application settings
19. Balance resource allocation

Common Performance Issues and Solutions

1. CPU Bottlenecks
2. Use top and htop to identify high CPU processes
3. Check process priorities with nice and renice
4. Monitor CPU scheduling with schedtool

5. Analyze system load with uptime

6. Memory Problems

7. Check memory usage with free and vmstat
8. Monitor swap usage
9. Analyze memory leaks

10. Tune kernel memory parameters

11. Disk I/O Issues

12. Monitor disk activity with iostat
13. Check disk health with smartctl
14. Analyze I/O patterns with iotop

15. Optimize filesystem parameters

16. Network Performance

17. Monitor bandwidth with iftop
18. Analyze connections with netstat
19. Check network errors with ethtool
20. Tune network parameters with sysctl

Quick Reference Commands

# System Overview
top                     # Process activity
htop                    # Interactive process viewer
uptime                  # Load average
dmesg                   # Kernel messages

# Memory Analysis
free -m                 # Memory usage
vmstat                  # Virtual memory stats
pmap                    # Process memory map
smem                    # Memory reporting

# Disk Operations
iostat                  # I/O statistics
iotop                   # I/O monitoring
fdisk -l                # Disk partitions
df -h                   # Filesystem usage

# Network Analysis
netstat                 # Network statistics
ss                      # Socket statistics
iftop                   # Network bandwidth
tcpdump                 # Packet analysis

# Process Management
ps aux                  # Process status
pstree                  # Process tree
nice                    # Process priority
renice                  # Change priority

# System Information
uname -a                # System information
lscpu                   # CPU information
lsblk                   # Block devices
lspci                   # PCI devices

Additional Resources

1. Documentation
2. Man pages for each tool
3. Linux kernel documentation
4. Distribution-specific guides

5. Tool-specific documentation

6. Performance Tuning Guides

7. Red Hat Performance Tuning Guide
8. Ubuntu Server Guide
9. Linux Performance website

10. Kernel documentation

11. Monitoring Tools

12. Nagios
13. Zabbix
14. Prometheus

15. Grafana

16. Training Resources

17. Linux Foundation courses
18. Distribution certification programs
19. Online tutorials
20. Community forums