When a critical outage hits a production environment, time is your most crucial resource. Systems going down, unresponsive microservices, kernel panics, or sudden I/O bottlenecks can quickly jeopardize core business operations. For systems administrators, site reliability engineers (SRE), and DevOps teams managing enterprise workloads on Red Hat Enterprise Linux 8 or 9 (RHEL), having a disciplined and reproducible standard operating procedure (SOP) is what separates a blink from a catastrophe.
This detailed troubleshooting guide provides a structured methodology for diagnosing and resolving high-severity production incidents across RHEL 8 and RHEL 9 environments. From rapid initial triage and systematic resource verification to core dump analysis and efficient Red Hat Support engagement, this guide covers everything required to restore stability and identify root causes swiftly.
This guide establishes a baseline framework for debugging critical infrastructure incidents, detailing operational methods to record system state, collect live diagnostic data, isolate resource constraints, and resolve outages efficiently while preserving vital evidence for root cause analysis (RCA). While individual enterprise deployments, compliance policies, or complex application architectures may require specific customizations, the triage pathways and data collection techniques detailed here provide a foundational toolkit for core focus areas across any RHEL ecosystem.
Incident severity classification and triage
To ensure swift, proportional action during an active incident, operations personnel must map observed symptoms against the standardized operational triage matrix before executing invasive diagnostic routines or restarting services:
| Severity level | Production impact | Immediate action required | Red Hat Support SLA |
|---|---|---|---|
| Severity 1 (Critical) | Complete loss of core business service or production cluster down. | Collect sosreport or live diagnostics immediately; execute panic/dump sequence if hung; notify TAM. | 1st Response < 1 hour |
| Severity 2 (High) | Operations severely degraded; critical functionality unavailable without workaround. | Capture performance baseline (sar, top, journalctl); log issue details; prepare support ticket. | 1st Response < 2 hours |
| Severity 3 (Medium) | Non-critical component degraded or isolated failure with workaround available. | Collect regular health checks and logs during business hours; attach output to case. | Standard Response |
| Severity 4 (Low) | General questions, documentation requests, or minor non-impacting issues. | Document background and attach diagnostic output using Customer Portal. | Standard Response |
Refer to https://access.redhat.com/support/offerings/production/sla for SLA definitions.
Comprehensive troubleshooting checklist for RHEL 8 and 9
When troubleshooting an active system, systematically work through these key diagnostic subsystems. The goal is to rapidly determine:
- What is failing?
- Which resource is constrained?
- When did the degradation begin?
- What evidence must be captured prior to taking intrusive corrective actions?
System identity and uptime verification
Objective: Confirm the exact target system identity, environment context, release baseline, and reboot timeline.
First, verify the system hostname and hardware deployment metadata:
Objective: Confirm the exact target system identity, environment context, release baseline, and reboot timeline.
First, verify the system hostname and hardware deployment metadata:
$ hostnamectlConfirm the OS release build:
$ cat /etc/redhat-releaseInspect active kernel release identifier:
$ uname -rGet the operational uptime and current load averages:
$ uptimeVerify system clock, synchronization state, and timezone setting:
$ timedatectlInspect reboot timeline and unexpected restart history:
$ last -x reboot | headKey observables and anomalies to target:
- Unexpected reboots or kernel panics correlating with reported issue timestamps
- Recent kernel upgrades or updates that do not match production baseline standards
- Clock skew, timezone misconfigurations, or unsynchronized system time across cluster nodes
CPU utilization, system load, and process state analysis
Objective: Identify CPU saturation, scheduling delays, runaway tasks, or processes stuck in I/O wait.
First, check system load metrics:
$ uptimeInteractive top process monitor:
$ topSample multi-core CPU utilization (this requires the sysstat package):
$ mpstat -P ALL 1 5List top CPU-consuming tasks:
$ ps -eo pid,ppid,comm,%cpu,%mem --sort=-%cpu | headCount total active tasks:
$ ps -e --no-headers | wc -lIdentify processes in uninterruptible sleep (D state):
$ ps -eo state,pid,ppid,comm,wchan:32 | awk '$1=="D"'Evaluate run-queue and CPU scheduling pressure:
$ vmstat 1 5Locate zombie tasks:
$ top -b1 -n1 | grep -i zombie
$ ps -A -ostat,pid,ppid | grep -e '\[zZ\]'Key observables and anomalies to target:
- High CPU usage or sustained load average consistently exceeding logical core counts
- Elevated
%wa(I/O wait time indicating storage bottlenecks) or%st(virtualization CPU steal time on cloud hypervisors) - Tasks blocked in
Dstate (uninterruptible I/O locks on SAN, NFS, or local disk systems) - High context-switching rates or severe run-queue backlogs
System load averages combine both actively running tasks and tasks stuck in uninterruptible sleep. It is not strictly equivalent to CPU utilization.
Memory saturation, swap activity, and OOM diagnostics
Objective: Evaluate RAM exhaustion, thrashing swap activity, memory leaks, and out-of-memory (OOM) killer terminations.
Check system memory allocation in human-readable format:
$ free -hRead detailed kernel memory counters:
$ cat /proc/meminfoInspect configured swap partitions and devices:
$ swapon --showMonitor swap-in (si) and swap-out (so) page rates:
$ vmstat 1 5Identify top resident set size (RSS) memory consumers:
$ ps -eo pid,comm,%mem,rss --sort=-rss | headAudit kernel journal for out-of-memory (OOM) killer invocations:
$ journalctl -k --since "-24 hours" | grep -Ei 'out of memory|oom-kill|killed process'Key observables and anomalies to target
- Critically low
MemAvailablevalues in/proc/meminfo(do not rely solely on rawfreememory statistics, because Linux caches aggressively) - Active, sustained page swapping (non-zero
si/sovalues invmstat) - Active kernel OOM-killer events terminating critical database or application processes
- Monotonic upward RSS growth over time indicating application-level memory leaks
Static swap space usage alone does not indicate an emergency. Systems frequently allocate swap for cold process pages without actively paging or experiencing performance degradation.
Storage subsystem: Disk capacity, inodes, and mounts
Objective: Identify full storage volumes, inode exhaustion, stale mounts, read-only filesystem remounts, and NFS connectivity locks.
Check filesystem disk utilization and type:
$ df -hTCheck inode usage across mounts:
$ df -ihInspect root filesystem specifically:
$ df -hT /Display active mount points and hierarchy:
$ findmntView configured persistent mounts:
$ cat /etc/fstabValidate /etc/fstab integrity and configuration correctness:
$ findmnt --verifyIsolate directory space usage under /var:
$ du -xhd1 /var 2>/dev/null | sort -hScan kernel logs for filesystem I/O or corruption errors:
$ journalctl -k --since "-24 hours" | grep -Ei 'I/O error|filesystem error|xfs|ext4|read-only|buffer I/O'Specialized diagnostic checks for NFS mounts:
$ findmnt -t nfs,nfs4
$ nfsstat -mKey observables and anomalies to target:
- Filesystems approaching 100% capacity
- Inode exhaustion (100% inode utilization prevents new file creation even with available disk space)
- Full partitions on
/,/var,/tmp, or/boot - Filesystems remounted as Read-Only following disk errors.
- Failed mount definitions in /etc/fstab blocking system boot or application startup.
Never execute mount -a or automated XFS/EXT4 repair tools blindly on production servers during an outage without full root cause evaluation.
Disk I/O, block devices, and storage performance
Objective: Diagnose storage latency, I/O queue bottlenecks, SAN/Multipath path failures, and LVM space allocation issues.
Extended disk I/O performance monitoring (sysstat required):
$ iostat -xz 1 5List block devices, mount points, and filesystem types:
$ lsblk -fPrint block device UUIDs and metadata:
$ blkidInspect LVM logical volumes, volume groups, and physical volumes:
$ lsblk ; lvs
$ pvs ; vgs ; lvs -a -o +devicesFilter kernel log for disk controller, multipath, or SCSI resets:
$ journalctl -k --since "-24 hours" | grep -Ei 'I/O error|reset|timeout|failed command|blk_update_request'Verify active SAN multipath maps (if configured):
$ multipath -llKey observables and anomalies to target:
- Elevated service wait times (
await) and high device utilization (%util) iniostat - High I/O queue depths causing process stalls in uninterruptible sleep
- Exhausted LVM Volume Groups or thin-pool storage limits
- SCSI layer timeouts, bus resets, or dropped paths on enterprise SAN storage
systemd service and application health
Objective: Verify systemd service states, detect failed units, review unit logs, and check active network socket bindings.
List all failed systemd units across the system:
systemctl --failedInspect specific service status (e.g., sshd, httpd, NetworkManager):
systemctl status <service>Check service startup enablement:
systemctl is-enabled <service>View service log journal for the current boot:
journalctl -u <service> -bReview recent service logs from the past hour:
journalctl -u <service> --since "-1 hour"Inspect listening TCP and UDP sockets and associated processes:
ss -lntup
ss -lntpReplace <service> with target unit names such as sshd, httpd, or NetworkManager.
Key observables and anomalies to target:
- systemd units in
failedorauto-restartloops - Missing listening sockets on expected application ports
An active (running) systemd state does not guarantee application health. Always validate application response through local HTTP endpoints, API probes, or functional health checks.
Network connectivity, routing, and DNS resolution
Objective: Isolate whether network issues stem from local interface failures, routing tables, name resolution, or upstream firewalls.
View brief network interface IP configuration:
$ ip -br addrInspect interface packet counters, errors, and drops:
$ ip -s linkDisplay current IPv4 routing table:
$ ip routeInspect ARP/neighbor cache entries:
$ ip neighCheck active DNS resolver configuration:
$ cat /etc/resolv.confTest hostname resolution via system NSS layer:
$ getent hosts <hostname>Basic ICMP network reachability test:
$ ping -c 4 <destination>Test specific TCP port connectivity using netcat:
$ nc -vz <hostname> <port>Display overall socket allocation statistics:
$ ss -sKey observables and anomalies to target:
- Network interfaces in
DOWNstate or accumulating drop/error counters - Missing default gateways or invalid static routes in
ip route - DNS lookup delays or timeouts in
getent hosts - Connection timeouts or refused sockets when attempting TCP handshake tests
A failed ICMP ping test does not definitively prove remote host failure, because ICMP traffic is frequently filtered by enterprise firewalls. Conversely, successful ping responses do not guarantee that application TCP ports are open and listening.
System logs, kernel ring buffer, and event correlation
Objective: Correlate system degradation timelines with kernel log messages, systemd events, and system log entries.
Review error-level log entries for the current boot:
$ journalctl -p err -bReview warning-level log entries and above:
$ journalctl -p warning -bInspect recent system logs from the past hour:
$ journalctl --since "-1 hour"Inspect kernel logs specifically for current boot:
$ journalctl -k -bPrint kernel ring buffer messages with human timestamps:
$ dmesg -T | tail -100Inspect recent SSH authentication logs:
$ journalctl -u sshd --since "-1 hour"List traditional system log files:
$ ls -ltr /var/log/
$ tail -100 /var/log/messagesAutomated high-priority pattern filtering:
$ sudo egrep -i \
"error|warning|fail|slow|crash|panic|delay|critical|respond|lockup|block|OOM" /var/log/messages
$ sudo dmesg -T | egrep -i \
"error|warning|fail|slow|crash|panic|delay|critical|respond|lockup|block|OOM"
$ sudo journalctl -p err..emerg -b --no-pagerAnalyze system startup bottlenecks and boot sequence delays:
$ systemd-analyze blame
$ systemd-analyze critical-chainKey observables and anomalies to target:
- Critical keywords:
error,warning,fail,slow,crash,not responding,delay,critical,lockup,block,OOM - Kernel oopses, stack traces, CPU soft lockups, or hardware machine check exceptions (MCE)
Timestamps generated using dmesg -T are converted from kernel uptime seconds and can drift following system sleep states or clock adjustments. For precision time correlation, rely on journalctl output.
Security controls: SELinux and file permissions
Objective: Detect access vector cache (AVC) denials, misconfigured security contexts, or file permission locks blocking application processes.
Verify current SELinux execution mode:
$ getenforceInspect detailed SELinux status and policy type:
$ sestatusCheck file permissions and SELinux security contexts:
$ ls -l <path>
$ namei -l <path>Audit setroubleshoot logs for automated policy remediation advice:
$ journalctl -t setroubleshoot --since "-1 hour"Search audit logs directly for recent AVC denials:
$ ausearch -m AVC,USER_AVC -ts recentKey observables and anomalies to target:
- SELinux AVC denials logged while running in
Enforcingmode - Mismatched file ownership, mode permissions, or path traversal restrictions
- Incorrect SELinux file labeling following manual file restores or moves
Do not disable SELinux globally as a quick troubleshooting workaround. Instead, inspect AVC denial events, test with Permissive mode if necessary, and apply proper context relabeling (restorecon) or custom booleans.
Time synchronization, scheduled cron jobs, and resource limits
Objective: Uncover hidden issues such as NTP clock drift, scheduled background batch jobs, open file descriptor saturation, or process ID limit exhaustion.
Check system clock synchronization state:
$ timedatectl statusCheck chrony tracking state and offset:
$ chronyc trackingList configured NTP time sources:
$ chronyc sources -vCheck root crontab definitions:
$ crontab -lInspect system-wide cron schedules:
$ ls -l /etc/cron.* /etc/cron.d/Audit recent execution logs for cron daemon:
$ journalctl -u crond --since "-1 hour"Check current shell user resource limits:
$ ulimit -aCheck system-wide file descriptor usage (allocated and max):
$ cat /proc/sys/fs/file-nrCheck maximum system process ID limit:
$ cat /proc/sys/kernel/pid_maxInspect resource limits for a specific target process PID:
$ cat /proc/<PID>/limitsKey observables and anomalies to target:
- Clock drift causing Kerberos, TLS, or database cluster synchronization failures
- Resource-heavy cron jobs firing during peak business hours
- Processes hitting
Max open filesorMax processeslimits defined in/etc/security/limits.confor systemd service unit files
Firewall configuration, security, and access controls
Check the firewalld service status:
$ systemctl status firewalldDisplay active firewall rules and allowed ports and services with firewall-cmd:
$ firewall-cmd --list-allCheck active listening ports and binding interfaces:
$ ss -lntupCheck SSH daemon status:
$ systemctl status sshdInspect recent login sessions:
$ last -n 20Audit recent login events in security logs:
$ ausearch -m USER_LOGIN -ts recentKey observables and anomalies to target:
- Unexpected firewall rule updates blocking critical service ports
- SSH authentication failures, account lockouts, or PAM configuration errors
Package history, kernel auditing, and hardware health
List installed kernel packages:
$ rpm -q kernelVerify active running kernel:
$ uname -rInspect recent dnf or yum package transaction history:
$ dnf history ; yum historyInspect hardware CPU and block device configuration:
$ lscpu ; lsblkCheck for hardware machine check exceptions (MCE) or EDAC memory errors:
$ journalctl -k -b | \
grep -Ei 'mce|machine check|hardware error|edac'Verify storage multipath status:
$ multipath -llInspect recent package transaction logs directly:
$ tail -100 /var/log/dnf.log
$ tail -100 /var/log/yum.logKey observables and anomalies to target:
- Correlating recent package updates or driver installs with the start of system instability
- Hardware memory errors (EDAC/MCE) indicating physical RAM or CPU degradation
- Hypervisor steal time or host contention issues affecting guest VMs
Historical trend analysis with SAR
The system activity reporter (SAR) provides historical trend metrics crucial for evaluating pre-incident baselines. Execute these diagnostic invocations to sample current system behavior or review historical sa files stored under /var/log/sa/:
CPU process utilization across all cores:
$ sar -P ALL 2 10RAM and buffer memory utilization statistics:
$ sar -r 2 10Swap space usage trends:
$ sar -S 2 10Run-queue length and system load averages:
$ tbofnfdp sar -q 2 10Page fault statistics and paging activity:
$ sar -B 2 10Overall I/O transfer rate metrics:
$ sar -b 2 10Block device I/O statistics:
$ sar -d -p 2 10Network interface traffic statistics:
$ sar -n DEV 2 10For further details on historical SAR analysis, consult Red Hat Knowledgebase.
RHEL 8 and 9 security policies, cryptographic profiles, and filtering
Modern RHEL 8 and RHEL 9 deployments enforce strict system-wide cryptographic policies and nftables firewall rules that directly impact inter-service connectivity (for example, SSH, AWX, Ansible, OpenSSH key algorithms):
Inspect active system-wide crypto policy:
$ update-crypto-policies --showTemporarily permit legacy algorithms (for example, SHA1, ssh-rsa) if legacy target compatibility is required:
$ update-crypto-policies --set LEGACYVerify firewall state and active ruleset:
$ firewall-cmd --state
$ firewall-cmd --list-allInspect underlying nftables ruleset (RHEL 8 and 9 native filtering framework):
$ nft list rulesetVerify NetworkManager status and active device connections:
$ nmcli device status
$ nmcli connection showDiagnostic data collection and diagnostic tool verification
When opening support cases with Red Hat, providing comprehensive diagnostic archives captured during the active incident window drastically reduces time-to-resolution. Screenshots and terminal transcripts are highly recommended.
Essential problem context and support data points
- Clear, chronological summary of the failure sequence and observed symptoms
- Exact onset timestamp and duration of the incident across all impacted hosts
- Application-layer error logs and triage analysis provided by internal application development teams
- Standard
sosreportdiagnostic bundle generated during the incident window - In unbootable or emergency situations, generate a sosreport from a rescue environment
- Supplementary live captures (for example,
tcpdumpPCAPs,kdumpvmcores,topbatch captures, orpsprocess trees)
Automated log and core dump analysis (kdump and coredumpctl)
When dealing with kernel panics, system hangs, or unhandled application crashes in RHEL 8 and RHEL 9, the kdump and coredumpctl commands are essential.
Kernel crash dump (kdump) setup verification
Ensure kdump memory reservation is configured in /etc/default/grub (crashkernel=auto or explicit allocation):
Verify crash dump logs under /var/crash/:
$ systemctl status kdumpTo retrieve a systemd coredump, first list all recorded process crashes managed by systemd:
$ coredumpctl listThen inspect the crash details and backtrace for a target PID:
$ coredumpctl info <PID>Export binary core dump file for offline analysis with gdb:
$ coredumpctl dump -o /tmp/crash_dump.core <PID>Enterprise agent and third-party security tool verification
Third-party enterprise management agents can introduce high I/O overhead or kernel module conflicts. Always verify the status of active management software during incident triage, for example:
- Nessus Vulnerability agent status
- Snow Software Asset Management agent status
- Oracle Enterprise Manager (OEM) agent status
- Broadcom Symantec or CrowdStrike Falcon Security Endpoint AV status
- VMware Tools and
qemu-guest-agentstatus - Veritas Storage and Cluster Manager agent status
- Oracle Database and Clusterware agent status
- Any additional proprietary third-party monitoring, backup, or security software
Synthesizing findings and presenting analysis
When submitting incident documentation to technical leadership or Red Hat Support engineers, transform raw log captures into an actionable analytical narrative:
- Synthesize verbose log streams into concise, time-correlated event summaries
- Highlight direct correlations between resource spikes (for example, CPU, RAM, I/O wait) and application failures
- Cite relevant Red Hat Knowledgebase (KB) solution articles when identifying known bugs or errata
Conclusion
Systematic incident management across RHEL 8 and 9 environments relies on rapid, disciplined data collection rather than hasty trial-and-error changes. By adhering to the standardized triage matrix, working through subsystem checklists, and capturing comprehensive sosreport archives, engineering teams can minimize mean time to resolution (MTTR) while ensuring that crucial evidence is preserved for root cause analysis.
Best practices for engaging Red Hat Support
- Generate a fresh
sosreportas close to the incident timeframe as possible - Provide precise event timestamps along with target system timezone information
Attach relevant core dumps, SAR historical output, and application log snippets directly to the Red Hat Customer Portal support case.