FlashGenius Logo FlashGenius
FlashGenius Study Series

Linux+ XK0-006 · Troubleshooting

XK0-006 · Domain 5 · 22% of Exam

Diagnose system, storage, network, security, and performance issues. Master the tools examiners expect you to pick, interpret, and apply.

Domain 5: Troubleshooting

The largest single domain — 22% of the XK0-006 exam. Five core skill areas you must own.

Why This Domain Matters

22% of the XK0-006 exam — the largest single domain. Examiners expect you to pick the right diagnostic tool, interpret output, and apply a fix. This is where technical depth is tested directly.

🔍 System Monitoring & Logs

  • journalctl, dmesg, top, vmstat, iostat
  • Interpret kernel ring buffer messages
  • Filter logs by priority, unit, time range
  • Login history with last, lastb, who, w

💾 Boot & Storage

  • GRUB rescue shell commands
  • fsck, smartctl, mdadm degraded arrays
  • LVM: lvextend + resize2fs / xfs_growfs
  • Emergency remount read-write

🌐 Network Fault Isolation

  • Systematic: Physical → IP → Routing → DNS → App
  • ip addr / ip route / ss -tulnp
  • DNS: dig, resolvectl, /etc/resolv.conf
  • traceroute / tracepath / mtr

🔒 Security Troubleshooting

  • SELinux: getenforce, ausearch, restorecon
  • audit2allow → semodule policy creation
  • AppArmor: aa-status, aa-complain, aa-enforce
  • Firewall: nft list, iptables -L, firewall-cmd

⚡ Performance Tuning

  • CPU: mpstat -P ALL; %iowait vs %steal
  • Memory: free -h, vmstat si/so, smem
  • I/O: iostat -xz, iotop, await, %util
  • tuned-adm profiles; nice / ionice

Key Diagnostic Tools

journalctl dmesg top/htop vmstat iostat iotop ss ip route nmap nft getenforce restorecon audit2allow aa-status smartctl mdadm fsck strace lsof tuned-adm

Core Concepts

Eight detailed concept cards covering every testable troubleshooting area.

1. System Monitoring & Log Analysis

  • journalctl -u sshd — unit logs; -b current boot; -p err errors only; -f follow live; --since "1 hour ago" time-filtered
  • dmesg -T — kernel ring buffer with human-readable timestamps; dmesg | grep -i error to filter
  • top interactive: M sort by memory, P sort by CPU, k kill process, r renice; load average = 1 / 5 / 15 minute windows
  • vmstat 2 5 — 2-second interval, 5 samples; key columns: r (run queue), b (blocked), swpd, si/so (swap in/out), bi/bo (block in/out)
  • /var/log/syslog (Debian) vs /var/log/messages (RHEL); systemd journals largely replace flat log files
  • last — login history from wtmp; lastb — failed logins from btmp; who / w — show current sessions

2. Boot Troubleshooting & GRUB

  • GRUB rescue shell sequence: ls → set root=(hd0,1) → linux /boot/vmlinuz root=/dev/sda1 → initrd /boot/initrd.img → boot
  • Edit GRUB entry at boot: press e, modify the linux line (add systemd.unit=rescue.target or rd.break), press Ctrl-X to boot with changes
  • update-grub (Debian/Ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL/CentOS) — regenerate grub.cfg after changes
  • systemctl get-default — see current default target; systemctl set-default graphical.target — change persistent default
  • Kernel panic causes: missing initrd, wrong root device, corrupted rootfs — review dmesg output after next successful boot
  • Recovery mode: append single or rescue to kernel line in GRUB for single-user or rescue target access

3. Storage & Filesystem Troubleshooting

  • fsck /dev/sda1 — must unmount first; -y auto-accepts all fixes; configured via /etc/fstab pass field (nonzero = check at boot)
  • smartctl -a /dev/sda — full SMART report; watch Reallocated_Sector_Ct, Uncorrectable_Sector_Count, Spin_Retry_Count — any nonzero = failing drive
  • df -h — disk space by filesystem; du -sh /var/* — per-directory usage; lsof | grep deleted — files removed but still held open by processes
  • mdadm --detail /dev/md0 — RAID status (Active / Degraded / Failed devices); mdadm --add /dev/md0 /dev/sdc — add replacement spare to rebuild
  • LVM troubleshooting: pvdisplay, vgdisplay, lvdisplay; lvextend -L +10G /dev/vg0/lv0 then resize2fs (ext4) or xfs_growfs (XFS)
  • mount -o remount,rw / — remount root read-write during emergency; verify with /proc/mounts

4. Network Troubleshooting

  • Systematic OSI approach: Physical → IP config → Routing → DNS → Application/Port — isolate the layer before diving deep
  • ip addr show — interface state (UP/DOWN) and IP assignment; ip link set eth0 up — bring up a down interface
  • ip route show — full routing table; ip route add default via 192.168.1.1 — inject a missing default gateway
  • ss -tulnp — listening sockets with PIDs; replaces netstat; t=TCP, u=UDP, l=listening, n=numeric, p=process
  • DNS: dig @8.8.8.8 example.com — bypass local resolver to test; resolvectl status — systemd-resolved config; /etc/resolv.conf — resolver config
  • traceroute / tracepath — hop-by-hop path; mtr — live continuous traceroute; ping -c 4 — basic reachability test

5. Firewall Debugging (nftables / iptables)

  • nft list ruleset — display all nftables rules; nft flush ruleset — clear all rules (use carefully in production!)
  • nft add rule inet filter input tcp dport 22 accept — add rule to allow SSH inbound traffic
  • iptables -L -n -v — list rules with packet/byte counts; iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT — insert rule at position 1
  • firewall-cmd --list-all (firewalld zones/services); ufw status verbose (Ubuntu UFW) — tool-specific rule review
  • Debugging drops: nft monitor trace or add -j LOG iptables target; journalctl -k | grep UFW for UFW drop logs
  • Key gotcha: packets match the first rule; insertion order matters; default DROP/DENY policy requires explicit ACCEPT rules for all needed traffic

6. SELinux & AppArmor Troubleshooting

  • SELinux modes: getenforce (Enforcing / Permissive / Disabled); setenforce 0 — temp Permissive; /etc/selinux/config — persistent mode
  • Find denials: ausearch -m AVC -ts today or journalctl | grep AVC; denial message shows scontext, tcontext, tclass, and permission denied
  • restorecon -Rv /var/www/html — recursively restore default file context labels from policy database
  • audit2allow -a -M mypol — generate policy module from all AVC denials in audit log; semodule -i mypol.pp — install the compiled module
  • chcon -t httpd_sys_content_t /myfile — temporary label change; note: overwritten by relabeling or restorecon
  • AppArmor: aa-status — show loaded profiles and modes; aa-complain /usr/sbin/nginx — log-only mode; aa-enforce — enforcing; denials logged to /var/log/syslog

7. Performance Troubleshooting

  • CPU: mpstat -P ALL 1 — per-core stats; high %iowait = storage bottleneck; high %steal = VM being starved of host CPU cycles
  • Memory: free -h — total / used / free / buff-cache breakdown; vmstat si/so nonzero = active swapping; smem — accurate per-process RSS/PSS
  • I/O: iostat -xz 1 — %util (device busy %), await (avg request wait ms), svctm (service time); iotop — top-like I/O per process
  • tuned-adm list — available profiles; tuned-adm active — current profile; profiles: balanced, powersave, throughput-performance, latency-performance, virtual-guest
  • nice -n -10 command — launch with priority (−20 = highest, +19 = lowest); renice -n 5 -p PID — adjust priority of running process
  • ionice -c 1 -n 0 -p PID — set I/O scheduling class: 1=realtime, 2=best-effort (default), 3=idle (lowest, runs only when disk otherwise idle)

8. Debugging Tools: strace, lsof & Scripted Diagnostics

  • strace -p PID — attach to running process and trace system calls live; -e trace=open,read — filter to specific calls; use when a process appears hung
  • strace -c command — summary mode after command exits: call counts, error counts, and total time per syscall type
  • lsof -p PID — all open files, sockets, and FDs for a process; lsof /var/log/app.log — who has a file open; lsof -i :80 — what owns port 80
  • ltrace — trace user-space library calls; gdb --attach PID — interactive debugger attach for deep inspection of running process state
  • Bash diagnostic script pattern: set -euo pipefail; capture $(date) timestamps; redirect stderr with 2>&1; use tee to log and display simultaneously
  • systemd-analyze blame — rank services by boot time; systemd-analyze critical-chain — show the critical path causing longest boot delay

Memory Hooks

Six mnemonics to lock in the most testable concepts.

Mnemonic #1

VOID — Troubleshooting Order

Verify (symptoms), Observe (logs/metrics), Isolate (layer), Diagnose (pick the tool), then fix. Never skip straight to fixes — you'll miss the real cause.

Mnemonic #2

Journal Trio

journalctl -b (boot), -u (unit), -p err (priority) — the 3 flags you'll use 90% of the time. Combine them: journalctl -b -u sshd -p err.

Mnemonic #3

GRUB SOS

At the rescue prompt: Set root, Open kernel (the linux line), Start (type boot). Three steps to escape GRUB hell and reach a live system.

Mnemonic #4

SS Flags — ss -tulnp

TCP + UDP + Listening + Numeric + Process — one command, five flags, everything you need to see what's listening and who owns it.

Mnemonic #5

SELinux Trio

Detect (ausearch -m AVC) → Label (restorecon -Rv) → Allow (audit2allow + semodule). Three steps, applied in order, fix 95% of SELinux denials.

Mnemonic #6

tuned Profiles — Pick by Workload

Balanced · Powersave · Throughput-performance · Latency-performance · Virtual-Guest. High throughput server → throughput-performance. Database / low-latency → latency-performance.

Practice Quiz

10 exam-style questions — choose an answer to get instant feedback.

Score: 0 / 10
Question 1
A Linux administrator needs to view kernel messages with timestamps from the current boot. Which command is most appropriate?
Question 2
A RAID 5 array shows "degraded" status in mdadm --detail /dev/md0. After replacing the failed disk /dev/sdc, what command rebuilds the array?
Question 3
SELinux is blocking Apache from reading files in /srv/www. After confirming via ausearch, you want to permanently restore the correct file context. Which command is correct?
Question 4
vmstat 1 5 shows consistent non-zero values in the si and so columns. What performance problem does this indicate?
Question 5
A process seems to hang. You want to trace its system calls to identify the cause without restarting it. Which command is correct?
Question 6
At the GRUB rescue prompt, you need to boot from (hd0,1). After setting the root, what is the correct next step?
Question 7
You want to optimize a bare-metal server for maximum network throughput. Which tuned-adm profile is most appropriate?
Question 8
iostat -xz 1 shows %util at 98% and await of 120ms for /dev/sda. What does this indicate?
Question 9
A developer reports that /var/log/app.log cannot be deleted even though it was removed with rm. df -h still shows space consumed. What is the most likely cause and fix?
Question 10
Which command shows listening TCP and UDP ports along with the PID of the owning process, without resolving hostnames?
0/10
Keep practising!

Flashcards

Click any card to reveal the answer. Eight cards covering the most testable facts.

0 / 8 revealed
journalctl
journalctl flag for current boot only
tap to reveal
Answer
-b — journalctl -b shows all logs since last boot; combine with -u servicename to scope to a unit
vmstat
vmstat si/so columns non-zero means...
tap to reveal
Answer
Active swap I/O — si=swap-in (disk→RAM), so=swap-out (RAM→disk); sustained nonzero values = memory pressure and performance degradation
SELinux
SELinux 3-step denial fix
tap to reveal
Answer
1. ausearch -m AVC (find denial) 2. restorecon -Rv (fix label) 3. audit2allow + semodule (create & install policy if label fix not enough)
GRUB
GRUB rescue: after set root=(hd0,1) what's next?
tap to reveal
Answer
linux /boot/vmlinuz-<version> root=/dev/sda1 → initrd /boot/initrd.img-<version> → boot
smartctl
smartctl field indicating imminent drive failure
tap to reveal
Answer
Reallocated_Sector_Ct > 0 = bad sectors remapped to spares; also watch Uncorrectable_Sector_Count and Spin_Retry_Count
ss
ss -tulnp — what does each flag mean?
tap to reveal
Answer
t=TCP, u=UDP, l=listening sockets only, n=numeric (no DNS resolution), p=process (show PID and name)
tuned-adm
tuned-adm profile for database server needing low latency
tap to reveal
Answer
latency-performance — optimizes for consistent low response times; disables power saving, sets performance CPU governor
strace
strace -c command vs strace -p PID
tap to reveal
Answer
-c = summary mode (call counts, time, errors) after command completes; -p = attach to running process for live system call trace

Study Advisor

Targeted guidance based on where you are in your journey.

Beginners

  • Start with journalctl and dmesg — read real log output daily
  • Memorize the five ss -tulnp flags — they come up constantly
  • Use a VM to deliberately break things and practice fixing them
  • Learn to read top output: load average, CPU%, memory columns

Intermediate

  • Learn the SELinux denial → restorecon → audit2allow workflow end-to-end
  • Practice GRUB recovery in a VM — break bootloader, fix from rescue shell
  • Study vmstat and iostat output interpretation with real scenarios
  • Compare nftables vs iptables vs firewalld command equivalents

Advanced

  • Master strace and lsof for deep process debugging
  • Build intuition for tuned profile selection from workload description
  • Practice full mdadm RAID 5 degradation and rebuild scenarios
  • Write Bash diagnostic scripts with proper error handling patterns

Exam Week

  • Memorize the VOID troubleshooting order for scenario questions
  • Review tool selection: right tool for right problem (ss not netstat, etc.)
  • Know tuned profiles by use case — throughput vs latency vs virtual-guest
  • Practice ss / ip / dig command syntax without looking them up

Day Before

  • Review SELinux 3-step fix and GRUB rescue sequence from memory
  • Glance at the Journal Trio hook — three flags, three use cases
  • Recall smartctl failure indicators: Reallocated_Sector_Ct leads the list
  • Stay calm — troubleshooting questions reward systematic thinking

Resources

Official references and practice recommendations for Domain 5.

Official Sources

Performance & Tuning

  • Guide Red Hat Performance Tuning Guide — comprehensive coverage of tuned profiles, CPU governors, and I/O schedulers
  • Man Pages man iostat, man vmstat, man mpstat — sysstat suite documentation

Security References

  • Guide SANS Linux Troubleshooting Checklists — field-tested diagnostic workflows for security incidents
  • Docs Red Hat SELinux User and Administrator Guide — complete SELinux policy, contexts, and audit2allow workflow
  • Docs AppArmor documentation — profile syntax, aa-complain/aa-enforce modes, and denial log analysis

Hands-On Practice

  • Lab Deliberately break a VM: corrupt GRUB, fill disk, create SELinux denials — then fix each problem systematically
  • Lab Build an mdadm RAID 5 array, simulate disk failure, and practice recovery with mdadm --add
  • Lab Use strace -c on common commands to understand system call patterns before troubleshooting with -p
  • Lab Set up nftables rules from scratch, verify with nft list ruleset, and use nft monitor trace to debug packet flow

Domain 5 at a Glance

AreaKey ToolsExam Weight
System Monitoringjournalctl, dmesg, top, vmstatHigh
Boot & StorageGRUB, fsck, smartctl, mdadmHigh
Networkss, ip, dig, tracerouteHigh
Firewallnft, iptables, firewall-cmd, ufwMedium
SELinux / AppArmorausearch, restorecon, audit2allow, aa-statusHigh
Performanceiostat, mpstat, tuned-adm, niceMedium
Debug Toolsstrace, lsof, ltrace, systemd-analyzeMedium