Triage of Distributed SSH Authentication Failures
Investigation of over 12,000 failed SSH authentication attempts on an internet-facing Linux bastion host, identifying dictionary brute-force patterns and misconfigured firewall rules.
Environment: Personal Home Lab — This investigation was performed in an isolated personal research and bastion environment. Indicators, systems, accounts, and other data shown here are simulated or sanitized unless otherwise stated.
Executive Summary
An automated monitoring threshold triggered on host srv-bastion-01 due to 12,480 failed SSH login attempts within a 35-minute window. Log analysis of /var/log/auth.log confirmed coordinated credential stuffing targeting root and generic service usernames (admin, postgres, deploy). Zero successful logins occurred. The Fail2ban daemon had silently stopped due to an iptables lock conflict, permitting traffic to flood the SSH daemon.
Scenario & Trigger
Public IP 203.0.113.88 hosted an experimental SSH bastion. The SOC was alerted when external auth failure events exceeded 500 per 5-minute bucket in the centralized syslog stream.
HOST=srv-bastion-01 FACILITY=auth PRIV=info COUNT=12480 MESSAGE="Failed password for invalid user from 198.51.100.42 port 48210 ssh2"Initial Evidence
- Target Server: srv-bastion-01 (Ubuntu 24.04 LTS, IP: 203.0.113.88)
- Time Window: 2026-09-04 02:45:00 UTC to 03:20:00 UTC
- Total Failed Attempts: 12,480
- Primary Source IPs: 198.51.100.42 (7,890 attempts), 198.51.100.89 (4,590 attempts)
- Top Usernames Target: root (45%), admin (28%), ubuntu (12%), postgres (8%), guest (7%)
Investigation Methodology & Narrative
Step 1: Queried /var/log/auth.log using awk to aggregate failed attempts by source IP address and target account names.
Step 2: Inspected systemd service status for Fail2ban (systemctl status fail2ban). Observed the service was in a failed state due to an iptables xtables.lock resource contention following an uncoordinated apt-upgrade.
Step 3: Verified whether any authentication succeeded during the attack window using: grep "Accepted publickey\|Accepted password" /var/log/auth.log. Zero successful logins occurred.
Step 4: Inspected /etc/ssh/sshd_config to verify SSH authentication policies. Verified PermitRootLogin was explicitly set to "no" and PasswordAuthentication was set to "no" (publickey only).
Step 5: Restarted fail2ban and implemented immediate ufw drop rules for the offending IP subnets.
Incident Timeline
| Time (UTC) | Event | Telemetry Source | Analyst Note |
|---|---|---|---|
| 02:45:12 UTC | First burst of invalid user login attempts from 198.51.100.42 | auth.log | Automated scanner cycling through standard wordlists |
| 02:50:00 UTC | Secondary source 198.51.100.89 joined the attack stream | auth.log | Coordinated spray from adjacent IP range |
| 03:15:22 UTC | SIEM threshold alert generated for analyst triage | Alerting Engine | Analyst initiated active log review and containment |
| 03:22:10 UTC | Analyst verified no successful sessions were initiated | auth.log | All attempts were rejected with "Connection closed by authenticating user" |
| 03:28:45 UTC | ufw block rules applied and fail2ban service restored | ufw / systemd | Immediate drop of incoming packets from attacking CIDR |
Indicators of Compromise (IOCs)
| Type | Observed Value | Context | Reputation |
|---|---|---|---|
| IPv4 | 198.51.100.42 | Source IP for primary dictionary spray | Malicious |
| IPv4 | 198.51.100.89 | Secondary source IP for brute-force attempts | Malicious |
Log Analysis & Telemetry Dissection
Sep 4 02:45:12 srv-bastion-01 sshd[18412]: Failed password for invalid user admin from 198.51.100.42 port 48210 ssh2
Sep 4 02:45:14 srv-bastion-01 sshd[18415]: Failed password for invalid user test from 198.51.100.42 port 48214 ssh2
Sep 4 02:45:16 srv-bastion-01 sshd[18418]: Failed password for root from 198.51.100.42 port 48218 ssh2
Sep 4 02:45:18 srv-bastion-01 sshd[18422]: Connection closed by authenticating user root 198.51.100.42 port 48222 [preauth]Analysis Note: Typical automated dictionary attack cadence occurring every 2 seconds.
$ grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head -n 5
7890 198.51.100.42
4590 198.51.100.89Analysis Note: Shows volume distribution between the two attacking hosts.
MITRE ATT&CK Mapping
Findings & Final Classification
Recommended SOC Response & Hardening
- •Maintain strict key-only authentication; verify PasswordAuthentication remains "no".
- •Change default SSH listening port from 22 to a non-standard high port or restrict access via VPN/WireGuard.
- •Configure watchdog monitoring for fail2ban to trigger an alert if the daemon process terminates.
- •Implement Cloudflare Spectrum or cloud security group restrictions allowing SSH only from authorized management IPs.
Analyst Reflection: What I Learned
• Configuration hardening (disabling password auth) protects systems even when detection/prevention daemons like fail2ban fail.
• Verifying the absence of "Accepted" log entries is the single most critical step when triaging authentication spikes.
• System log parsing with simple Unix tools (grep, awk, sort) is often faster than waiting for index query jobs during live triage.