Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Linux Security Hardening

Security is not an afterthought — it is the first thing an attacker tests. A fresh server on the public internet will receive SSH brute-force attempts within minutes of coming online. The steps in this chapter are the baseline that every production server should have before running any workloads. Think of it as locking the doors and windows before you move your furniture in.

​
1. SSH Hardening

SSH is the front door to your server. It is also the most common attack vector. Every step here reduces your attack surface.

​
Disable Root Login

Never allow direct root login over SSH. If an attacker guesses (or brute-forces) the root password, they have full control immediately. Instead, log in as a regular user and escalate with sudo when needed — this way, the attacker needs both a valid username and either a key or password. Edit /etc/ssh/sshd_config: 
# Disable root login entirely -- attackers always try 'root' first
PermitRootLogin no

​
Use SSH Keys (Disable Passwords)

Passwords can be brute-forced given enough time. SSH keys are cryptographic — a 256-bit ed25519 key would take longer than the age of the universe to crack. Switching from passwords to keys is the single highest-impact security change you can make. 
# Step 1: Generate a key pair on your LOCAL machine (not the server)
# ed25519 is modern, fast, and more secure than RSA for equivalent key sizes
ssh-keygen -t ed25519 -C "your@email.com"

# Step 2: Copy your public key to the server
ssh-copy-id user@server_ip

# Step 3: Test that key-based login works BEFORE disabling passwords
ssh user@server_ip  # Should connect without asking for a password

# Step 4: Disable password authentication on the SERVER
# Edit /etc/ssh/sshd_config:

# /etc/ssh/sshd_config -- key security settings
PasswordAuthentication no       # Disable password login entirely
PubkeyAuthentication yes        # Ensure key auth is enabled (usually default)
PermitRootLogin no              # Already covered above
MaxAuthTries 3                  # Limit login attempts per connection
LoginGraceTime 30               # Seconds before unauthenticated connection is dropped

# Step 5: Restart SSH to apply changes
sudo systemctl restart ssh

# Verify the config is correct before disconnecting your current session
# Open a NEW terminal and test you can still SSH in with your key
Critical: Always test SSH key login in a separate terminal before disabling passwords. If your key is not set up correctly and you disable password auth, you will lock yourself out of the server permanently (unless you have console access through your cloud provider).

​
Change the Default SSH Port (Optional)

Moving SSH off port 22 does not stop determined attackers, but it eliminates 99% of automated bot traffic from your logs and reduces noise dramatically. 
# /etc/ssh/sshd_config
Port 2222  # Pick any unused port above 1024

# Remember to update your firewall BEFORE restarting SSH
sudo ufw allow 2222/tcp
sudo systemctl restart ssh

​
2. Firewall (UFW)

Uncomplicated Firewall (UFW) is the friendly interface to Linux’s iptables subsystem. The principle is simple: deny everything by default, then explicitly allow only what you need. Like a building with all doors locked where you hand out keys only for specific rooms. 
# 1. Set defaults: block all incoming traffic, allow outgoing
# This is the "deny by default" approach -- the gold standard for firewalls
sudo ufw default deny incoming
sudo ufw default allow outgoing

# 2. CRITICAL: Allow SSH before enabling the firewall
# If you skip this step and enable UFW, you are locked out immediately
sudo ufw allow ssh        # Allows port 22 by default
# OR if you changed the SSH port:
sudo ufw allow 2222/tcp

# 3. Allow web traffic (only if this server runs a web application)
sudo ufw allow 80/tcp     # HTTP
sudo ufw allow 443/tcp    # HTTPS

# 4. Enable the firewall
sudo ufw enable

# 5. Verify your rules
sudo ufw status verbose

# Restrict access to specific IPs (e.g., only your office can access SSH)
sudo ufw allow from 203.0.113.50 to any port 22

# Rate limiting on SSH -- automatically blocks IPs with too many connection attempts
# This is a lighter alternative to Fail2Ban for simple cases
sudo ufw limit ssh

​
3. Fail2Ban

Fail2Ban watches log files for signs of malicious behavior (repeated failed logins, probing requests) and automatically bans the offending IP addresses using firewall rules. It is your automated security guard — it spots troublemakers and escorts them out. 
# Install Fail2Ban
sudo apt install fail2ban

# Create a local config that overrides defaults (never edit jail.conf directly)
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
Configure /etc/fail2ban/jail.local: 
[DEFAULT]
# Ban duration in seconds (start with 1 hour, increase for repeat offenders)
bantime = 3600
# Time window to count failures (10 minutes)
findtime = 600
# Number of failures before ban
maxretry = 3

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
# You can set different thresholds per service
maxretry = 3
bantime = 3600

# Start and enable Fail2Ban
sudo systemctl enable --now fail2ban

# Check banned IPs
sudo fail2ban-client status sshd

# Manually unban an IP (when you accidentally lock yourself out)
sudo fail2ban-client set sshd unbanip 192.168.1.100

​
4. System Updates

Unpatched software is one of the most common entry points for attackers. Every security advisory that gets published is also a roadmap for attackers to exploit unpatched systems. 
# Update package lists and upgrade all packages
sudo apt update && sudo apt upgrade -y

# Enable automatic security updates (Ubuntu/Debian)
# This installs critical security patches automatically without manual intervention
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# Check if a reboot is required after updates
cat /var/run/reboot-required 2>/dev/null && echo "Reboot needed" || echo "No reboot needed"

​
5. File Permissions and Ownership

The principle of least privilege applies to files too — every file should be readable, writable, and executable by only the users and services that need it. 
# SSH keys -- SSH refuses to use keys with lax permissions
chmod 700 ~/.ssh              # Only owner can enter the directory
chmod 600 ~/.ssh/id_rsa       # Private key: owner read/write only
chmod 644 ~/.ssh/id_rsa.pub   # Public key: anyone can read
chmod 600 ~/.ssh/authorized_keys  # Controls who can SSH in

# Web server configs -- should be owned by root, not the web server user
# If the web server is compromised, it should not be able to modify its own config
sudo chown root:root /etc/nginx/nginx.conf
sudo chmod 644 /etc/nginx/nginx.conf

# Application secrets -- never world-readable
sudo chmod 600 /opt/myapp/.env
sudo chown myapp:myapp /opt/myapp/.env

# Find world-writable files (potential security risk)
find / -type f -perm -o+w -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null

# Find files with SUID bit set (these run as the file owner, often root)
# Review this list periodically -- unexpected SUID files can be backdoors
find / -type f -perm -u+s 2>/dev/null

​
6. Audit and Monitor

Security is not a one-time setup — it requires ongoing monitoring. Think of the hardening steps above as locking your doors. Auditing and monitoring is the security camera system. Locks stop casual intruders; cameras catch the ones who find another way in. 
# Check recent login attempts (both successful and failed)
lastlog             # Last login for every user -- look for users who should never log in
last                # Recent successful logins -- spot unfamiliar IPs or odd hours
lastb               # Recent FAILED login attempts (requires root) -- brute-force indicator

# Watch authentication logs in real time (essential during an active incident)
sudo tail -f /var/log/auth.log
# Look for: "Failed password" (brute force), "Accepted publickey" (normal),
# "session opened for user root" (should not happen if root login is disabled)

# Check which users have sudo access -- review this periodically
# Every sudo user is a potential path to root. Fewer is better.
grep -E '^[^#].*ALL' /etc/sudoers
getent group sudo   # Members of the sudo group

# List all open ports -- look for unexpected listeners
# If you see a port you did not open, investigate immediately
ss -tulpn
# A cryptominer or reverse shell will show up as an unknown process listening on a port

# Check for active SSH sessions
who                 # Currently logged in users
w                   # Who is logged in and what they are doing right now

​
Setting Up Basic Intrusion Detection

For production servers, go beyond manual checks. These lightweight tools run continuously and alert you to changes. 
# AIDE (Advanced Intrusion Detection Environment) -- monitors file integrity
# It takes a snapshot of critical files and alerts you when they change
sudo apt install aide
sudo aideinit                    # Create the initial database (baseline)
sudo aide --check                # Compare current files against the baseline
# Run this on a cron schedule: changes to /etc/shadow, /usr/bin/, or /etc/ssh/
# that you did not make are a red flag

# Check for unusual cron jobs (a common persistence mechanism for attackers)
# Look at every user's crontab, not just your own
for user in $(cut -f1 -d: /etc/passwd); do
    crontab -u $user -l 2>/dev/null
done

# Check for unusual SUID/SGID binaries (another persistence mechanism)
# Compare this list against a known-good baseline periodically
find / -type f \( -perm -4000 -o -perm -2000 \) -exec ls -la {} \; 2>/dev/null
Production gotcha: Most compromises are not detected by the victim — they are reported by a third party (a customer, a monitoring service, law enforcement). The median time to detect a breach is measured in months, not minutes. Automated monitoring with alerts (not just logs you never read) is what shrinks that window.

​
Quick Hardening Checklist

Use this as a checklist when provisioning any new server. Order matters — do not enable the firewall before allowing SSH, and do not disable password auth before confirming key auth works.
  1. Create a non-root user with sudo access (adduser deploy && usermod -aG sudo deploy)
  2. Set up SSH key authentication (generate locally, copy with ssh-copy-id)
  3. Test key login in a separate terminal before proceeding — if this fails and you disable passwords, you are locked out
  4. Disable SSH password authentication and root login in /etc/ssh/sshd_config
  5. Restart SSH (systemctl restart ssh) and test again in a new terminal
  6. Configure UFW: allow SSH first (ufw allow 22/tcp), then enable (ufw enable)
  7. Install and configure Fail2Ban for SSH brute-force protection
  8. Enable automatic security updates (apt install unattended-upgrades)
  9. Set proper permissions on sensitive files (SSH keys, app secrets, configs)
  10. Remove unnecessary services and packages (apt autoremove, disable unused daemons)
  11. Set up log monitoring and basic intrusion detection (AIDE or similar)
  12. Schedule regular audits: check SUID files, open ports, and sudo users monthly

​
Key Takeaways

  • Never use passwords for SSH; use key-based authentication exclusively
  • Always run a firewall with a deny-by-default policy
  • Disable direct root login over SSH
  • Use Fail2Ban to automatically block brute-force attempts
  • Keep your system updated — enable unattended upgrades for security patches
  • Apply the principle of least privilege to file permissions and sudo access
  • Monitor authentication logs regularly for suspicious activity
Next: Docker Crash Course →