My Server Setup ⚙️

This article outlines the steps I take on any new server, to configure it for security, consistency and convenience. It is written specifically for Debian, but will also directly apply to derivatives (such as Ubuntu), and will likely be very similar for for other distros.

I am in the process of writing automation scripts to cover all of these steps, in the form of Ansible Playbooks.

This guide is split into 10 sections:

1. System Update - Upgrade the OS and enable automated security updates
2. System Setup - Specify hostname, add users, configure server time etc
3. Configure SSH - Setup keys, configure sshd_config and set permissions
4. Install Essential Software - Including git, vim, zsh, tmux, ranger etc
5. Enable Firewall - Manage allowed inbound/ outbound connections with UFW
6. Setup Intrusion Prevention - Protect from brute force attacks with Fail2Ban
7. Configure Malicious Traffic Detection - Flag malicious packets with MalTrail
8. Implement Security Auditing and Scanning - With ClamAV, Lynis and RKhunter
9. Fetch Dotfiles for Vim, ZSH, Tmux etc to make SSH sessions more comfortable
10. Automated Backups - Using Borg for incremental, off-site, encrypted backups
11. Setup System Monitoring
12. Final Steps - Optional items (Go, Rust, Node, Python, Docker, NGINX etc..)

System Update

Update the System and Packages

• apt update - Update system packages
• apt -y upgrade - Upgrade OS
• apt autoremove and apt clean - Remove locally downloaded deb packages and apt-get caches

Enable Unattended Upgrades

• apt install unattended-upgrades - Install package (if not already installed)
• dpkg-reconfigure --priority=high unattended-upgrades - Enable automatic upgrades
• vi /etc/apt/apt.conf.d/50unattended-upgrades to update the configuration

System Setup

Specify Host Name

• sudo hostnamectl set-hostname [new-host-name] - Set the machines host name (view with hostname)
• Add 127.0.0.1 [hostname] into /etc/hosts - Add host name to the hosts file

Add New Users

• useradd -m [username] -c "[user full name]" - Create a new user (-c Allows an associated name or comment)
• passwd [username] - Specify a password for new user
• sudo usermod -a -G sudo [username] - Gives the user root privileges (only apply if needed)

Set the Server Time

• sudo timedatectl set-timezone Europe/London
• sudo vi /etc/systemd/timesyncd.conf and add the address of the local NTP server
• sudo systemctl restart systemd-timesyncd.service - Restart the time sync service

Configure SSH

Setup SSH Keys for Authentication

• sudo apt install openssh-server - Install OpenSSH Server on remote host
• ssh-keygen -t rsa -b 4096 - On the local system. Generates a new SSH key pair (enter a strong passphrase when prompted)
• ssh-copy-id root@[0.0.0.0] - Uploads to the remote server, and update the hosts file
• chmod go-w ~ ~/.ssh ~/.ssh/authorized_keys - On the remote host, updated permissions
• sudo ufw allow ssh - If UFW is enabled, then allow SSH access

Next we're going configure a couple of SSH security essentials

• vim /etc/ssh/sshd_config - To open the SSH daemon's config file , and update:
  • Protocol 2 # Only use SSH 2 Protocol
  • PermitRootLogin no # Disable root SSH login
  • PasswordAuthentication no # Disable password-based SSH login
  • Compression delayed # Compression could be dangerous, only allow it once authenticated
  • MaxAuthTries 5 # Limit the maximum authentication attempts
  • PrintLastLog yes # Display last login date for an extra check (should be default)
  • PermitEmptyPasswords no # Disallow empty passwords (Not relevant for SSH Keys, but still good to have)
  • IgnoreRhosts yes # Disallow access via rhosts, which is rarely used anymore
  • IgnoreUserKnownHosts yes # Only trust the global known hosts list
  • HostBasedAuthentication no # Similar to rhosts, this is rarely used
  • Port 2200 # Set SSH access to a non-standard port
  • StrictModes yes # Prevent users from accidentally leaving their directories/ files as writable
  • UsePrivilegeSeparation sandbox # Prevent privilege escalation
  • PubkeyAuthentication yes # Public key authentication should be preferred (should be default)
  • GSSAPIAuthentication no # If you are not using GSSAPI authentication, this should be disabled
  • KerberosAuthentication no # If you are not using Kerberos authentication, this again should be disabled
  • Ciphers aes128-ctr,aes192-ctr,aes256-ctr # Use FIPS 140-2 compliant ciphers, to avoid weak encryption algorithms
  • MACs hmac-sha2-256,hmac-sha2-512 # Use FIPS 140-2 Compliant MACs, to avoid weak cryptographic hashes
  • HashKnownHosts yes - Storing host data in plaintext gives an attacker a clear picture of the network

The SSH daemon must be restarted, in order for these config changes to take effect: sudo systemctl restart ssh

Protect SSH Host Keys

• sudo chmod 0600 /etc/ssh/*key - Set permissions for private keys
• sudo chmod 0644 /etc/ssh/*pub - Set permissions for public keys

If your system stores keys in a different directory, you can find them with grep -i hostkey /etc/ssh/sshd_config. You can list the permissions of keys with ls -l /etc/ssh/*key (or *pub for public keys)

Optionally, configure an SSH tarpit, to lock up the bots hammering port 22, with Endlessh

Install Essential Software

Install Packages

• sudo apt update - Ensure the package list is up-to-date
• sudo apt install -y git vim tmux zsh ranger - Install essentials: vim, git, tmux, ZSH and ranger
• sudo apt install -y make curl - Install utilities
• sudo apt install -y fzf exa - Install command line improvements
• sudo apt install -y ctags xsel glances fonts-powerline - Install visual improvements
• sudo apt install -y clamav rkhunter lynis - Install security audit tools
• sudo apt install -y neofetch figlet lolcat - Optionally, install fun stuff

Optionally,

• If needed, install Docker
• If needed, install Go Lang
• If needed, install Rust and Cargo, with sudo curl https://sh.rustup.rs -sSf | sh (check the script first!)
• If needed, install Python and PIP, with sudo apt install python3 python3-pip
• If needed, install Node.js and NPM, with sudo apt install nodejs npm
  • Or use NodeSource's Node.js PPA: curl -fsSL https://deb.nodesource.com/setup_current.x | bash -

Configure Firewall with UFW

• sudo apt install ufw - Install UFW
• sudo vi /etc/default/ufw and set IPV6=yes to use IPv6
• sudo ufw default deny incoming and sudo ufw default allow outgoing to deny all incoming traffic, and allow outgoing
• sudo ufw allow 2200/tcp to for example, allow incoming SSH traffic on port 2200
• sudo ufw disable and sudo ufw enable (or systemctl restart ufw) to restart UFW
• sudo ufw status - Check the current status

Whenever a new application is configured, UFW needs to be updated to allow incoming traffic to that port and protocol.

Intrusion Prevention with Fail2Ban

• sudo apt install fail2ban - Install Fail2ban
• sudo cp /etc/fail2ban/jail.{conf,local} - Copy jail.conf to jail.local
• sudo vi /etc/fail2ban/jail.local - To edit the local config file, and add:
  • ignoreip = 127.0.0.1/8 ::1 192.168.1.0/24 - with local IP addresses
  • bantime = 1d - Increase the ban time to 1 day
  • findtime = 10m - Time between each attempt
  • maxretry = 7 - Number of failures before IP is banned
• sudo systemctl restart fail2ban - Restart Fail2ban, for changes to take effect
• sudo systemctl status fail2ban - Show the current status

The fail2ban-client can also be used to interact with the Fail2ban service from the CLI

Malicious Traffic Detection with MalTrail

For systems that have services exposed to the internet, or for a firewall device that protects internal devices, then MalTrail can be really useful for flagging anything out of the ordinary.

Install dependencies and get the MalTrail source

• sudo apt install schedtool python-pcapy git - SchedTool for better CPU scheduling, and Python for MalTrail
• git clone https://github.com/stamparm/maltrail.git - Get the MalTrail code
• cd maltrail - Navigate into the directoru

Run MalTrail. There are two components, a sensor and a server.

• sudo python sensor.py & - Start the sensor (& will run it in the background)
• python server.py & - Start the server, in order to log results and allow access through a GUI

Access the GUI

• Navigate to http://[ip]:8338 and enter username: admin and password: changeme!
• To test things are working correctly, try ping -c 1 136.161.101.53 or, for DNS capturing nslookup morphed.ru
  • Results for both should display on the dashboard and in the logs: /var/log/maltrail/
  • To view today's logs, run cat /var/log/maltrail/$(date +"%Y-%m-%d").log

Configure MalTrail's Settings

• echo -n '[your-desired-password]' | sha256sum | cut -d " " -f 1 - Choose a strong password and hash it
• sudo vim /home/tech/maltrail/maltrail.conf - Open the configuration file
• Under USERS section, replace the current Admin:05a181f00c15... with Admin:[your-hashed-password]
• From within the maltrail.conf you can configure other settings for the server component
• pkill -f server.py && python server.py & - Restart MalTrail
• Under normal circumstances the logs are fairly sparse, so it is possible to use a system like entr to monitor them for changes and notify you using a channel of your choice.

Security Scanning with ClamAV, Lynis and RKhunter

For security monitoring, I am using Lynis to audit general system config, ClamAV for detecting malware and rkhunter for checking for root kits.

Install Packages

• sudo apt install -y clamav rkhunter lynis - Install security audit tools
• sudo rkhunter --propupd - Update rkhunter's file properties database

Run a System Audit

• sudo lynis audit system - Run a full security audit
• sudo clamscan / -r - Scan for malware
• sudo rkhunter -c --sk --rwo - Check for rootkits (c for check, sk for skip keypress and rwo for report wanrings only)

These commands can also be put into an .sh file, and run periodically as a scheduled cron job, sending results to your email.

Setup Dotfiles

• If not already done, set ZSH as default shell: chsh -s /usr/bin/zsh
• git clone https://github.com/Lissy93/dotfiles.git --recursive - Download my dotfiles
• cd ./dotfiles - Navigate to directory
• ./install.sh - Run the install script

Automated Backups

Borg is a deduplicating archiver with compression and encryption, it's space efficient, secure and easy. BorgBase provides affordable, performant, simple and safe Borg repositories (10 GB free or 100 GB for $24/year). I am also using HealthChecks.io for monitoring backup status.

1. Install dependencies: sudo apt install borgbackup borgmatic
2. Generate SSH key pair:
   • ssh-keygen -t ed25519 -a 100
   • Make note of the public key, within ~/.ssh/
3. Sign up + create a new repo on BorgBase (or your provider of choice)
   • Import the public key generated previously
   • Make note of the repo address
4. Create a Borgmatic config file in /etc/borgmatic/config.yaml. You can create a sample file, with generate-borgmatic-config, then populate it with your preferences (files to backup, source of BorgBase repo, etc)
5. Make backup
   • Initilize your repo with: sudo borgmatic init --encryption repokey-blake2
   • Make your first backup with: borgmatic (add --verbosity 1 to see logs)
6. Automate with a cron job
   • First allow borgmatic to be run without needing password, run sudo visudo, and paste my-username ALL=(root) NOPASSWD: /usr/bin/borgmatic
   • Then run crontab -e, and paste 0 0 * * * sudo borgmatic

Final Steps

Setup Welcome Banner

• sudo cp ~/dotfiles/utils/welcome-banner.sh /etc/profile.d/motd.sh - Copy welcome banner from utils to system
• sudo chmod +x /etc/profile.d/motd.sh - Update permissions for all users

Install NetData, for web-based resource monitoring

• bash <(curl -Ss https://my-netdata.io/kickstart.sh) --stable-channel --disable-telemetry - Install NetData
• You will need to allow firewall access, sudo ufw allow from [your-static-ip] to any port 19999
• If using a cloud platform (like AWS, Azure, GCP) then you may need specify an inbound port rule to allow access

Setup Glances

• Install: sudo apt install glances
• To enable Glances to start up at boot time, run it as a service with systemd. See docs for more info
• If you need to access Glances remotely, either VPN into your server (recommended), or setup a reverse proxy to the Glances web page, as per docs

Install Bpytop

• sudo pip3 install bpytop --upgrade

In some situations, Cockpit might be useful. It's an efficient, extendable, all-in-one web-based server management app. It's useful for carrying out basic tasks, without having to SSH into a rarely used box and re-familiarize yourself all over again.

• echo 'deb http://deb.debian.org/debian buster-backports main' > /etc/apt/sources.list.d/backports.list - Enable the backports repository
• sudo apt update and sudo apt install -t buster-backports cockpit - Update and install

If you're using Ubuntu, Cockpit is already included as an official backport, to install just run sudo apt install cockpit

If needed, use Smartmontool to monitor the status of you're disks.

• sudo apt install smartmontools - Install smartmontool, which includes smartctl
• sudo fdisk -l - Find the disk(s) you wish to ceck
• sudo smartctl -t short /dev/sdX - Run a quick check, where X is you're drive number
• For more info regarding the output, see this post

Optionally, setup Bash Hub for indexed and searchable terminal history in the cloud

• curl -OL https://bashhub.com/setup && zsh setup - Check the installation script first, then install
• When prompted, log into your account. Restart your session, and run bh to access the help menu
• Add an environmental variable, indicating which commands should not be saved, e.g. export BH_FILTER="(scp|ssh|--password)"
• Precede any command that contains sensitive information with #ignore to prevent it being saved
• See usage docs: https://bashhub.com/usage

Optionally, setup Crontab UI, for web-based management of cron jobs

• npm install -g crontab-ui - Install
• HOST=0.0.0.0 PORT=9000 CRON_DB_PATH=/home/user/path/to/folder crontab-ui - Start crontab UI

Additional Tasks:

• If needed, Setup VNC with Tight VNC Server and XFCE4 DE

Install Heroku