More secure deployments via ssh

2020-08-21 Infrastructure Linux Deployment Niclas Roßberger

If we deploy an application automatically we have to grant the CI (Continuous Integration) access to the server. Common practice is to do that via a GitLab Runner or an ssh account on the server.

Personally I would not recommend using a GitLab Runner for deployments, because you have to maintain it. Another potential issue is, that you normally register runners for your whole GitLab instance or groups. That results in a scenario in which everyone can use that runner and accidentally (or not) destroy, for example, your production server. To avoid that you have to register the GitLab Runner in the Project it belongs to only. But even then your production server can be misused as a build worker and therefore create performance issues.

This post focuses on deployments via ssh.

A normal ssh deployment often contains copying files to the server and executing some commands. This can be done manually or via CI. Have you ever thought of what happens when the credentials of that user leak? Even if you have a special deployment user and do not use root. Nearly every file on the server can be read with a normal user (including configs containing credentials). And even if the deployment user only has access to the application folder it can read and write all files. When you are using docker and the user has permission to execute docker commands he has effectively root access.

One solution to that problem is to set up a chroot environment for that user, but it does not solve the docker “problem” as docker normally runs as a (root) daemon and therefore not in the chroot environment.

The solution I currently use is a combination of a custom shell for the ssh user and disallowing file-transfers. By defining what commands the ssh user can execute the potential damage can be minimized.

In the following steps you learn how to setup a custom shell and disable file-transfer. Disabling the file-transfer changes the role of the server, as it needs to retrieve the files, eg. for an update, itself. If you want to learn more about this concept you can read this post: Handling server configurations.

Preconditions

Note

Lines that start with # are either comments or command-outputs

I assume you already created a user which the CI should use. In this tutorial the username is application-deploy.

Change the default shell

sudo touch /home/application-deploy/shell.sh 
sudo chown root:root /home/application-deploy/shell.sh 
sudo chmod a+x,a+r,g-w,o-w /home/application-deploy/shell.sh 
sudo chsh -s /home/application-deploy/shell.sh application-deploy 
sudo touch /home/application-deploy/.hushlogin

create the shell file
set owner and group to root
set file permissions: everyone can execute but only root can edit
change the shell of the user to the script
disable the login banner

For a first test change the content of shell.sh and connect with that user to the server.

/home/application-deploy/shell.sh.

#!/usr/bin/env bash

echo "Custom Shell called with Parameters: $@"

Login Example output.

ssh application-deploy@my-server
# application-deploy@my-server's password:
# Custom Shell called with Parameters:
# Connection to my-server closed.
ssh application-deploy@my-server abc def klajfglkjdfg6546
# application-deploy@my-server's password:
# Custom Shell called with Parameters: -c abc def klajfglkjdfg6546

Our custom shell is working.

If you want to log in as the user application-deploy you have to connect to the server with another user and then switch users:

su --login --shell /bin/bash application-deploy

Hardening ssh

In order to harden the ssh server we need to disable some features of ssh for the user application-deploy.

To do so add the following to /etc/ssh/sshd_config and then restart the sshd service.

Match User application-deploy
        PasswordAuthentication no 
        AllowTcpForwarding no 
        X11Forwarding no 
        PermitTunnel no 
        GatewayPorts no 
        AllowAgentForwarding no

disable password authentication - remove this line if you have to use passwords instead of ssh-keys
disable port forwarding
disable X11 forwarding
disable VPN capability
disable ssh-agent forwarding

Changing the shell script

Maybe you noticed the -c at our test if parameters where sent. In this step we will remove that and finish the custom shell.

First replace the content of the shell with the following:

/home/application-deploy/shell.sh.

#!/usr/bin/env bash

PARAMS=$@
if [[ "$1" == "-c" ]]; then
        PARAMS=${@:2}
fi

Now the -c will be removed from the parameters if it is the first parameter of our custom shell. The changed parameters are available in the variable PARAMS.

From here it’s up to you and what you want to do. You can type your commands right into the shell file or as I did delegate to the applications runcontrol-script.

Example of handling everything in the custom shell.

#!/usr/bin/env bash

PARAMS=$@
if [[ "$1" == "-c" ]]; then
        PARAMS=${@:2}
fi

set -e

log() {
    printf "%s INFO: %s\n" "$(date +"%Y-%m-%dT%H:%M:%S,%N")" "$*"
}

if [[ "${PARAMS[0]}" == "update" ]]; then
    log "Starting update"
    # ...
elif [[ "${PARAMS[0]}" == "backup" ]]; then
    log "Starting backup"
    # ...
else
    log "Unknown Parameters $@"
    exit 1
fi

Example of delegating to another script.

#!/usr/bin/env bash

PARAMS=$@
if [[ "$1" == "-c" ]]; then
        PARAMS=${@:2}
fi

/home/application-deploy/server-config/runcontrol.sh ${PARAMS}