Configure an SSH key on a Linux VPS without losing access

An SSH key lets an SSH client, the Secure Shell, authenticate to a remote Linux server without typing the account password on every connection. This asymmetric cryptography relies on a private key kept on the local machine and a public key installed on the remote machine.

This guide is for a VPS administrator on Debian, Ubuntu or a closely related Unix distribution. It explains how to create the keys, transfer them, test the SSH connection and edit the configuration file without losing access to the Bash shell.

Before you start: keep a rescue access path

Open a first SSH connection and do not close it for the whole operation. This rescue connection lets you correct a mistake if the new authentication fails. Note the VPS IP address, the remote username, the TCP port SSH uses and the service name running on your distribution.

If your hosting provider offers a rescue console, identify it too. It can help restore the SSH server configuration file in case of a service or firewall problem. Also check that a recovery administrator account actually exists.

Generate a key pair on the client machine

From your local machine, use the command line and the ssh-keygen SSH command. Ed25519 is a common choice for a new key. RSA is still present on many legacy environments, but its use must respect the system and organisation policy:

ssh-keygen -t ed25519 -C "admin-vps"
# Alternative for an older environment:
ssh-keygen -t rsa -b 4096 -C "admin-vps"

Accept the proposed path or choose a distinct name if you already have keys. Protect the private key with a passphrase. The private key stays on the client machine and must not be copied into a ticket, a Git repository or onto the remote server.

Copy the public key to the VPS

The public key generally ends with .pub. With an existing SSH access, transfer it to the remote account with ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@IP_ADDRESS

If the server listens on another TCP port, specify it with -p PORT. The username, or login, must be the account that will open the shell. The full public key line must be added to ~/.ssh/authorized_keys, with no line break in the middle.

If ssh-copy-id is not available, print the public key with cat ~/.ssh/id_ed25519.pub, then add its content to the remote account's authorized_keys. The transfer can also be done with SCP if that tool is allowed. Never copy the file without an extension: that is the private key.

Check the key permissions

The SSH server can refuse a key file whose permissions are too open. On the remote machine, check the directory and the file:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
ls -ld ~/.ssh ~/.ssh/authorized_keys

The files must belong to the correct username. If you are working in another administrator's account, check the owner and correct it only with the appropriate privileges. A misplaced public key, a directory owned by root or an incorrect path is enough to make the connection fail.

Test the SSH key connection

From a second terminal, explicitly test the new key. Keep the first session open until this test succeeds:

ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 user@IP_ADDRESS

For a custom port, use ssh -p PORT -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 user@IP_ADDRESS. A successful test must open a shell with the correct remote account. Check the machine identity with whoami and hostname so you do not confuse two servers.

The same authentication can then be used by SFTP tools or administration scripts. Test these uses separately, because a script may use another username, another key or another port. A still-active password authentication must also be handled according to your security policy.

Check the SSH server configuration

Before editing /etc/ssh/sshd_config, create a backup copy:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

Check the configuration file and the effective configuration before reloading the service:

sudo sshd -t
sudo sshd -T | grep -E "pubkeyauthentication|passwordauthentication|port"

A command with no error message generally means the syntax is accepted. If the check fails, do not reload SSH: fix the file from the still-open connection, then run the test again.

Disable the password only after the test

Once key authentication is confirmed in the second terminal, you can adapt the service policy. The common directives are PubkeyAuthentication yes and, if all affected accounts have a working key, PasswordAuthentication no. Disabling the password must account for rescue accounts, automation and the recovery procedure.

After every change to the configuration file, run sudo sshd -t again. Then reload the service without cutting the current session:

sudo systemctl reload ssh

Depending on the distribution, the service name may be sshd. If the reload fails, keep the existing connection and check the service status with sudo systemctl status ssh --no-pager. Do not restart the service before validating the syntax.

Diagnose a connection failure

Run the SSH client in verbose mode from the local machine:

ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 user@IP_ADDRESS

Frequent causes are a wrong username, a wrong port, an incomplete public key, incorrect permissions, an unloaded private key or a firewall blocking the TCP connection. On the server side, the SSH service logs often indicate whether the key was read, refused or looked up in the wrong path.

Do not publish these logs: they can contain account names, IP addresses or details about the remote machine. If the server is completely unreachable, first follow a network diagnostic procedure before editing the SSH file any further.

Good practices after configuration

• Keep a protected, documented backup key, without sharing it.
• Remove old keys from authorized_keys only after identifying their owner.
• Use a named administrator account and limit direct root use according to your policy.
• Document the SSH port, the username, the rescue console and the recovery procedure.
• Periodically test an SSH connection and an SFTP transfer from the intended machine.

To go further with your preparation, also see the first security settings for a dedicated server, the Docker on a VPS guide, diagnosing an unreachable VPS, the Ubuntu VPS page and dedicated servers.