Introduction
Administering large numbers of SSH keys and servers can be very difficult as your organization grows. Correctly identifying valid keys and removing invalid keys throughout an organization can be fraught with errors and have huge consequences on your server security.
In addition, when there are server changes, sometimes your users will receive warnings about being unable to establish the authenticity of your server. Most users will not double-check the key fingerprint of the server before connecting, allowing someone to potentially spoof the server and execute a man-in-the-middle attack.
A project called monkeysphere was created to address these issues. It does this by leveraging GPG keys and the web of trust model to both validate a server’s credentials, and provide easy user management.
In this guide, we will discuss how to set up monkeysphere in order to validate your server to users. This will solve the problem of users having to guess whether the server they are connecting to is actually the one they’re attempting to access. Usually, when you connect to a server for the first time, you will see something that looks like this:
The authenticity of host '107.170.43.127 (107.170.43.127)' can't be established.
ECDSA key fingerprint is 10:14:75:d6:42:a3:c5:59:d1:83:6d:cf:52:61:4a:52.
Are you sure you want to continue connecting (yes/no)?
We will work in this article to avoid showing these messages to our users. In a future guide, we will discuss how to solve the problem of easily identifying and authenticating users.
We will be using Ubuntu 12.04 VPS instances to configure this system. We will have an SSH server that we will attempt to validate to our users. We also need two other machines to demonstrate (whether local computers or VPS instances). One will be the administrator’s computer, and another will a client that we will use as a test machine to see if our web of trust allows it to verify the identity of the server.
Basic Strategy
The entire monkeysphere system relies on GPG keys and key servers to function. You will have to configure these keys for each system before we can begin. Much of the actual configuration will be done through these keys as well.
Before you begin working with GPG, you may want to look at our article on how to use GPG keys. We will be giving you the specific commands needed, but a deeper knowledge of what is going on may help you troubleshoot in case of issues.
Our three machines and their uses will be:
- server.example.com: The SSH server that we want to validate to clients.
- admin.example.com: Our administrator computer that will be using to configure access. This is usually your home computer, but we’ll be using another droplet in our guide.
- client.example.com: This is the client that we will be using to test our validation against. We want this computer to be able to connect to the server without receiving an unknown host warning.
The SSH server machine should have a publicly accessible domain name. Use this guide to set up domain names on DigitalOcean.
For the first portion, we want to generate keys on both the admin and client computers. We will then upload these keys to a centralized keyserver. We will pull down the admin’s key from the keyserver and sign and trust it using the client key, indicating that our client trusts our admin.
This is the basis of the entire verification scheme. Instead of trusting a computer or a key, we are leveraging GPG’s web of trust model. If you consider a person to the knowledgeable and trustworthy about the server you are trying to connect to, you can defer to that person to let you know if the server is legitimate. In this case, our client will trust that the server administrator can validate the server’s identity.
We can then configure monkeysphere to use the GPG framework to pull down information to verify that the administer that we trust vouches for the server we are trying to connect to.
Set Up GPG Keys and Authentication
Luckily, GPG is installed on Ubuntu by default.
We will begin by creating some GPG keys on both our client system and our administrator’s system. These keys are associated with a single user and are used to identify that user globally. Each person that needs access to your SSH server should have GPG keys that identify them. This is how we build our web of trust.
Generating GPG Keys
We need to create a GPG key on both the client machine and the admin machine. Take the following actions on both of these computers:
gpg --gen-key
This will prompt you with a number of questions to create your key pair. First, it asks which type of keys you wish to create. Select “1” to create two RSA keys. Accept the default value in the next question to make a 2048-bit key. Select “0” to make the key never auto expire, and then type “Y” to confirm that the information is correct.
Next, you’ll be asked for your information for each of these users. We will use the name “admin” and the email address “[email protected]” for our administrator’s keys. For our client, we will use the name “client” and the email “[email protected]”. You have the opportunity to add an optional comment, and then you should type “O” to indicate that the information is okay.
Enter and confirm a passphrase to protect the keys.
The computer will now create the key using random pieces of data collected from the system. This is called “entropy” and is used to create a truly random key. It may take awhile. Sometimes it is helpful to log in with a separate SSH session to do some work in order to speed up the process.
When the key has been generated, it will be stored within your GPG keyring. You can see it by typing this:
gpg --list-keys
/root/.gnupg/pubring.gpg ------------------------ pub 2048R/08D014B3 2014-03-14 uid client <[email protected]> sub 2048R/4C73683E 2014-03-14
The part that is in red above is a shortened key ID. We can use this hash to refer to this key.
We should upload our keys to keyservers. These keyservers are replicated around the world and allow anybody to pull down our key information. This is what we want, because it allows our two users and the SSH server to interact with each other and establish trust relationships.
To upload our keys to keyservers, on our client and admin computers, we will need to type something like this. We will need the key ID that we mentioned above:
gpg --send-key key_id
So for our client key above, we could upload this to the keyserver by typing:
gpg --send-key 08D014B3
Signing the Keys
Now that both our administrator and our client have GPG keys and have uploaded them to the keyservers, they will begin to propagate to other keyservers throughout the world. It may take some time for each of the keys to be copied to GPG servers around the world, so you may have to wait a few minutes for these steps to work.
After waiting a few minutes, you can try to pull down the opposite key on each of your computers. This means, on the client computer, try to pull down the administrator’s key. You should also pull down the client key for the administrator computer.
We are going to be signing these keys, meaning that we consider them valid and have verified that they are the correct keys for the person we are trying to identify. To make absolutely sure that you are getting the correct key, we will do this without searching and by specifying the exact key by identifying it by its fingerprint.
On your client server, get the fingerprint of your GPG key by typing this. The “client” is the name or email address you selected when building your key:
gpg --with-colons --fingerprint client
tru::1:1394819815:0:3:1:5 pub:u:2048:1:4B3F73E208D014B3:2014-03-14:::u:client <[email protected]>::scESC: fpr:::::::::85ECDB498FB0CAB5F02989E64B3F73E208D014B3: sub:u:2048:1:254105194C73683E:2014-03-14::::::e:
The portion of the output that is highlighted above is the fingerprint that you are looking for.
Now that you have this fingerprint, on your administrator’s computer, you can tell GPG to look for a key with that fingerprint and pull it down to the local computer, like this:
gpg --recv-keys 85ECDB498FB0CAB5F02989E64B3F73E208D014B3
This will connect to the default keyserver and ask for the key that is identified by that fingerprint. It will be transferred into our local system.
Now that we have access to the key, we can sign it to indicate that we, as the administrator, trust the key that belongs to the client:
gpg --sign-key 85ECDB498FB0CAB5F02989E64B3F73E208D014B3
Now, we have signed the key on our local system. We now should send the client’s key back to the keyserver. The keyserver will update its information on the client key to indicate that our administrator account has signed the key and considers it valid.
To send the signed key back to the keyserver, we type:
gpg --send-key 85ECDB498FB0CAB5F02989E64B3F73E208D014B3
The keyserver now has our administrator’s key and the client key, just like before. The difference is that the keyserver also now has a signature from the administrator key on the client key.
We need to do the opposite operation now (sign the administrator’s key with the client key). To do this, on the administrator’s machine, we get the administrator key fingerprint:
gpg --with-colons --fingerprint admin
Now, on the client machine, we use the fingerprint that we collected from the output to pull down, sign, and then upload the administrator’s key, just as we did before:
gpg --recv-keys 7C873BB244245CB13BFEFC31F7C66E2FF945A061
gpg --sign-key 7C873BB244245CB13BFEFC31F7C66E2FF945A061
gpg --send-key 7C873BB244245CB13BFEFC31F7C66E2FF945A061
Afterwards, on both machines, we need to refresh our keys. This is because each of our users have had their keys signed, but they don’t have that signature on their own computer, just on the opposite computer and in the keyserver. You will want to wait a few minutes here to let your changes propagate, and then type:
gpg --refresh-keys
You need to verify that the update was processed. The output should have a line that says “new signatures: 1”. If this is not present, retry until it shows up.
Now, each of your computers should have both signed keys.
Trusting the Administrator’s Judgement
In order for our web of trust to actually work, we’re going to not only have to have the client sign the administrator’s key, we are also going to have to establish that the client “trusts” the administrator’s judgement.
This means that when the administrator says that the SSH server is the machine it says it is, that we, as the client, can trust the signatures that another person, the administrator, makes.
First, we can see the current settings on our client by typing:
gpg --check-trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 1 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1 valid: 1 signed: 0 trust: 1-, 0q, 0n, 0m, 0f, 0u
This is a very confusing set of information. The first line states the trust model that we are operating under. Basically, if we have a key that is signed by one user that we completely trust, then we will consider that key valid as well. We can also trust a key if it is signed by at least 3 people we only marginally trust.
The second line tells us about our “depth 0” trust level. This is basically information about our own key. We consider it valid and signed. And the “trust” portion tells us that the key is in the “u” category, meaning ultimately trusted.
The third line tells us about keys at “depth 1”. These are keys that we’ve personally signed. We can see that we consider one key valid (we signed the admin key to make it valid), and that the admin key has not signed any additional keys that we care about.
In the trust section of that line, we have a field that has “1-“. This means that the one key that we have at this level (the admin key) has not been given a trust setting. We don’t know what level of trust to put in this key.
We want to fully trust the admin key. This way, any key that the admin signs (like the SSH server we are trying to verify the identity of) will be considered legitimate by us. To do this, we can update our trust database on our client machine by typing:
gpg --update-trustdb
You will asked to assign a trust rating to any key you’ve signed that does not currently have a trust value. It will look like this:
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 1 trust: 0-, 0q, 0n, 0m, 0f, 1u
No trust value assigned to:
2048R/49E95F19 2014-03-14
"admin <[email protected]>"
Primary key fingerprint: A612 56B8 5307 B7ED 9AD8 D93E 9E06 881E 49E9 5F19
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
s = skip this key
q = quit
Your decision?
We want to fully trust this key, so type “4”.
Now, since we trust the administrator’s judgement, we can be reasonably sure that the keys the admin signs are legitimately associated with the user or service in question (typically we’d use real names instead of client or administrator).
Install Monkeysphere and Configure SSH Server
Now that we have our client and administrator computers set with their GPG keys, we can get started on the actual monkeysphere installation and configuration.
There is an monkeysphere package in Ubuntu’s default repositories. It contains both the server and client utilities necessary for SSH to use GPG for validation. As such, we will need to install the package on each of our computers.
On all of your computers in this model (administrator, client, SSH server), install monkeysphere by typing:
sudo apt-get update
sudo apt-get install monkeysphere
We now have the components necessary to put these pieces together. On the SSH server, we will generate a special GPG key using a wrapper utility included in monkeysphere. The generated key will be based off of the ssh_host_rsa_key that is used to identify a server to clients.
On the SSH server, type:
monkeysphere-host import-key /etc/ssh/ssh_host_rsa_key ssh://server.example.com
The second component above should indicate the SSH server’s domain. This will help clients find the correct server key when trying to verify the server connection.
Now, we can upload the key we just created to the public keyserver again. This time, we will again use a wrapper program included in the monkeysphere suite:
monkeysphere-host publish-key
This will upload your server key and make it available to clients wishing to connect.
Validating the Server Key as the Server Administrator
Now, we have the SSH server’s key being propagated to the world’s keyservers. But how does this help us?
Well, the way that GPG works is through establishing what it calls a “web of trust”. Simply put, it works by establishing a personal network of people that you know and whose identity you can verify to a high degree of certainty. It then uses those trust relationships to allow you to offload the responsibility for knowing information onto people you trust.
In our case, we (as the client for this example) are offloading the responsibility of knowing whether the server is legitimate onto our administrator, who we trust. The server’s administrator should have a good idea whether the server’s credentials check out.
So what we need to do now is sign the SSH server’s GPG key as the server administrator, this will allow our client user to verify the server’s identity by trusting administrator.
On the SSH server, get the GPG fingerprint by typing:
monkeysphere-host show-key
pub 2048R/0D281337 2014-03-14
uid ssh://fakedomain.com
OpenPGP fingerprint: E06A426459E584F272DB708AD2D462790D281337
ssh fingerprint: 2048 61:1e:a7:66:1d:04:64:80:3f:27:81:34:31:78:8d:df (RSA)
The “OpenPGP fingerprint” is the same as our GPG fingerprint. This is what we will use to pull down the key to our administrator’s computer and sign it.
On our administrator’s computer, type this to obtain the SSH server’s key. Again, you may need to wait a few minutes:
gpg --recv-key E06A426459E584F272DB708AD2D462790D281337
We will be signing the key, just as we did above with the client key:
gpg --sign-key E06A426459E584F272DB708AD2D462790D281337
Finally, we need to remember to upload the signed key back to the keyserver:
gpg --send-key E06A426459E584F272DB708AD2D462790D281337
Verifying the Identity of the SSH Server from the Client
Now, we have everything in place to validate the identity of the SSH server from the client computer.
To do this, we will have to wrap the SSH commands in a monkeysphere utility. This will tell our client to use monkeysphere to check for a GPG key of the server we are trying to ssh into. It will then see if we trust anyone who’s confirmed that the server is valid.
We will be doing this manually the first time to show you what is happening. Afterwards, we can add this to a file to do this automatically.
On the client, type the manual command like this:
ssh -oProxyCommand='monkeysphere ssh-proxycommand %h %p' server.example.com
-------------------- Monkeysphere warning ------------------- Monkeysphere found OpenPGP keys for this hostname, but none had full validity. An OpenPGP key matching the ssh key offered by the host was found: pub 2048R/0D281337 2014-03-14 uid [ unknown] ssh://server.example.com sig!3 0D281337 2014-03-14 ssh://fakedomain.com RSA key fingerprint is 61:1e:a7:66:1d:04:64:80:3f:27:81:34:31:78:8d:df. -------------------- ssh continues below -------------------- The authenticity of host 'server.example.com (<no hostip for proxy command>)' can't be established. ECDSA key fingerprint is 78:50:80:60:2a:a3:51:51:37:9d:25:8b:d4:0c:d1:15. Are you sure you want to continue connecting (yes/no)?
Type “no” here.
As you can see, we have the usual message about how the authenticity of the host we’re trying to connect to cannot be established. We have not fixed the problem yet and haven’t verified the identity of the server, so we must type “no” to protect ourselves from connecting to the wrong host.
However, we have also been given an additional section of information under the “Monkeysphere warning” section header. It tells us that it was able to retrieve the key, but it couldn’t verify the server because it didn’t have the trust relationship necessary.
If we look at our keys in the client computer’s key chain, we will notice that we now have the SSH server’s key:
gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub 2048R/87791BD0 2014-03-14
uid client <[email protected]>
sub 2048R/3294D31D 2014-03-14
pub 2048R/54AD641F 2014-03-14
uid admin <[email protected]>
sub 2048R/A87CADCB 2014-03-14
pub 2048R/0D281337 2014-03-14
uid ssh://fakedomain.com
So we have the key in our system now. Now we just need to refresh our keys. This will pull in any signatures for the keys in our system:
gpg --refresh-keys
You should again see a line that says “new signatures: 1“. We can see this by checking the signatures available on the SSH server’s key:
gpg --list-sigs ssh://server.example.com
pub 2048R/0D281337 2014-03-14 uid ssh://server.example.com sig 3 0D281337 2014-03-14 ssh://server.example.com sig 54AD641F 2014-03-14 admin <[email protected]>
As you can see, we now see a “sig” line listing the administrator’s key. Since our client trusts the administrator, we can now, by proxy, trust the SSH server.
Let’s try our command again:
ssh -oProxyCommand='monkeysphere ssh-proxycommand %h %p' server.example.com
[email protected]'s password:
As you can see, we have now been given the password prompt without asking us to verify the validity of the SSH server. That’s because we have verified its identity through GPG.
To avoid having to type this long command in every time we want to SSH into the server, we add this to our configuration file on our client computer:
nano ~/.ssh/config
Host *
ProxyCommand monkeysphere ssh-proxycommand %h %p
This will allow us to connect through SSH as normal, and will do all of the monkeysphere verifying in the background:
ssh server.example.com
To double check that this is working, feel free to delete your current known_hosts file:
rm ~/.ssh/known_hosts
Now, retry the ssh command:
ssh server.example.com
You’ll be logged in or prompted for a password, never asking if you accept the host. You can also see that the known_hosts file has been recreated automatically and that host has been added:
cat ~/.ssh/known_hosts
server.example.com ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQC9aTHZmHZSgwNtwichF0AqDI74bCMtI29kqPDZaNn2r86NGIElRUlQiRImmZXs5oEjF0o8VaW6s1cIj0hC5ziDPShJ3VzZTWz9RmJ9xfPPcAPw2JbV1c1Q1bplstQqCZmFcRZyofztnP55HqOiJ4htLMxH+a9lM4AydDZtGHhzU+usxUjHniVbxCUVntpunlwtMk+Mtk9eysVdnJCJyV02/W89HExiO9QRpv+EugKN1eCQYrGvNbKWQKq4gSJ0RDwOSKNgkY/Ii0MsGJ2HuioO9np6IEdeZdgSGHPA23+zZe8asrN62iLUBADDkyIR6FAonCvfh99hbFxpNz2N8Mdb MonkeySphere2014-03-21T21:30:44
Conclusion
If you configure your organization to use monkeysphere for SSH, SSH users will never have to question the legitimacy of a host within your organization. Once your users trust the server administrators and have configured their SSH to rely on monkeysphere, and assuming the administrators are vigilant about signing new hosts, your users should never be asked to verify a hosts identity.
Blindly accepting hosts as valid is a huge risk and most users will not have the capabilities to legitimately check the identity of a server without the administrators help anyways. So using Monkeysphere, we can cut out this entire process for the safety of the entire organization.
This might seem like a lot of work, but setting up a trust network will become less work as you go and will allow you to avoid dangerous man-in-the-middle style attacks.
In the next article, we will discuss how to use monkeysphere to validate your users to the SSH server.