Setting up a VPN in AWS
I need to perform some networking experiments revolving around using commercial cellular (4G/5G) to connect to a VPN for access to protected services and to enable client-to-client communication through the VPN. The VPN must also support machine-to-machine communication — a client must be able to automatically authenticate and connect to the VPN without user intervention. Authentication using digital certificates (for mutual authentication) is my goal. The clients will run a variety of operating systems, but Red Hat Linux is the main one.
Who Else Does This Help?
There are other use cases for this type of communication network. Any distributed system (IOT swarm, collection of remote servers, for example) that don’t have a common network to run on but that do have backhaul to the Internet and can run a VPN client. This is a mix and match setup that is very flexible and resilient, since the VPN clients can be set to auto-connect enabling them to rejoin the network after a loss of connectivity, power loss, etc.
Initial Research
While doing initial research, I found that OpenVPN (https://openvpn.net ) has the client-to-client communication feature I need. Clients are also available for many operating systems (https://openvpn.net/vpn-client/ ). Furthermore, the OpenVPN client works with the Amazon Web Services VPN. I have set up an AWS Free Tier account to experiment with these features and verify my plan will work.
To meet my requirements, the first thing I need is an AWS Virtual Private Cloud (VPC). I used the new Create VPC wizard and it was pretty simple to establish a VPC since good default values are provided with every option. This provided public and private subnets, an internet gateway, and a secure gateway to S3 storage. The public subnet is connected to the internet gateway using a routing table (automatically created). I found out later that AWS created a default VPC for me automatically, so mine was redundant.
The next step is to create the Client VPN endpoint which will be the hub of communications for the clients and the entryway to my VPC.
I need digital certificates for mutual authentication. These can be self-signed, and the steps are here:
https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#mutual
To get the certs into the AWS Credential Manager (ACM), I recommend using the ACM console. You can use the AWS CLI client but that takes a little more work unless you have it all set up.
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
It took over 4 hours just to get VPN set up to the point of being able to connect a client. Hopefully these notes will help you do it quicker. The last section, Diversions, covers my mistakes so you can avoid them.
Self-Signed Certificates
Generating your own CA, server certificate and key, and one or more client certificates and keys is straightforward and well-documented here:
https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#mutual
A word about security. The commands for cert generating all specify nopass, which means do not encrypt the private key (default is encrypted). The easyrsa command will prompt you to enter PEM pass phrase. In some cases that may be required (unattended login, a GUI client that does not know how to ask you for a password, etc.) Just pay attention to your needs.
Next, upload the certs using the ACM console at https://console.aws.amazon.com/acm
One thing to note is that when you cut and paste in your PEM encoded cert files, you need to include the header ( — — -BEGIN CERTIFICATE — — -) and footer ( — — -END CERTIFICATE — — — ) for the cert data.
I imported the server cert and key, plus the ca.crt file and my first client cert and key. I went back to create a new client cert and found a minor problem with the instructions. Before doing this be sure to copy your client cert and private key to a new file (rename) or location!!
They say to create a new client cert, just re-run the command:
./easyrsa build-client-full client1.domain.tld nopassThis causes the following error (file collision):
Request file already exists. Aborting build to avoid overwriting this file. If you wish to continue, please use a different name or remove the file. Matching file found at: /Users/eizdepski/github/easy-rsa/easyrsa3/pki/reqs/client1.domain.tld.req
What you need to do is change the output file name for the new client cert. I simply made mine client2, as follows:
./easyrsa build-client-full client2.domain.tld nopassI copied the new cert pair to my directory containing all the certs and imported the new one using the ACM. The ACM also let me add a key/value pair for the cert so I gave them names indicating what they were for.
Creating the Client VPN Endpoint
Now that the prerequisites are met, time to create the Client VPN Endpoint. This time when I look at the Server Certificate ARN drop down list I see the new certs I imported!
I also want to ensure only certificates I issue and provision in the VPN Server are allowed. Not sure if that is a feature or not, but will do some testing.
Logs and CloudWatch
Since I am doing this for testing, I turned on CloudWatch Logs. This required that I first create a CloudWatch Log Group Name. I went to the CloudWatch service page (https://console.aws.amazon.com/cloudwatch ) and selected Logs: Log Group from the page menu and then clicked the “Create log group” button. I set log retention to 30 days since just testing. You should also create a log stream. Refreshing the dropdown on the Client VPN page showed me the group and log stream I just created so I selected them.
Below is an example of cloudwatch logs for a successful VPN client connection.
{
"connection-log-type": "connection-attempt",
"connection-attempt-status": "successful",
"connection-attempt-failure-reason": "NA",
"connection-id": "cvpn-connection-blah",
"client-vpn-endpoint-id": "cvpn-endpoint-blah",
"transport-protocol": "tcp",
"connection-start-time": "2022-03-10 19:10:05",
"connection-last-update-time": "2022-03-10 19:10:05",
"client-ip": "10.0.0.131",
"common-name": "client1.domain.tld",
"device-type": "mac",
"device-ip": "72.83.89.146",
"port": "60521",
"ingress-bytes": "0",
"egress-bytes": "0",
"ingress-packets": "0",
"egress-packets": "0",
"connection-end-time": "NA",
"connection-duration-seconds": "0"
}Optional (but important)
There are some important optional settings. I switch the TLS Transport Protocol to TCP (I am using cellular connections, so I want the reliability). I also turned on split tunneling so not all my traffic will go through the VPN and enabled the self-service portal for retrieving clients and VPN config files easily (turns out the portal does not work when mutual authentication is selected).
Associating the VPN to a VPC
What about the VPC ID? I had made a VPC, but when I checked my console, I found I had TWO. One was listed as the default. The other was the one I made and named. This is a little strange. I selected the default VPC.
I did a little more reading and found that AWS creates a default VPC for you, so I deleted the one I had created since it was useless. It also had an overlapping CIDR range which prevents association from working.
The Client VPN console says the status is “pending-associate”. In the console I chose the Associations tab (on the bottom) and the association process started. The process took about 10 minutes to complete.
Connecting To The VPN
The VPC dashboard now had a new button- Download Client Configuration. This gets you the OpenVPN config file needed for your client. It includes things like the DNS name for the VPN endpoint and the CA Cert I created. But it is incomplete- nothing about the client cert.
From the Client VPN console page, I re-downloaded my opvn config file (and deleted the others). I also tried the DNS name (ping) for the VPN endpoint and it was not active yet. [By the way, ping is ignored by the VPN server but the command will show you if the DNS name is resolving to an IP address or not, so still useful.]
While investigating those issues I found an example ovpn config file which explained where to put my client cert and key data. Be sure to include the header and footer for the data.
<ca>
Contents of CA
</ca>
<cert>
Contents of client certificate (.crt) file
</cert>
<key>
Contents of private key (.key) file
</key>I added my cert and key for client authentication and tried to connect. It failed and said the DNS name for my VPN endpoint was not resolving. That can take a few hours to propagate so I will try tomorrow.
I tried my OpenVPN client the next day and it still could not connect. I figured this was taking too long to be just a DNS name propagation problem. I found an interesting piece of EXTRA INSTRUCTIONS here, which goes into how to change your OpenVPN config file in case of DNS errors:
The downloaded config file has a remote-random-hostname setting that apparently does not work in OpenVPN (on a Mac using version 3.3.3 (4163)).
remote cvpn-endpoint-blah-blah-blah 443
remote-random-hostnameWhat this is supposed to do is pre-pend a random string to the DNS name to prevent DNS caching, but it does not. I added ‘hala.’ to the remote line where the DNS name is, and now I can connect:
remote hala.cvpn-endpoint-blah-blah-blah 443
remote-random-hostnameClient-to-Client Comms
I also need to change the VPN Server configuration to allow client-to-client communications and test with multiple clients. I plan to connect them using different access networks (cellular and home FIOS). The instructions are here: https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/scenario-client-to-client.html
Now let’s see how well they work! The instructions worked without a hitch. Just be sure to pay attention to which CIDR range you are using. Sometimes it is the client range, sometimes the VPC range.
I connected two computers to the VPN and tried to ping them from each other. I could go from Windows to Mac, but not the other way around without turning off the Windows firewall. Bottom line is they are able to talk to through the VPN. I also launched a webserver on the Windows machine and after changing the Apache config to listen on the current VPN IP address, I could view the webpages through the VPN client-to-client comms on the Mac.
Adding EC2 Instance to VPC
I want to verify everything so will also set up a simple web server in the AWS VPC subnet I am using so I can test connectivity from a client through the VPN to the web server. Instructions are here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html#initiate-instance-launch
When launching an instance, be sure it is launching in the correct VPC and on the subnet that your Client VPN is associated with. The Instance Details section of the instance launch wizard shows the VPC and subnet and allows you to edit them. Once launched you can verify the instance private IP is in the CIDR Range for your Client VPN Endpoint.
I launched my instance and tried to connect via SSH. I could connect over SSH using its public DNS name and the key pair I created during instance launch and did not need to be on the VPN to do this. Not really what I want, since I want to have access ONLY via the VPN, but for testing this isn’t bad. I did notice that your SSH username for redhat has to be ec2-user vice root (docs are wrong). Your key file permissions can also not be too ‘open’. World-readable is not allowed. I reset my permissions using chmod 400 on my key file.
The next challenge is using SSH to access the private DNS (or IP) on the VPC subnet while connected over the VPN. This didn’t work at first, because when I authorized access from the Client VPN to the VPC subnet, I transposed part of the IP address. Once I fixed that I was able to jump right on using SSH and the private IP address of the EC2 instance. The EC2 instance DNS name did not work, though. Note that ping will probably NOT work due to the EC2 instance firewall, so use other means to verify connectivity.
The EC2 launch instance does allow inbound SSH traffic by default- you can check those settings in the EC2 console page for the instance in its Security tab. The firewall appears set to ‘deny all’ so any other testing requires adding the appropriate inbound traffic rules.
Web Server Testing
I installed Apache on the EC2 instance. Takes 3 commands (only 2 if you don’t want it to autostart) to get it running.
sudo yum install http
sudo chkconfig httpd on
sudo server httpd startBe sure to add an HTTP inbound traffic rule to the security group for the EC2 instance!!
I accessed it over the Client VPN using its private IP address and got the RHEL Test Page. I still don’t have DNS working for VPN clients, though, for accessing servers on the VPC subnet, nor for themselves. I would also like DNS names for the VPN clients to be automatically applied based on properties in the client digital certificate. This will make it much easier for clients to find each other since the client IP changes when you reconnect.
Additional Testing
I did some other testing. I made a profile with no client cert and it cannot connect as expected. I have the original profile that is missing the random DNS string part, and it does not work (again, as expected). I also made a profile with a new client cert NOT uploaded to the ACM, and it connected. This means the VPN is accepting anyone with a client cert signed by the CA I created.
I tried the Windows 10 OpenVPN client and it works properly with the config files I created. I then removed the DNS prefix to see if remote-random-hostname worked with this client and it does! This is good, since while the prefix fixes the Mac client problem, with DNS caching you could still get a situation where the IP was changed by Amazon and your local DNS cache pointed to the old IP. The random prefix prevents the cache from working, since it won’t find an entry in the cache and will always query the DNS server.
I also tested everything over a 4G hotspot (my ultimate target network) through the Client VPN and it all worked.
DNS in the VPC
While this setup will work without DNS, it is tedious to lookup IP addresses for connections to the VPN server, or use IP addresses for services in the subnet I am using for the EC2 instance. So how to fix that?
The DNS server in a VPC subnet is at the .2 address for the subnet. My subnet is 172.31.0.0 so that means the DNS server should be at 172.31.0.2. I went on the VPN and tried an nslookup command using the private DNS name of my web server and it worked.
eizdepski@eclector ~ % nslookup ip-172-31-73-3.ec2.internal 172.31.0.2
Server: 172.31.0.2
Address: 172.31.0.2#53Non-authoritative answer:
Name: ip-172-31-73-3.ec2.internal
Address: 172.31.73.3
There’s just one problem- I don’t know how to provide the DNS server IP to the OpenVPN client. I tried one way, but the logs still show no DNS Server is set. I went back to Client VPN setup and remembered that when I set it up I did not enable DNS since I did not know the VPC DNS Server IP address at the time. I went back and fixed that.
I reconnected the VPN client and the logs now show my DNS server. I tried the web server with the private DNS name and I get the RHEL Test Page as desired.
Summary
This took a bit of time (~9 hours overall, but that included this write-up) so I hope it will help you go a little quicker. There is still one thing I want to achieve; namely, providing DNS names for the VPC clients so they can find each other easier. At least with DNS support now on in the VPC clients can easily connect to a central server, allowing workarounds for getting the IP address of all nodes in the network. Other things to consider are further lockdowns of the environment but you can only go so far before you lock yourself out so be careful. The default rules that only allow SSH traffic are good.
Diversions
This is the section about all the mistakes I made! Might save you some time so look it over.
Typos
Watch those CIDR ranges!
Custom VPC
If you make your own VPC, be sure your CIDR range for IP addresses does not overlap with the range used by the Client VPN. It won’t work, and you can’t change the range, meaning you have to start over and create a new VPC.
Self-Service Page
I tried the self-service page but got an error message. After a little more reading I found that the portal is not available for those using mutual authentication.
Using CLI to upload certificates
Note- this is Mac OS specific
I was using Version 1.10.62 of the AWS CLI (run “aws — version” to check), so decided to upgrade to version 2. Versions below 1.20 are based on Python 2.7, which is deprecated, plus I am using 3.7.4 First, I had to remove the old CLI but got an error on my Mac.
This will remove the cli (unless you installed with pip)
sudo rm -rf /usr/local/awssudo rm /usr/local/bin/aws
If you installed with pip, then uninstall this way. Hinte: to find out if you installed it with pip, type ‘pip freeze’ and look for the awscli entry.
eizdepski@eclector ~ % pip3 uninstall awscliI received an error.
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrunThe following command will fix that: xcode-select –install
Next, I installed CLI Version 2 and verified it.
eizdepski@eclector ~ % curl “https://awscli.amazonaws.com/AWSCLIV2.pkg" -o “AWSCLIV2.pkg”
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 27.1M 100 27.1M 0 0 5985k 0 0:00:04 0:00:04 — : — : — 5985k
eizdepski@eclector ~ %
eizdepski@eclector ~ % sudo installer -pkg ./AWSCLIV2.pkg -target /
installer: Package name is AWS Command Line Interface
installer: Installing at base path /
installer: The install was successful.
eizdepski@eclector ~ % which aws
/usr/local/bin/aws
eizdepski@eclector ~ % aws --version
aws-cli/2.4.23 Python/3.8.8 Darwin/21.3.0 exe/x86_64 prompt/off
eizdepski@eclector ~ %This is the command to import the certs:
eizdepski@eclector halavpn % aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crtHowever, trying to upload still didn’t work. I didn’t have permission to upload (probably since I upgraded my aws CLI to version 2) and have not completely setup the new CLI installation .An error occurred (UnrecognizedClientException) when calling the ImportCertificate operation:
The security token included in the request is invalid.I switched to the ACM Console- much simpler!
