Bootstrapping UniFi Controller on AWS

As part of a recent project to upgrade some legacy infrastructure and split services across both Azure and AWS I found myself needing to migrate a UniFi instance.

Previously hosted on a Windows 2019 Azure VM I was hoping to achieve a few things –

  • Migrate to AWS
  • Migrate to Linux (Ubuntu specifically)
  • Fully IaC and bootstrapped

I wont go into too much detail about the first two steps as they are relatively straightforward and lots of documentation already exists online. However, the bootstrapping side is a lot more interesting…

First, you may be asking “what is bootstrapping?”, bootstrapping is the first piece of code that runs when a machine starts, generally used to trigger the installation of software or make configuration changes. It allows the server to be rebuilt to the same state every time its provisioned using the bootstrap, allowing for repeatable deployments and working towards immutable infrastructure. AWS refers to this as “user data” and Azure calls it “custom data”.

In this case I want the end result to be –

  1. Run terraform apply
  2. Open a browser to the provisioned public IP and be greeted with a UniFi setup page OR login page depending on if an existing backup is found

Thats it! A completely repeatable set of code designed to be run in an immutable way for upgrades.

Of course there is a *little* bit more too it than that, high level technical steps should roughly look like –

  1. S3 storage bucket provisioned if it doesn’t already exist
  2. EC2 instance provisioned
  3. Update / upgrade Ubuntu & set hostname
  4. Prerequisites (MongoDB etc.) installed
  5. UniFi installed
  6. Restore UniFi from latest backup if any exist
  7. Configure some elements of UniFi e.g guest portal ports
  8. Sync UniFi backups to S3
  9. Start UniFi

Knowing this we can start to break each point down and look at how technically it can be done.

I am going to skip points 1 & 2 in this post as they are not included in the bootstrap and are standard Terraform resources, although note that the EC2 resource will need to have the script we create below passed to it as user data using a Terraform templatefile, for example:

  user_data = templatefile("../_bootstrap/bootstrap.sh.tpl", {
                                  server_name       = var.server_name
                                  env_name          = var.env_name
                                  unifi_version     = var.unifi_version
                                  mongo_version     = var.mongo_version
                                  portal_https_port = local.unifi_portal_https_port
                                  portal_http_port  = local.unifi_portal_http_port
                                  bucket_name       = aws_s3_bucket.unifi.id
                                })

The full Terraform code, including bootstrap, can be found on my Github – https://github.com/mhosker/terraform-unifi-bootstrap

3 .Update / upgrade Ubuntu & set hostname

This is quite simple:

# ---------------------------------------------------------
# Update / Upgrade
# ---------------------------------------------------------

echo "Updating / Upgrading..."
sudo apt-get update
sudo apt-get upgrade -y

# ---------------------------------------------------------
# Change Hostname
# ---------------------------------------------------------

echo "Changing hostname..."
hostname "${server_name}-${env_name}"
echo "${server_name}-${env_name}" > /etc/hostname

The first two commands perform the update and upgrade, followed by the second two commands which set the relevant hostname based off parameters set in the Terraform, in this case the server name followed by the environment name e.g SERVER-PROD

4. Prerequisites (MongoDB etc.) installed

UniFi relies on a number of prerequisites, most notably MongoDB. The below code installs them:

# ---------------------------------------------------------
# Install Prerequisites
# ---------------------------------------------------------

echo "Installing prerequisites..."
sudo apt-get install ca-certificates apt-transport-https awscli jq -y

# ---------------------------------------------------------
# Install MongoDB
# ---------------------------------------------------------

echo "Installing MongoDB ${mongo_version}..."

# https://www.mongodb.com/download-center/community/releases

# Install libssl1 as required for MongoDB
sudo wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
sudo apt-get install -f ./libssl1.1_1.1.1f-1ubuntu2_amd64.deb -y

# Loop through Ubuntu distros that are used in MongoDB download URLs
# Note the order of newest first... so we only install the version for the newest distro.
for distro in jammy focal bionic xenial trusty precise
do
    # Test wget for version on distro
    # Continue to next distro if 404
    sudo wget -q https://repo.mongodb.org/apt/ubuntu/dists/$distro/mongodb-org/$( echo ${mongo_version} | sed "s/\.[^.]*$//")/multiverse/binary-amd64/mongodb-org-server_${mongo_version}_amd64.deb 2>&1 || continue

    # Will only reach this point if wget above was sucessful
    # So... if it was sucessful then we install the MongoDB .deb that wget downloaded
    sudo apt-get install -f ./mongodb-org-server_${mongo_version}_amd64.deb -y
    sudo systemctl enable mongod
    # And break as we dont need to test any more distros...
    break
done

A couple of points of note about this code –

Firstly, the ca-certificates and apt-transport-https packages are required by UniFi as listed in their documentation, awscli and jq however are additional packages that are required by this bootstrap script specifically.

Secondly, when installing MongoDB we first need to install libssl1 as it is required for the downloading of the MongoDB package, furthermore we loop through a list of Ubuntu distros starting with the newest first in order to select the latest version of the MongoDB installer for the requested version.

5. UniFi installed

This is obviously an important step and is actually very straightforward, taking into account the requested UniFi version:

# ---------------------------------------------------------
# Install UniFi
# ---------------------------------------------------------

echo "Installing UniFi ${unifi_version}..."

# Install requested version of UniFi
sudo wget https://dl.ui.com/unifi/${unifi_version}/unifi_sysvinit_all.deb
sudo apt-get install -f ./unifi_sysvinit_all.deb -y

6. Restore UniFi from latest backup if any exist

This step is quite important from an immutability point of view, as once the initial UniFi install is done, in order to retain full immutability, I do not want to be SSHing to the server to perform upgrades etc. So a method is needed to completely rebuild the solution as it was, config included, with the only persistent element being the S3 bucket containing the backups.

This is where the code starts to get complex, due to the way UniFi handles the restoring of backups, a task typically performed via the web UI and with no command line options available. This meant some reverse engineering was required.

# ---------------------------------------------------------
# Restore UniFi
# ---------------------------------------------------------

echo "Restoring UniFi..."

# Copy backup files from S3 to tmp location
sudo aws s3 sync s3://${bucket_name}/backups/ /tmp/unifi/backups/

# Get the latest backup file - we will use this to restore from...

# Set as zero - this is used later for greater than numeric comparison
newestbackup[1]=0

# Merge the two UniFi backup .json files into one - autobackup and manual backups
cat /tmp/unifi/backups/meta.json /tmp/unifi/backups/autobackup/autobackup_meta.json > /tmp/unifi/allbackups.json

# Loop through each backup - format in loop of backupname=backupdatetime
for backup in $(jq -r 'keys[] as $k | "\($k)=\(.[$k] | .datetime)"' /tmp/unifi/allbackups.json)
do
    # Split the backup name / datetime into an array [0] / [1]
    backup=($${backup//=/ })

    # Convert backup time to UNIX epoch
    backup[1]=$(date --date=$${backup[1]} +"%s")

    # Check if current backup UNIX time is greater than or equal to the current newest backup UNIX time
    if [ $${backup[1]} -ge $${newestbackup[1]} ]
    then
        # If the current backup is newer then we set it as the newest backup
        newestbackup[0]=$${backup[0]}
        newestbackup[1]=$${backup[1]}
    fi
done

# Check if file DOES NOT end in .unf
if ! [[ $${newestbackup[0]} == *.unf ]]
then
    # If it does not... add .unf
    newestbackup[0]="$${newestbackup[0]}.unf"
fi

# Upload backup file to UniFi
# NOTE: The "X-Requested-With: XMLHttpRequest" is really important!
# We use jq to parse the JSON and grab the backup ID we will use next to perform the restore
backup_id=$(curl https://127.0.0.1:8443/upload/backup --form file=@$(find /tmp/unifi/backups/ -type f -name "$${newestbackup[0]}") --header "Content-Type: multipart/form-data" --header "X-Requested-With: XMLHttpRequest" --insecure | jq -r '.meta.backup_id')

# Restore backup
curl https://127.0.0.1:8443/api/cmd/backup --request POST --header "Content-Type: application/json" --data "{\"cmd\":\"restore\",\"backup_id\":\"$${backup_id}\"}" --insecure

# Copy ALL backups to UniFi
cp -r /tmp/unifi/backups/* /var/lib/unifi/backup/

# Remove UniFi tmp folder
rm -r /tmp/unifi

# Set file owner properly for UniFi backups
# This is (very!!) important to allow new backups to write to meta.json & autobackup_meta.json files
sudo chown -R unifi:unifi /var/lib/unifi/backup/

The code comments explain in most detail exactly what each line is doing, however from a high level –

  1. Copy in any existing backup files from S3
  2. Work out the latest backup file. We do this by using jq to parse the JSON format UniFi uses for storing backup metadata, including time generated.
  3. Check if the file needs to have .unf appended to make it a valid backup file we can restore from.
  4. Upload the backup file ready to be restored. Saving the returned backup ID as required for restore. This is the first request in the restore flow as performed from the web UI.
  5. Using the backup ID tell UniFi to perform the restore using the file.
  6. Copy all of the available backups, including historic, to the UniFi backup folder.
  7. Finally, set the backup directory owner as required to allow UniFi to write new backups.

7. Configure some elements of UniFi e.g guest portal ports

Additional config can be added here, however I am only changing the default UniFi captive portal HTTPS port –

# ---------------------------------------------------------
# Configure UniFi
# ---------------------------------------------------------

echo "Configuring UniFi..."

# Set guest portal https / http ports
# NOTE: Must be over 1024 as UniFi does not run as root so cannot bind to priveleged ports (<1024)
sed -i "/^portal.https.port=/d" /var/lib/unifi/system.properties # Remove existing https port
echo "portal.https.port=${portal_https_port}" >> /var/lib/unifi/system.properties # Add https port
sed -i "/^portal.http.port=/d" /var/lib/unifi/system.properties # Remove existing http port
echo "portal.http.port=${portal_http_port}" >> /var/lib/unifi/system.properties # Add http port

In this case I used 2083 for HTTPS and 8880 (default) for HTTP, allowing me to then proxy this traffic through CloudFlare.

8. Sync UniFi backups to S3

This step is important to keep UniFi auto backups and any manual backups taken via the web UI in sync with the S3 bucket, so upon a rebuild the latest backup can always be copied and used.

# ---------------------------------------------------------
# UniFi Sync
# ---------------------------------------------------------

echo "Configuring UniFi Sync..."

# Create a script to sync the UniFi backup folder to an S3 bucket
echo "#!/bin/sh
while true
do
    sudo aws s3 sync /var/lib/unifi/backup s3://${bucket_name}/backups --only-show-errors --delete
    sleep 300 # Wait 5 mins
done" > /usr/local/bin/unifi-sync

# Make UniFi sync script executable
sudo chmod +x /usr/local/bin/unifi-sync

# Create systemd service for unifi-sync
echo "[Unit]
After=network.target

[Service]
ExecStart=/usr/local/bin/unifi-sync

[Install]
WantedBy=default.target" > /etc/systemd/system/unifi-sync.service

# Set required permissions for service
sudo chmod 664 /etc/systemd/system/unifi-sync.service

# Start the service
sudo systemctl daemon-reload
sudo systemctl enable unifi-sync

Note that we run the sync script as a service, this is so that even after a reboot the sync will start and continue to work.

9. Start UniFi

Finally, we are ready to start UniFi!

# ---------------------------------------------------------
# Start UniFi & UniFi Sync
# ---------------------------------------------------------

echo "Starting UniFi and syncing backups..."

# Start UniFi
sudo service unifi start

# Run UniFi sync script
unifi-sync

We also start the sync script here too.

And that’s it! UniFi can now be deployed on AWS in a completely repeatable and immutable way.

Full code, including Terraform can be found on my GitHub – https://github.com/mhosker/terraform-unifi-bootstrap