Bootstrapping DigitalOcean Servers

I manage multiple DigitalOcean servers. There are a number of configurations to be done in order to bring a secure DO droplet. For example, firewall, alerts, etc. I missed a step years ago that caused a low priority security alert lately. So, I automated most of the steps while configuring any DigitalOcean server (new or old).

Like most other things, I open-sourced the project in Github. Please check it out at https://github.com/pothi/doctl-automation .

The first step is to get an API key and to get doctl command line interface tool. Then you can initialize doctl with the API key as follows…

# you will be asked the access token
# <NAME> could be "team-name"
doctl auth init --context <NAME>

# Authentication contexts let you switch between multiple authenticated accounts.
doctl auth list
doctl auth switch --context <NAME>

# validate doctl
doctl account get

Firewall

Firewall is obviously the most important of any server configuration. With DigitalOcean, you don’t have to rely on the server-level firewall such as ufw. With a server-level firewall, it is possibly to lock yourself out of the server. With DigitalOcean (and most other cloud providers), the firewall can be configured and updated on the host level, a layer above the server or the operating system installed in it. Here I used a simple firewall with the name “Basics” that covers the basic stuff…

Firewall_Name=Basics
# create a firewall using minimal outbound rules
Outbound_Rules="protocol:icmp,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:0,address:0.0.0.0/0,address:::/0 protocol:udp,ports:0,address:0.0.0.0/0,address:::/0"
doctl compute firewall create --name $Firewall_Name --outbound-rules "$Outbound_Rules"

# Get FirewallID using FirewallName
Firewall_ID=$(doctl compute firewall ls --format ID,Name --no-header | grep $Firewall_Name | awk '{print $1}')

# Add tags, standard inbound rules and any custom inbound rules
doctl compute firewall add-tags $Firewall_ID --tag-names live,prod

Inbound_ICMP="protocol:icmp,address:0.0.0.0/0,address:::/0"
Inbound_HTTP="protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0"
Inbound_HTTPS="protocol:tcp,ports:443,address:0.0.0.0/0,address:::/0"
Inbound_SSH="protocol:tcp,ports:22,address:0.0.0.0/0,address:::/0"

doctl compute firewall add-rules $Firewall_ID --inbound-rules $Inbound_ICMP
doctl compute firewall add-rules $Firewall_ID --inbound-rules $Inbound_HTTP
doctl compute firewall add-rules $Firewall_ID --inbound-rules $Inbound_HTTPS
doctl compute firewall add-rules $Firewall_ID --inbound-rules $Inbound_SSH

# delete a firewall rule
doctl compute firewall remove-rules $Firewall_ID --inbound-rules=$Inbound_SSH

Internal Firewall

If you’d like to allow traffic between servers, you may use the following…

# Internal Firewall
Firewall_Name=InternalNetwork

Internal_10="protocol:tcp,ports:0,address:10.0.0.0/8"
Internal_10_udp="protocol:udp,ports:0,address:10.0.0.0/8"
Internal_172="protocol:tcp,ports:0,address:172.16.0.0/12"
Internal_172_udp="protocol:udp,ports:0,address:172.16.0.0/12"
Internal_192="protocol:tcp,ports:0,address:192.168.0.0/16"
Internal_192_udp="protocol:udp,ports:0,address:192.168.0.0/16"

doctl compute firewall create --name $Firewall_Name --inbound-rules "$Internal_10 $Internal_10_udp $Internal_172 $Internal_172_udp $Internal_192 $Internal_192_udp"

Once the firewalls are created, you may attach them to any existing droplets or new droplets (while creating the droplets) using tags. DigitalOcean has some powerful tagging system that works nicely.

Monitoring

A big part of any server setup is monitoring and alerting us upon any usual activities. While monitoring is a complex topic, DigitalOcean allows us to monitor the resources such as the disk space, memory usage, CPU spikes, etc. All these things can be configured in a flash using the following code that becomes applicable to all droplets (existing and new)…


ADMIN_EMAIL=$(doctl account get --format "Email" --no-header)

doctl monitoring alert create --compare "GreaterThan" --value "90" --emails $ADMIN_EMAIL --type "v1/insights/droplet/cpu" --description "CPU is running high"
doctl monitoring alert create --compare "GreaterThan" --value "75" --emails $ADMIN_EMAIL --type "v1/insights/droplet/disk_utilization_percent" --description "Disk Usage is high"
doctl monitoring alert create --compare "GreaterThan" --value "90" --emails $ADMIN_EMAIL --type "v1/insights/droplet/memory_utilization_percent" --description "Memory Usage is high"

What’s still not implemented?

While the existing API is matured enough, it doesn’t have all the functionalities that can be done on the DigitalOcean dashboard. For example, a few months ago, DigitalOcean introduced uptime monitoring that can not be configured via doctl, yet!

Nevertheless, DO API is stable and I highly recommend it if you plan to use DigitalOcean to host your sites or your clients’ sites.

Happy Hosting!

DigitalOcean Hosting Review

No affiliate links are inserted in this post. I did not have and do not have any monetary benefits in posting a positive review about DigitalOcean (or any other hosts in this blog), either. No host is perfect. So, I am posting only what I like about DigitalOcean. There may be some cons of using any host. If you search the internet, you may find them. I use Google Compute Engine for hosting this site, though.

Logo - DigitalOcean

Time to Restart the Whole Server

Whenever I build a server from scratch, I make sure that the automatic security updates are enabled. Most security updates are applied automatically. However, certain updates, particularly, the updates related to Linux kernel needs a restart of the entire server. This means a short downtime of the site/s hosted in that particular server. No one likes a downtime. At least, Google Search Bots do not.

The server restart and the resulting downtime is the primary reason for the popularity of containers such as Docker, LXC, LXD, etc. Of course, there are other benefits of using a container. Not everyone can afford to use containers, load balancers, etc. Maintaining them cost a lot more than maintaining a single server without the container eco-system. Most individual site owners and small businesses opt for single server system to lower the cost of running a server.

In shared hosts, the Linux kernel updates are applied on-the-fly, using one of the multiple technologies available, such as kpatch, livepatch (commercial), etc. Most of them are commercial services. The free services come with a big warning such as the following…

WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss may occur!

So, most vendors (shared hosts) opt for paid commercial services to apply Linux kernel updates without a complete restart of the server. If you use a shared host, then no worries. This post is valid only for VPS servers and dedicated servers. The term VPS may also mean cloud servers with certain hosts. Some popular VPS providers are AWS (Amazon Web Services), Google Compute Engine (GCE), Linode, DigitalOcean, Vultr, etc.

The time to restart a server (VPS or dedicated server) depends on various factors. The number of running processes, the number of running programs, the choice of web server, the database server, etc. Every tiny thing matters. Even the hypervisor used by the host can have an impact on the time to restart the whole server.

Choosing a host is hard. There are lots of factors to consider. Price, the location of data centers, the Operating Systems provided, the bandwidth or the network speed, overall value for money, etc. Among these, you might also want to consider the time to restart a server when choosing a host. Because, some hosts take more than 5 minutes to restart a server, resulting in a downtime that may be a considerable amount for Google Search Bots.

In my experience working with various hosts, DigitalOcean restarts the servers in the shortest time possible, often less than 30 seconds for a server with a memory of up to 32GB. An idle server with no sites hosted on it, may restart much more quickly, like less than 15 seconds.

There are other advantages of choosing DigitalOcean. I will update this post with those, if it is worth mentioning.

Happy Hosting!

css.php