Advanced Guide: Creating 3 servers VPC - Part 2 (Reddit Clone)

In this 2-part tutorial, we will show you how to create a cluster of servers using 3 CloudAfrica server instances.

Previous blog post (Part 1)

In Part 1 of this 2-part tutorial, we created 3 servers and linked them up to a VPC (Virtual Private Cloud).

In case you missed it, it is strongly advised that you read through that tutorial to understand the tutorial below. The link can be found here:

Advanced Guide: Creating 3 servers linked via VPC - Part 1

Part 2

We will now go through an actual setup where a Frontend, Application and Database servers are setup.

To simplify matters, we will be installing a Reddit clone called Drum: https://github.com/stephenmcd/drum/ (Drum is written in Python using Django)

The instructions involve a little familiarity with deploying web applications in dynamic languages like Ruby/Python. However, all of the instructions can be copied/pasted and the Drum application should run.

Getting Started

Before proceeding, you need to have already created a VPC and need to know how to create a server.


Our setup will look like so:


  • Frontend: Nginx - handles incoming web traffic and redirects to application-server (can also act as load-balancer)
  • Application: Drum - Reddit clone that allows users to post links/comments to a website
  • Database: PostgreSQL - popular SQL database

Naming

Although we will refer to each server based on its purpose (Frontend, Application, Database), for purposes of this tutorial, the server names we used to create the actual servers on the CloudAfrica dashboard are:


  • Frontend = tut3-frontend
  • Application = tut3-app
  • Database = tut3-db

Creating 3 servers

Based on Part1, you should have 3 servers existing. Frontend will have a public IP-address, whereas Application and Database will only have internal IP addresses and will be accessed via the VPC (for production-servers, the isolation of app/database is worth considering in terms of security. For the purposes of this tutorial however, we created public IP addresses for all the servers so that we could run updates for them)

If you haven't done so already, please go ahead and create the 3 servers. You can name them whatever you wish, but we suggest naming them as we have done so above.

The size/distro used for each server is listed below:


  • Frontend: 2GB/Ubuntu16.04
  • Application: 1GB/Ubuntu16.04
  • Database: 1GB/Ubuntu16.04


For the Application and Database servers, we will need to access them via the frontend server. To do this, we will need to generate an SSH key on Frontend and use the public-key on Application and Database.

Steps to achieve SSH access to internal servers

  1. SSH into Frontend (tut3-frontend)
  2. Follow the instructions of points 3. and 4. from this link: SSH Keys to generate an SSH keypair ON Frontend
  3. You can add/name the public-key of Frontend as: tut3-internal-frontend
  4. When creating Application and Database, create the servers using "tut3-internal-frontend" as the SSH key

Testing connections

Our first test is to make sure we can successfully SSH into Frontend

ssh -i /path/to/.ssh/myprivatekey root@[frontend-public-IP]

Now that we have access to the frontend server, we can try pinging our other 2 servers:

tut3-app: ping [app-private-IP]

tut3-db: ping [db-private-IP]

If you followed Steps 1-4 above, you should be able to SSH into the internal servers as well:

ssh -i /path/to/.ssh/frontendpvtkey root@[private-IP]

Now that we are successfully able to access all the servers, we can get started installing the required software for each of them.

Frontend

This will be our Nginx/proxy server. Let us install the necessary software:

 # running some updates

sudo apt update

sudo apt upgrade

 # installing the required software

sudo apt install nginx ufw

# limiting access to nginx on Port 80 only

sudo ufw allow 'Nginx HTTP'

If everything is working correctly, you should be able to visit/see the server at the following URL:

http://[frontend-public-IP]

Application

This is the server where we will install Drum. Instructions for installation are:

# running some updates

sudo apt update

sudo apt upgrade

# Start: Python section

sudo apt install python3-pip

pip3 install virtualenv

virtualenv drum

cd drum/

source bin/activate

pip install drum gunicorn psycopg2

# End: Python section

Database

This is where we will install PostgreSQL. Instructions:

# running some updates

sudo apt update

sudo apt upgrade

# Installing PostgreSQL

sudo apt-get install postgresql postgresql-contrib

# Setting password for postgres user

sudo -u postgres psql postgres

\password postgres

# Use password '12345'

# Create database 'redditclone'

sudo -u postgres createdb redditclone

Configuring it all together

In this part, we will make all the servers "talk" to each other.


The basic structure will look like so:

Incoming web request (Frontend) => Processed by Application Server <=> data sent back and forth to Application (Database)

In tut3-app(Application Server):

source drum/bin/activate

mezzanine-project -a drum reddit-clone

cd drum/reddit_clone/reddit_clone/

vim settings.py

Now in vim mode, we need to add our database configuration details:

DATABASES = {
"default": {
# Add "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
"ENGINE": "django.db.backends.postgresql_psycopg2",
# DB name or path to database file if using sqlite3.
"NAME": "redditclone",
# Not used with sqlite3.
"USER": "postgres",
# Not used with sqlite3.
"PASSWORD": "12345",
# Set to empty string for localhost. Not used with sqlite3.
# "HOST": "[db-private-IP]", -- example below:
"HOST": "10.1.44.68",
# Set to empty string for default. Not used with sqlite3.
"PORT": "5432",
}

}

In the same settings.py , we need to go a few lines down the file and find the ALLOWED_HOSTS section. In here, we add the following:


ALLOWED_HOSTS = ['[nginx-private-IP]', '[nginx-public-IP]']


It should look something like:


ALLOWED_HOSTS = ['10.1.44.27', '42.85.92.160']


Now do the following:

cd ..

python manage.py createdb --noinput

The application and database should now be provisioned. Although we will not be able to reach the internal application just yet (we need to configure the application-server and then make the webserver point to it)

Collecting the static files:

python manage.py collectstatic

This will create a copy of the static files to the following path:

/root/drum/reddit_clone/static

Gunicorn

We will now configure a systemd service to make the gunicorn server run automatically.

Start with this command:

vim /etc/systemd/system/gunicorn.service

You will now be in the vim editor. Press `i` and you will now be in Editor mode. Copy/Paste the following text:


[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/root/drum/reddit_clone/
ExecStart=/root/drum/bin/gunicorn --access-logfile - --workers 3 --bind 0.0.0.0:8080 reddit_clone.wsgi:application

[Install]
WantedBy=multi-user.target


Press Escape. Then type the following: `:wq` and hit Enter. This will save the file.

You can now start the server:

systemctl start gunicorn

Check the status:

systemctl status gunicorn

Enable the gunicorn process to start each time the server boots:

systemctl enable gunicorn

Nginx

We will now be working in tut3-frontend(Frontend Server).

We firstly need to copy the static content from tut3-app(Application Server) to tut3-frontend(Nginx), like so:


# Instructions

# scp -i /root/.ssh/frontendpvtkey -r root@[tut3-app-private-IP]:/root/drum/reddit_clone/static /var/www/

# The instructional command above copies the static/ folder from tut3-app recursively into /var/www/ , creating the path: /var/www/static/

# example being:


scp -i /root/.ssh/frontendpvtkey -r root@:10.1.44.150/root/drum/reddit_clone/static /var/www/


Now that we have the static content ready, we need to configure the nginx config file to serve the app and the static content.

Access the nginx-config file like so:

vim /etc/nginx/sites-available/default

Copy/Paste the following code:


server {

listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;

# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;

server_name _;

location /static/ {
root /var/www/;
}

location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://10.0.35.117:8080;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log /var/log/nginx/access_forwarded.log;

}

}


You can copy/paste the code over the existing server {} section.


Save the file (ESP + `:wq` + ENTER) and restart Nginx by doing the following:

systemctl restart nginx

If you now navigate to the public IP address of the Frontend Sever (tut3-frontend), you should see Drum running, as indicated by this demo: drum.jupo.org

Notes

This tutorial provides very rudimentary introduction to running a multi-server environment. This setup is not recommended for production environments. Adopting a modular and automated architecture to your systems is the preferred method to use.