Cloud Ops

Overview

This repo controls all of the cloud operations for Titans Of Eden. This includes:

  1. Website HTML/CSS/JS

  2. Online Demo

  3. Blender Video-Rendering & Formatting

  4. Database Creation & Management

  5. Web-based API

  6. Email Listserv Management

Cloud Infra

The following infrastructure was setup for this project:

  1. (Production) File Server / Static Website

  2. Dev File Server / Static Website

  3. Batch Account (for parallel computing / video rendering)

  4. MySQL Server (as a general database)

  5. Container Registry

  6. CDN (for public-facing website)

  7. Linux VM (which provides all API functionality)

All infrastructure resides on Azure.

Cloud VM

This project has a cloud VM. All development work should be carried out on this VM (e.g. by SSH’ing into it).

The VM’s IP is whitelisted, which allows it to access the database and other cloud services. Additionally, credentials are securely stored on this VM.

If the cloud VM ever goes down and needs to be re-provisioned, once the VM is up again, you can re-provision by:

  1. Placing a MySQL connection string in ~/secrets/titans-mysql

  2. Whitelisting the VM’s IP

  3. Running the command:

    curl https://raw.githubusercontent.com/lakes-legendaries/titans/main/webserver/provision.sh | bash

This will install all dependencies, establish credentials for a secure HTTPS connection, launch the API, and schedule regular updates and upgrades.

Website

All (most?) website code is contained in titans/website.

Please note that I am not a frontend developer: Cut me some slack if this doesn’t look perfect!

HTML

HTML code is contained in the subfolder html. I wrote my own compiler for HTML code: Comments that include the name of a file substitute in that file’s contents on compilation. E.g. the comment <!-- html/partial/include.html --> in html/constructed.html substitutes in the contents of html/partial/include.html on compilation.

Instructions on how to run the compiler are below.

CSS

All CSS code is in the subfolder style, including code for dynamic scaling to mobile / desktop screens.

JS

All JS code is located in the subfolder script. JQuery was used heavily. This code could definitely use some TLC by a real frontend developer (e.g. within the React framework).

Compiling

To run the compiler, simpy run:

python titans/website/compile.py

and all html files will be compiled and output to titans/website/site.

Assets

There are a handful of image files that are NOT managed by this repo, but are expected to be uploaded to the dev fileserver by other processes. This includes various card images, instructional images, and rules PDFs, among some other scattered files. The code to upload those files lives with each of the respective files.

(Some other assets, like videos and online demo code, are discussed below.)

Deploying

To Dev

Once all code has been compiled, you can deploy from local to dev by running:

python titans/website/deploy.py

This will only deploy the HTML/JS/CSS files, and will NOT deploy other website files (e.g. online demo files, video files).

To Production

After reviewing the files on the dev fileserver, you can deploy from dev to production by running

python titans/website/deploy.py --prod

Unlike the dev deployment code, this will copy over all files from dev to production, including demo files and video files. (The rationale is that, when you move from local to dev, with this script, you’re testing out HTML code; and, when you move from dev to prod, you are testing out all files and code on the server.)

Online Demo

Code for the online demo lives in titans/demo. To upload to the dev fileserver, simply run:

python titans/demo

This code was written in Phaser 3. Potential future upgrades include upgrading the AI system; enabling P2P versus, and including additional cards. However, as the goal of this demo is to serve as a try-before-you-buy, this does NOT have a high priority.

Videos

Several blender videos have been created to demo this project. These use Blender to render, and have been setup to work using Azure Batch, which allows for massively-parallel rendering. (At one point, these were all running sequentially on my desktop, and would take several weeks to finish; now, they complete in mere hours.)

A Dockerfile has been provided for use by Azure Batch, so that each pod doesn’t have to install its own dependencies. To build the docker image and store it on a container registry, run:

python titans/videos/build_img.py --verbose

Once this completes, you can launch various rendering tasks via the command:

python titans/videos

The main commands that should be run, in order, are:

python titans/videos animate
python titans/videos render
python titans/videos convert

Each command should only be run after all jobs on Azure complete. Several of these jobs string together multiple jobs that all depend on one another. (These could have all been strung together into a single job; however, I wanted to encourage users to look at the output of each job before moving onto the next.)

The first command, animate, creates video frames as png files. The second, render, combines pngs to create videos. The third, convert, transforms videos into modern codec formats.

Each of these commands has many command-line options for rerunning only certain subsets. Refer to the documentation of each for help.

Databases

This repo creates and manages a MySQL database with three tables: contacts, comments, and creds.

Comments and creds are tables meant to store user input from the website. These were created via the python command:

python titans/sql/comments.py --create
python titans/sql/contacts.py --create

Some db management is available via flags on those arguments. Use with caution.

Various credentials are also stored in this database. This is probably not best practice, but works, as all dev work happens on this project’s cloud VM, which is the only IP able to access the database. This credentials table was made, and can be updated, via:

python titans/sql/creds.py

Refer to the docstrings for additional information.

API

I setup a cloud VM to act as an API for this project. This API will one day be upgraded to handle the decisions made by the AI for the online demo (which will eventually be expanded into an online client).

The API was setup with FastAPI, and it handles all database queries and insertions made from the website. This includes:

  1. Subscribing to the email list (which adds a user’s name and email to the contacts table)

  2. Unsubscribing (removing name/email)

  3. Commenting (saving comment, and optionally email, to the comments table)

Entries and scanned, sanitizied, and validating before insertions take place.

Emails are additionally automatically sent by the API whenever a comment is left on the website, or whenever a new subscriber joins the email list.

The API can be accessed here. FastAPI docs for this API are located here.

Updates

If you want to test changes to the API before going live, you can update the code, and then run (on the webserver):

webserver/test-api.sh

This will spin up a copy of the API on the webserver at the port 1024, which can be accessed here.

To update the real API, push your changes to GitHub, then run:

webserver/run-service.sh

This command is automatically run every time the webserver restarts.

Emails

This package handles sending emails to all of our subscribers, all from the command line. This uses the MS 365 Graph API to execute sending.

Emails are additionally automatically sent whenever a comment is left on the website, or whenever a new subscriber joins the email list.

Authenticating

For first-time use, you must authenticate with Office365 by following these steps:

In the Azure portal:

  1. Go to Azure Active Directory -> App Registrations

  2. Create a new app registration. Set the redirect URI to http://localhost.

  3. Go to Certificates & secrets, then create a new client secret. Copy the Value, as you’ll only be able to see this once.

  4. Go to API Permissions, and from Microsoft Graph the Delegated Permission of Mail.Send.

  5. Create a ~/secrets/titans-email-creds file that looks like:

    {
    "tenant": "<directory (tenant) id>",
    "client_id": "<application (client) id>",
    "client_secret": "<certifiacte & secrets value>",
}

In this repo:

  1. Make sure you have a SECRETS_DIR environmental variable set, e.g. ~/secrets.

  2. Run titans/email/get-code.sh. Follow the website it points you to, and authenticate with your office account. It’ll redirect you to an error page. On that page, look at the url, and copy the code parameter from the URL (i.e. the portion that reads &code=...&). Paste that code into a $SECRETS_DIR/titans-email-code file in this repo.

  3. Run auth/get-token.sh. This will create a $SECRETS_DIR/titans-email-token file, containing your authentication token.

  4. Finally, update the creds table in sql by running:

    python titans/sql/creds.py

Sending Emails

To send emails, run:

python titans/email yamlconfig

where yamlconfig is a configuration yaml file that is unpacked to initialize titans.email.sender.SendEmails. An example configuration yaml file would look like:

subject: We're Back!
body: 2022-07/body.html
attachments: [
  2022-07/box.png,
  2022-07/divider.png,
  2022-07/final_judgment.png,
  2022-07/logo.png,
]

where 2022-07/body.html is the name of an html file you want to send, and the attachments are names of attachments you want to attach.

Please note that any attachments can be referenced and inserted in body via their cid:

<img src="cid:logo">

and the string #|EMAIL|# will be replaced with the receipient’s email address.

Dev: Testing

To send a test email, run:

python tests/titans/api/subscribe_test.py

Please note that this test will fail unless it’s run on the whitelisted VM!

Polls

Periodically, we run polls out of the email service. To do so:

  1. Create a new table in the database:

    python titans/sql/polls.py table_name

    This table will store the poll responses.

  2. Build the email that will contain this poll. This email needs to have links that the recipient can click on that query the API. These responses have to look like:

    https://titansapi.eastus.cloudapp.azure.com/poll/POLL_NAME?email=#|EMAIL|#&response=RESPONSE

    which includes the parameters:

    1. POLL_NAME`: The name of the poll, which is the name of the table created in the previous step.

    2. EMAIL: The email of the responder, which will be auto-inserted by the email client. (The string #|EMAIL|# is a special string that is processed by the email client.)

    3. RESPONSE: The response to the poll, which should be hard-coded for each clickable link.

  3. Send the email, and check the results as they roll in!