059 - Deploying Again

March 22, 2020

In the last journal entry I wrote about how I'd be moving to Kubernetes.

I'm inexperienced when it comes to the best practices for modern deployments - so landing on Kubernetes came down to choosing something that met all of my current deployment needs in a way that I wouldn't be too coupled to in the long term.

I mention that because I'm hearing that "micro VMs", whatever those are, are gaining traction these days so I'll try to keep my ears open over the coming years to different ways that I can continue to simplify my deploy process while keeping security in mind.

Kind

My main goal with moving to Kubernetes was seeking greater dev-prod parity. I want to continue to gain the confidence that if everything is working on my development machine that everything will be working in production.

I don't actually run the full game very often locally. I'd say that 90%+ of my development time is spent just writing small tests and making them pass.

I'll fire the game up when I want to interact with and feel something out - but even then I just run a script that starts the game on my host machine, not in any docker containers.

So the dev-prod parity that I seek is more of a "If I can get my infrastructure working locally it will also work when I deploy it to production."

I began with minikube which was mostly nice - but had some aspects that left me desiring more.

Mainly the long cluster start up time and that you had to run some script in order to mount volumes. There was also a not-too-ergonomic process for getting local docker images into your cluster - which was my main deal breaker.

I eventually stumbled on kind and it solved all of my problems. Going from nothing to a fully working cluster takes about 2 minutes if my crate dependencies and my docker layers are already cached.

There's room to speed this up in the future if I need to by trying to parallelize some of the setup, but for now a roughly 2 minute cold start is totally fine.

Most of that time is spent running kind load to add different docker images from my local host machine to the cluster, and my docker images are based on debian:buster-slim which is 40 Mb, much larger than the actual binaries themselves.

If I can move to a smaller base image in the future I'd imagine that the cluster startup and preparation time would drop to sub one minute, but this isn't a concern at the moment.

Now I run ak k8s create-new-cluster to start up my cluster locally.

$ ak k8s --help
ak-k8s 0.0.1
Work with our Kubernetes clusters

USAGE:
    ak k8s <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    create-local-cluster    Start a k8s cluster locally using kind
    help                    Prints this message or the help of the given subcommand(s)

I don't expect to use Kubernetes locally very often as just running binaries directly on my host machine has been fine and fast to date.

Really my main use case for the local cluster now for when I want to try something out and get something working before applying it to the production cluster.

In the future I might also have a staging cluster, but I don't need that just yet.

Deploying Again

Last week I mentioned an error where the game was working locally but in production it would mysteriously crash whenever a player logged out.

This ended up being due to me not running the latest database migrations.

I updated the code path that tries to load players from the database to log an error if anything goes wrong.

I also refactored the entity spawning and removal systems to have only a notion of entities instead of treating human players as a separate concept.

This led to things being much cleaner and extensible - so I'm happy with that investment.

I added and tweaked some new and old command in the ak CLI for building production distributions of the different services and uploading binaries to S3 and docker containers to AWS ECR.

I've turned off continuous deployment for now as I need to move our CI/CD pipeline to use these new commands.

I'll give it a little bit of time before I re-jig it so that I can focus on getting some basic functionality deployed after spending quite a bit of time working on non-visible underlying technical progress.

In the meantime I'll be deploying to the production cluster by running the release commands on my host machine.

Other Progress

  • Fixed some issues with our RenderSystem around the rendering of terrain and user interfaces

Next Week

I'm dedicating this week's work to things that players can see and interact with.

I'm starting off my getting some of the user interface in place.

Then I'll create some terrain textures and add more terrain and meshes into the game.

So we'll finally start having screenshots and other visuals in the journal entries again.

Cya next time!

- CFN