048 - December 29, 2019

A yak shaving adventure

This week started off with a simple goal - but very rapidly morphed into a yak shaving adventure.

I originally set out to finish up getting pathfinding working between arbitrarily sized sets of tiles (so pathfinding between a 2x2 entity to a 3x3 entity, for example) and after a good bit of elbow grease things started falling into place.

Pathfinding tests passing Got all of the pathfinding tests passing at my parents house on the night of Christmas - put me right to sleep.

After that my goal was to start rendering scenery into the game - reading from the scenery.yml file that I talked about last week.

The game sever doesn't know about scenery (it just knows about the tiles in the world that the scenery is making unreachable, but it doesn't know or care why) but scenery and entities both use the same renderers on the client side - so naturally they share some data structures.

This is where the yak shaking deep dive began. The game server would previously send down a Renderable3dComp to the client - a component that describes how to render an entity.

This broke one of our architectural rules - the game server should not knowing anything about the 3D nature of the clients. From a data perspective - the game is entirely two dimensional (just a big tile grid with lots of different entities). Our clients take that information in order to render a 3D world - but the server should never need to know about that.

While modifying the codebase to render our scenery I decided to fix this knowledge leak. I introduced a RenderableId enum that is auto generated from build script that reads from a yaml file.

Here are some snippets showing how that's put together.

###################################################################################################

A map of RenderableId -> Render3dComp

###################################################################################################

RenderableId
Snail:
  RenderDescription
  Absolute:
    mesh_names:
      - Snail

SnailBody:
  Relative:
    relative_to: Snail
    rotation:
      - 0.0
      - 0.0
      - 1.5708
    scale: 0.85
    disable_animation: true
// One of our build scripts
fn main() {
    // ... snippet ...
    generate_renderable_id_enum();
    // ... snippet ...
}
//! RenderableId

include!(concat!(env!("OUT_DIR"), "/renderable_id_enum"));

One slightly pesky thing about using the include! concat! technique for code generation is that intellij-rust doesn't currently auto-complete the code - but hey it gives me so many other powerful features that I'm certainly not going to complain.

Now all our server knows is that an entity can have a RenderableId - accomplishing our goal of minimizing the amount of information that the server needs to know about the nature of our clients.

Our client then uses these RenderableIds to look up Render3dComps when rendering the game.

User Equipment

When loading up a player into the game I was previously stuffing their Render3dComp with some MeshName enums for their head, torso, legs, etc.

This broke down when the server no longer knew about the MeshName enum (because we don't want the server to know about anything related to drawing the game).

So I did something I've been meaning to do and introduced a user_equipment table (where we persist your worn equipment) and some data structures around equipment.

The server know sends down an EquipmentComp component with different EquipmentIds in the EquipmentSlots and the client calls EquipmentId.into::<RenderableId>() in order to figure out how to render it.

By now I take for granted how easy serde makes it to share data structures / code between the client and serve .. but as I type this journal entry I'm feeling a sense of appreciation for that amazing project.

Specs ECS

Back in mid October we started gearing up to use specs ECS in our game-server - as it was very clearly superior to my own hand rolled ECS in every way.

commit 3fb152f3c6e454f369c3bde729448463c0e9678f
Author: Chinedu Francis Nwafili <frankie.nwafili@gmail.com>
Date:   Tue Oct 15 10:49:33 2019 -0400

    Add specs dependency

    Need to start using it in a future PR. Just adding the dep for now

Since then it's been such a pleasure to work with and re-shaped how I structure and approach my code so nicely that I'm ready to start using it in on the game client side.

I've introduced the dependency to the game-app crate in my workspace (powers our web browser client and in the future will power clients on other targets such as mac or windows) and will be slowly moving move things into my specs::World over time.

Fortunately there are no design constraints that force me to make this move all at once like there were on the server side - so I can just casually migrate over the coming months as I touch different parts of the client.

Introducing our first browser test

I've been in a test driven development cadence since around August or September of 2019 and I'm excited as I see my code quality rising to the next level.

These days the only time I spend debugging is when I'm getting a new test that I'm writing to work or when I'm touching old code that I wrote earlier in the codebase's life before I was testing.

So - now that I'm gearing up to work on the renderer and make some client side changes I wanted to extend this TDD to the client side (I've been heavy on the server side for the last couple of months).

When I'm working on the game server I never run it outright. I'm always either running a unit test or running an integration test (so far there's only one major test in our game-server-integration-tests crate that plays through the first quest in the game - but there are some smaller integration tests sprinkled around.)

On the web client though - it was a different story. I wrote about using test-driven development on our game UI back in September - but even then I eventually had to boot up the game to see if it all looked right to a human eye. (For example - does the spacing look too large, too small?)

My vision around this is to instead be able to automatically generate a screenshot of a UIElem that I'm working on - or any other visual aspect of the game that I'm touching such as when I'm working on shaders.

I write some unit tests - then when I'm ready to see the thing I call some function that will produce a rendered screenshot of whatever I need to see.

Note that I'm being general as I describe this. I'm not looking to prescribe a solution just yet. I just know that as I dive into the client side I'll be automating the process of being able to visualize whatever I'm working on (even if it's in baby steps over time) and I'm excited for whatever I end up landing on.

A step in the right direction

The first step in this direction is a new crate in the workspace called web-client-integration-tests.

Whenever I'm ready to deploy a bigger change to the game I'll usually fire it up and do a quick sanity check that it loads.

This is a violation of my longer term goal of never needing to run the game myself to check if things that I already built are still working (I should mainly only run the game when I want to feel out the new gameplay or just generally feel out the overall gameplay experience).

To combat this we used rust-headless-chrome to create our first browser test.

It more or less just fires up the game in Chromium and verifies that the canvas renders.

/// Start a game-server
///
/// Start a static server to serve:
///   1. The web-client HTML, Wasm and JS
///   2. The game assets such as mesh and armature data
///
/// Compile the web-client in release mode with debug symbols enabled
///
/// Load the game in Chromium and wait until the canvas gets painted.
/// The first time the canvas is painted the web-client appends an element to the DOM as
/// confirmation - so we wait for that element to appear.
///
/// Check the console and verify that there are no errors or warnings of any kind.
///
/// Take a screenshot of the canvas and verify that we rendered to the canvas.
#[test]
fn load_game_without_errors() -> Result<(), failure::Error> {
    thread::spawn(|| start_game_server());
    thread::spawn(|| start_static_server());

    compile_web_client();

    let browser = init_browser()?;
    let tab = browser.wait_for_initial_tab()?;
    tab.enable_log()?;

    let errors_and_warnings = Arc::new(Mutex::new(vec![]));
    begin_monitoring_chrome_console(&tab, errors_and_warnings.clone())?;

    tab.navigate_to(web_client_url().as_str())?;

    wait_for_first_canvas_render(&tab)?;

    assert_no_errors_or_warnings(errors_and_warnings);

    assert_webgl_rendering_occured(&tab)?;

    Ok(())
}

Fail if there are console errors or warnings After I got the test running there was an (expected) failed assertion around there being warnings in the console. I needed to add a Favicon and not try to render meshes until the textures were downloaded.

I like to lean on as many unit tests as possible and only write a heavier integration test for critical code paths - so I can't say when or how often we'll be adding new browser tests.

Ideally there will be very few browser tests - and most of our server/client integration testing will happen outside of a browser (we already have one now for the first quest in the game - it connects to the game server using a crossbeam channels).

Here's one of our integration testing data structures that doesn't use a browser:

/// A player that is connected to our game server via a mpmc crossbeam channel,
/// similar to how a real player might be connected via websocket.
///
/// ## Persisting Player To Database
///
/// We don't currently persist our simulated player to the database - although in the future
/// we might want to add a way to do that in case we want to have integration tests that
/// verify that the database is updated correctly.
pub struct ConnectedPlayer {
    /// The user_id for the player
    ///
    user_id: u32,
    /// Every game tick the server sends state to all connected players.
    /// This is how we receive that state.
    state_receiver: Receiver<Vec<u8>>,
    /// Used to send ConnActivity::* to the game server.
    ///
    /// For example, this might be a new ClientRequest, or a ClientDisconnect.
    conn_activity_sender: Sender<ConnActivity>,
    /// Receive an acknowledgement that our connection activity was received and processed.
    conn_ack_receiver: Receiver<()>,
}


I always enjoy writing the first integration test for some major code path as it typically exposes a lot of unnecessary coupling or inflexible code and makes for a good learning experience and spring cleaning.

But I definitely default to carefully thinking about whether I can accomplish what I want to accomplish with unit tests before I dip my hand into the integration testing realm. It's something to be used sparingly/appropriately in my opinion as it's inherently much more coupled to your codebase.

Other Changes that I remember

  • Fix the impl Iterator For TilesBetweenWorldPoints implementation - it was broken in a few cases.

  • Center entities within their TileBox when rendering on the client.

  • Allow the game to load player data from a provided HashMap - useful for our integration tests so they don't need to hit the database.

This Week

This week I'll be starting to populate the client side specs::World - with our first client side components, resources and systems revolving around rendering our game scenery.

After that I can finally get back to finishing modeling Pookie's house in Blender and getting that scenery rendered in the game.


Cya next time!

- CFN