High performance vector graphic video games

For a few years I’ve been trying to solve a hard problem: “How can I use vector graphics as the backing image model in realtime systems?”

Yes, the image model that uses points, lines, and equations to encode image data. Of which, several common encoding formats exist, such as like SVG and PDF.

The internet has done this once meaningfully in the past. In the 2000s, a herculean effort to optimize the performance of interactive vector graphics brought an advent of vector games through Adobe’s Shockwave Flash player. But in the wake of SWF’s deprecation from major browsers, there wasn’t a worthy successor, and the era of web games died off. Unfortunately, efforts like HTML5 and WebGL were never able to replace what was lost.

Hence, I believe the first games to understand and successfully use this image model under the constraints of major browsers could benefit from content-delivery savings, resolution independence, and offer novel applications of interactivity, physics, and non-destructive transformations.

Onto the papercuts

Vector images are notoriously unfit for modern GPU architecture because of an inherent locality issue. In contrast to raster graphics where color and fill information is explicitly encoded, rendering a pixel in a vector image requires knowledge of the entire image. Efficiently solving the fill of every pixel in realtime poses a unique challenge. With the rising accessibility of compute kernels and low-level GPU architecture access over the past few years, especially from projects like wgpu, friction with general purpose GPU computing is fading. Projects like vello are pioneering the 2d vector graphics space, and furthering the experimental research inspired by projects like pathfinder.

The high-level strategy used by these new hardware-accelerated renderers is to create a compute-centric pipeline with a sort-middle architecture to parallelize the problem space efficiently. This is a technical leap in recent years and its efficiency will be hard to challenge.

Vong

The technical leap was implemented in vello (formerly piet-gpu), started by Raph Levien and my colleagues in the linebender community. The Vello API depends on WebGPU to provide an abstraction layer to targets, while offering more accessibility to low-level hardware like compute shaders. It’s for these reasons rust, webgpu, and vello would make a perfect combination for cross-platform vector games with one code-base.

This spurred on my race to prove the lost benefits are still relevant. In a sudden case of reviving-the-horse syndrome, I worked with a few colleagues from my previous tenure at NASA to understanding the unknowns of why I’ve never seen a modern Kurzgesagt vector game. Hence, the decision to start by creating the classic game of Pong was deliberate, providing a foundational exploration of Vello’s capabilities in rendering vector graphics dynamically. It felt appropriate to call the project Vong, joining “Vector” and “Pong”.

Matching webgpu, vello, and rust’s strengths, bevy was the obvious choice for a cross-platform open source game engine. It also uses webgpu, and was developed open-source and code-first.

Rendering integration

Standing in the way of seeing my first Ghostscript Tiger (the Hello, World! of vector graphics) was an integration to render vector assets in bevy with vello. Sebastian Hamel (@seabassjh) and I (@simbleau) developed that integration in the open, bevy-vello. Just like any other ordinary asset in bevy, such as a png, there are no surprises:

// Bevy 0.12
fn setup_assets(
    mut commands: Commands,
    asset_server: ResMut<AssetServer>
) {
    // Load image of egg
    commands.spawn(
        VelloVectorBundle {
            vector: asset_server.load("egg.svg"),
            origin: bevy_vello::Origin::Center,
            transform: Transform::from_scale(Vec3::splat(0.1)),
            ..Default::default()
        }
    )
}

Architecturally, bevy-vello was built with two backends. The first is an .svg ETL library called vello-svg which loads the SVG using the vello SceneBuilder API. The second is vellottie, a parser and runtime for Airbnb’s .json lottie format, and a rewrite of Chad Brokaw’s lottie runtime, “velato”.

Strategically, it made sense to pursue both formats, but a painful lesson learned was that the .svg and .json lottie specifications are huge, with plenty of feature flags that aren’t supported by vello. While writing a parser wasn’t too challenging using the serde parsing library, dealing with animation inconsistencies and rendering artifacts remains a difficult task. Currently, vello is writing CPU shaders to help triage these artifacts. But for the future, I’m inclined to support only lottie formats, since any SVG could be reformed into a 0 frame-rate lottie animation. One of the biggest obstacles in commercial viability is artist tooling, so I’m keeping an eye on companies like LottieLab, Phase, and Rive.

Physics

Once rendering was solved, another thorny issue with pong was physics: collision detection between arbitrarily curved shapes is really hard. While some algorithms exist, none today are obviously good.

Eventually I settled on tessellation as a decent proxy for physics. The technique is quite simple in concept.

First, curves are flattened into line segments with configurable accuracy (aka “tolerance”).

Attribution: Image from Lyon, licensed under MIT/Apache 2.0

Then, vertices are paired to generate a triangle mesh. The resulting triangle mesh is used to derive a convex hull hitbox.

Attribution: Image from Lyon, licensed under MIT/Apache 2.0. Modified by Spencer Imbleau.

This is obviously not true collision detection between arbitrary curves with infinitessimal precision, but was rather a compromise given a lack of algorithmically efficient collision detection between curves.

As with most things in game development, this is a hack, but a good one! Even when using a liberal tolerance for curve flattening, physics yield a high level of collision fidelity, nearly indistinguishable from the true underlying curves. Granted, this only works great since my egg is convex in nature.

The lyon library was used for tessellation and bevy_rapier was used for collision detection, linear velocity, and angular momentum.

Demo

Is the frame immediately below solid black? Vong uses compute shaders. Make sure your browser is updated to Chrome M113 or another browser compatible with WebGPU!
Edit 1/31: It seems Firefox Nightly, even with the WebGPU flag, is having issues. For now, please use Chrome.

Controls:

• Up/Down (Arrow keys): Scale camera
• PgUp/PgDown/Home/End: Free camera
• W/S: Left bacon
• I/K: Right bacon
• C: Watch egg intensely

You may find the source code here. You may also build and play this natively with cargo run.

What now?

I look forward to following up with future development on vector games. I’m committing this year to publishing more about where this is all leading, but for now, please keep in touch. The next blog planned is on WebRTC to bring our games to life with multiplayer.