angelo.dini.dev Blog

Fretfall Player, a Note Highway in Plain JavaScript

Sat, 26 Sep 2026 00:00:00 GMT

Notes that fall towards you

A tab tells you which fret to press on which string. It's bad at telling you when, and it says nothing at all about which finger. You can read every number on the page and still play the song like someone reading a recipe out loud.

I play guitar. It was on the list of habits I tracked back in 2020. What a static page of tab can’t give you is the song and the next note at the same moment. Arcade-style note highways do exactly that: the notes come down a lane towards you and you play them when they reach the line. So I built one that opens ordinary tab files, the Guitar Pro and MusicXML kind that are already everywhere.

It’s called Fretfall Player. It runs at fretfall.com, the code is on GitHub under MIT, and it’s published to npm as @fretfall/player. This post is a tour of how it works under the hood, because the interesting parts turned out to be the maths and not the UI.

What it does

You open a Guitar Pro file (.gp, .gp3 to .gp5, .gpx) or MusicXML (.musicxml, .mxl) and the notes start falling down a 3D fretboard. Hand positions are drawn on the neck, every note carries the finger that should play it, chord names sit beside the shapes, and the fret numbers are painted on the floor of the highway like road markings.

Then you can drop the band’s recording on it (.mp3, .m4a, .ogg, .wav or .flac) and the player works out on its own where in that recording the tab starts. From then on you play along with the real song instead of a synth.

Beyond the highway there are two more views: a tablature, scrolling or a page at a time, with the rhythm written over the staff; and a notation page, which sets the tab as systems of whole bars with the technique marks, chord names and lyrics. For practice there are phrase loops, a speed control with a speed trainer, and a metronome. And there’s a band mode that opens every other part of the song in a window of its own, all playing in time.

The house rules

Two rules are written in the README and enforced by CI, and they shape everything else:

No framework and no runtime dependencies. Plain HTML and ES modules. The whole player is about 70 kB gzipped.
No pull request may make it slower or bigger.

The second one is checked properly. scripts/bench.mjs builds the player and the base branch, then runs both in turns on the same runner, so the machine’s ups and downs hit them alike. It measures frame time and draw calls of the highway, the tablature and the notation page over a whole song, how long lining up a recording takes, the fingering, and the gzipped size. More than 10% worse on frame or sync time, 5% on draw calls or 1% on size, and the build fails. A regression taken on purpose needs the perf-ok label.

The canvas in that benchmark is a Proxy that draws nothing and only counts calls, so the number it reports is the time spent in the player’s own code and not in the GPU. It also means the benchmark runs in plain Node, no browser needed.

There is one honest asterisk on “no dependencies”: reading and synthesising the tab files is done by alphaTab, which the page loads from jsDelivr at runtime, together with its SoundFont. It’s not bundled and not in package.json, but the player can’t open a Guitar Pro file without it. Writing a Guitar Pro parser from scratch was never on my list, and alphaTab does it well.

The only dev dependencies are esbuild and oxlint. Tests are node --test with plain node:assert, no framework.

The modules

There’s no build step for development beyond esbuild serving src/ on port 4410. The repository is small enough to describe in a table:

Module	What it does
`player.js`	the page: opening files, alphaTab, the transport, practice tools, settings, band windows, the public hook
`highway.js`	all three views drawn on a canvas: the 3D highway, the tablature and the notation page
`music.js`	pure helpers: tempo maps, tunings, note names, repeated chords
`fingering.js`	hand positions and fingers for charts that don’t say
`sync.js`	lining a recording up with the chart
`themes.js`	the looks: colours, fonts, note shapes
`mount.js`	the player in an element of someone else’s page

music.js, fingering.js and sync.js touch no DOM at all, which is why they each have a test file next to them that runs in plain Node. The two big ones are player.js, the glue, at about 3,500 lines, and highway.js at about 2,800.

The highway is a canvas, not WebGL

The 3D highway is a 2D canvas with a hand-rolled perspective camera. The comment at the top of highway.js describes it better than I can: a camera looks down the highway from just behind the strike line, so the strings there read as a fretboard and the notes rise out of the distance. World units are fret widths, and time runs along the z axis.

Every frame, a few constants turn time into distance. Notes travel 28 fret widths per second, and the highway shows 84 fret widths ahead of the strike line, so with the default settings you see three seconds of music coming at you:

const VH = 900, HIGHWAY = 84, NOTE_SPEED = 28;

export const lookAhead = (view) =>
  (HIGHWAY * (view.drawDistance ?? 1)) / (NOTE_SPEED * (view.noteSpeed ?? 1));

The projection itself is a one-liner per point: subtract the eye, take the dot product with the camera’s forward vector, and divide by it. Because that function runs thousands of times a frame, the camera vectors are unpacked into plain local variables before the loop instead of being read out of arrays each time. It’s the kind of thing you’d never write in a tutorial and it shows up straight away in the benchmark.

The camera follows the hand. Each hand position coming up in the next three seconds gets framed, a big move zooms out to show both the old and the new position, and then it eases back in. The easing is a critically damped spring, the same maths as Unity’s SmoothDamp, so the camera never lurches even when the target jumps.

Things a canvas is slow at

Most of the comments in highway.js are about what was too slow, which is my favourite kind of comment:

Shadow blur. A glow with shadowBlur is a separate pass for every shape, and it halved the frame rate. A glowing shape now gets its outline stroked three times underneath it, wider and fainter each time, which falls off the way a blur does.
Big gradients. Filling the floor with a gradient was the slowest thing to draw without a GPU. The floor is now an image one pixel wide with bands of flat colour, stretched over the floor without smoothing: one draw call where there used to be 128.
Text. Short text is set at quarter-pixel sizes and measured once per font and string, because a font at a size the canvas hasn’t seen before is slow to set up.
Reading state back. Reading g.font back from the context is slow, so the code keeps track of what it last set.

None of that is clever on its own. It’s just the list of things a profiler finds once you have one, and adding ?perf to the URL turns on a frame profiler for exactly that reason.

Fingering, worked out

Most tabs say which fret, but not which finger, and a note highway without fingers is only half the help. fingering.js fills that in with about 60 lines of dynamic programming.

The model is the classic one-finger-per-fret position. The index finger rests on the position fret and fingers one to four cover four frets:

const SPAN = 4;

export const fingerFor = (fret, position) =>
  Math.min(4, Math.max(1, fret - position + 1));

The harder question is which position to be in. Notes are grouped by what’s played together (within five milliseconds of each other), open strings are skipped because they don’t pin the hand, and each group gets the list of positions that would reach all of its notes. Then the whole song is solved at once: every possible position of every group gets a cost, a shift costs 1 + 0.3 per fret moved, a shift during a rest of more than 0.4 seconds costs half, and there’s a small preference for putting the index finger on the lowest note when it fits. Walk the cheapest path back from the end and you have a hand position for every moment of the song.

It isn’t how a teacher would finger every passage, and I wouldn’t claim it is. But it keeps shifts few and short, which is most of what I want when I’m learning something new, and a chart that does say which finger wins over the guess.

Lining up the recording

This is the part I’m most pleased with. You drop an MP3 on the player and it has to answer two questions: at what second of the recording does the tab start, and is the band playing at exactly the tempo the tab says? Live recordings drift, tabs are transcribed by people, and a count-in or an intro of applause moves everything.

sync.js models the answer as a straight line:

audio time = offset + ratio × chart time

and looks for the offset and ratio that put the chart’s notes on the recording’s attacks. It goes like this:

Decode small. The file is decoded with an OfflineAudioContext at 11,025 Hz and mixed to mono. The comment says 11 kHz is plenty to find attacks, and it is.
Build an onset envelope. The samples are cut into frames of 1,024 with a hop of 128, each frame goes through a Hann window and a radix-2 FFT (written inline, about 25 lines), and the spectral flux between 150 Hz and 5 kHz says how sharply the sound changes at each moment. Whatever sticks out above the surrounding 0.8 seconds is an attack.
Coarse search. For every tempo ratio from 0.9 to 1.1 in steps of 0.0025, the chart’s onsets become spikes on a grid, and their cross-correlation with the envelope is computed with FFTs. One pass scores every possible offset at once, which is the trick that makes it fast enough to run in a browser while you wait.
Fine search. Around the best coarse result, ratio and offset are searched directly in much smaller steps.
Confidence. How far the best fit stands above the envelope’s average. Below about 2, the recording probably isn’t the song the tab describes, and the player says so instead of pretending.

The player then wraps the <audio> element in a small clock object, so everything else in the app (the loop, a seek, the band windows) asks for chart time and never sees the recording’s own:

get time() {
  return (el.currentTime - offset) / ratio - songOffset / 1000;
},
seek: (t) => {
  el.currentTime = Math.max(0, offset + ratio * (t + songOffset / 1000));
},

songOffset is a nudge in milliseconds on top, for when the fit is close but your ears disagree. fretfall.snap() sets it automatically: it puts the tab’s first note on the loudest attack within 0.3 seconds of where the fit expects it, or on the recording’s first loud sound if the fit wasn’t sure. The comment admits that a count-in can win that second case, which is true and a little annoying.

The audio element has preservesPitch on, so slowing the recording down for practice doesn’t drop it into a lower key.

Practice tools

The practice tools are the point of the whole thing, and each of them is small.

Phrase loops. Above the highway, every bar of the song is a bar in a strip, and its height is how many notes are in it. Busy bars stand out at a glance, which is usually where a loop is needed. Drag across the strip and the loop snaps to whole bars (hold Shift for a free range). When playback passes its end, it jumps back to the start with one second of run-up, so you’re not dropped straight onto the first note.

Speed and the speed trainer. Speed goes from 10% to 150% in steps of 10%. The speed trainer only switches on with a loop and a speed of 90% or less, and then it adds 10% every time the loop comes round, up to 100%, where it switches itself off. That’s the whole feature:

if (loop && player?.playing && t >= loop.end && !following) {
  player.seek(Math.max(0, loop.start - 1)); // one second of run-up
  if (training) setSpeed(speed + 0.1); // the speed trainer: faster each time round
}

The metronome. It clicks on the chart’s beats, not on a fixed tempo, so it follows tempo changes in the song. The clicks are short oscillator blips (1,600 Hz on the first beat of a bar, 1,000 Hz otherwise) scheduled on the Web Audio clock a little ahead of when they sound. The frame loop asks for the beats in the next tenth of a second and books them in advance, so the click keeps time however the frames fall. It follows the volume but not the mute, so you can mute the song and play against the click alone.

Band mode

Band mode opens every other part of the song in a window of its own, and one window leads. The windows talk over a BroadcastChannel called fretfall-band: the leader sends the song, play, pause, seek, speed and its clock, and the others follow. It’s the simplest thing that works, and the code says so in a comment, with the obvious next step if it ever has to reach another machine:

// ponytail: windows of one browser on one computer (BroadcastChannel), a server if players join from elsewhere
const band = new BroadcastChannel("fretfall-band");

Nobody has asked for a server yet, so there isn’t one.

Building on it

The player starts with no song of its own. The page around it opens the first one, which is what the demo’s front door does with its demo tab (an original practice piece of about two minutes that uses every notation the highway draws, from palm mutes to golpe). Once it’s ready, the player exposes window.fretfall:

if (!window.fretfall)
  await new Promise((ready) =>
    addEventListener('fretfall:ready', ready, { once: true })
  );

addEventListener('fretfall:song', (e) => console.log(e.detail.title, e.detail.parts));
await fretfall.open(files, { audio: recording });
fretfall.part = 1;

Most of what the controls do is settable on that object (playing, time, speed, loop, metronome, view) and one fretfall:controls event fires whenever any of it changes. For a page that isn’t the player’s, a React app say, mount(element) from @fretfall/player/mount.js puts the whole player into an element, with its styles scoped to it, and fretfall.dock(element) moves the header’s controls into the page’s own bar. There’s no unmount: the modules run once per page, so you hide the element instead of removing it. That’s written down in the README, so nobody has to discover it the hard way.

Desktop apps are a different story, as I found out with Tauri. This one is a static folder: npx serve node_modules/@fretfall/player and you have it.

What still bugs me

The fingering is a model and not a teacher. It counts frets, so it knows nothing about stretches that are fine high up the neck and painful at the nut, or about a hand that would rather stay put and reach. The sync fits one straight line through the whole song, one offset and one tempo ratio, so a band that speeds up in the chorus is something it can only average over. And because alphaTab and its SoundFont come from a CDN, the first file you open waits on that download.

The README says a bug report with a tab that reproduces it is as useful as a patch, and I mean it. If you play and have Guitar Pro files lying around, try it at fretfall.com and send me the tab that breaks it.

‘Till next time!

A Desktop App with Tauri 2

Sat, 22 Aug 2026 00:00:00 GMT

The browser was never going to do this

Take a photo of a printed sheet-music part, get it back transposed into a different key and clef, with every accent and repeat intact, and print it. That's the whole product. It needs a 200 MB recognition engine written in Java, a PDF rasteriser, a local web server for the phone, and the ability to spawn and kill processes. None of that runs in a tab.

So I built a desktop application. It’s called Semitone, it’s not public yet, and it’s a React app inside Tauri 2 with a Rust backend that does everything the browser can’t. This post is the Tauri half: what a React developer needs to know, what I’d do again, and the bits that ate whole days.

The example that started the project is a 1. Flügelhorn in B part from a brass-band folder that has to be played from bass clef in C. Today the options are transposing by hand, re-entering it in a notation editor, or not playing. I’ll leave the music side for another post, because the desktop side is plenty.

What Tauri actually is

Tauri gives you a native window with a system webview in it (WebKit on macOS, WebView2 on Windows) and a Rust process behind it. Your frontend is whatever you’d build for the web. Mine is React 19, Vite, Tailwind v4 and shadcn/ui, with Vitest for tests, and if you removed Tauri from the repository it would still start in a browser and render every screen. The Rust side exposes functions the frontend can call, and that’s the entire contract.

npm create tauri-app@latest

That scaffolds both halves. The frontend lives in src/, the native side in src-tauri/, and tauri.conf.json describes the window, the bundle and the security policy. The dev loop is npm run tauri dev, which starts Vite on a fixed port and opens a window pointed at it. Hot reload works. Rust changes recompile and restart the window, which takes a few seconds and is the one place where the loop is slower than a web project.

Two Vite settings matter and the template sets both. The port is fixed and strict, because Tauri expects to find the dev server where it said it would be, and Vite is told to ignore src-tauri/ so a Rust edit doesn’t trigger a frontend rebuild.

Tauri or Electron

The question every React developer asks first, so here is the answer before the details. I chose Tauri and would again, but not for the reasons the comparison articles lead with.

	Tauri 2	Electron
Renders with	the webview the OS already has: WebKit on macOS, WebView2 on Windows, WebKitGTK on Linux	its own copy of Chromium, bundled into every app
Backend	a Rust process, functions exposed as commands	a Node.js process, the same JavaScript as the frontend
What you ship	your app and a thin runtime; Semitone’s own binary, frontend, fonts and engraver come to 8.4 MiB compressed	your app plus a browser and a Node runtime, so the installer is heavy before you have written a line
Rendering	can differ per platform, because three engines	identical everywhere, because one engine
Mobile	iOS and Android builds from the same project	desktop only
Permissions	denied by default, granted per window in a capabilities file	the renderer can be locked down, but you do the locking
Age	younger; signing, updater and plugins work, and you will read their source	older; the signing and update story has been walked by thousands of apps

Size and memory are the numbers everyone quotes, and they are real: Electron carries a copy of Chromium and Node in every app, Tauri borrows the browser the OS already has. For Semitone the argument barely registers. A 200 MB recognition engine sits next to the binary either way, and nine tenths of every download is Java. That table would have looked much the same with Electron, give or take a browser.

What decided it was the backend. Everything hard in this app is process management, file handling and a PDF rasteriser, and I wanted that in a language whose compiler takes ownership seriously and refuses to build a data race. Rust was the point, not a consequence of picking Tauri.

Where I would pick Electron: a UI that must look identical on every machine, because three webviews are three renderers and the CSS fix you make on macOS is a new bug on Windows; or a backend that leans on the Node ecosystem, where the library you need exists in npm and not on crates.io. Neither applied here. The rendering differences are the honest downside in that table, and they are why the web build of this app is a plausible future rather than a promise.

Commands, the door between the two worlds

A command is a Rust function with an attribute on it:

#[tauri::command]
pub fn host_info() -> HostInfo {
    HostInfo {
        platform: std::env::consts::OS.to_string(),
        app_version: env!("CARGO_PKG_VERSION").to_string(),
    }
}

import { invoke } from '@tauri-apps/api/core';

const info = await invoke<HostInfo>('host_info');

Arguments and return values go through JSON, so anything that derives Serialize on the Rust side arrives as a plain object. The naming convention differs between the two languages, so nearly every struct I return carries #[serde(rename_all = "camelCase")], and errors are enums tagged with a code field so the frontend can switch on them rather than parse sentences.

Long work must not run on the thread that owns the window. Marking a command async puts it on Tauri’s runtime, and CPU-bound work goes one step further to the blocking pool:

#[tauri::command]
pub async fn export_pdf(svg_pages: Vec<String>, destination: String) -> Result<(), String> {
    tauri::async_runtime::spawn_blocking(move || {
        let pdf = svg_pages_to_pdf(&svg_pages).map_err(|e| e.to_string())?;
        std::fs::write(&destination, pdf).map_err(|e| e.to_string())
    })
    .await
    .map_err(|e| e.to_string())?
}

Laying out a twenty-page part is not something to do on a thread other commands are waiting on, and the frontend’s progress bar is one of those commands.

For the other direction, Rust to frontend, there are events. The recognition engine runs for minutes and writes a log, so the Rust side follows the log and emits progress events that the React side subscribes to with listen. Same mechanism when the phone submits a photo, and when a second document is double-clicked while the app is already open.

The rule I’d keep even if I threw everything else away

Nothing in the frontend imports @tauri-apps/* except one folder, src/platform/. It exports a Host interface, every invoke in the application lives behind it, and the rest of the app asks for getHost() and doesn’t know what answers.

The reason is that a web version is a plausible future, and with this boundary it is one module swap: an HTTP-backed host instead of a Tauri-backed one. Without the boundary it’s a rewrite. So it is enforced by ESLint rather than remembered:

{
  files: ['src/**/*.{ts,tsx}'],
  ignores: ['src/platform/**'],
  rules: {
    'no-restricted-imports': ['error', {
      patterns: [{
        group: ['@tauri-apps/*'],
        message: 'Reach the host through src/platform instead of calling Tauri directly.',
      }],
    }],
  },
}

The second lint rule in that config is the one I’d recommend to anyone shipping a product in two languages. react/jsx-no-literals with noStrings: true makes a hardcoded user-facing string a lint error. Every string comes from a catalog through t(), the English catalog is the authoritative set of keys, and the German type is derived from it, so a missing translation is a type error rather than a blank label found by a user. I wrote about switching to Biome last year and I still like it. This project stayed on ESLint because both of those rules are the reason the codebase has the shape it has, and I wasn’t going to give them up for a faster linter.

Capabilities, or why your dialog doesn’t open

Tauri 2 replaced the old allowlist with capabilities. A JSON file per window says which permissions the webview gets, and by default it gets almost nothing:

{
  "identifier": "default",
  "windows": ["main"],
  "permissions": [
    "core:default",
    "dialog:allow-save",
    "dialog:allow-open",
    "dialog:allow-ask",
    "dialog:allow-message",
    "core:window:allow-destroy"
  ]
}

This is the first thing to look at when a plugin call silently does nothing. Your own commands don’t need entries here, but every plugin API does, and the error when you forget is not always loud. The model is sound: the webview is running content you control today and possibly content you don’t tomorrow, and a save dialog it wasn’t granted is a save dialog it can’t open.

The content security policy lives next to it in tauri.conf.json. Mine needed 'wasm-unsafe-eval' in script-src because the engraving library is WebAssembly, blob: in worker-src because it runs in a worker, and the asset: protocol in img-src so the app can show a page it wrote to disk. Each of those fails silently when it’s missing: a blank pane and nothing in the console.

Bundling a 200 MB Java program you are not allowed to touch

Recognition is done by Audiveris, an open-source optical music recognition engine. It is written in Java, it ships with its own trimmed runtime, and it is AGPL. Semitone is not.

That combination is workable only under strict process separation, and I wrote the rules down before writing the code. Audiveris runs as a subprocess. Arguments go in on the command line, MusicXML comes back out of a directory, and that is the whole interface. No JVM linked into my process, no calls into its classes, no parsing its project files, no source patches, no forks. The launcher and its runtime are vendored into the bundle by a script that downloads the exact upstream release, checks it against the digest upstream publishes, and unpacks it without modification.

npm run audiveris:vendor

It is deliberately absent from the licence inventory, because the inventory describes what is compiled or bundled into Semitone and the whole position rests on the answer being “none of it”. The position itself is written up for a lawyer to read before anyone pays for this. I’m a developer, not a lawyer, and the honest version of that sentence is that I built the process boundary as narrow as it can be and then stopped guessing.

On the Rust side the interesting problems are the ones every subprocess has and nobody plans for. A run takes minutes, so the log is followed to make the wait legible. The user can cancel, so the child is in its own process group and the signal reaches the whole tree; on Windows that means taskkill, an external program that can be missing, with Child::kill as the fallback. Quitting during a run must not leave an orphan JVM behind. And the working directory is removed on every path out, so a cancelled job leaves nothing on disk.

The engraver, and a different licence

Engraving, turning MusicXML into a picture of music, is done by Verovio, compiled to WebAssembly and running in a Web Worker so the UI stays responsive. Verovio is LGPL, and its WASM is inlined into its JavaScript as a string, so bundling it into the app chunk would be a lot closer to static linking than to dynamic.

The answer is to not bundle it. A script copies the two files verbatim out of node_modules into public/, where Vite treats them as static assets loaded at runtime, and it fails the build if the version in package.json isn’t pinned exactly. It runs on predev and prebuild, so nobody has to remember it.

I mention both of these because licensing is the part of a desktop app that a web developer has never had to think about. On the web you serve, you don’t distribute. The moment you ship a binary, every dependency is something you’re conveying, and the difference between “an aggregate” and “a combined work” decides whether you have to publish your own source.

The licence gate

Which is why there’s a CI job for it. Two scripts generate an inventory of every npm package and every Rust crate that ends up in the bundle, with its licence text, and a third checks them against a list of what we permit. Not what we recognise, what we permit. AGPL, SSPL, Commons Clause, non-commercial terms and packages with no licence declared all fail the build, and so does anything nobody has thought of yet, because it isn’t on the list.

npm run licences:npm
npm run licences:cargo
npm run licences:check

The committed inventories have to match the tree, so CI regenerates them and diffs. The app shows the whole inventory under a Licences entry in the header, which is about 750 kB of licence text in a chunk of its own that Vite would otherwise warn about on every build.

It also shaped the Rust dependencies more than I expected. The PDF rasteriser is hayro, a pure Rust crate, because the obvious choices are MuPDF and Poppler and both are copyleft, and the third is a C++ blob I’d have to vendor per platform. Most crates have default-features = false, partly for size and partly because the default feature set of an image library is twenty decoders, every one of them a parser running on a file a stranger sent you.

Things I measured instead of assumed

The release profile has panic = "unwind" where the template had abort. Abort is smaller. But the app runs two parsers over files that arrive from outside, images and PDFs, and the “we show a message rather than crash” behaviour only exists if a panicking task can be caught. With abort, the same panic takes the process down with the user’s corrected part in it, unsaved since the last thirty-second autosave. I measured the cost: 9.27 MB to 11.05 MB, plus 19 percent on my binary. Against the 200 MB engine sitting next to it, that’s under one percent of the download. Crash safety on hostile input is worth nine tenths of one percent.

The updater was the other one. Tauri’s updater doesn’t patch, it replaces the whole bundle. So an update from 0.1.0 to 0.2.0 downloads 101.5 MiB, of which 93 MiB is a recognition engine that hasn’t changed. Semitone’s own binary, frontend, fonts and engraver come to 8.4 MiB compressed. The update is twelve times larger than it needs to be, and for now that stands, because splitting the engine into its own installable piece is a real build and the release cadence doesn’t justify it yet. The updater itself added 4.6 MiB to the binary, which is what an HTTP client, a TLS stack and a tar reader weigh. It’s written down with the numbers so the next person doesn’t re-measure it.

CI, and the release pipeline that runs with no secrets

Four jobs. A fast one on Ubuntu with no Rust toolchain: format, lint, typecheck, tests, the fixture and inventory checks, the licence gate. A Rust one, also on Ubuntu because it’s the cheapest place to run Clippy with -D warnings and the unit tests, including a golden-PDF test that catches print regressions nobody would otherwise notice until they printed. A build matrix for the two platforms we ship, macOS on Apple Silicon and Windows x64, which vendors the engine, builds the installer, weighs it against a committed size budget and uploads it. And a nightly job on macOS that runs the real engine on a real picture of music and checks that MusicXML comes out, because that’s too slow and too large for every push. The Windows build only exists in CI. You cannot build it on a Mac.

I’ve written about containerising a Next.js app and the shape here is the same one: the artefact you test is the artefact you ship, or the test is theatre.

The release workflow is the part I’m most pleased with, for a reason that has nothing to do with code. It runs end to end with no secrets configured at all. No signing certificate, no Apple account, no updater key. It builds both installers, weighs them, labels them UNSIGNED in the artefact name, and refuses to publish. A release pipeline nobody can exercise until the certificates arrive is a pipeline nobody has tested, and the day the certificates arrive is the worst possible day to find out it doesn’t work. The label has three states, not two, because signed-not-notarized on macOS still stops at Gatekeeper with a different message and the same outcome.

Two small things from that pipeline that I’d have paid to know in advance. The macOS runner’s bash is 3.2, where expanding an empty array under set -u is an error, which the modern bash on your own machine will never show you. And Tauri’s .dmg step drives Finder over AppleScript to lay out the window, which stalls on a permission prompt the first time you run it on a machine. CI=true skips that cosmetic step, and the runner sets it for you, so this only ever bites locally.

The phone

One feature that’s worth a paragraph because the obvious approach is wrong. You can photograph a part with your phone and have it land in the app: the app starts a small web server on the LAN, shows a QR code, and the phone opens the page. My first instinct was a live camera preview with getUserMedia. That requires a secure context, a phone hitting http://192.168.x.x:port doesn’t have one, and a self-signed certificate produces a full-page warning that reads as a broken app. A plain <input type="file" capture="environment"> needs no secure context and uses the phone’s own, better, camera app. Less code and a nicer photo 🎺.

What isn’t done

The thing I should say plainly: the go/no-go for this whole product, whether correcting recognition output on real photographs actually beats transposing by hand, has not been run. It needs eight to ten real photographed band parts and a half-day against a protocol that’s written and ready. Everything downstream of it is built. That order is backwards and I know it. The MusicXML-in, transposed-part-out path stands on its own if the answer is no, which is the excuse I’m using.

Two release blockers are open and named: the updater has no public key yet, and the AGPL written offer inside the bundle still names a placeholder address. Both need a person, not a build.

Would I use Tauri again? Yes, without much hesitation. The window is the OS’s own webview, the binary is small, the command boundary is simple, and Rust on the other side of it is a good deal more pleasant than I expected for a job that is mostly “run this program and read its output”. The parts that cost time were never Tauri. They were the parts of shipping software to strangers’ machines that the web had let me forget about.

‘Till next time!

Godot for Web Developers

Fri, 31 Jul 2026 00:00:00 GMT

A genre I had no business starting with

On the 23rd of July I opened an empty folder and decided to build a real-time strategy game. Not a prototype of a mechanic, an actual RTS: base building, an economy, fog of war, an AI opponent that has to find you. I have written a lot of software. None of it had a frame budget.

I picked Godot 4, which I had never used, and GDScript, which I had never written. Eight days later the repository has 316 commits, 91 test files and a match you can lose. This post is what a web developer learns in the first week, written while it’s still fresh and I still remember what confused me.

The game is called Hammerfest. It’s flat, isometric and single player, in the spirit of the Command & Conquer games I grew up on and the Kingdom Rush look I still like. The plan is that it eventually ships on Steam. The plan for the first slice is much smaller: one level, and an honest answer to whether expanding a base under pressure is fun for twenty minutes.

Why Godot and not a browser

My instinct was TypeScript and a canvas library, because that is what I know and because everything would be text. I talked myself out of it in an afternoon.

Godot hands you the parts I would otherwise write badly: an editor for maps, audio, particles, a profiler, and an export to desktop that is one button. I estimated two to four months of plumbing saved, and given how slow I am at anything involving a GPU, that is probably optimistic in my favour. The cost is an editor that a lot of your work goes through, which I’ll come back to, because it turned out to matter less than I feared.

GDScript over C# was easier. It has no compile step, it’s what almost all Godot 4 material is written in, and with static typing switched on it stops being the loose scripting language people remember from Godot 3.

var speed: float = 3.0
var goal_cell: Vector2i = Vector2i.ZERO

func apply(world: World) -> void:
	...

Type every declaration. The project settings have warnings for untyped declarations and unsafe property access, and I set both to error on day one. GDScript’s type inference is fine, but an untyped var in a function that runs thirty times a second for every unit is a Variant box being allocated and torn down thirty times a second for every unit, and the profiler showed me that within a day. More on that below.

One thing that surprised me: GDScript is indented with tabs by convention, and the editor inserts them. I gave in. It’s a tab project now.

The one rule the whole thing hangs on

If I had to keep a single decision from the first week it would be this: the simulation does not know Godot exists.

Everything in sim/ is a plain RefCounted class. No Node, no scene tree, no physics, no input, no wall clock, no unseeded randomness. Godot’s node classes read from the simulation and draw it. They never own state. In web terms it’s “the business logic knows nothing about React”, which every React developer nods at and few of us actually enforce.

I enforce it with a shell script in CI. It greps sim/ for the banned symbols (PhysicsServer, CharacterBody, NavigationServer, Input., Time.get_ticks, randi, and so on), and it checks that every extends in that folder resolves to RefCounted, Object, or a class declared inside sim/ itself. Anything else fails the build. Banning Node subclasses by name is a losing battle, Godot has dozens, so the allowlist runs the other way. There’s an escape hatch comment, # purity-ignore, and it has been used zero times.

An invariant nobody enforces is a promise, and this one is very easy to break by reflex. Area2D overlap queries are right there, they are how every tutorial finds “what is near this unit”, and they are exactly the thing you cannot use if you want the next section to be true.

What the split buys

It sounds like architectural fussiness. It is the cheapest thing in the project, and it bought four concrete things in week one:

Tests run headless, in milliseconds, with no window. The whole suite runs on every commit on a Linux runner with no GPU.
Save and load is “serialise the world”. There is no per-feature save code, and there never will be.
Two AIs can play each other at a few hundred times real time, which turns “is the AI any good” from an opinion into a number.
The renderer got swapped twice, and sim/ did not change by one line either time. I’ll get to that.

Fixed timestep, and why your game is otherwise a different game on every machine

Web developers meet requestAnimationFrame and learn to multiply movement by the delta since last frame. That works for a spinner. It’s wrong for a simulation.

If the world advances by however much real time has passed, a 144 Hz machine and a 40 Hz machine run different games. Floating point doesn’t associate, so the positions drift apart, so the fights resolve differently, so a save from one laptop loads into a slightly different match on another. Replays are impossible. Bugs are irreproducible by construction.

The simulation in Hammerfest advances in fixed ticks, thirty per second, and never by a delta. A Node in the presentation layer accumulates real elapsed time and spends it in whole ticks:

class_name MatchRunner
extends Node

const MAX_TICKS_PER_FRAME: int = 8

var world: World = null
var _pending: Array = []
var _accumulator: float = 0.0

func _process(delta: float) -> void:
	_accumulator += delta
	var ticks: int = 0
	while _accumulator >= World.TICK_DELTA and ticks < MAX_TICKS_PER_FRAME:
		world.step(_pending)
		_pending.clear()
		_accumulator -= World.TICK_DELTA
		ticks += 1

The pending array is the commands issued since the last tick, which is the only way anything gets into the world. More on that in a moment.

The cap on ticks per frame is not tidiness. Without it one long stall queues a burst of ticks, which itself takes long enough to queue more, and the game freezes forever trying to catch up. The usual name for this is the spiral of death. Dropping simulated time instead means a bad frame turns into a moment of slow motion, which nobody can see.

The rendering runs at whatever rate the machine manages and interpolates between ticks. At thirty ticks a second you do notice if it doesn’t.

Everything is a command

The second rule: nothing mutates the simulation except a command.

A click becomes a MoveCommand. The AI’s decision becomes a MoveCommand, the same class, through the same door. The debug panel that spawns a thousand units for a stress test issues SpawnEntityCommands. There is no back door for tooling, because a back door added “just for debugging” is a back door the AI could use, and I’d never be sure it wasn’t.

class_name Command
extends RefCounted

func type_code() -> int:
	push_error("Command.type_code() must be overridden")
	return 0

func apply(_world: World) -> void:
	push_error("Command.apply() must be overridden")

func to_array() -> Array:
	push_error("Command.to_array() must be overridden")
	return []

Every command is serialised to a positional array with an integer type id, appended to a log with its tick number. seed + ordered command log reconstructs any match exactly, and there is a test that plays a match, replays the log into a fresh world, and asserts the two state hashes are equal. That test is the backbone of the project, because determinism breaks silently. A stray randi() or an iteration over a Dictionary whose order you assumed will not throw. It will quietly make every save, replay and benchmark wrong, and you find out weeks later.

Two habits keep it honest. All randomness goes through one seeded generator owned by the world, and I never iterate an unordered collection where the order affects the result. Entities are processed in ascending slot order, always.

The AI drives the same command interface as the player and reads the world through a filtered view that only contains what its units can currently see. No cheating, and more usefully, no accidental cheating, which is the kind that actually happens.

Handles, not references

Every RTS hits the dangling-target bug. Unit A is attacking unit B. B dies, its slot is reused for a harvester, and A cheerfully keeps shooting at a harvester it has never seen, possibly its own.

The fix is old and simple: entities are stored in dense arrays and every cross-entity reference is a generational handle, an index plus a generation counter packed into one 64-bit integer. When a slot is reused, the generation increments. A stale handle resolves to null instead of resolving to the wrong thing.

static func make(index: int, generation: int) -> int:
	return ((generation & 0xFFFFFFFF) << 32) | (index & 0xFFFFFFFF)

static func index_of(handle: int) -> int:
	return handle & 0xFFFFFFFF

Generations start at 1 so that zero is never a valid handle. So far it has done its job, which for a first RTS feels like cheating.

Pathfinding is where RTS projects die

The obvious approach is A* per unit. Order fifty units to one spot and you get fifty near-identical searches, then fifty units trying to walk the same line and shoving each other into walls.

A flow field inverts it. Once per destination, compute a direction arrow for every passable cell, flooding outward from the goal. Any number of units then read the arrow in their cell and follow it. One computation serves the whole group, and it gets relatively cheaper the more units share it. The flood is integer breadth-first search, so the routing input is exactly reproducible. Units bumping into each other is a separate, local problem, and for now it’s a plain “push away from close neighbours” force.

The first version built the field at full resolution, 256 by 256 cells. That is 65,536 cells, and GDScript took 330 ms to flood them. Inlining the hot loop got it to 117 ms. Against a 5 ms tick budget that is still unusable by a factor of twenty, and it is not an algorithm bug. The search is correctly linear in cells. It’s the interpreter’s per-operation cost over a lot of cells, and inlining is the floor.

The fix was a coarse routing grid, 64 by 64, decoupled from the fine collision grid: about 7 ms per build, inside a frame. Routing coarse and collision fine is how actual RTS engines separate the two problems, which I did not know until I’d rediscovered it the expensive way.

The performance gate that lied to me

This is the part of the week worth reading twice, because the bug was in the instrument.

I had a gate that spawned 600 units, sent them across the map, and checked the tick against the 5 ms budget. It printed 4.18 ms, PASS. Comfortable.

But look at what those 600 units were. Bare entities: no player, no weapons, no cargo. So combat hit its early-out and returned. Vision stamped one player’s fog. The economy found nothing to do. The gate measured movement honestly and everything else not at all, then printed a verdict about the whole tick. The same 600 units actually fighting cost 17.1 ms, three and a half times the budget.

A gate that measures a subset and reports on the whole is worse than no gate, because a green light gets believed. The fix was not to optimise. It was a probe that runs the real systems on the real level, two armies meeting mid-map with both economies running, and times each system separately:

System	Before
Combat	8.16 ms
Economy	2.40 ms
Vision	2.36 ms
Movement	1.94 ms

Combat costing four times what movement cost was not what I would have guessed, and every instinct I had about where a simulation spends its time was wrong about half the time. The biggest single win was embarrassing: iterating the world allocated a fresh Array[Entity] on every call, and the systems call it about seventeen times a tick. Caching it took the economy from 2.40 ms to 0.86 ms.

What GDScript actually costs, in one sentence: the wins come from deleting calls and lookups, not from cleverer arithmetic. Packed*Array is the only unboxed storage the language has, and anything a loop touches every tick belongs in one.

A rock was scouting for you

My favourite bug of the week. Harvesters, freshly spawned, walked straight to the nearest contested resource node even though nobody had scouted it yet.

The obvious cause was the AI’s target search ignoring what its owner had seen. Adding that check did nothing. The real cause was one layer down: resource nodes are entities, entities need an owner to spawn, so the nodes were owned by player 0 “because nothing reads it”. Vision read it. The fog pass stamped a sight disc for every entity, rocks included, so every node on the map revealed itself to player 0 from tick zero, permanently. Player 0 could see every deposit for free. Player 1, the AI, could see none.

Silently asymmetric, and in the play scene player 0 is the human. It showed up in the numbers the moment it was fixed: contested ore mined across a match went from 170 against 20 to 170 against 155. Deposits now have a vision range of zero. A rock is scenery, not a scout.

The renderer changed twice and the simulation never noticed

On the 26th the game rendered in 2D. On the 27th I ported it to 3D under an orthographic isometric camera, because I thought I wanted real height and painted buildings as billboards. On the 28th a tileset I had bought settled the question the other way: cliffs are tiles, buildings are placeables, the whole thing is a 2D tilemap. So the 3D renderer was deleted. 7,157 lines removed, 305 added.

sim/ did not change. Not one line, in either direction. The entire UI layer used exactly three things from the camera, a ground-point lookup, a to-screen projection and the zoom, so the minimap, HUD and overlays each needed one type name changed. That’s the bill a narrow interface pays, and it is legible in the diff. I wrote the invariant down on day one hoping it would hold. It has now been tested twice, which is better than asserted.

Testing

gdUnit4 is the test framework, vendored into addons/. Tests mirror the source layout, extend GdUnitTestSuite, and run headless:

extends GdUnitTestSuite

func test_stale_handle_does_not_resolve() -> void:
	var store: EntityStore = EntityStore.new()
	var handle: int = store.spawn(Entity.new())
	store.despawn(handle)
	assert_object(store.get_entity(handle)).is_null()

./tools/ci/run_tests.sh

One sharp edge is worth knowing about. Godot’s registry of class_name declarations lives under the gitignored .godot/ folder and is populated by a filesystem scan that godot --headless -s script.gd does not perform. On a fresh clone the test runner failed with “Could not find type GdUnitTestCIRunner”, and worse, Godot’s script runner still exited 0, silently running zero tests. The runner script now imports the project first, unconditionally, and it also fails the run if no tests actually executed. Green with zero tests is the worst colour.

Most of the suite is simulation tests that run in milliseconds. The ones I trust most are the determinism tests: play a match, replay its command log into a fresh world, compare the hashes. Every system that landed this week landed with one of those.

Every mechanic must be visible

Headless tests prove correctness. They don’t find the problems you didn’t think to test for, and in a real-time game that’s most of them.

So every system ships with a debug overlay you can toggle at runtime (flow-field arrows, paths, collision radii, aggro ranges, the AI’s current stance) and with a sandbox scene where that one mechanic can be poked in isolation. Spawn units, click, drop a wall, watch. The single most useful key in the whole project is ., which advances the simulation by one tick while paused. Most gameplay bugs are one bad tick, invisible at thirty a second.

They feel like extra credit next to the “real” work. They aren’t. The rock above was found by watching harvesters walk, not by a test.

About the editor

The thing I feared most about Godot was that a chunk of the work would live inside a GUI, away from text. In practice, almost everything is a .gd file and the scenes are plain text .tscn files that diff acceptably. The editor is where I lay out a level and look at the result. It is not where the logic lives, and it would be a bad place for it.

Two things worth knowing. Godot writes .uid sidecar files next to every script and they belong in version control, which the commit history shows me forgetting at least once. And it will happily rewrite project.godot when you touch a setting, so commit that separately or you’ll never find your own change in the diff.

Who typed all this

Not all me, and I’d rather say so. I’ve written about Claude Code twice this year, and this project is where I’ve leaned on it hardest. The repository has a CLAUDE.md whose first instruction is “you are a critical collaborator, not a cheerleader”, with a standing order to argue the case against my plan before helping with it. There are three advisor agents, a designer, an engineer and a director who resolves the two, and I go to the director for decisions. Each day ends with a journal entry in docs/journal/, and one of the branches merged this week is literally called overnight-autonomous.

What I do is decide, review, play and reject. The sim purity script, the perf gate that lied and the probe that replaced it, the rock: all of those came out of me reading what was built and pushing back. The 316 commits are a measure of how fast the loop runs when one side of it never sleeps, not of how much I typed.

What’s next

The 20-minute fun gate is still open. Matches currently last about fourteen minutes against a target of twenty, and the AI needs to attack rather than trickle. The map editor exists but is two days old. I have not drawn a single piece of art I like.

I’m also aware that a lot of this reads like confidence. It’s not. I’ve been in this engine for eight days and I have a folder of daily journal entries whose most common heading is some variation of “what I got wrong”. Godot has been pleasant, GDScript is faster to write and slower to run than I expected, and the architecture is holding. Ask me again after the fun gate.

‘Till next time!

AI Skills

Sat, 20 Jun 2026 00:00:00 GMT

A folder my agent knows how to read

For a couple of years, teaching an AI assistant your workflow meant pasting the same paragraph of instructions into a different box for every tool you used. Rewrite it for the terminal agent, rewrite it again for the editor, watch both drift out of date. Agent Skills fix that by being boring: a folder, a Markdown file, and a spec that a couple of dozen tools have agreed to read.

I wrote about Claude Code skills a couple of months ago, back when they still looked like a Claude Code feature. They aren’t one any more. Anthropic published the format as an open standard, the rest of the industry picked it up faster than anyone expected, and there is now a registry layer sitting on top of it. That changes the calculation. A skill you write today is not a bet on one vendor’s roadmap.

This post is about the format itself: what the spec requires, what it merely suggests, how the loading model works, and where the sharp edges are.

What is actually in one

A skill is a directory with a SKILL.md in it. The file starts with YAML frontmatter and continues with plain instructions. That is the entire required surface.

---
name: changelog
description: Generate a changelog entry from staged git changes.
  Use when the user asks to write a changelog or a release note.
---

## Writing a changelog entry

1. Run `git diff --staged` to see what changed.
2. Group changes under Added, Changed, Fixed, Removed.
3. Write each entry in past tense, user-facing language.
4. Skip internal refactors that don't affect behaviour.
5. Prepend the entry to `CHANGELOG.md` under an `## [Unreleased]` heading.

That is a working skill. No SDK, no build step, nothing to register. The agent reads the description, decides the skill applies, loads the instructions and follows them.

A skill can carry more than the one file, and the spec suggests three conventional directories for it:

my-skill/
├── SKILL.md          # required: metadata and instructions
├── scripts/          # executable code the agent can run
├── references/       # documentation loaded on demand
└── assets/           # templates, schemas, images

None of those are mandatory, and the agent will happily read anything else you put in the folder. The convention exists so that a skill someone else wrote is legible at a glance. That bundling is the real difference between a skill and a saved prompt: a prompt is text, a skill is text plus the script that does the fiddly part and the reference file you would otherwise have to summarise badly.

The frontmatter, field by field

Two fields are required and four are optional. It is a short spec, which I appreciate.

Field	Required	Notes
`name`	yes	Max 64 characters, lowercase letters, numbers and hyphens. Must match the parent directory name.
`description`	yes	Max 1024 characters. What it does and when to use it.
`license`	no	A licence name, or a pointer to a bundled licence file.
`compatibility`	no	Max 500 characters. Environment requirements: needs Docker, needs network access, and so on.
`metadata`	no	Arbitrary string-to-string map for anything the spec doesn’t cover.
`allowed-tools`	no	Space-separated list of pre-approved tools. Still experimental, support varies.

The name rules trip people up more than they should. No uppercase, no leading or trailing hyphen, no consecutive hyphens, and it has to match the folder. pdf--processing is invalid and the error message you get depends entirely on which agent you are running.

The description is the field to spend time on, because it is the only part of your skill the agent sees until it decides to use it. “Helps with testing” tells it nothing. “Use when the user asks to write or fix a Vitest unit test, or when a test file is open and failing” gives it a trigger it can match against. Write it for a matching algorithm, not for a README.

Individual agents add their own fields on top. Claude Code has disable-model-invocation, argument-hint and a few others; those are not in the spec and other tools will ignore them. That is fine as long as you know which half of your frontmatter is portable.

Why fifty skills don’t cost fifty skills

Skills load in three stages, and this is the design decision that makes the whole thing viable.

Discovery. At startup the agent reads only name and description for every installed skill. Budget roughly a hundred tokens each.
Activation. When a task matches, the full SKILL.md body is read into context. The spec recommends staying under 5000 tokens, or about 500 lines.
Execution. Anything in scripts/, references/ or assets/ is pulled in only when the instructions point at it.

So a skill you never use costs you a sentence. That is why you can have several dozen installed without the context window turning to soup, and it is also why the “keep SKILL.md short, push detail into reference files” advice is about cost rather than tidiness. Every token sitting in an activated skill is a token not being spent on your actual code.

One practical rule from the spec that is easy to miss: keep file references one level deep from SKILL.md and use relative paths. Chains of references that point at further references are exactly as fun to debug as they sound.

Who reads them

The format came out of Anthropic and was released as an open standard at agentskills.io, with the specification and a reference validator developed in the open on GitHub. The spec document is not the interesting part. Adoption is.

Tool	Vendor	Skills support
Claude Code and Claude	Anthropic	Native
ChatGPT and Codex	OpenAI	Native
Gemini CLI	Google	Native
VS Code and GitHub Copilot	Microsoft	Native
Cursor	Anysphere	Native
Kiro	AWS	Native
Junie	JetBrains	Native
Goose	Block	Native
OpenCode, Roo Code, Amp, Factory	Various	Native

The client showcase on agentskills.io runs to several dozen entries at this point, including some genuinely unexpected ones (Laravel Boost, Spring AI, Snowflake’s Cortex Code). Competitors who agree on very little agreed on this within about two days of the spec landing, which tells you how much everybody wanted the problem to go away.

The practical upshot is mundane and useful. If your team encodes its deployment checklist as a skill, it works whether a given colleague lives in a terminal or an IDE, and it keeps working when half of them switch tools next quarter. Your ChatGPT custom instructions have never done that for you.

Getting them onto your machine

Agents scan conventional directories. Paths vary slightly, but the Claude Code layout is representative:

Scope	Path	Applies to
Personal	`~/.claude/skills/<name>/SKILL.md`	Everything you work on
Project	`.claude/skills/<name>/SKILL.md`	One repository

Personal skills are for habits that belong to you: how you like commit messages written, how you want code explained. Project skills belong in version control, which is the part I care about more. Commit them and every fresh checkout arrives with the team’s playbooks already attached. Nobody has to be told about the deployment procedure, because it is sitting in the repository.

Installing a skill by hand means dropping a folder in the right place. That works fine until you want to share one.

Running them

Two ways in.

The first is explicit: type the skill’s name as a command, /changelog, and the agent follows that playbook and no other. The second is automatic: the agent reads descriptions, decides yours matches what you just asked for, and loads it without being told. Most of the time the automatic path is the one you want, and it is also the one that makes a good description pay for itself.

For anything with side effects, turn the automatic path off. Deploying, deleting, sending a message to a customer: these should only ever happen because you typed the command. In Claude Code that is a frontmatter flag:

---
name: deploy
description: Deploys the app to production.
disable-model-invocation: true
---

Read-only skills can be automatic. Destructive ones should be explicit. I have not found a case where the reverse was defensible.

Writing one

The best first skill is the workflow you have explained to an agent more than twice. Open a folder, write the file, and keep it lean.

Lead with a description that says what and when. It is a trigger, not a summary.
Write imperative numbered steps. Agents follow a list more reliably than they follow a paragraph.
Bundle the script or the template instead of describing it in prose.
Validate before you publish. The reference library will check your frontmatter and naming for you: skills-ref validate ./my-skill.

The mental adjustment that helped me most: you are not prompt-engineering, you are writing onboarding documentation for a colleague who reads very quickly and remembers nothing between sessions.

The registry layer

Once skills became portable folders, somebody was always going to build npm for them. Vercel did, in January 2026, and skills.sh is the one you will run into first. It is a directory with leaderboards: all-time installs, what is trending today, which agents a given skill is compatible with. Installation is a one-liner.

npx skills add <owner/repo>

The directory itself is open source, and the compatibility tags cover a couple of dozen agents. Treating skills as packages rather than as folders you email around is what makes any of this scale to a team, let alone an ecosystem.

Now the caveat, because it matters more than the convenience. A registry is not a quality gate. These catalogues are open, anyone can publish, and a skill is instructions plus executable scripts your agent will run on your machine. That is a real trust surface and a fairly appealing one to attack. Before installing anything from a stranger, read the SKILL.md, open whatever is in scripts/, and treat the install count as weak evidence rather than an endorsement. It is the same caution you would apply to an npm package with a postinstall hook, and for the same reason.

Where a skill ends and a context file begins

The two mechanisms get muddled constantly, so: persistent project knowledge goes in an always-loaded context file, and on-demand procedures go in skills.

The CLAUDE.md in this site’s repository is a decent example of the first kind. It says which npm scripts exist, where blog posts live on disk, that webpack resolves src/ as a module root so imports look unusual, and which SCSS conventions the project follows. All of that is true in every session, so it should be loaded in every session.

A skill answers a narrower question. How do I scaffold a new post. How do I write a commit message here. What does our release checklist look like. Those are only relevant while you are doing that specific thing, and paying for them the rest of the time is waste.

If you find yourself unsure which bucket something belongs in, ask whether an agent would need it to answer “what is this project?”. If yes, context file. If it only helps once you have already decided what to do, skill.

What I’d write first

Pick the one thing you re-explain most often. Not the most impressive workflow, the most repeated one. Write it, use it for a week, and then go back and sharpen the description until the agent reaches for it at the right moments without prompting. That second pass is where most of the value is, and it is the step people skip.

After that, go and look at what other people have published before writing your own version of a changelog generator. The good ones are already there.

The bit I am still unconvinced by is discovery. Leaderboards rank by installs, installs reward whoever published first and tweeted loudest, and there is currently no way to tell a carefully maintained skill from one that was generated in bulk and never opened again. That will need solving, and a star count is not going to be the thing that solves it

‘Till next time!

Building AI-Powered Apps with the Vercel AI SDK

Sat, 23 May 2026 00:00:00 GMT

Making the model a parameter

Every provider ships its own SDK, and every one of them has a different opinion about what a stream looks like, how a tool call is shaped, and what comes back when the model refuses to answer. Write against OpenAI directly and you have written against OpenAI forever. The Vercel AI SDK is the layer that turns the provider into a parameter instead of an architecture decision you have to live with.

The catch is that the SDK moves. Version 5 rewrote the React hooks from the ground up, version 6 deprecated the function most people learned first, and one of its three layers has carried an “experimental” label for its entire life. A post that reprints the quickstart would be stale by autumn. So what follows is a map of the pieces, with an opinion attached wherever I have one about which of them I would actually build on.

If you have been following the rise of AI agents, this is the plumbing underneath that story. Nothing here is conceptually hard. Most of the difficulty is in knowing which API survived the last major version.

Three layers, one of which I ignore

The SDK is really three packages wearing a trench coat.

AI SDK Core is the part that matters. generateText, streamText, embed, embedMany and friends behave identically no matter which model you point them at. It runs anywhere TypeScript runs: a route handler, a cron job, a CLI, a worker.

AI SDK UI is the framework layer. useChat for conversations, useObject for progressively filling structured data. React, Svelte and Vue are supported; useObject in particular is still flagged experimental and is not available everywhere.

AI SDK RSC streams React Server Components straight from the server as the model produces them. It is the flashiest of the three and the one I would leave alone. More on why later.

You install what you need, nothing more:

npm install ai zod

That is genuinely all it takes now. The provider packages (@ai-sdk/openai, @ai-sdk/anthropic and the rest) are thin adapters, and since the AI Gateway became the default provider you can skip them entirely for the common models and pass a plain string instead.

Calling a model

The smallest useful call gives you a model and a prompt and hands back text:

import { generateText } from 'ai';

const { text, usage, finishReason } = await generateText({
  model: 'openai/gpt-5',
  prompt: 'Explain quantum entanglement in two sentences.',
});

console.log(text);
console.log(`Used ${usage.totalTokens} tokens`);

Moving to Anthropic is a string edit:

const { text } = await generateText({
  model: 'anthropic/claude-opus-4-7',
  prompt: 'Explain quantum entanglement in two sentences.',
});

provider/model strings resolve through Vercel’s AI Gateway, which reads AI_GATEWAY_API_KEY from the environment. If you would rather talk to a provider directly, import its package and pass openai('gpt-5') instead. Both routes land on the same call signature, and the shape that comes back (text, usage, finishReason, warnings) does not change either. System prompts, message history, temperature and stop sequences all work the way you would expect.

Streaming

For anything a human is waiting on, streamText replaces generateText:

import { streamText } from 'ai';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: 'openai/gpt-5',
    system: 'You are a helpful assistant.',
    messages,
  });

  return result.toUIMessageStreamResponse();
}

toUIMessageStreamResponse() returns a Response whose body speaks the SDK’s own stream protocol, which is what the client hook knows how to parse. If you want the raw material instead, result.textStream gives you nothing but text deltas, and result.fullStream gives you every event that passes through: text, tool calls, tool results, finish reasons.

Structured output, and the rug pull

Getting typed JSON out of a model is the single most useful thing the SDK does, and it is also where version 6 caused the most grumbling. generateObject and streamObject were the headline feature for two years. They are now deprecated in favour of an output option on the ordinary text functions:

import { generateText, Output } from 'ai';
import { z } from 'zod';

const { output } = await generateText({
  model: 'openai/gpt-5',
  output: Output.object({
    schema: z.object({
      name: z.string(),
      ingredients: z.array(
        z.object({ name: z.string(), amount: z.string() })
      ),
      steps: z.array(z.string()),
      prepMinutes: z.number(),
    }),
  }),
  prompt: 'Generate a recipe for sourdough focaccia.',
});

for (const item of output.ingredients) {
  console.log(`${item.amount} ${item.name}`);
}

There is a defensible reason for the change. Output.object() composes with tool calling, whereas generateObject was a dead end that could not do both at once. Output.array(), Output.choice() and Output.json() round out the set, and the streaming equivalent renamed partialObjectStream to partialOutputStream. Fine. It is still a lot of churn for a function that was doing its job, and if you follow a tutorial written before December 2025 you will be writing deprecated code without any warning.

Underneath, the SDK still uses whatever native structured-output mode the provider offers and validates the result against your schema. A response that does not fit throws, so wrap it and retry rather than assuming the model behaved.

Tools

A tool is a function you let the model call. The SDK normalises the shape across providers so you write it once:

import { streamText, stepCountIs, tool } from 'ai';
import { z } from 'zod';

const result = streamText({
  model: 'openai/gpt-5',
  messages,
  tools: {
    getWeather: tool({
      description: 'Get the current weather for a city',
      inputSchema: z.object({
        city: z.string().describe('The city name, for example "Zurich"'),
        units: z.enum(['celsius', 'fahrenheit']).default('celsius'),
      }),
      execute: async ({ city, units }) => fetchWeather(city, units),
    }),
    refundOrder: tool({
      description: 'Refund an order in full',
      inputSchema: z.object({ orderId: z.string() }),
      needsApproval: true,
      execute: async ({ orderId }) => refund(orderId),
    }),
  },
  stopWhen: stepCountIs(5),
});

The model reads the descriptions and the schemas, picks a tool, and the SDK runs your execute and feeds the return value back as a tool result. stopWhen caps how many rounds of that the model gets. Version 6 defaults it to twenty steps, which is a much better default than the old one but is still twenty chances to call a paid API in a loop, so set it deliberately.

needsApproval is the piece I would not skip. Mark any tool that spends money, deletes something or sends a message, and the SDK pauses instead of executing, handing you an approval request to surface in the UI. Before version 6 you did this by leaving execute off and reimplementing the round trip yourself, which worked but was fiddly enough that plenty of people quietly didn’t bother.

Write the .describe() calls on your schema fields, by the way. The model reads them, and vague field names are the most common reason a tool gets called with nonsense arguments.

useChat, after the rewrite

useChat is the workhorse of the UI layer, and version 5 changed it enough that almost every example on the internet is now wrong. The hook no longer owns your input state, handleSubmit and handleInputChange are gone, and the endpoint is configured through a transport rather than an api string.

'use client';

import { useState } from 'react';
import { useChat } from '@ai-sdk/react';
import { DefaultChatTransport } from 'ai';

export function Chat() {
  const [input, setInput] = useState('');
  const { messages, sendMessage, status, stop } = useChat({
    transport: new DefaultChatTransport({ api: '/api/chat' }),
  });

  return (
    <div className="flex flex-col gap-4">
      {messages.map((message) => (
        <div key={message.id} className={message.role === 'user' ? 'text-right' : ''}>
          {message.parts.map((part, index) => {
            if (part.type === 'text') {
              return <span key={index}>{part.text}</span>;
            }
            if (part.type === 'tool-getWeather') {
              return <WeatherCall key={index} part={part} />;
            }
            return null;
          })}
        </div>
      ))}

      <form
        onSubmit={(event) => {
          event.preventDefault();
          if (!input.trim()) return;
          sendMessage({ text: input });
          setInput('');
        }}
      >
        <input
          value={input}
          onChange={(event) => setInput(event.target.value)}
          placeholder="Ask something"
          disabled={status !== 'ready'}
        />
        {status === 'streaming' && (
          <button type="button" onClick={stop}>
            Stop
          </button>
        )}
      </form>
    </div>
  );
}

I was annoyed by the input-state change when I first hit it, and I have come round. Owning the useState yourself means you can prefill from a query string, keep a draft in local storage or drive the box from a keyboard shortcut without fighting the hook. The old convenience was hiding about four lines of code.

The parts array is the other thing worth internalising. A single assistant message can carry text, tool calls, tool results, reasoning and files, and each tool gets its own part type named tool-<toolName>. So a weather tool produces tool-getWeather parts, each with a state that walks from streaming input through to output-available or output-error. Render by part type and by state, and the awkward intermediate moments (a tool that has been called but has not returned) stop being a special case.

status moves through submitted, streaming, ready and error. Use it for the disabled state and for a stop button, because long responses that cannot be interrupted feel broken even when they are working.

To restore a saved conversation, pass id and an initial messages array from the server. The old initialMessages prop is gone.

Generative UI, and why I skip it

The RSC layer lets the model choose a React component, and the server streams that component to the browser. streamUI picks a tool, the tool’s generate function yields a skeleton, fetches data, then yields the real thing. Written down it sounds wonderful, and the demos are genuinely impressive.

It is also, still, experimental, and Vercel’s own documentation says as much and points you at AI SDK UI for production. The known problems are not cosmetic: components remounting and flickering mid-stream, Suspense boundaries falling over, and data transfer that grows quadratically as the stream goes on. The API surface has also shifted between releases, and error recovery is nowhere near as mature as the client-hook path.

My take is that generative UI is a real idea implemented too early. You can get most of the benefit without RSC by having the model call a tool, rendering that tool’s part on the client with a component you chose, and keeping the server doing what servers are good at. That is one indirection more than streamUI and roughly infinitely fewer surprises at 2am.

Embeddings

The SDK is not only for chat. embedMany is the least ceremonious way I know to get vectors:

import { embedMany } from 'ai';
import { openai } from '@ai-sdk/openai';

const { embeddings } = await embedMany({
  model: openai.embedding('text-embedding-3-large'),
  values: docs.map((doc) => doc.content),
});

await db.documents.insertMany(
  docs.map((doc, index) => ({ ...doc, embedding: embeddings[index] }))
);

Use embed for a single query vector and run a similarity search against your store. Both Drizzle and Prisma handle pgvector columns now, so a normal Postgres database is usually enough and you can put off adding a dedicated vector service until you actually have the volume to justify one.

The gateway

Because every call takes the same shape, choosing a model can be ordinary application logic:

function pickModel(task: 'fast' | 'smart' | 'cheap') {
  switch (task) {
    case 'fast':
      return 'google/gemini-2.5-flash';
    case 'smart':
      return 'anthropic/claude-opus-4-7';
    case 'cheap':
      return 'openai/gpt-5-mini';
  }
}

const { text } = await generateText({ model: pickModel('smart'), prompt });

Those strings all go through the AI Gateway, which gives you one API key instead of four, per-request analytics, one invoice, and failover to a model you nominate when the upstream provider falls over. That last point is the one that sold me. Provider outages are frequent enough to plan for, and turning one into a logged warning rather than a paged engineer is worth a small amount of added latency.

If you would rather not route through Vercel, nothing forces you to. Keep the provider packages and the rest of your code is unchanged.

Agent loops

When a task needs several rounds of tool use, ToolLoopAgent packages the loop:

import { ToolLoopAgent, stepCountIs } from 'ai';

const supportAgent = new ToolLoopAgent({
  model: 'anthropic/claude-opus-4-7',
  instructions: 'You are a support engineer. Investigate with the tools before answering.',
  tools: { searchTickets, readLogs, queryDatabase },
  stopWhen: stepCountIs(10),
});

const { text, steps } = await supportAgent.generate({
  prompt: 'Investigate why customer 4271 cannot log in.',
});

If you used this in version 5 it was called Experimental_Agent and took system rather than instructions. Same idea, renamed.

The genuinely valuable return value is steps. Each entry holds the reasoning, the tool calls and their results for one round of the loop. Log it. When an agent does something baffling in production, that array is the difference between a debugging session and a shrug.

Middleware and telemetry

Every model call can be wrapped, which is where caching, redaction and logging belong:

import { wrapLanguageModel } from 'ai';

const model = wrapLanguageModel({
  model: 'openai/gpt-5',
  middleware: {
    wrapGenerate: async ({ doGenerate, params }) => {
      const key = hashParams(params);
      const cached = await redis.get(key);
      if (cached) return JSON.parse(cached);

      const result = await doGenerate();
      await redis.set(key, JSON.stringify(result), 'EX', 3600);
      return result;
    },
  },
});

Middleware can also implement transformParams to rewrite a call before it leaves, and wrapStream for the streaming path.

For telemetry, pass experimental_telemetry: { isEnabled: true } and the SDK emits OpenTelemetry spans covering the request, the tool calls and the token usage. Point those at whatever you already run. I send mine to PostHog, which has an LLM analytics view that turns the spans into per-feature cost, and it took about ten minutes.

Is the layer worth it?

Concern	Provider SDK directly	Vercel AI SDK
Changing model	Rewrite the call site	Edit a string
Streaming	Provider-specific SSE	One protocol, one parser
Tool calling	A different shape per provider	One `tool()` helper
Structured output	Hand-rolled schema handling	`Output.object()` with Zod
Approvals	Build the round trip	`needsApproval: true`
React	Write the hook yourself	`useChat`
Multi-step loops	Write the loop yourself	`ToolLoopAgent` + `stopWhen`

The cost is an abstraction between you and the model, and a maintainer who has shown they will deprecate things. The benefit is that almost none of your application code cares which model is behind it.

For most projects that trade is easy. Where I would think twice: research code that needs a provider feature the week it ships, and a one-file script that calls one model once and exits. The raw openai package is perfectly pleasant for that, and adding a framework to it is a waste of everyone’s afternoon.

A handful of habits

Things I would tell myself before the first project rather than after it.

Set stopWhen explicitly on every tool-enabled call, even though the default is now sane.
Put needsApproval: true on anything destructive before you write the happy path, not after the first accident.
Show tool calls in the interface instead of hiding them. People trust an agent they can watch.
Persist messages server-side from onFinish in the route handler. Do not trust the client to hand back a complete transcript.
Turn telemetry on from day one. Working out where the money went is much harder retroactively.
Cache anything whose prompt does not depend on user input. A summarisation call behind a middleware cache costs a fraction of what it did unwrapped.

What I’d watch next

The two things I want are boring ones. Better long-term stability, because three major versions in eighteen months is a lot of migration work for a dependency this central. And an honest decision about the RSC layer, which has been experimental long enough that it should either graduate or be retired rather than sitting in the docs tempting people.

Neither of those is a reason not to use it. Core alone earns the install, and useChat saves an afternoon per project. Just pin your version, read the migration guide before you upgrade, and treat anything labelled experimental as exactly that

‘Till next time!

Writing Skills for Claude Code

Sat, 18 Apr 2026 00:00:00 GMT

Explaining the same thing every morning

You start a session, and before anything useful happens you type the same paragraph you typed yesterday. Our commits look like this. The migrations live over there. Run the linter before you push, and no, not that linter. It is the software equivalent of repeating your coffee order to someone who has been making it for a year.

Claude Code has a mechanism for this, and it’s about as low-tech as it could be: a folder with a markdown file in it. Write the instructions down once, and they get loaded when they’re relevant. That’s a skill.

I’ve written about AI in the editor before, back when the argument was still whether autocomplete counted as cheating. That argument is over. The interesting question now is how much of your team’s tacit knowledge you can hand over, and skills are the sharpest tool I’ve found for it.

What the thing is, briefly

Claude Code is Anthropic’s agentic coding tool. You start it in a project directory:

claude

and then talk to it. The word doing the work is agentic: it doesn’t propose a diff for you to apply, it reads files, edits them, runs the tests, and comes back with what happened. One request routinely becomes thirty tool calls.

What it can do, roughly:

Read and navigate the whole project, including how the pieces connect
Edit across as many files as a change touches, so renaming a component updates the imports
Run builds, tests, linters, and anything else in your shell
Stage, commit, branch, and open pull requests
Spawn parallel subagents for work that splits cleanly, with a lead agent coordinating
Run in the terminal, in VS Code and JetBrains, in a desktop app, and in the browser

It also reads CLAUDE.md from your project at the start of every session. That file is the standing brief: architecture, conventions, the commands that matter. Skills are the opposite shape, loaded only when needed.

A skill is a folder with a markdown file in it

Minimum viable skill: a directory containing SKILL.md, which is YAML frontmatter followed by instructions in plain English.

The mechanism that makes this scale is what the spec calls progressive disclosure. At startup Claude reads only each skill’s name and description. The body stays on disk until the skill is actually used. So thirty skills cost you thirty short descriptions of context, not thirty documents, and there’s no reason to be precious about how many you install.

Two ways in. You type /skill-name and it loads. Or Claude reads the description, decides it’s relevant to what you just asked for, and loads it itself. Both routes matter, and as we’ll see, sometimes you want to close one of them.

Where they live

Location	Path	Applies to
Personal	`~/.claude/skills/<name>/SKILL.md`	All your projects
Project	`.claude/skills/<name>/SKILL.md`	This project only
Plugin	`<plugin>/skills/<name>/SKILL.md`	Wherever the plugin is enabled

Personal skills are for habits that follow you around: how you like commits written, how you want code explained back to you. Project skills go in version control and encode things the team agreed on, which means a new colleague inherits them by cloning the repository rather than by asking someone.

When the same name exists at more than one level, personal wins over project, and either wins over a skill that ships with Claude Code. A code-review skill in your project directory replaces the built-in one. Plugin skills are namespaced as plugin-name:skill-name and can’t collide.

In a monorepo, skills also load from nested .claude/skills/ directories underneath your working directory, so a package can carry its own. If the names clash, both stay available and the nested one gets a qualified name like apps/web:deploy. Handy, and also the sort of thing you discover by accident when the wrong deploy skill fires.

The frontmatter

Here’s a skill that does something concrete:

---
name: review-component
description: Reviews React components for accessibility, performance,
  and adherence to project conventions. Use when reviewing PRs or
  checking component quality.
---

When reviewing a React component, check the following areas:

## Accessibility
- All interactive elements have appropriate ARIA attributes
- Images have alt text
- Colour is not the only means of conveying information
- Keyboard navigation works correctly

## Performance
- No unnecessary re-renders (check dependency arrays)
- Large lists use virtualisation
- Images are optimised and lazy-loaded

## Conventions
- Component uses TypeScript with proper prop types
- Follows the project's naming conventions
- Uses Tailwind utility classes, not inline styles
- Tests cover the main interaction paths

Provide specific line references for any issues found.

Two fields is often all you need. The rest are there for control:

---
name: deploy-production
description: Deploys the application to production. Runs tests,
  builds the Docker image, and pushes to the container registry.
disable-model-invocation: true
allowed-tools: Read, Grep, Glob, Bash(npm run *)
argument-hint: "[environment]"
model: inherit
---

description is the field that decides whether a skill ever gets used. Claude only sees this line when choosing, so it has to say both what the skill does and when to reach for it. “Helps with components” is useless. Keep it tight, too: long descriptions get truncated in the listing, and the truncated half is the part you cared about.

name is less important than it looks. For a personal or project skill it’s a display label; the command you type comes from the directory name. ~/.claude/skills/deploy-staging/SKILL.md gives you /deploy-staging whatever the frontmatter says. Only in plugin skills does name set the final segment of the command.

disable-model-invocation: true stops Claude from loading the skill on its own. Anything with side effects wants this, and I’d argue for it more broadly than most people do. The failure mode isn’t dramatic, it’s just that a skill you wrote for occasional use fires during unrelated work because the description happened to match.

allowed-tools pre-approves tools for the turn that invoked the skill, so you’re not clicking through permission prompts mid-workflow. The grant expires when you send your next message. You can scope it narrowly, as Bash(gh *) rather than all of Bash.

argument-hint shows up in autocomplete. model accepts the same values as /model, plus inherit to leave the session model alone. Reaching for a cheaper model on a mechanical skill is a reasonable instinct, though I’d measure before assuming it saves anything.

Who can invoke what

Configuration	You	Claude
Default	Yes	Yes
`disable-model-invocation: true`	Yes	No
`user-invocable: false`	No	Yes

The third row is for background knowledge that shouldn’t clutter the / menu: house API conventions, a naming policy, something Claude should know when it’s relevant and you’d never type by hand.

Arguments

Whatever follows the skill name lands in $ARGUMENTS:

---
name: fix-issue
description: Reads a GitHub issue and implements the fix
argument-hint: "[issue-number]"
---

## Issue context
Read GitHub issue #$ARGUMENTS and understand the problem.

## Implementation
1. Find the relevant code
2. Implement the fix
3. Write tests
4. Create a commit with message: "fix: resolve #$ARGUMENTS"

/fix-issue 42 substitutes 42 everywhere the placeholder appears. If the skill doesn’t contain $ARGUMENTS at all, what you typed gets appended at the end anyway, so nothing is silently dropped.

For several arguments there’s positional access, $ARGUMENTS[0] or the shorter $0, $1 and so on. Multi-word values need quoting the way a shell would expect: /migrate-component "Search Bar" React Vue.

Injecting live context

This is the feature that changed how I write skills. The !`command` syntax runs a shell command before the skill is handed to Claude, and the output takes the placeholder’s place:

---
name: pr-summary
description: Summarises the current pull request
context: fork
agent: Explore
allowed-tools: Bash(gh *)
---

## Pull request context
- Changed files: !`gh pr diff --name-only`
- PR description: !`gh pr view --json body -q .body`
- Review comments: !`gh pr view --comments`

## Task
Summarise this pull request. Focus on:
1. What changed and why
2. Any concerns raised in review comments
3. Suggested improvements

Worth being precise about what’s happening, because it’s easy to misread. This is preprocessing. The commands run first, their output is pasted in, and Claude receives a finished prompt containing the actual diff. It never sees the commands, and it isn’t deciding to run them.

Some sharp edges. Substitution happens once over the file, so output containing another placeholder won’t be expanded again. The inline form only triggers when ! starts a line or follows whitespace, which means KEY=!`cmd` is left as literal text and quietly does nothing. For several commands, open a fenced block with a ! right after the backticks instead of stringing inline placeholders together.

There’s also a disableSkillShellExecution setting that turns the whole feature off, replacing each command with a note that policy blocked it. If you’re introducing skills somewhere with a security team, know that this exists before they ask.

Running a skill in a subagent

Research burns context. A skill that reads forty files to answer one question will leave your main conversation full of file contents you’ll never refer to again. context: fork runs it somewhere else:

---
name: deep-research
description: Performs thorough codebase research on a topic
context: fork
agent: Explore
---

Research the following topic in this codebase: $ARGUMENTS

Trace through all relevant code paths. Map the architecture.
Identify patterns, abstractions, and dependencies.
Return a summary with the 10 most important files
to understand this area of the codebase.

The skill body becomes the subagent’s prompt. It gets the codebase and the tools, but not your conversation history, and only its answer comes back.

That last point has a consequence worth knowing: the built-in Explore and Plan agents deliberately skip CLAUDE.md to keep their context small. So a forked skill using agent: Explore sees your skill content and nothing else about the project. If it needs a convention to do its job, the convention has to be written into the skill.

The other trap is forking a skill that has no task in it. If your skill is a page of guidelines with no instruction, the subagent receives the guidelines, has nothing to do, and returns nothing useful. context: fork is for skills that ask for something.

More than one file

A skill doesn’t have to be a single document. It can be a directory of supporting material:

review-security/
├── SKILL.md              # Main instructions
├── owasp-checklist.md    # Reference material
├── examples/
│   ├── good-review.md
│   └── bad-review.md
└── scripts/
    └── scan.sh

SKILL.md stays short and points at the rest, which gets read only if it’s needed. Use ${CLAUDE_SKILL_DIR} when referring to bundled files so the paths don’t depend on where the session started.

Templates work well for anything where the shape of the output matters more than the wording:

<!-- template.md -->
## Security review: {{component_name}}

### Summary
{{one_paragraph_summary}}

### Findings
{{findings_table}}

### Risk level
{{low|medium|high|critical}}

### Recommended actions
{{numbered_action_items}}

Then, in SKILL.md, a line telling it to fill in template.md and leave nothing blank. A filled template is far more reliable than a paragraph asking for consistent formatting.

The ones that ship with it

Claude Code comes with bundled skills you can read for reference. /code-review and /batch and /loop are the ones I’ve opened most. /batch in particular is a decent lesson in decomposition: it splits a large change into independent units, runs them in separate git worktrees, and opens a pull request per unit. /loop re-runs a prompt on an interval, which is a surprisingly good way to babysit a deploy.

Three that earn their keep

Commits

---
name: commit
description: Creates a conventional commit from staged changes.
  Analyses the diff, determines the commit type, and writes a
  descriptive message.
disable-model-invocation: true
---

Create a git commit following the Conventional Commits specification.

## Steps
1. Run `git diff --staged` to see what's being committed
2. Analyse the changes to determine the type:
   - `feat`: New feature
   - `fix`: Bug fix
   - `refactor`: Code restructuring
   - `docs`: Documentation only
   - `test`: Adding or updating tests
   - `chore`: Maintenance tasks
3. Write a commit message: `type(scope): description`
   - Scope is the primary area affected (component name, module)
   - Description is imperative mood, lowercase, no full stop
   - Mark a breaking change with `!` after the type, as in `feat!:`,
     or with a `BREAKING CHANGE:` footer
   - Add a body if the "why" isn't obvious from the diff
4. Include `Co-Authored-By: Claude <noreply@anthropic.com>`
5. Show the message and ask for confirmation before committing

This is conventional commits written down where something else can follow them. Note step 5: skills that touch git should show their work before doing it.

Scaffolding a component

---
name: create-component
description: Generates a new React component with TypeScript,
  tests, and a Storybook story following project conventions.
argument-hint: "[ComponentName]"
---

Generate a new React component named `$ARGUMENTS`.

## Files
- `src/components/$ARGUMENTS/$ARGUMENTS.tsx`
- `src/components/$ARGUMENTS/$ARGUMENTS.test.tsx`
- `src/components/$ARGUMENTS/$ARGUMENTS.stories.tsx`
- `src/components/$ARGUMENTS/index.ts` (barrel export)

## Component conventions
- TypeScript with an exported Props interface
- `forwardRef` for components that render DOM elements
- Tailwind for styling, no CSS modules
- JSDoc comment on the Props interface
- Named export, not default

## Story conventions
- CSF3 with `satisfies Meta<typeof Component>`
- Include `tags: ['autodocs']`
- Stories for Default, WithProps, EdgeCase

Every project has this shape written down somewhere nobody reads. Here it produces files. The story conventions line up with what I covered in the Storybook post.

A read-only explorer

---
name: explore
description: Read-only codebase exploration. Use when you want to
  understand code without any risk of modification.
allowed-tools: Read, Grep, Glob
context: fork
agent: Explore
---

Explore this codebase to answer: $ARGUMENTS

You are in read-only mode. Search files, read code, and trace
execution paths. Do not suggest changes. Focus entirely on
understanding and explaining what exists.

Return:
1. A clear explanation of what you found
2. Key files involved, with paths and line numbers
3. An ASCII architecture diagram if it helps

The one I use most, and the one I’d write first on an unfamiliar codebase.

CLAUDE.md or a skill?

The split is simple enough. CLAUDE.md is loaded every session and should contain facts that are always true: what this project is, how it’s laid out, which commands to run. A skill is loaded on demand and should contain a procedure.

The test I use: if it reads as a paragraph of description, it belongs in CLAUDE.md. If it reads as numbered steps, it’s a skill. When a section of CLAUDE.md grows a numbered list, that’s usually the moment to move it out. /init will generate a first CLAUDE.md from your codebase, which is a reasonable starting point and a poor finishing one.

project/
├── CLAUDE.md
└── .claude/
    └── skills/
        ├── commit/SKILL.md
        ├── review-pr/SKILL.md
        └── create-component/SKILL.md

What makes a skill work

One job per skill. A deploy-and-notify-and-update-docs skill is three skills that would compose better separately, and it’s harder to tell which third of it went wrong.

Keep SKILL.md short, a few hundred lines at most, and push detail into files it can reference. Everything in the skill is context that isn’t being spent on your actual code.

Write the description for the moment of selection, not for a catalogue. It should answer “when would I want this?”, because that’s the question being asked of it.

Test with more than one model. A skill that a larger model infers its way through can leave a smaller one guessing, and the fix is usually explicitness rather than length.

Portability, with an asterisk

Skills follow the Agent Skills spec, which Anthropic released as an open standard and which a long list of tools have since adopted, Cursor, Gemini CLI, Codex and GitHub Copilot among them. That’s genuinely useful. A folder with a SKILL.md in it travels.

The asterisk is that the spec is narrower than Claude Code’s implementation. It defines six fields: name, description, license, compatibility, metadata and allowed-tools. Everything else in this post, argument-hint, context: fork, disable-model-invocation, is Claude Code’s own. Packaging a skill for the Skills API or uploading it to claude.ai with those fields present fails outright rather than ignoring them:

Unexpected key(s) in SKILL.md frontmatter: argument-hint.

The body features don’t travel either. !`command` injection is Claude Code preprocessing, and elsewhere it’s just text. So “write once, run anywhere” holds for the plain instructional skills and not for the clever ones. Worth knowing before you build a library on the assumption.

The part that gets left out

A skill is a prompt, not a program. Nothing enforces it. Claude reads your numbered steps and generally follows them, and occasionally it decides step three doesn’t apply today and it is, annoyingly, sometimes right about that. If a step must happen every time without exception, it belongs in a shell script that the skill calls, or in a git hook, not in prose asking nicely.

That’s not a complaint so much as the thing to calibrate against. Skills are good at capturing judgement, the stuff that’s hard to encode: which cases matter in a review, what a decent commit message looks like here, where the bodies are buried in this module. They’re bad at guarantees. Anything you’d write a test for, write a test for.

I’m still poking at the edges of the forked-subagent behaviour, mostly trying to work out how much project context a skill has to restate when agent: Explore skips CLAUDE.md. Not a clean answer yet, so I’ll leave it there.

‘Till next time!

Docker for Frontend Developers

Fri, 13 Mar 2026 00:00:00 GMT

The frontend grew a database

There was a time when handing a frontend project to someone meant sending them a zip file and telling them which version of Node to install. Then the frontend grew a database, a cache, a queue and four environment variables that are different on everybody's laptop, and "works on my machine" stopped being a joke and started being a Tuesday.

This site has shipped in a container since 2020, when I wrote up putting it on Divio Cloud. That Dockerfile was one stage, installed the Gatsby CLI globally and copied everything in one go. It worked. It was also about a gigabyte, and I didn’t think about it again for three years.

What changed isn’t Docker. It’s that a Next.js app is now a server, and servers have neighbours. This post covers containerising one properly: the production image, the local stack, the file-watching mess on macOS, and the case for not bothering.

The four pieces

Four concepts and then we can move on.

An image is a read-only filesystem plus the metadata to start a process in it. A container is one running instance of an image. Nothing more mysterious than a class and an object.

A Dockerfile is the recipe for an image. Every instruction produces a layer, and layers are cached by content, which is the single fact that determines whether your builds take ten seconds or ten minutes.

Compose is the thing that starts several containers together with a network between them, described in a compose.yml. For local development it’s the piece you’ll actually live in.

One choice worth making early: base your images on node:22-alpine rather than plain node:22. Alpine is a minimal Linux distribution built around musl instead of glibc, and it drops the image from roughly 1.1 GB to roughly 150 MB before you’ve added a single file of your own.

Building the image

Standalone output

Before the Dockerfile, tell Next.js to produce a self-contained build:

// next.config.ts
import type { NextConfig } from 'next';

const nextConfig: NextConfig = {
  output: 'standalone',
};

export default nextConfig;

With standalone, Next.js runs @vercel/nft over your build to trace which files are genuinely reachable at runtime, then writes them into .next/standalone along with a small server.js. The point is that you no longer need node_modules in the production image. Only the modules your code actually imports come along.

Two directories are left out of the trace deliberately, because they’re normally served from a CDN: public/ and .next/static/. If you forget to copy them the app boots fine and then serves a page with no CSS and no images, which is a confusing five minutes.

The production Dockerfile

Three stages: fetch dependencies, build, then assemble a runtime image that contains neither the dependencies nor the build tools.

# Stage 1: Install dependencies
FROM node:22-alpine AS deps
RUN apk add --no-cache libc6-compat
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

# Stage 2: Build the application
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

# Stage 3: Production runner
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
RUN mkdir .next
RUN chown nextjs:nodejs .next

COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/public ./public

USER nextjs
EXPOSE 3000
ENV PORT=3000
ENV HOSTNAME="0.0.0.0"

CMD ["node", "server.js"]

A few of those lines are load-bearing and not obvious:

libc6-compat is the glibc compatibility shim. Some native packages, sharp in particular (which next/image uses for image optimisation), ship binaries built against glibc and fall over on musl without it. This is the standard Alpine tax.

npm ci rather than npm install. It installs strictly from the lockfile and wipes node_modules first, so the build can’t quietly pick up a floating minor version and produce something that doesn’t match what you tested.

HOSTNAME="0.0.0.0" is the one that catches people. Without it the server binds to loopback inside the container, the port mapping looks right, and nothing answers. I have lost time to this more than once.

The nextjs user exists because a process running as root inside a container is one escape away from root on the host. It costs four lines.

A development image

Development wants the opposite of all that. Keep it in the same Dockerfile as an extra stage so Compose can target it:

FROM node:22-alpine AS dev
RUN apk add --no-cache libc6-compat
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
EXPOSE 3000
CMD ["npm", "run", "dev"]

No standalone output, no pruning, devDependencies included. You want the dev server, not a small image.

The local stack

This is where Docker earns its place for frontend work. One file describes the app, the database and the cache, and they come up together on a network where they can find each other by name:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: dev
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/myapp
      - REDIS_URL=redis://redis:6379
      - WATCHPACK_POLLING=true
    volumes:
      - .:/app
      - /app/node_modules
      - /app/.next
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy

  db:
    image: postgres:16-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: myapp
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres-data:
  redis-data:

Notice there’s no version: key at the top. Compose has ignored it for years and now warns about it, so if you’re copying an old file around (mine on this site still carries a version: '2.3' from 2020) you can delete that line.

The DATABASE_URL points at db:5432, not localhost. Inside the Compose network every service is reachable by its service name, and this trips up everyone once, usually while wondering why Prisma can’t connect. If you’re wiring up the database side, I covered that in using Prisma with Next.js.

condition: service_healthy is the part I’d argue is mandatory rather than nice to have. Plain depends_on only waits for the container to start, and Postgres takes a couple of seconds after starting before it accepts connections. Without the health check your app boots into a connection refused error roughly one time in three, which is exactly the kind of intermittent failure that eats an afternoon.

The named volumes keep your data across restarts. Leave them out and every docker compose down wipes the database, which is either a disaster or a feature depending on the day.

Then the whole thing is:

docker compose up

No installing Postgres, no matching Node versions, no page of README instructions that went stale in November.

Hot reload, and why macOS makes it hurt

Getting file changes to reach the dev server inside a container is the part of this that genuinely annoys me. On Linux it mostly just works. On macOS, Docker runs inside a virtual machine, your source lives on the host, and filesystem events have to cross that boundary. They often don’t.

The three volumes

volumes:
  - .:/app              # source code from the host
  - /app/node_modules   # keep the container's copy
  - /app/.next          # keep the container's build cache

The first line is the useful one. The other two are anonymous volumes that exist purely to stop the host from shadowing directories that must stay Linux-native. Your macOS node_modules contains a sharp built for Darwin on arm64; mount that over the container’s and the app dies on startup with an error about missing bindings.

Polling

Next.js watches files through Watchpack, which prefers native filesystem events. When those don’t arrive, you edit a file and nothing happens at all, with no error to tell you why. The fix is to stop waiting for events and ask instead:

environment:
  - WATCHPACK_POLLING=true

Polling costs some CPU and adds a small delay, usually a few hundred milliseconds. That’s fine. Silent non-reloading is not fine.

Compose Watch

Since Compose v2.22 there’s a better answer than bind mounts, which is to let Compose push changes in rather than having the container watch for them:

services:
  app:
    build:
      context: .
      target: dev
    ports:
      - "3000:3000"
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json
        - action: sync+restart
          path: next.config.ts
          target: /app/next.config.ts

sync copies changed files into the container and lets Next.js handle the reload. rebuild rebuilds the image, which is what you want when package.json changes. sync+restart copies the file and bounces the container, for config that’s only read at boot.

docker compose watch

On macOS this is noticeably quicker than a bind mount, because it’s a targeted file copy instead of a shared filesystem doing translation on every stat call.

How small it gets

Multi-stage builds are the whole game for image size. Each FROM starts fresh and only the final stage ships, so everything you needed to build the app is thrown away.

Stage	Purpose	What survives
`deps`	Install dependencies	`node_modules`
`builder`	Compile the app	`.next` output
`runner`	Serve in production	Standalone server, static and public files

And the base image choice, for context:

Base image	Approximate size
`node:22` (Debian)	1.1 GB
`node:22-slim`	220 MB
`node:22-alpine`	150 MB

Alpine plus standalone output plus three stages usually lands a Next.js app somewhere around 150 to 200 MB, against well over a gigabyte for the copy-everything approach. Worth doing, though be honest about why: it’s mostly about how fast the image pulls on deploy, not about disk space, which is free.

CI with GitHub Actions

The same Dockerfile builds in CI unchanged, which is most of the appeal:

name: Build and Push

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha
            type=ref,event=branch

      - name: Build and push
        uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The two cache lines are what separate a two-minute build from a nine-minute one. Runners are ephemeral, so without them every push reinstalls every dependency from scratch. With type=gha the layers go into GitHub’s cache, and a commit that only touches source code reuses the deps layer.

Build-time values, the NEXT_PUBLIC_* sort that get baked into the client bundle, have to be passed in as build arguments, because they’re needed while the image is being built rather than when it runs:

- name: Build and push
  uses: docker/build-push-action@v7
  with:
    context: .
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    build-args: |
      NEXT_PUBLIC_API_URL=${{ vars.API_URL }}

Anything secret stays out of build args. They’re visible in the image history.

Things worth getting right

Instruction order

Docker caches layers in sequence, and one invalidated layer invalidates everything below it. So put what rarely changes first:

# 1. Base image (rarely changes)
FROM node:22-alpine
WORKDIR /app

# 2. Dependencies (change when package.json changes)
COPY package.json package-lock.json ./
RUN npm ci

# 3. Source code (changes constantly)
COPY . .
RUN npm run build

Move COPY . . above npm ci and every typo you fix triggers a full dependency reinstall. This is the most common Dockerfile mistake I see, and it’s the one with the largest daily cost.

.dockerignore

Everything in the build context gets shipped to the daemon before the build starts. Without a .dockerignore that includes your node_modules, your entire git history and any .env file lying around:

node_modules
.next
.git
.env*
*.md
coverage
.vscode
.DS_Store
docker-compose*.yml

Faster builds, and, more importantly, secrets that don’t accidentally end up in a layer that someone can docker history their way into later.

Security

Run as a non-root user, as in the Dockerfile above. Pin base images to something specific: node:22.14.0-alpine3.21 rather than node:22-alpine, which quietly moves under you. Keep credentials in runtime environment variables and never in the image. And put a scanner in the pipeline, either docker scout cves <image> or Trivy, because your base image will pick up CVEs whether or not you touched it.

Setting NEXT_TELEMETRY_DISABLED=1 is also worth doing, if only so your CI containers stop phoning home on every build.

Docker Desktop and the alternatives

Docker Desktop is the default and it’s fine, but it needs a paid licence for commercial use once a company passes 250 employees or 10 million dollars in revenue, and on macOS it’s a heavy neighbour.

OrbStack is what I’d point macOS users at first. It starts in a couple of seconds, sits well under a gigabyte of memory, and behaves like a native app. Free for personal use.
Colima runs from the terminal with no GUI at all, idles around 400 MB, and is fully open source.
Podman is daemonless and rootless by default, and CLI-compatible enough that aliasing docker to podman works for most everyday commands.

All of them run standard Dockerfiles and Compose files, so this isn’t a decision you have to get right first time.

When I’d skip it

Docker is not free, and pretending otherwise is how teams end up with a slow, resented development setup.

Reach for it when your app has neighbours: a database, a cache, a queue, a worker. Reach for it when your CI builds images anyway, so local and production share one definition. Reach for it when new people join often enough that onboarding time is a real cost.

Skip it for a static site or a pure SPA that talks to an API somebody else runs. Skip it when you’re one person on one machine and the setup is five minutes of npm install.

What I actually do most of the time is neither: Postgres and Redis in Compose, the Next.js dev server running natively on the host, pointed at localhost:5432. The services are the part that’s annoying to install and easy to containerise. The dev server is the part that’s trivial to run and painful to containerise. Splitting them that way gets you the consistency without ever thinking about Watchpack again.

For production, of course, everything goes in the image. That’s the one place the container is unambiguously the right answer.

Where this leaves my own setup

The Dockerfile for this site is still a two-stage build that produces static files and hands them to nginx, which is about as simple as it gets and needs none of the standalone machinery above. I’ve been meaning to pin the base image properly for a while now, having just written a whole section telling you to do exactly that 🙃

‘Till next time!

Component-Driven Development with Storybook

Sat, 21 Feb 2026 00:00:00 GMT

The button you can never get to

Every frontend project eventually grows a component you cannot look at. It only renders when a request fails, or when the user has no permissions, or when the list comes back empty on page four of the results. To see it you log in as the right user, click through five screens and then hand-edit the network response in devtools. Storybook exists so you can just open it instead.

The last time Storybook turned up here it was a supporting act, one piece of a screenshot testing setup alongside Jest and Puppeteer. It has grown a lot since then. What used to be a component explorer is now a second build of your UI that you can point tests, screenshots and accessibility checks at. That growth is mostly good and partly annoying, and I’ll get to the annoying part at the end.

This post is about getting it running on a Next.js project and the parts of it I actually use. For the wider picture of how component testing fits together, see testing strategies in React.

What a story is

A story is one rendered example of a component with a specific set of props. That’s the whole idea. Button gets a Default story, a Destructive story, a Loading story, and each one shows up as its own entry in the sidebar.

What you gain from that is mostly boring and mostly valuable:

You can build a component before the screen it belongs to exists, and before the API that feeds it exists.
The states nobody looks at, empty, loading, error, “user has a 40-character name”, get a permanent home instead of being checked once during development and then forgotten.
A published Storybook is something a designer can open in a browser without cloning the repository or running a dev server.
Stories are importable, so the same file that documents a state can drive a test of that state.

The last one is the reason I keep coming back. Test fixtures and design documentation are the same data, and Storybook is the only tool I’ve used that treats them that way.

Getting it running with Next.js

The installer detects your framework and writes the config for you:

npm create storybook@latest

(The older npx storybook@latest init still works and does the same job. The create form is what the docs point at now.)

You’ll be asked to pick a builder: Webpack via @storybook/nextjs, or Vite via @storybook/nextjs-vite. Take Vite. The builds are faster and, more to the point, the Vitest integration that powers everything in the testing sections below only works on the Vite framework. Picking Webpack here means opting out of half the tool.

The installer creates a .storybook directory, drops a few example stories in and adds scripts to package.json. Next.js specifics are handled for you: next/image, next/font, next/navigation, CSS Modules and the path aliases from your tsconfig.json all work without configuration. That sounds like a small thing until you’ve tried to render a Next.js component in a plain React Storybook and spent an afternoon stubbing out the router.

For a Tailwind project, import your stylesheet in the preview file so every story gets the same base styles as the app:

// .storybook/preview.ts
import '../src/globals.css';

const preview = {
  parameters: {
    controls: {
      matchers: {
        color: /(background|color)$/i,
        date: /Date$/i,
      },
    },
  },
};

export default preview;

Those matchers are worth setting up on day one. Any prop whose name ends in color gets a colour picker, anything ending in Date gets a date picker, across every story in the project. It saves a lot of repeated argTypes boilerplate later.

shadcn/ui components need nothing special. They’re ordinary React components with Tailwind classes on them, so as long as the CSS variables from globals.css are loaded they render exactly as they do in the app.

Writing stories

Stories use Component Story Format, which is a fancy name for “an ES module with a default export and some named exports”. The default export describes the component, each named export is a story.

// Button.stories.tsx
import type { Meta, StoryObj } from '@storybook/nextjs-vite';
import { Button } from './Button';

const meta = {
  component: Button,
  tags: ['autodocs'],
  argTypes: {
    variant: {
      options: ['default', 'destructive', 'outline', 'ghost'],
      control: { type: 'select' },
    },
  },
} satisfies Meta<typeof Button>;

export default meta;
type Story = StoryObj<typeof meta>;

export const Default: Story = {
  args: {
    children: 'Click me',
    variant: 'default',
  },
};

export const Destructive: Story = {
  args: {
    children: 'Delete',
    variant: 'destructive',
  },
};

export const Loading: Story = {
  args: {
    children: 'Saving...',
    disabled: true,
  },
};

Note the import path. In Storybook 9 the types come from the framework package, @storybook/nextjs-vite, not from @storybook/react. Older tutorials and half the answers on Stack Overflow still show the renderer package, and it will look like it works until it doesn’t.

args maps straight onto props, so anything you can pass to the component you can pass to a story. The satisfies Meta<typeof Button> is doing real work here: pass an arg the component doesn’t accept and TypeScript complains at compile time rather than leaving you with a silently ignored prop.

Decorators

Some components only make sense inside something else. A dropdown needs room to open, a sidebar item needs a sidebar. Decorators wrap a story in whatever context it needs:

export const InSidebar: Story = {
  decorators: [
    (Story) => (
      <div style={{ width: 250, padding: 16 }}>
        <Story />
      </div>
    ),
  ],
  args: {
    children: 'Sidebar Action',
    variant: 'ghost',
  },
};

They stack in three layers: on a single story, on the meta so every story of that component gets them, or globally in .storybook/preview.ts. Theme providers and i18n providers belong in the global layer. Layout wrappers usually belong on the story, because the whole point is that this particular story renders in a narrow column and the others don’t.

Controls

Controls are the panel underneath the story where you can change props by hand. Storybook infers them from your TypeScript types, so a boolean becomes a toggle and a union of string literals becomes a dropdown, with no configuration at all.

Where the inference isn’t good enough, argTypes takes over:

const meta = {
  component: Card,
  argTypes: {
    backgroundColor: { control: 'color' },
    padding: {
      control: { type: 'range', min: 0, max: 100, step: 4 },
    },
    size: {
      options: ['sm', 'md', 'lg'],
      control: { type: 'radio' },
    },
    icon: {
      control: false, // no sensible widget for a ReactNode
    },
  },
} satisfies Meta<typeof Card>;

Turning a control off with control: false is underrated. Storybook will happily generate a JSON editor for a prop that takes a React node, and nobody has ever typed valid JSX into it.

Play functions

A play function runs after the story renders. It’s the same idea as a Testing Library test, except the setup is the story you already wrote:

import { expect, fn, userEvent, within } from 'storybook/test';

export const SubmitForm: Story = {
  args: {
    onSubmit: fn(),
  },
  play: async ({ canvas, args, step }) => {
    await step('Fill in the form', async () => {
      await userEvent.type(
        canvas.getByLabelText('Email'),
        'user@example.com'
      );
      await userEvent.type(
        canvas.getByLabelText('Password'),
        'secretpassword'
      );
    });

    await step('Submit', async () => {
      await userEvent.click(
        canvas.getByRole('button', { name: 'Sign In' })
      );
    });

    await expect(args.onSubmit).toHaveBeenCalledOnce();
  },
};

canvas gives you Testing Library queries scoped to the story, userEvent does clicks, typing, hovering and keyboard navigation, and fn() creates a spy you can assert against. Note the import: everything comes from storybook/test now, a subpath of the core package. It used to be @storybook/test, and before that @storybook/jest plus @storybook/testing-library. Three homes in three majors.

The step calls are optional and I’d still write them. They group the interactions into collapsible sections in the Interactions panel, which turns a red test into something you can step through, pause and rewind, rather than a stack trace you squint at. For a five-step checkout flow that difference is the whole debugging session.

These same stories run headless in CI through Vitest. The addon writes a storybook project into your Vitest config, so the script is:

{
  "scripts": {
    "test-storybook": "vitest --project=storybook"
  }
}

Then npm run test-storybook in your pipeline. Every story is a test even without a play function, because rendering without throwing is already an assertion worth having. Stories with play functions get their interactions checked on top.

Visual testing

Play functions catch behaviour. They will not notice that a container overflows at 375px or that someone bumped a margin and shunted the whole card grid down by four pixels. That needs pixels compared against pixels, which I’ve written about before in a different context.

Chromatic is made by the Storybook team and plugs straight in:

npx storybook add @chromatic-com/storybook

On every push it screenshots each story across browsers, diffs against the accepted baseline and surfaces the changes on the pull request. You either accept the diff, which makes it the new baseline, or you don’t, which fails the build.

# .github/workflows/chromatic.yml
name: Visual Tests
on: push

jobs:
  chromatic:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: npm ci
      - uses: chromaui/action@latest
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}

The fetch-depth: 0 matters. Chromatic works out what changed by walking the git history, and a shallow clone leaves it comparing against nothing useful.

Be warned that visual tests are noisy until you tame them. Anything with a timestamp, a random avatar or an animation mid-flight will diff against itself forever. Freeze that data in the story rather than accepting diffs out of habit, because a team that clicks “accept all” every morning has a screenshot service, not a test suite.

Accessibility

The a11y addon runs axe-core against every story:

npx storybook add @storybook/addon-a11y

An Accessibility panel appears next to Interactions, listing violations, passes and the checks axe couldn’t decide about, each with a link to what to do about it. Storybook’s own docs put the ceiling at “up to 57% of WCAG issues”, which is the figure Deque quote for automated checking generally. It’s a floor for your process, not a certificate. Contrast, missing labels and bad ARIA get caught; whether your focus order makes sense to somebody using a screen reader does not.

Rules can be adjusted per story or per component:

export const WithCustomA11y: Story = {
  parameters: {
    a11y: {
      test: 'error',
      config: {
        rules: [
          { id: 'color-contrast', enabled: true },
          { id: 'landmark-one-main', enabled: false },
        ],
      },
    },
  },
};

The test key decides what happens in CI: 'error' fails the run, 'todo' records the violation without failing, 'off' skips the story. Starting a legacy project on 'todo' and moving components to 'error' as you fix them is far more likely to succeed than switching the whole thing on and drowning in red.

Autodocs

Add tags: ['autodocs'] to the meta and Storybook generates a documentation page: every story rendered in sequence, plus a props table pulled from the TypeScript types.

const meta = {
  component: Button,
  tags: ['autodocs'],
  parameters: {
    docs: {
      description: {
        component: 'Primary UI button with multiple variants and sizes.',
      },
    },
  },
} satisfies Meta<typeof Button>;

For anything more involved, Storybook takes MDX files, so you can write prose around live examples and drop in doc blocks like <Canvas>, <Controls> and <Source> where you want them.

The value here isn’t that the docs are pretty. It’s that they’re generated from the same source as the components, so renaming a prop updates the table by itself. Every hand-written component page I’ve maintained in a wiki was wrong within two months.

Addons worth keeping

The addon list is long and most of it you will never open. These are the ones I leave installed:

Viewport for checking layouts at phone and tablet widths without resizing the window.
Themes for flipping between light and dark, which is where about half of my contrast bugs turn up.
Backgrounds for putting a component on a dark surface, because “looks fine on white” is not the same as fine.
Measure & Outline for when the spacing is off by an amount you can see but not name.

Everything else I’ve installed at some point and removed again. Addons go in .storybook/main.ts and most need no configuration beyond being listed.

One story, several kinds of test

Because stories are plain modules, other tools can import them. That gives you layers of checking over a single definition:

Test type	Tool	What it catches
Render	Stories themselves	Crashes, missing providers
Interaction	Play functions via Vitest	Broken flows, handlers that never fire
Visual	Chromatic	Layout shifts, contrast, overflow
Accessibility	a11y addon (axe-core)	Missing labels, bad roles, contrast

End-to-end tools can point at the story URL directly, which is handy for testing a single component under a real browser without booting the whole app:

import { test, expect } from '@playwright/test';

test('form submission works', async ({ page }) => {
  await page.goto('/storybook/iframe.html?id=forms-login--submit-form');
  await expect(page.getByText('Welcome back')).toBeVisible();
});

What it costs

Now the annoying part I promised. Storybook is a second build of your application, and second builds need feeding. The config drifts from the app config, a Next.js upgrade breaks the framework package before the framework package catches up, and the import paths move on every major version. The testing utilities alone have lived in three different packages across three releases, and each move meant a morning of codemods and confused imports.

There’s also the rot problem. Stories only stay honest if somebody looks at them. A component library with 200 stories and no visual testing has 200 files that used to be true, and nobody notices until a designer opens the published build and asks why half of it looks nothing like production. Wiring up Chromatic or the Vitest run on day one is what keeps that from happening, not good intentions.

So it isn’t free, and on a small project with two components it isn’t worth it. Where it pays is a shared component library, or any codebase where more than one person edits the UI and nobody can hold all the states in their head.

Next thing on my list is getting the Vitest run fast enough to live in a pre-commit hook instead of only in CI. Not there yet

‘Till next time!

Looking back at 2025

Sat, 31 Jan 2026 00:00:00 GMT

A year spent in the same stack

2025 was a quiet year, and after the last few I'll happily take quiet. The monthly writing rhythm held, which I'm pleased about. Reading the list back, though, it's obvious I had one topic stuck in my head for twelve months straight.

Auth, payments, emails, server actions, styling, linting, type-safe APIs, analytics, documentation. Written out like that it looks deliberate, as if I’d sat down in January and drawn up a curriculum. I didn’t. Each month I wrote about whatever I happened to be wrestling with, and it turns out I spent the whole year wrestling with the same application.

What I’d change

Two of those are the odd ones out. The agents piece and the frameworks piece are me speculating about where things are heading, and they’re the two I’m least happy with. Predictions age badly and I have no particular advantage at making them.

So for 2026: fewer forecasts, more write-ups of things I’ve actually installed, run and broken.

See you all in 2026 🎉

‘Till next time!

Beautiful Documentation with Fumadocs

Sat, 20 Dec 2025 00:00:00 GMT

Docs that live in the same app

Documentation usually ends up in a second repository, with a second build system, a second deployment target and a second set of design decisions that slowly drift from the product it describes. It works, and it is nobody's favourite part of the week. Fumadocs takes the other route: the docs are routes in your existing Next.js app, sharing its bundler, its Tailwind config and its components.

I wrote about Docusaurus a while back and ended by saying that MkDocs with Material is almost always enough. I still think that. Fumadocs is the exception to it, and only for one specific case: when the thing you are documenting is itself a Next.js or React project, and you want the documentation to be able to render the real component rather than a screenshot of it.

Everything below is against the 16.x line of fumadocs-core and fumadocs-ui, with fumadocs-mdx at 14. The API moved around quite a bit through 2025, so version numbers matter more than usual here.

What makes it different

Docusaurus is its own application. It has a build system, a routing model and a theming approach that are all its own. It does not care that you already have a Next.js project; it sits next to one. Nextra grew up in the Pages Router era and has spent a while catching up with the App Router.

Fumadocs was written for React Server Components from the start. Your pages render on the server and ship almost no client JavaScript; the only interactive parts are the search dialog and the sidebar toggles. If you already run Next.js with Tailwind, adding docs is adding a route group, not adopting a second framework.

The adoption list is the part that made me look properly. Zod, Better Auth, shadcn/ui and several of Vercel’s own SDK docs run on it, and those are projects whose documentation people actually read.

The other structural choice is that it splits into a headless core and a styled layer:

fumadocs-core does the work with no opinions about appearance: the Source API, search adapters, Markdown utilities.
fumadocs-ui is the theme. Sidebar, table of contents, breadcrumbs, code blocks, callouts, built on Tailwind CSS v4.
fumadocs-mdx turns MDX files on disk into typed data, with validated frontmatter and a generated table of contents.
create-fumadocs-app scaffolds the whole thing.

You can take the core alone and build your own presentation. Almost nobody does at first, and the option matters mostly because it means the styled layer is not load-bearing. You can replace a piece of it without forking anything.

Getting it running

npm create fumadocs-app

That scaffolds a working Next.js project. In an existing one:

npm install fumadocs-core fumadocs-ui fumadocs-mdx

The MDX pipeline is a Next.js plugin, so next.config.ts has to know about it:

// next.config.ts
import { createMDX } from 'fumadocs-mdx/next';

const withMDX = createMDX();

export default withMDX({});

Declaring the content

source.config.ts at the project root describes what exists and what shape it has:

// source.config.ts
import { defineDocs, frontmatterSchema } from 'fumadocs-mdx/config';
import { z } from 'zod';

export const docs = defineDocs({
  dir: 'content/docs',
  docs: {
    schema: frontmatterSchema.extend({
      category: z.enum(['guides', 'api', 'tutorials']).optional(),
      audience: z.enum(['user', 'developer']).default('developer'),
    }),
  },
});

Two things to get right here. schema takes an actual schema object, not a bag of field definitions. Plenty of tutorials show a plain object literal there, and it will not validate. And you extend frontmatterSchema rather than replacing it, because that is what already defines title and description for the rest of the pipeline.

The payoff is that a typo in a frontmatter key fails at build time with a message naming the file, instead of rendering an empty sidebar entry that nobody notices for two months. Zod is the obvious choice but any Standard Schema library works.

The loader

// lib/source.ts
import { loader } from 'fumadocs-core/source';
import { docs } from '@/.source';

export const source = loader({
  baseUrl: '/docs',
  source: docs.toFumadocsSource(),
});

.toFumadocsSource() is the bit people leave off. The generated docs export is a collection of entries; the loader wants a Source, and that method converts one into the other. Skip it and TypeScript will tell you, at some length.

What comes back is the Source API: getPage(slugs), getPages(), getPageTree(), generateParams() for static generation. That object is the only interface between your content and your pages.

Layout and page

// app/docs/layout.tsx
import { DocsLayout } from 'fumadocs-ui/layouts/docs';
import { source } from '@/lib/source';
import type { ReactNode } from 'react';

export default function Layout({ children }: { children: ReactNode }) {
  return (
    <DocsLayout
      tree={source.getPageTree()}
      nav={{ title: 'My Docs' }}
      sidebar={{ defaultOpenLevel: 1 }}
    >
      {children}
    </DocsLayout>
  );
}

// app/docs/[[...slug]]/page.tsx
import { source } from '@/lib/source';
import { DocsPage, DocsBody } from 'fumadocs-ui/layouts/docs/page';
import { notFound } from 'next/navigation';

export default async function Page({
  params,
}: {
  params: Promise<{ slug?: string[] }>;
}) {
  const { slug } = await params;
  const page = source.getPage(slug);

  if (!page) notFound();

  const MDX = page.data.body;

  return (
    <DocsPage toc={page.data.toc}>
      <DocsBody>
        <MDX />
      </DocsBody>
    </DocsPage>
  );
}

export function generateStaticParams() {
  return source.generateParams();
}

That is the whole integration. Drop MDX files into content/docs/ and they appear, with navigation, breadcrumbs and a table of contents built from the headings.

Writing in it

Code blocks go through Shiki at build time, so there is no highlighting library in the client bundle. Line ranges, titles and copy buttons come from the fence:

```typescript title="api/users.ts" {3-5}
import { db } from '@/lib/db';

export async function getUsers() {
  return db.user.findMany();
}
```

Callouts, tabs, steps and cards are components you can use directly in MDX:

<Callout type="warn">
  This deletes data and does not ask twice.
</Callout>

<Tabs groupId="package-manager" items={['npm', 'pnpm', 'yarn']}>
  <Tab value="npm">npm install fumadocs-ui</Tab>
  <Tab value="pnpm">pnpm add fumadocs-ui</Tab>
  <Tab value="yarn">yarn add fumadocs-ui</Tab>
</Tabs>

The groupId on tabs is the detail I did not expect to care about and now would not give up. Every tab group sharing that id switches together, and the choice persists as the reader moves between pages. A pnpm user picks pnpm once at the top of the install guide and never sees an npm install again. It is a tiny thing that removes a constant low-grade annoyance from every documentation site that has ever existed.

For library documentation there is TypeTable, which renders a props or options table without you hand-maintaining a Markdown grid:

import { TypeTable } from 'fumadocs-ui/components/type-table';

<TypeTable
  type={{
    name: {
      type: 'string',
      description: 'The user display name',
      required: true,
    },
    role: {
      type: "'admin' | 'user'",
      description: 'User role',
      default: "'user'",
    },
  }}
/>

Search

Orama is the default and it is self-hosted, meaning it runs in your own route handler with no external service:

// app/api/search/route.ts
import { source } from '@/lib/source';
import { createFromSource } from 'fumadocs-core/search/server';

export const { GET } = createFromSource(source);

Four lines, and Cmd+K works. Fumadocs indexes your content structurally (headings, sections, paragraphs) rather than dumping page text into a blob, so results land you on the right section rather than the top of a long page.

If you outgrow it, there is an Algolia adapter, though it works differently from what you might expect. It is not a drop-in replacement for the route handler: you write a sync script that runs after a build and pushes records up.

// scripts/sync-algolia.ts
import { algoliasearch } from 'algoliasearch';
import { sync } from 'fumadocs-core/search/algolia';
import { source } from '@/lib/source';

const client = algoliasearch(
  process.env.ALGOLIA_APP_ID!,
  process.env.ALGOLIA_ADMIN_KEY!
);

void sync(client, {
  indexName: 'docs',
  documents: source.getPages().map((page) => ({
    _id: page.url,
    title: page.data.title,
    description: page.data.description,
    url: page.url,
    structured: page.data.structuredData,
  })),
});

There is an Orama Cloud adapter too if you want hosted search without Algolia’s pricing conversation. For anything under a few hundred pages, plain Orama is the right answer and adding a search vendor is work you have invented for yourself.

API reference from OpenAPI

fumadocs-openapi is a separate package (npm i fumadocs-openapi shiki, plus one more stylesheet import), and worth mentioning because most documentation frameworks make you find a third-party plugin for this and then maintain it. You point createOpenAPI from fumadocs-openapi/server at an OpenAPI 3.0 or 3.1 document and it generates reference pages: endpoints, parameter tables, request and response schemas, example payloads, styled like the rest of the site rather than like an embedded Swagger UI.

The value is not the rendering. It is that the reference is generated from the same spec your server validates against, so it cannot drift. Hand-written endpoint tables always drift.

Theming

fumadocs-ui is built on Tailwind CSS v4, which means the CSS-first configuration I wrote about in the Tailwind v4 post applies directly. Eight colour themes ship with it: neutral, black, vitepress, dusk, catppuccin, ocean, purple and solar. You pick one by importing its stylesheet.

@import 'tailwindcss';
@import 'fumadocs-ui/css/neutral.css';
@import 'fumadocs-ui/css/preset.css';

@theme {
  --color-fd-primary: oklch(0.55 0.2 250);
  --color-fd-background: oklch(0.98 0 0);
}

Every token is prefixed --color-fd-*, which keeps them out of the way of your application’s own variables. That is a small piece of foresight, and it saves a genuinely annoying afternoon when the docs and the app share a stylesheet.

There is also a shadcn.css that maps the Fumadocs tokens onto shadcn/ui’s colour system. If your app already uses shadcn/ui, that one import makes the documentation look like it belongs to the product rather than next to it.

Beyond colour, the layout handles RTL, collapsible sidebar folders with configurable open depth, root-level tabs for splitting “Guides” from “API Reference”, banners for version notices, and a mobile navigation that you do not have to think about.

Multiple languages

i18n is configured once and threaded through the loader:

// lib/i18n.ts
import { defineI18n } from 'fumadocs-core/i18n';

export const i18n = defineI18n({
  languages: ['en', 'de', 'fr'],
  defaultLanguage: 'en',
  hideLocale: 'default-locale',
});

// lib/source.ts
export const source = loader({
  baseUrl: '/docs',
  source: docs.toFumadocsSource(),
  i18n,
});

A page tree is built per language, missing translations fall back to the default, and the search index is built per locale so people search in the language they are reading. hideLocale: 'default-locale' keeps /docs/getting-started rather than /docs/en/getting-started, which matters if you have existing URLs and an SEO person.

Living in Switzerland, I have opinions about translation workflows that are not really about tooling. Fumadocs does the mechanical part properly. Keeping four languages actually in sync is a staffing problem, not a framework problem.

The sidebar comes from the directory structure. Nested folders become nested sections. Where you want to override the ordering or the label, drop a meta.json into the folder:

{
  "title": "Getting Started",
  "pages": ["installation", "quickstart", "configuration"]
}

This is one of the few places I would have designed it differently. Ordering lives in meta.json, page metadata lives in frontmatter, and when a page is in the wrong place you have to remember which file to open. It is a small tax and it never quite stops being one.

Performance

MDX compiles to Server Components, so a documentation page ships the HTML and essentially nothing else. generateStaticParams() means every page can be prerendered at build time, and Orama’s index can be prerendered too, which leaves you with a directory of static files and no server runtime at all.

The Source API works from in-memory structures built during the build. There is no database, no external service, nothing to provision. For a documentation site that is exactly the right amount of infrastructure.

Where it doesn’t fit

	Fumadocs	Docusaurus	Nextra
Runs inside	Your Next.js app	Its own build	Next.js
Rendering	RSC, minimal JS	Client-side	Pages or App Router
Styling	Tailwind v4	CSS Modules	Tailwind or CSS
Content	MDX with a typed schema	MDX	MDX
Search	Orama, Orama Cloud, Algolia	Algolia	Flexsearch
API reference	`fumadocs-openapi`	Plugin	Roll your own
Customising	Headless core plus UI layer	Swizzling	Theme options

Docusaurus has a much larger community, more plugins and more Stack Overflow answers, and that is worth real money when something breaks at an awkward hour. Fumadocs is younger and moves faster: the API churn through 2025 was not trivial, and reading a tutorial written six months ago will hand you imports that no longer exist.

And the complaint I had about Docusaurus applies here twice over. It is React all the way down. A broken MDX import fails your production build, and now it fails your product’s production build, because they are the same build. That is arguably correct and it is not the trade I would make for a Python service’s documentation, where Sphinx or MkDocs will serve you better and nobody has to learn anything.

Where I’ve landed

For a Next.js project whose docs need to render live components: Fumadocs, without much hesitation. The docs inherit the design system, the components are the real ones, and there is no second deployment to forget about.

For everything else, my answer from the Docusaurus post stands. MkDocs with Material is almost always enough, and “almost always enough” remains an underrated property in a documentation tool.

What I want to try next is pointing fumadocs-openapi at a spec generated from a running server, so the reference pages fail the build when an endpoint changes shape. That feels like the version of this that would actually keep documentation honest

‘Till next time!

Product Analytics with PostHog

Sat, 22 Nov 2025 00:00:00 GMT

One vendor instead of five

The analytics stack for a small product tends to accrete rather than get designed. Something for events, something else for session recordings, a third thing for feature flags, a fourth when somebody wants an A/B test, and by then nobody can answer "did the users who saw variant B actually convert" without exporting two CSVs and joining them in a spreadsheet.

PostHog’s pitch is that all of it belongs in one database. Analytics, replays, flags, experiments and surveys share the same event stream, which means the join is a filter rather than an export. It is open source, roughly 30,000 stars on GitHub, and the free tier is generous enough that a side project will never see an invoice.

This is about wiring it into a Next.js App Router project, which has a few sharp edges that the marketing pages do not mention.

Why this one

Count the vendors in a typical setup: Google Analytics or Mixpanel for product analytics, Hotjar for recordings, LaunchDarkly for flags, Optimizely for experiments, Typeform for surveys. Five contracts, five scripts on the page, five dashboards, and five different ideas about what a “user” is.

The open source part matters to me more than it probably should. The core repository is MIT licensed, with the ee/ directory under a separate commercial licence, so you can read exactly what the SDK collects rather than trusting a minified bundle. And there is an EU region if your client’s legal department asks where the data lives, which in Switzerland they generally do.

The free tier at the time of writing covers a million analytics events, 5,000 session recordings, a million feature flag requests and 1,500 survey responses a month. That is enough to run a small product on indefinitely, which is a rare thing to be able to say about a free tier.

Getting it in

Two packages, and they do different jobs. posthog-js runs in the browser, posthog-node runs on the server, and (this is the bit worth knowing up front) they do not automatically know about each other.

npm install posthog-js posthog-node

Environment variables:

# .env.local
NEXT_PUBLIC_POSTHOG_KEY=phc_your_project_api_key
NEXT_PUBLIC_POSTHOG_HOST=https://us.i.posthog.com

Since Next.js 15.3 there is an instrumentation-client.ts file that runs before your application code hydrates, which is exactly where SDK initialisation belongs:

// instrumentation-client.ts
import posthog from 'posthog-js';

posthog.init(process.env.NEXT_PUBLIC_POSTHOG_KEY!, {
  api_host: '/ingest',
  ui_host: 'https://us.posthog.com',
  defaults: '2025-05-24',
});

That defaults value deserves an explanation, because it looks like a version string somebody forgot to remove. PostHog uses dated defaults so they can change behaviour without breaking existing installs: you opt into the behaviour as of a given date. '2025-05-24' switches pageview capture to 'history_change', which means client-side navigations in the App Router are captured automatically.

That matters because most PostHog tutorials for Next.js still show a hand-written PostHogPageView component that watches usePathname() and useSearchParams() and fires a $pageview on every change. You needed that once. With dated defaults you do not, and the hand-rolled version has a subtle bug anyway. useSearchParams() forces the component into client-side rendering unless you wrap it in a Suspense boundary, which is easy to forget.

For the React hooks you still need a provider, in a client component:

// app/providers.tsx
'use client';

import posthog from 'posthog-js';
import { PostHogProvider } from 'posthog-js/react';

export function Providers({ children }: { children: React.ReactNode }) {
  return <PostHogProvider client={posthog}>{children}</PostHogProvider>;
}

Wrap {children} in your root layout with that, and you are collecting.

A note on imports, because this changed recently and will bite somebody: PostHog has published the React bindings as a standalone @posthog/react package, and switched their documentation over to it in the last couple of weeks. posthog-js/react still works. Check which one the docs page you are reading assumes before you spend twenty minutes on an import error.

Autocapture, and its limits

Without writing a line of tracking code, PostHog records clicks on interactive elements, form submissions, pageviews and page leaves, scroll depth, and the four Core Web Vitals: LCP, CLS, FCP and INP. (If you see an older guide mentioning FID, it predates INP replacing it in early 2024.)

The value of autocapture is not that it replaces deliberate instrumentation. It is that when someone asks in three months “how many people ever clicked that button”, the answer exists retroactively, because you were capturing it before anyone thought to ask. That is genuinely useful and I have stopped arguing about it.

The limit is that autocaptured events are described by DOM structure. Rename a CSS class, restructure a component, and your “clicks on .btn-primary in the pricing page” series quietly becomes two series. Anything you plan to report on for more than a quarter should be a named custom event.

You can narrow what gets captured:

posthog.init(process.env.NEXT_PUBLIC_POSTHOG_KEY!, {
  api_host: '/ingest',
  defaults: '2025-05-24',
  autocapture: {
    dom_event_allowlist: ['click', 'submit'],
    url_allowlist: ['https://yourdomain.com/app/*'],
    css_selector_allowlist: ['[data-track]'],
  },
});

And anything carrying a ph-no-capture class is excluded outright. Put it on payment fields and anything showing another person’s data.

Custom events

'use client';

import posthog from 'posthog-js';

export function PricingCard({ plan }: { plan: string }) {
  const handleUpgrade = () => {
    posthog.capture('upgrade_clicked', {
      plan,
      source: 'pricing_page',
    });
  };

  return <button onClick={handleUpgrade}>Upgrade to {plan}</button>;
}

Properties are arbitrary. Spend a few minutes deciding on naming before you have two hundred of them: snake_case, verb in the past tense, and the same property key meaning the same thing everywhere. Nobody enforces this and everybody regrets not doing it.

Identifying people

'use client';

import posthog from 'posthog-js';
import { useEffect } from 'react';

export function IdentifyUser({
  user,
}: {
  user: { id: string; email: string } | null;
}) {
  useEffect(() => {
    if (user) {
      posthog.identify(user.id, { email: user.email });
    }
  }, [user]);

  return null;
}

identify merges the anonymous session into the known person, so the pages they read before signing up stay attached to them. Call posthog.reset() on logout, or the next person to use that browser inherits the identity, which on a shared machine is a data protection problem rather than just a messy chart.

Feature flags

Create a flag in the dashboard, read it in a component:

'use client';

import { useFeatureFlagEnabled } from 'posthog-js/react';

export function Dashboard() {
  const showNewDashboard = useFeatureFlagEnabled('new-dashboard');

  if (showNewDashboard) return <NewDashboard />;
  return <OldDashboard />;
}

Three kinds exist: boolean flags for on/off, multivariate flags for several named variants with rollout percentages, and remote config for values you want to change without a deploy. Targeting works on person properties, cohorts, percentage rollouts, geography, and dependencies on other flags.

On the server you use posthog-node:

// lib/posthog-server.ts
import { PostHog } from 'posthog-node';

export const posthogServer = new PostHog(
  process.env.NEXT_PUBLIC_POSTHOG_KEY!,
  { host: 'https://us.i.posthog.com' }
);

import { posthogServer } from '@/lib/posthog-server';
import { getSession } from '@/lib/auth';

export default async function Page() {
  const session = await getSession();
  const flags = await posthogServer.getAllFlags(session.user.id);

  return <div>{flags['new-header'] ? <NewHeader /> : <OldHeader />}</div>;
}

The hook returns undefined on first render while flags load, which produces a visible flicker: the old header paints, then swaps. The fix is bootstrapping: evaluate the flags on the server and hand them to posthog.init as bootstrap: { featureFlags }, so the client has values before it renders anything. It is more wiring than it should be, and it is the difference between a flag you can use above the fold and one you cannot.

Session replay

Replays are DOM snapshots, not video, so they are small and the text in them is selectable.

posthog.init(process.env.NEXT_PUBLIC_POSTHOG_KEY!, {
  api_host: '/ingest',
  defaults: '2025-05-24',
  session_recording: {
    maskAllInputs: true,
    maskTextSelector: '.sensitive-data',
  },
});

Masking happens in the browser before anything is sent, which is the only design that survives a conversation with a privacy officer. You can also limit when recording happens: only on certain URLs, only after an error event, only for users in a given flag, or a percentage sample to keep within quota.

The reason replay is worth having in the same tool as everything else is the cross-referencing. Watch the recordings of the twelve people who dropped out of your signup funnel at step three. That is one filter, not an export. Twelve recordings will tell you more than a month of staring at the funnel chart.

Experiments

Experiments sit on top of multivariate flags:

'use client';

import { useFeatureFlagVariantKey } from 'posthog-js/react';

export function CheckoutButton() {
  const variant = useFeatureFlagVariantKey('checkout-experiment');

  if (variant === 'test') {
    return <button className="btn-green">Complete Purchase</button>;
  }

  return <button className="btn-blue">Checkout</button>;
}

You define the success metric in the dashboard and PostHog does the statistics. It defaults to a Bayesian analysis, so the result reads as “a 96% probability that variant B is better” rather than a p-value that half the people in the room will interpret incorrectly. A frequentist mode is available if your organisation prefers it.

You can run one control against up to nine variants, test at group level rather than per person (useful when the unit of decision is a company account, not an individual), and set aside a holdout group to check that the accumulated wins are real.

Surveys

Most surveys need no code at all: build a popover in the dashboard, target it at a URL pattern or a person property, and it appears. When you want to trigger one yourself:

'use client';

import posthog from 'posthog-js';

export function FeedbackButton() {
  return (
    <button
      onClick={() =>
        posthog.displaySurvey('survey-id', { displayType: 'popover' })
      }
    >
      Give Feedback
    </button>
  );
}

Lowercase 'popover'; 'inline' is the other option and needs a selector telling it where to render. renderSurvey still exists and still works, but it was deprecated a couple of months ago in favour of displaySurvey, so start with the new one.

Question types cover free text, ratings on emoji or numeric scales, and single or multiple choice. Targeting can key off a flag, a URL, a device type, person properties, or an event that happened earlier in the session, which is how you ask “what went wrong?” only of people who actually hit the error.

The proxy is not optional

A meaningful share of visitors run an ad blocker, and every one of them blocks requests to i.posthog.com. Route the traffic through your own domain instead:

// next.config.js
const nextConfig = {
  async rewrites() {
    return [
      {
        source: '/ingest/static/:path*',
        destination: 'https://us-assets.i.posthog.com/static/:path*',
      },
      {
        source: '/ingest/array/:path*',
        destination: 'https://us-assets.i.posthog.com/array/:path*',
      },
      {
        source: '/ingest/:path*',
        destination: 'https://us.i.posthog.com/:path*',
      },
    ];
  },
};

Three rules, and the order is load-bearing: the catch-all must come last or it swallows the other two. The /array/ one serves remote config and is the one people leave out, then wonder why their flags never arrive. Set api_host: '/ingest' (as above) and ui_host to the real PostHog URL so the toolbar links still work. Swap us for eu on the EU region.

Tracking from the server

Anything that happens without a browser (a webhook, a Stripe callback, a cron job) needs posthog-node:

'use server';

import { posthogServer } from '@/lib/posthog-server';

export async function purchaseAction(formData: FormData) {
  const session = await getSession();
  const plan = formData.get('plan') as string;

  await processPurchase(plan);

  posthogServer.capture({
    distinctId: session.user.id,
    event: 'purchase_completed',
    properties: { plan, source: 'upgrade_page' },
  });
}

Note the explicit distinctId. Nothing links the browser’s anonymous ID to your server events for you: if you pass a different identifier on the server than the one you passed to posthog.identify on the client, you get two person records for the same human, and every funnel that crosses the boundary breaks. Pick the identifier once, put it in a helper, and use it everywhere.

The other server-side trap is flushing. posthog-node batches and sends on an interval, which is fine on a long-lived process and useless in a serverless function that gets frozen the moment it returns. On Vercel, wrap the capture in the after() API and call flush() there, or shutdown() if the process really is ending. Events you never flushed do not show up as an error anywhere. They just are not there.

What the data is for

Once events are flowing, the dashboard gives you trends over time, funnels through a multi-step process, retention cohorts, path analysis, lifecycle segmentation into new/returning/resurrecting/dormant, and stickiness. All of it filters on person properties, flag state, geography and anything you attached to an event. When the built-in charts run out, HogQL lets you write SQL directly against the event table.

The part that took me longest to appreciate is that you should build fewer of these than you think. A dashboard with thirty charts is a dashboard nobody reads. Three numbers somebody checks on a Monday will change more decisions.

Self-hosting, honestly

You can run the whole thing yourself:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)"

Note the form: the script prompts you interactively, so piping it into bash lets the pipe swallow its own input and it sits there looking hung.

It wants something like 4 vCPUs, 16GB of RAM and more than 30GB of disk. And PostHog are unusually direct that self-hosting is unsupported: no help if it breaks, none of the paid features, and the data is your problem. Given that it is ClickHouse, Kafka, Redis and Postgres in a trench coat, I would take them at their word unless data residency leaves you no choice, and the EU cloud region probably already solves that.

Where I’d stop

PostHog’s breadth is the reason to use it and the reason to be a little sceptical of it. LaunchDarkly’s flag targeting is more sophisticated. Hotjar’s heatmaps are more mature. Mixpanel’s cohort analysis goes deeper. If one of those is genuinely core to how your team works, PostHog will feel like the second-best tool for that job.

For everyone else, having one event stream that flags, replays and experiments all agree on is worth more than any individual feature being best in class. That is the actual argument, and the vendor consolidation is a side effect.

What I would still like is a decent story for identity across the client/server boundary that does not involve me threading a distinct ID through by hand. PostHog have been circling something for Next.js specifically; when it lands I will happily delete a helper file

‘Till next time!

What's New in Next.js 15

Sat, 25 Oct 2025 00:00:00 GMT

The release that took a default back

Frameworks rarely admit that a default was wrong. Next.js 15 does, twice. Fetches are no longer cached behind your back, GET route handlers are no longer static by accident, and the request APIs you have been calling synchronously for years now hand you a promise. It is the least glamorous kind of release and, I think, the most useful one Next.js has shipped.

Next.js 16 landed four days ago, which makes writing about 15 feel slightly odd. It is not, though: most of the codebases I touch are on some 15.x, and they will be for a good while yet. Everything below is about that line, from 15.0 through the point releases that quietly fixed half of it.

The two changes worth reading carefully are the async request APIs and the caching reversal. Both require you to do something. The rest is upside.

Turbopack, now by choice

The Rust bundler went stable for development in 15. You opt in on the dev script:

next dev --turbopack

Vercel published numbers from their own application, which is a large real-world Next.js codebase rather than a benchmark repository:

up to 76.7% faster local server startup
up to 96.3% faster code updates with Fast Refresh
up to 45.8% faster initial route compile, with no disk caching at that point

15.2 pushed compile times a further 57.6% down against 15.1, with roughly 30% less memory used during local development. My own experience is less dramatic than those headlines but consistently better, and the gap widens the bigger the project gets.

Turbopack takes a different bet from Vite. Vite serves modules unbundled over native ESM in dev, which starts instantly and then pays a little on every request. Turbopack bundles, but lazily, compiling only what the route you are looking at actually needs. You get fast startup without the dev/production divergence that unbundled ESM occasionally produces.

Under the hood it is SWC for JavaScript and TypeScript, Lightning CSS for stylesheets. Webpack loaders work through the turbopack.rules config (that key lived under experimental.turbo until 15.3, so check which version you are reading docs for). Webpack plugins do not work at all, and there is no compatibility layer coming. If your build depends on a plugin, that is your blocker, and it is worth finding out on day one rather than after you have rewritten your dev script.

Everything request-shaped is now a promise

This is the breaking change. These are all asynchronous:

cookies()
headers()
draftMode()
params, in layouts, pages, route handlers, default.js, generateMetadata and generateViewport
searchParams, in pages

So this, from 14:

import { cookies } from 'next/headers';

export default function Dashboard() {
  const cookieStore = cookies();
  const token = cookieStore.get('session');

  return <div>Dashboard for {token?.value}</div>;
}

becomes this:

import { cookies } from 'next/headers';

export default async function Dashboard() {
  const cookieStore = await cookies();
  const token = cookieStore.get('session');

  return <div>Dashboard for {token?.value}</div>;
}

Props follow the same rule:

export default async function Page({
  params,
  searchParams,
}: {
  params: Promise<{ slug: string }>;
  searchParams: Promise<{ query: string }>;
}) {
  const { slug } = await params;
  const { query } = await searchParams;

  return <div>Slug: {slug}, Query: {query}</div>;
}

The reasoning is worth understanding, because otherwise this looks like churn for its own sake. In traditional server rendering, nothing happens until a request shows up. But plenty of your component tree does not care about the request at all: a header, a nav, a marketing block. If the framework can tell which components need request data and which do not, it can render the second group ahead of time. Making the request APIs async is how it tells. That is the foundation for Partial Prerendering, further down.

A codemod does most of the work:

npx @next/codemod@canary next-async-request-api .

It handles direct call sites well and gives up on anything indirect: a helper that takes cookies() as an argument, a params object destructured three levels deep, anything behind a conditional. Expect to finish by hand. The synchronous forms still work in 15 with a deprecation warning, so you can migrate over a few pull requests rather than in one heroic commit.

Nothing is cached until you say so

Next.js 14 cached aggressively by default, and the resulting confusion was the single most common complaint I heard about the App Router. You would ship a fix, refresh, and see the old data. 15 reverses the defaults.

Fetches

// Not cached (the 15 default)
const data = await fetch('https://api.example.com/posts');

// Explicitly cached
const data = await fetch('https://api.example.com/posts', {
  cache: 'force-cache',
});

// Cached, revalidated hourly
const data = await fetch('https://api.example.com/posts', {
  next: { revalidate: 3600 },
});

GET route handlers

In 14, a GET handler that used no dynamic function was rendered once at build time. In 15 it is dynamic unless you say otherwise:

export const dynamic = 'force-static';

export async function GET() {
  return Response.json({ data: 'cached' });
}

Metadata files are the exception. sitemap.ts, opengraph-image.tsx, icon.tsx and friends stay static by default unless they reach for something dynamic.

The client router cache

Page segments now have a staleTime of 0, so navigating to a route fetches its data again. Layouts still are not refetched, which keeps partial rendering intact, and back/forward navigation still restores from cache so the browser can put your scroll position back. loading.js stays cached for five minutes.

If you liked the old behaviour:

// next.config.js
const nextConfig = {
  experimental: {
    staleTimes: {
      dynamic: 30,
    },
  },
};

I would not reach for that config, personally. Stale data is a horrible class of bug to debug because it never reproduces on your machine, and you spend the first hour convinced the problem is in your query. Choosing correctness first and adding caching deliberately where the profile says it matters is simply the better order to work in.

React 19 underneath

15.0 shipped the App Router on React 19 RC; 15.1 moved to React 19 stable, and the Pages Router kept working with React 18 for anyone not ready. In practice that means everything from the React 19 release is available: Server Actions as real form handlers, useActionState, useFormStatus, useOptimistic, and use for reading a promise during render.

React 19 also added sibling pre-warming for Suspense. When one boundary suspends, React begins work on its siblings instead of sitting idle, which shows up as less staggered loading on pages with several independent data sources.

`after()`

Some work has no business blocking a response. Logging, analytics, syncing a row to some other system. On a serverless platform you cannot simply fire a promise and walk away, because the function is torn down the moment the response closes.

import { after } from 'next/server';

export default function Page() {
  after(() => {
    analytics.track('page_view', { page: '/dashboard' });
  });

  return <Dashboard />;
}

It arrived as unstable_after behind an experimental flag in 15.0 and went stable in 15.1, dropping the prefix. It works in Server Components, Server Actions, route handlers and middleware. The important difference from a bare waitUntil is that the framework owns the lifecycle, including on self-hosted servers, so the callback still runs when the client disconnects mid-stream.

The `<Form>` component

next/form is a small thing that removes a nuisance:

import Form from 'next/form';

export default function SearchForm() {
  return (
    <Form action="/search">
      <input name="query" placeholder="Search…" />
      <button type="submit">Search</button>
    </Form>
  );
}

It prefetches the target layout and loading UI once the form scrolls into view, navigates client-side on submit so shared layouts and state survive, and falls back to a plain full-page form post if JavaScript has not loaded yet. Getting all three by hand meant a useRouter, an onSubmit that reassembles FormData into a query string, and a router.prefetch in an effect. I have written that component more than once. I would rather not write it again.

Worth being clear about the scope: this is for forms that navigate, like search. Forms that mutate still want a Server Action.

`instrumentation.ts`

Stable in 15, and the experimental.instrumentationHook flag can come out of your config. Put the file at the project root:

// instrumentation.ts
export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const { NodeSDK } = await import('@opentelemetry/sdk-node');
    new NodeSDK().start();
  }
}

export async function onRequestError(
  err: Error,
  request: { path: string; method: string },
  context: { routerKind: string; routeType: string }
) {
  await errorTracker.captureException(err, {
    path: request.path,
    routeType: context.routeType,
  });
}

onRequestError was designed with Sentry, and the context it hands you is the part that matters: which router, and whether the error came from a Server Component, a Server Action, a route handler or middleware. Stack traces from a server render are otherwise not much help in telling you where you were when it blew up.

Errors you can actually read

Spread across the point releases, and easy to overlook in a changelog:

15.0: hydration errors now show the offending source with a suggested fix, instead of the old diff-of-two-trees message that told you nothing.
15.1: source maps use the ignoreList property, so frames from your dependencies collapse out of the way in both the browser overlay and the terminal. Terminal output was reformatted to match the browser. Ignored frames also disappear from browser profiles, which makes profiling a Next.js app far less noisy.
15.2: a redesigned overlay built on React’s captureOwnerStack, which points at the subcomponent that actually created the bad element rather than the intermediate that passed it along. A new dev indicator shows rendering mode, Turbopack compile status and active errors in one place.

None of this is a feature you would put on a slide. All of it is time you get back.

Metadata stopped blocking

In 14, an async generateMetadata had to resolve before any HTML went out, because the <title> belongs in the <head> and the head comes first. One slow CMS call and your entire page waited on it.

From 15.2 the initial UI streams immediately and metadata is streamed into the head when it resolves. Bots and crawlers still get the complete document up front, detected by user agent, and you can adjust which agents get that treatment with htmlLimitedBots in next.config.

`next.config.ts`

TypeScript config, with a NextConfig type:

import type { NextConfig } from 'next';

const nextConfig: NextConfig = {
  experimental: {
    reactCompiler: true,
  },
  turbopack: {
    rules: {
      '*.svg': {
        loaders: ['@svgr/webpack'],
        as: '*.js',
      },
    },
  },
};

export default nextConfig;

reactCompiler still lives under experimental in the 15 line. It runs through Babel, so builds get slower when you turn it on; there is more on the trade-off in my post on the React Compiler.

Server Actions got quieter

Two changes, both automatic. Unused Server Actions are removed by dead code elimination during the build, so an exported action nobody calls no longer sits there as a live public endpoint. Actions that are used get non-deterministic IDs, recalculated between builds, which means you cannot guess an action URL and you cannot replay one from a previous deployment.

That is a genuine improvement, and it is not a security model. A Server Action is still a public HTTP endpoint that anyone can call with any payload. Validate the session and the input inside the action itself, every time: the same discipline I described in the post on Server Actions.

Self-hosting

Vercel gets accused of neglecting the self-hosted path. 15 pushes back on that a little:

Custom Cache-Control headers are no longer overwritten with framework defaults, so your CDN configuration finally survives contact with the app.
The stale-while-revalidate window is configurable as expireTime in next.config, replacing the old experimental.swrDelta. Its default moved to one year, which is what most CDNs need to apply SWR semantics properly.
sharp no longer needs installing by hand. next start and standalone output use it automatically, which removes a deployment gotcha that bit people for years.

Partial Prerendering, still experimental

PPR is what the async request APIs are for. A page gets split into a static shell and dynamic holes:

import { Suspense } from 'react';

export const experimental_ppr = true;

export default function Page() {
  return (
    <main>
      <h1>Dashboard</h1>
      <Sidebar />
      <Suspense fallback={<Skeleton />}>
        <DynamicContent />
      </Suspense>
    </main>
  );
}

The heading and sidebar are prerendered and served from the edge instantly. DynamicContent renders per request and streams into its slot. You enable it per route with that export, having first switched it on in the config:

const nextConfig = {
  experimental: {
    ppr: 'incremental',
  },
};

It is still experimental, and I would not put it on a client project yet. But it explains the whole shape of the release: without knowing which components need the request, the framework cannot decide what to prerender.

Upgrading from 14

npx @next/codemod@canary upgrade latest

That updates dependencies and offers the relevant codemods. Four things to check afterwards:

Async APIs. The codemod gets the direct cases. Hunt for helpers that pass cookies() or params around.
Caching. Anything that was fast because it was accidentally cached is now hitting your database on every request. Add force-cache or revalidate deliberately, and watch your query counts after deploy.
Config renames. serverComponentsExternalPackages is now serverExternalPackages, bundlePagesExternals is now bundlePagesRouterDependencies.
Node.js. Minimum is 18.18.0.

The migration I would budget real time for is the second one. Type errors announce themselves; a fetch that used to be free and now is not will only show up as a slightly unhappy database.

What I’d do

If you are on 14 and staying put, do the async request API migration anyway. It is the one piece of work that carries forward, and 16 has already removed the synchronous fallbacks.

The thing still keeping Turbopack out of older projects is the webpack plugin gap, and no amount of startup-time percentage fixes that. Check your plugin list before you promise anyone a faster dev server

‘Till next time!

End-to-End Typesafe APIs with tRPC

Sat, 27 Sep 2025 00:00:00 GMT

The same shape, written twice

Every full-stack TypeScript project I have worked on ends up with the same quiet tax. The server knows what a user looks like. The client also knows what a user looks like. Those two pieces of knowledge live in different files, and sooner or later they disagree with each other. The disagreement never announces itself at compile time. It shows up in production, as undefined is not a function.

tRPC is a fairly blunt answer to that. Instead of describing your API in a schema and generating types from it, you write server functions and let TypeScript’s inference carry their signatures across to the client. No SDL, no codegen step, no openapi.json sitting in the repo going stale. It pairs particularly well with a Next.js app that already talks to a typed database layer such as Prisma, because at that point the types run all the way from a column definition to a button handler.

I want to walk through tRPC v11 in a Next.js App Router project, because v11 changed the React integration enough that most of the tutorials you find are subtly wrong.

Why bother

Take the ordinary REST case. You add a GET /api/users/:id route. On the client you write an interface User so that fetch() returns something better than any. Now there are two definitions of a user in the codebase and nothing at all connecting them. Rename a field on the server and the client compiles happily, right up until it renders undefined.

GraphQL fixes the connection problem by making the schema the single source of truth. It charges you for it, though: a schema language, resolvers, a codegen step, and a build pipeline that has to run in the right order or your types are from last Tuesday. For a large team serving several clients that’s a fair trade. For one person shipping a Next.js app it is a lot of ceremony.

tRPC removes the middle layer entirely. There is no schema because the router is the schema, and the client reads its type. Change a procedure’s input and the call sites go red immediately, in the editor, before you have saved anything. I found that hard to give up again after a week of it.

The obvious catch is that both ends have to be TypeScript. tRPC gives you nothing if a Swift app or a partner’s Python service needs to call your API. For those, REST with an OpenAPI document or GraphQL are still the right answers. But if the API only ever serves your own Next.js frontend, the polyglot flexibility you are paying for is flexibility you will never use.

The pieces

There are five nouns and you need all of them:

Procedures are the endpoints. A query reads, a mutation writes, a subscription streams.
Routers group procedures under a namespace, and nest as deep as you like.
Context is per-request state (the session, a database handle) that every procedure receives.
Middleware wraps procedures, and crucially can narrow the context type for everything downstream.
Validators check input and, optionally, output.

That fourth point is the one people underestimate. I will come back to it.

Wiring it into Next.js

1. Install

npm install @trpc/server @trpc/client @trpc/tanstack-react-query \
  @tanstack/react-query zod

The React bindings live in @trpc/tanstack-react-query now. If you find a tutorial importing createTRPCReact from @trpc/react-query, that is the v10 integration. It still works in v11, but it is not the one being developed.

2. Initialise tRPC

One instance, created once, exporting the builders everything else uses:

// server/trpc.ts
import { initTRPC, TRPCError } from '@trpc/server';
import type { db } from '@/lib/db';

export type Context = {
  user: { id: string; name: string; role: string } | null;
  db: typeof db;
};

const t = initTRPC.context<Context>().create();

export const router = t.router;
export const publicProcedure = t.procedure;
export { TRPCError };

3. Write a router

// server/routers/users.ts
import { z } from 'zod';
import { router, publicProcedure, TRPCError } from '../trpc';

export const usersRouter = router({
  list: publicProcedure.query(async ({ ctx }) => {
    return ctx.db.user.findMany();
  }),

  byId: publicProcedure
    .input(z.object({ id: z.string() }))
    .query(async ({ ctx, input }) => {
      const user = await ctx.db.user.findUnique({
        where: { id: input.id },
      });

      if (!user) {
        throw new TRPCError({
          code: 'NOT_FOUND',
          message: 'User not found',
        });
      }

      return user;
    }),

  create: publicProcedure
    .input(
      z.object({
        name: z.string().min(1),
        email: z.string().email(),
      })
    )
    .mutation(async ({ ctx, input }) => {
      return ctx.db.user.create({ data: input });
    }),
});

The .input() call does double duty. At runtime it validates and rejects bad payloads; at compile time it types input inside the handler as { name: string; email: string }. You write the schema once and get both. Zod is the common choice but not a requirement, since v11 accepts anything implementing Standard Schema (Valibot, ArkType and friends all work).

4. Merge into an app router

// server/routers/index.ts
import { router } from '../trpc';
import { usersRouter } from './users';
import { postsRouter } from './posts';

export const appRouter = router({
  users: usersRouter,
  posts: postsRouter,
});

// Export the TYPE, never the implementation
export type AppRouter = typeof appRouter;

That last line matters more than it looks. export type is erased at build time, so importing AppRouter in a client component pulls in exactly zero bytes of server code. Export the value by accident and you have just shipped your database queries to the browser.

5. Serve it

// app/api/trpc/[trpc]/route.ts
import { fetchRequestHandler } from '@trpc/server/adapters/fetch';
import { appRouter } from '@/server/routers';
import { createContext } from '@/server/context';

const handler = (req: Request) =>
  fetchRequestHandler({
    endpoint: '/api/trpc',
    req,
    router: appRouter,
    createContext,
  });

export { handler as GET, handler as POST };

One catch-all route, and every procedure in the tree is reachable.

6. Set up the client

This is the part that changed in v11. createTRPCContext hands you a provider and a hook, and the client itself comes from @trpc/client:

// lib/trpc.ts
'use client';

import { createTRPCContext } from '@trpc/tanstack-react-query';
import type { AppRouter } from '@/server/routers';

export const { TRPCProvider, useTRPC, useTRPCClient } =
  createTRPCContext<AppRouter>();

// app/providers.tsx
'use client';

import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { createTRPCClient, httpBatchLink } from '@trpc/client';
import { useState } from 'react';
import { TRPCProvider } from '@/lib/trpc';
import type { AppRouter } from '@/server/routers';

export function Providers({ children }: { children: React.ReactNode }) {
  const [queryClient] = useState(() => new QueryClient());
  const [trpcClient] = useState(() =>
    createTRPCClient<AppRouter>({
      links: [httpBatchLink({ url: '/api/trpc' })],
    })
  );

  return (
    <QueryClientProvider client={queryClient}>
      <TRPCProvider trpcClient={trpcClient} queryClient={queryClient}>
        {children}
      </TRPCProvider>
    </QueryClientProvider>
  );
}

Note trpcClient, not client. I lost a while to that one.

Calling it from a component

v11 dropped the old trpc.users.list.useQuery() style in favour of handing you plain TanStack Query option objects. It reads as slightly more verbose and is much less magical:

'use client';

import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query';
import { useTRPC } from '@/lib/trpc';

export function UserList() {
  const trpc = useTRPC();
  const queryClient = useQueryClient();

  const { data: users, isPending } = useQuery(trpc.users.list.queryOptions());

  const createUser = useMutation(
    trpc.users.create.mutationOptions({
      onSuccess: () => {
        queryClient.invalidateQueries({ queryKey: trpc.users.list.queryKey() });
      },
    })
  );

  if (isPending) return <div>Loading…</div>;

  return (
    <div>
      <ul>
        {users?.map((user) => (
          <li key={user.id}>{user.name}</li>
        ))}
      </ul>
      <button
        onClick={() =>
          createUser.mutate({ name: 'New User', email: 'new@example.com' })
        }
      >
        Add User
      </button>
    </div>
  );
}

The win over the v10 hooks is that useQuery here is the real TanStack hook. Anything you know about select, enabled, placeholderData or suspense applies directly, and you are not waiting for tRPC to wrap each new Query option. queryKey() giving you the exact key tRPC used is the small detail that makes invalidation stop being guesswork.

Context, and the middleware trick

Context is built once per request:

// server/context.ts
import { auth } from '@/lib/auth';
import { db } from '@/lib/db';

export const createContext = async (opts: { req: Request }) => {
  const session = await auth(opts.req);

  return {
    user: session?.user ?? null,
    db,
  };
};

Note that user is nullable, because most requests are anonymous. Middleware is where that gets fixed:

// server/trpc.ts
export const authedProcedure = publicProcedure.use(async (opts) => {
  if (!opts.ctx.user) {
    throw new TRPCError({ code: 'UNAUTHORIZED' });
  }

  return opts.next({
    ctx: { user: opts.ctx.user },
  });
});

export const adminProcedure = authedProcedure.use(async (opts) => {
  if (opts.ctx.user.role !== 'admin') {
    throw new TRPCError({ code: 'FORBIDDEN' });
  }

  return opts.next();
});

Now build the router out of those instead of publicProcedure:

export const usersRouter = router({
  list: publicProcedure.query(async ({ ctx }) => ctx.db.user.findMany()),

  create: authedProcedure
    .input(z.object({ name: z.string(), email: z.string().email() }))
    .mutation(async ({ ctx, input }) => ctx.db.user.create({ data: input })),

  delete: adminProcedure
    .input(z.object({ id: z.string() }))
    .mutation(async ({ ctx, input }) =>
      ctx.db.user.delete({ where: { id: input.id } })
    ),
});

Here is the bit I actually like. Inside authedProcedure, ctx.user is typed non-null, because opts.next({ ctx }) narrows the context for everything downstream. Inside adminProcedure you can reach ctx.user.role without a single optional chain. The authorisation check and the type narrowing are the same line of code, which means you cannot skip the check and still compile. That is a much stronger guarantee than a linting rule about calling requireAuth() first.

Middleware also does the boring cross-cutting work:

const loggedProcedure = publicProcedure.use(async (opts) => {
  const start = Date.now();
  const result = await opts.next();
  console.log(`${opts.path} completed in ${Date.now() - start}ms`);
  return result;
});

When things go wrong

TRPCError takes a code from a fixed list, each mapped to an HTTP status:

import { TRPCError } from '@trpc/server';

throw new TRPCError({ code: 'NOT_FOUND', message: 'Post not found' }); // 404
throw new TRPCError({ code: 'BAD_REQUEST', message: 'Invalid input' }); // 400
throw new TRPCError({ code: 'FORBIDDEN', message: 'Access denied' }); // 403
throw new TRPCError({ code: 'CONFLICT', message: 'Slug already taken' }); // 409

An input that fails Zod validation becomes a BAD_REQUEST on its own; you do not need to catch and rethrow. On the client the error arrives through React Query’s normal channel:

const { data, error } = useQuery(trpc.users.byId.queryOptions({ id }));

if (error) {
  // error.data?.code is 'NOT_FOUND', 'UNAUTHORIZED', …
  return <div>Error: {error.message}</div>;
}

And for everything you would rather see in your logs than in a user’s browser, the handler takes an onError:

fetchRequestHandler({
  endpoint: '/api/trpc',
  req,
  router: appRouter,
  createContext,
  onError: ({ error, path }) => {
    console.error(`tRPC error on ${path}:`, error);
  },
});

One thing worth knowing: in production tRPC strips the stack trace out of INTERNAL_SERVER_ERROR responses and replaces the message. That is the right default, and it does mean onError is the only place you will ever see the real cause.

Prefetching from a Server Component

Client components fetching on mount produce the waterfall everyone complains about: HTML arrives, React hydrates, then the request goes out. You can hoist the fetch into the Server Component that renders them.

The server-side proxy needs to be told how to build a context, which router to call, and where to put the results:

// lib/trpc-server.ts
import 'server-only';
import { cache } from 'react';
import { createTRPCOptionsProxy } from '@trpc/tanstack-react-query';
import { appRouter } from '@/server/routers';
import { createContext } from '@/server/context';
import { makeQueryClient } from './query-client';

export const getQueryClient = cache(makeQueryClient);

export const trpc = createTRPCOptionsProxy({
  ctx: createContext,
  router: appRouter,
  queryClient: getQueryClient,
});

cache() is not optional decoration. Without it you get a fresh QueryClient per call and nothing you prefetch ends up in the one that gets dehydrated.

// app/users/page.tsx
import { dehydrate, HydrationBoundary } from '@tanstack/react-query';
import { trpc, getQueryClient } from '@/lib/trpc-server';
import { UserList } from './user-list';

export default async function UsersPage() {
  const queryClient = getQueryClient();

  void queryClient.prefetchQuery(trpc.users.list.queryOptions());

  return (
    <HydrationBoundary state={dehydrate(queryClient)}>
      <UserList />
    </HydrationBoundary>
  );
}

Because the proxy calls the router directly rather than over HTTP, there is no network hop at all on the server. The data is serialised into the HTML, and UserList hydrates with isPending already false. The void is deliberate: not awaiting means the page can stream while the query resolves.

Batching, streaming and subscriptions

httpBatchLink collects every tRPC call that happens in the same tick and sends them as one request. Render six components that each want their own query and you get one round trip instead of six. It is on by default in most setups and it quietly removes a whole category of performance problem.

Its downside is head-of-line blocking: the batch resolves when the slowest procedure in it resolves. httpBatchStreamLink fixes that by streaming each result as it lands.

import { createTRPCClient, httpBatchStreamLink } from '@trpc/client';

createTRPCClient<AppRouter>({
  links: [httpBatchStreamLink({ url: '/api/trpc' })],
});

v11 also added subscriptions over Server-Sent Events, which is a big deal for anyone who was avoiding real-time features because they did not want a WebSocket server to operate. SSE is a plain HTTP response that stays open. Your existing route handler, your existing deployment, no second process.

And if your router tree has grown large enough to hurt cold starts, lazy defers loading a subtree until something calls into it:

import { lazy } from '@trpc/server';
import { router } from './trpc';

export const appRouter = router({
  users: lazy(() => import('./routers/users')),
  reports: lazy(() => import('./routers/reports').then((m) => m.reportsRouter)),
});

The types stay exactly as they were. Only the loading is deferred.

Validating what goes out

Input validation gets all the attention. Output validation is the one that saved me from a bad afternoon.

const userOutput = z.object({
  id: z.string(),
  name: z.string(),
  email: z.string().email(),
});

export const usersRouter = router({
  byId: publicProcedure
    .input(z.object({ id: z.string() }))
    .output(userOutput)
    .query(async ({ ctx, input }) => {
      return ctx.db.user.findUnique({ where: { id: input.id } });
    }),
});

Add a passwordHash column to the users table, forget that this procedure does a bare findUnique, and without .output() you have just published password hashes to every client. With it, the response fails validation and tRPC returns an INTERNAL_SERVER_ERROR instead: loudly, in your logs, rather than silently in the payload. It costs a little runtime and it is worth it on anything touching user records.

Testing without a server

Procedures are functions that take a context and an input, so you can call them directly. No listening port, no supertest, no fixture app.

import { appRouter } from '@/server/routers';

describe('users router', () => {
  it('creates a user', async () => {
    const caller = appRouter.createCaller({
      user: { id: '1', name: 'Test', role: 'admin' },
      db: mockDb,
    });

    const result = await caller.users.create({
      name: 'New User',
      email: 'test@example.com',
    });

    expect(result.name).toBe('New User');
  });

  it('refuses an anonymous delete', async () => {
    const caller = appRouter.createCaller({ user: null, db: mockDb });

    await expect(caller.users.delete({ id: '1' })).rejects.toThrow(
      'UNAUTHORIZED'
    );
  });
});

Middleware runs, so that second test genuinely exercises the auth path rather than mocking it away. There is also a createCallerFactory helper if you would rather build the caller once and reuse it across a suite.

Where it doesn’t fit

	tRPC	REST	GraphQL
Type safety	Inferred, no config	Manual, or OpenAPI codegen	Schema plus codegen
Schema artefact	None	OpenAPI (optional)	SDL (required)
Code generation	None	Optional	Usually required
Non-TS clients	No	Yes	Yes
Best for	Full-stack TS apps	Public and partner APIs	Large data graphs, many clients

The honest summary is that tRPC’s greatest strength and its only real weakness are the same property. Skipping the schema is what buys you the instant feedback, and it is also why nobody outside your TypeScript codebase can talk to the thing. If there is any chance of a public API later, either put REST in front of the same service layer, or accept that you will be writing an adapter.

What I’d watch for

Two things bother me, and neither is a reason not to use it.

The first is compile speed. Inference this deep is not free, and a router with a few hundred procedures will make tsc and your editor think about it. Splitting into sub-routers and keeping return types simple helps.

The second is that it encourages you to treat the API as an implementation detail, because for a while it is. Then someone wants a mobile app, and the “API” turns out to be four hundred procedures with no versioning, no documented contract, and Zod schemas as the only description of what anything accepts. Keep the business logic in plain functions and let procedures be thin wrappers over them. Future you will need those functions.

Next on my list is pointing tRPC at Server Actions rather than a route handler, since a lot of what I use mutations for could be a form post. I suspect the answer is “pick one”, but I would like to find out properly first.

‘Till next time!

Biome

Sat, 23 Aug 2025 00:00:00 GMT

One binary instead of six packages

Open the devDependencies of any frontend project older than a year and count how many of them exist purely to make the linter and the formatter stop arguing. On this site the answer is six: eslint, prettier, eslint-config-prettier, eslint-plugin-prettier, stylelint and stylelint-prettier, plus a handful of plugins underneath. None of that is doing work I asked for. It is doing the work of reconciling two tools that both have opinions about where a line break goes.

Biome is the attempt to remove that entirely. One binary, written in Rust, that formats, lints and sorts imports, configured by one file. Version 2 landed in June 2025 and is the release where I stopped filing it under “interesting, ask again next year”.

Where it came from

Biome is what is left of Rome, the toolchain Sebastian McKenzie started after Babel and Yarn. Rome was going to replace everything in the JavaScript build pipeline at once. It was written in TypeScript, then rewritten in Rust for speed, and backed by a company.

The company failed. When the layoffs came, the core contributors found they had been locked out of the infrastructure they had been maintaining: no npm registry access, no Discord, no website. In August 2023 they forked it and shipped under a new name, made by fusing “bis” onto “Rome”. Second Rome.

I have a mild fondness for that origin story. The project people remember is the venture-funded one that stopped; the one that is still shipping releases is the fork the maintainers made when they had no infrastructure and no salary.

The tax it is removing

The ESLint and Prettier pairing works, and I have run it for years. The friction is not in any one piece:

Two configuration files, and a third package whose only job is switching off the ESLint rules that fight Prettier.
Two tools with overlapping jurisdiction, which is why that third package needs to exist.
JavaScript performance, on a workload that is entirely parsing and walking trees.
Plugin arithmetic. A typical React setup pulls in @typescript-eslint/parser, @typescript-eslint/eslint-plugin, eslint-plugin-react, eslint-plugin-import and eslint-plugin-jsx-a11y, each with its own peer dependency range, all of which have to agree before anything runs.

That last one is the one that actually costs time. Nobody minds a slow linter as much as they mind a major ESLint release forcing a coordinated bump across five plugins that update on five different schedules. It is the same complaint I ended up with when writing about Conventional Commits: the tooling around the convention kept demanding more attention than the convention itself.

Speed

Biome’s own benchmark is formatting 171,127 lines across 2,104 files on an Intel Core i7 1270P, where it comes out roughly 35 times faster than Prettier. Take the multiplier with the usual pinch of salt, since it is the vendor’s own number on the vendor’s own hardware, but the order of magnitude is not in dispute.

Where you feel it is not the full run. It is the pre-commit hook and the editor. A formatter that takes 800ms to start up before it does anything is a formatter you eventually configure to run less often, and then you stop trusting the repo to be formatted. Biome keeps a daemon alive for editor requests, so there is no per-invocation startup at all.

Getting it in

Install it pinned, because formatter output does drift across versions and an unpinned formatter turns one person’s save into a hundred-line diff:

npm install --save-dev --save-exact @biomejs/biome

Then generate a config:

npx @biomejs/biome init

Three commands cover daily use:

# Format files
npx biome format --write .

# Lint files
npx biome lint --write .

# Format, lint, and apply assist actions in one pass
npx biome check --write .

check is the one that ends up in your scripts. It runs the formatter, applies the safe lint fixes, and runs the assist actions, which in v2 is where import sorting now lives.

The config file

Everything goes in biome.json, or biome.jsonc if you want comments. A version tuned for a React and TypeScript project:

{
  "$schema": "https://biomejs.dev/schemas/2.0.0/schema.json",
  "formatter": {
    "enabled": true,
    "indentStyle": "space",
    "indentWidth": 2,
    "lineWidth": 100
  },
  "linter": {
    "enabled": true,
    "rules": {
      "recommended": true,
      "complexity": {
        "noUselessFragments": "warn",
        "noForEach": "off"
      },
      "style": {
        "noNonNullAssertion": "warn"
      },
      "suspicious": {
        "noExplicitAny": "warn"
      }
    }
  },
  "assist": {
    "actions": {
      "source": {
        "organizeImports": "on"
      }
    }
  },
  "javascript": {
    "formatter": {
      "quoteStyle": "single",
      "semicolons": "always",
      "trailingCommas": "all"
    }
  },
  "vcs": {
    "enabled": true,
    "clientKind": "git",
    "useIgnoreFile": true,
    "defaultBranch": "main"
  },
  "files": {
    "includes": ["src/**", "tests/**"]
  }
}

No parser configuration anywhere. Biome works out that a .tsx file is TypeScript with JSX without being told, which after years of parserOptions blocks feels like something is missing.

Two things moved in v2 and will bite you if you are upgrading from v1 rather than starting fresh. files.include and files.ignore were replaced by a single files.includes list that takes negated globs with !. And organizeImports is no longer a top-level key; it is an assist action, as above. Running npx @biomejs/biome migrate --write after a version bump rewrites the config for you, which is a nice habit to have even when nothing has changed.

The formatter

Biome formats JavaScript, TypeScript, JSX, TSX, JSON, JSONC, CSS and GraphQL. It passes 97% of Prettier’s test suite, which is how it won the Prettier Challenge back in 2023.

The 3% is worth knowing before you run it on a repository full of other people’s code:

The default indent style is tabs, where Prettier defaults to spaces. Set "indentStyle": "space" if you would rather not explain the diff.
Object property unquoting follows ES2015 identifier rules rather than Prettier’s ES5 ones, so a few more keys lose their quotes.
The parser is stricter. Things Prettier’s Babel parser waves through, like duplicate modifiers, are errors here.

Expect one enormous reformatting commit and then nothing. Put its hash in .git-blame-ignore-revs and move on.

Opting out of formatting for a block uses a comment:

// biome-ignore format: complex template literal alignment
const query = `
  SELECT * FROM users
  WHERE  id = ${userId}
  AND    active = true
`;

The linter

Rules are grouped into eight domains: accessibility, complexity, correctness, nursery, performance, security, style and suspicious. Nursery is opt-in and holds whatever is still settling.

Most rules are ports of existing ones from ESLint core, typescript-eslint, eslint-plugin-react, eslint-plugin-jsx-a11y and eslint-plugin-unicorn. The naming convention changed, though: Biome uses camelCase where ESLint uses kebab-case, and often renames on the way, so no-unused-vars arrives as noUnusedVariables. Any suppression comment you have memorised is wrong here, and so is any rule name in your team’s coding guidelines.

Type-aware rules without the TypeScript compiler

This is the headline of v2 and the reason it is worth a second look. typescript-eslint’s type-aware rules are the slowest part of most lint runs, because they need a full TypeScript program in memory. Biome v2 does its own type inference across files instead, so rules like noFloatingPromises work with no tsconfig.json wiring at all:

// Biome catches this without the TypeScript compiler
async function fetchData() {
  return await fetch('/api/data');
}

// Error: noFloatingPromises - This promise must be awaited or handled
fetchData();

The coverage is partial. Biome puts it at roughly three quarters of what typescript-eslint catches, which is the honest way to describe it and also the thing to check before you delete anything. If your team relies on a specific type-aware rule, look it up rather than assuming.

Safe and unsafe fixes

Every fix is classified:

# Apply safe fixes only (semantics-preserving)
npx biome lint --write .

# Apply safe AND unsafe fixes (may change behavior)
npx biome lint --write --unsafe .

A safe fix cannot change what your program does: dropping an unused import, collapsing a redundant expression. An unsafe fix might, and rewriting a for loop into Array.map is the example that makes the distinction obvious. Having the two separated by a flag rather than by rule documentation is the difference between running the fixer in CI and not.

Imports

Import organising was rebuilt in v2. It merges duplicate imports from the same module, sorts the specifiers inside each one, understands import attributes, and handles exports as well:

// Before
import { useState } from 'react';
import { z } from 'zod';
import { useEffect } from 'react';
import type { FC } from 'react';
import { db } from '@/lib/db';

// After (biome check --write)
import type { FC } from 'react';
import { useEffect, useState } from 'react';

import { db } from '@/lib/db';

import { z } from 'zod';

It also respects blank lines you put in deliberately, treating them as group boundaries rather than noise to be tidied away. That single behaviour removes the reason most people reach for eslint-plugin-import or a Prettier sorting plugin in the first place.

Migrating

Biome reads your existing configuration and converts it:

# Migrate ESLint config
npx @biomejs/biome migrate eslint --write

# Migrate Prettier config
npx @biomejs/biome migrate prettier --write

The ESLint side handles both the legacy .eslintrc format and flat config, resolves shared configs and plugins, and maps rules onto their Biome equivalents where they exist. Rules with no equivalent are dropped, so read the output rather than trusting it.

If the result is a wall of violations, you do not have to fix them before adopting the tool. Write suppression comments for everything currently failing and work through them later:

npx biome lint --write --suppress --reason="suppressed during Biome migration"

Then the uninstall, which is the satisfying part:

npm uninstall eslint prettier eslint-config-prettier eslint-plugin-react \
  @typescript-eslint/parser @typescript-eslint/eslint-plugin \
  eslint-plugin-import eslint-plugin-jsx-a11y

Eight packages, and every one of them was a version constraint you had to keep satisfied.

Editors and CI

First-party extensions exist for VS Code, IntelliJ and Zed, with community ones for Vim, Neovim and Sublime Text. In VS Code the settings are:

{
  "editor.defaultFormatter": "biomejs.biome",
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "quickfix.biome": "explicit",
    "source.organizeImports.biome": "explicit"
  }
}

For CI there is a purpose-built command that checks everything and fixes nothing:

npx biome ci .

It reports formatting, lint and assist violations and exits non-zero. On a pull request you rarely want the whole repository, and Biome knows about your VCS:

# Only check files changed since the default branch
npx biome check --changed

# Only check staged files, for a pre-commit hook
npx biome check --staged

Which makes for a short workflow file:

name: Code Quality
on: [pull_request]

jobs:
  biome:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx biome ci .

What it will not touch

Language	Format	Lint
JavaScript / TypeScript	Yes	Yes
JSX / TSX	Yes	Yes
JSON / JSONC	Yes	Yes
CSS	Yes	Yes
GraphQL	Yes	Nursery rules
HTML	Experimental	No

The absences matter more than the table suggests. SCSS, Sass and Less are not supported, only plain CSS. Vue, Svelte and Astro single-file components are on the roadmap and not here yet. If your project contains either of those, Biome is an addition to your toolchain rather than a replacement for it, and the whole argument for adopting it gets weaker.

What v2 brought

Beyond type-aware linting, the June release added a plugin system built on GritQL for writing your own pattern-matching rules, though the distribution story for sharing them is still being worked out. Assist actions arrived as a category separate from lint diagnostics, covering things like useSortedKeys and useSortedAttributes that are preferences rather than problems. Suppressions gained a range form with // biome-ignore-start and // biome-ignore-end, which spares you a comment per line when you need to exempt a block. Monorepos got nested configuration files that inherit from the root. And the HTML formatter appeared, experimental and off by default.

Would I switch this site?

No, and the reason is the third row from the bottom of that table.

This blog is Gatsby with Bootstrap and a stack of hand-written SCSS, linted by Stylelint with a handful of plugins on top of it. Biome does not do SCSS. So switching would mean replacing ESLint and Prettier with Biome, keeping Stylelint anyway, and ending up with two tools and two config files, which is precisely the thing I would be switching to avoid. The dependency count barely moves.

For a project that is TypeScript and plain CSS, I would use it without hesitating, and I would do it on day one rather than migrating later. That is not really a caveat about Biome, it is a caveat about my own repositories, most of which predate the assumption that CSS is something you write directly.

The thing I will actually do is smaller. The next time an ESLint major forces me to bump five plugins in lockstep, I am going to open that language support table again and check whether the SCSS row has moved

‘Till next time!

React Compiler

Sat, 26 Jul 2025 00:00:00 GMT

Memoisation you no longer write by hand

Two years ago I wrote a post about how useCallback does nothing in most of the places people put it. The React Compiler is the React team's answer to the same problem, arrived at from the other direction: instead of teaching everyone when memoisation pays off, take the decision away from us and let a build step make it, every time, everywhere.

It has been called React Forget, then an experimental release, then a beta, and since April it has been a release candidate. It is not finished, but it is well past the stage where trying it is an adventure. Meta runs it in production on Facebook, Instagram, Threads and the Quest Store, and the version you can install today is close to what will ship as 1.0.

The problem it is solving

React’s model is that a state change re-renders a component and everything below it. That simplicity is why React is pleasant to work with and also why it does redundant work: a child whose props are identical still re-renders, and rebuilds its own children while it is at it.

The manual fix has been three APIs. useMemo caches a computed value, useCallback keeps a function reference stable between renders, and React.memo skips a re-render when props compare equal. They work. They are also a tax that gets paid on every component you write, in attention rather than milliseconds. You have to decide what to wrap, keep the dependency arrays honest, and remember that memoising the callback accomplishes nothing unless the component receiving it is also memoised. I went through those trade-offs in detail in Optimising React Components and the conclusion was uncomfortable: most useCallback calls in most codebases are decoration.

Meta measured this before building the compiler. Only about 8% of their React pull requests used manual memoisation at all, and those pull requests took 31 to 46% longer to author. That is a real cost for an optimisation most people get partly wrong. A missing dependency in useMemo produces stale data, which surfaces as a bug that reproduces once a week. Too much memoisation burns memory and cache-comparison time to avoid work that was cheap anyway.

What it actually does

The compiler is a Babel plugin. It reads your source before anything else has touched it, converts each component into an internal representation built around a control flow graph, works out which values depend on which, and emits the memoisation you would have written if you had infinite patience.

Take a component with nothing interesting in it:

// Your source code
export default function Greeting({ name }: { name: string }) {
  const message = `Hello, ${name}!`;

  return <div>{message}</div>;
}

Here is roughly what comes out the other side:

// Compiled output (simplified)
import { c as _c } from "react/compiler-runtime";

export default function Greeting({ name }: { name: string }) {
  const $ = _c(2);
  let t0;

  if ($[0] !== name) {
    const message = `Hello, ${name}!`;
    t0 = <div>{message}</div>;
    $[0] = name;
    $[1] = t0;
  } else {
    t0 = $[1];
  }

  return t0;
}

$ is a cache slot array sized to the number of values worth caching. First render fills it, subsequent renders compare and reuse. It is the same idea as useMemo, applied to every intermediate value and every JSX element rather than the two or three you happened to worry about.

The interesting part is what it can do that you cannot. Hooks must be called unconditionally at the top level, so anything after an early return is off-limits to useMemo by construction. The compiler has no such restriction, because the cache is an array it indexes rather than a hook call it has to keep in order. It also tracks mutation across a function body properly, which is the bit people get wrong when they hand-write dependency arrays.

Getting it running

Install the plugin, pinned:

npm install --save-dev --save-exact babel-plugin-react-compiler@rc

--save-exact is not paranoia. The memoisation the compiler emits changes between versions, and a floating range means two developers on the same branch can produce different output from identical source.

For Next.js the framework does the wiring:

// next.config.js
const nextConfig = {
  experimental: {
    reactCompiler: true,
  },
};

module.exports = nextConfig;

You still need babel-plugin-react-compiler installed; the flag only tells Next.js to use it. On recent versions Next.js runs an SWC pass first to work out which files contain components at all, so Babel only touches the files that need it. That matters more than it sounds, because adding a Babel step to an otherwise SWC-based build is the single biggest cost of adoption.

For Vite, hang it off the React plugin’s Babel config:

// vite.config.js
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [
    react({
      babel: {
        plugins: ['babel-plugin-react-compiler'],
      },
    }),
  ],
});

And for anything else with a Babel config, the plugin goes first in the list. Not somewhere in the list. First:

// babel.config.js
module.exports = {
  plugins: [
    'babel-plugin-react-compiler', // must be first
    // ... other plugins
  ],
};

The reason is that the compiler needs to see your code as you wrote it. Once another transform has rewritten your JSX or shuffled your arrow functions, the analysis it relies on stops being trustworthy, and rather than guess it will bail out and leave the component alone.

If you are not on React 19

This is the part I care about most, because plenty of code out there is not on 19 and will not be soon. This blog, for instance, still runs Gatsby 4 on React 17, and Gatsby 4 is not moving.

The compiler targets React 19 by default, but it will compile for 17 and 18 if you give it the runtime polyfill and tell it what to aim at:

npm install react-compiler-runtime@rc

// babel.config.js
module.exports = {
  plugins: [
    ['babel-plugin-react-compiler', { target: '18' }],
  ],
};

The polyfill supplies the cache primitive that React 19 has built in. It is not a compatibility hack bolted on late; backwards compatibility was one of the things the release candidate specifically added, which suggests the team understands that “upgrade to 19 first” is not advice most teams can act on this quarter.

The rules it depends on

The compiler is only correct if your components follow the Rules of React. These are not new rules and the compiler did not invent them. What is new is that breaking them used to produce a subtle bug and now produces a subtle bug that is harder to find, because the compiler made a caching decision based on a promise your code did not keep.

Components must be pure. Same input, same output, no side effects on the render path.

// Bad: side effect during render
function Counter({ count }: { count: number }) {
  document.title = `Count: ${count}`; // side effect in render!
  return <div>{count}</div>;
}

// Good: side effect in useEffect
function Counter({ count }: { count: number }) {
  useEffect(() => {
    document.title = `Count: ${count}`;
  }, [count]);

  return <div>{count}</div>;
}

Props and state are immutable. The compiler decides a cached value is still good by comparing references. Mutate the thing behind the reference and the comparison lies to it:

// Bad: mutating state directly
function TodoList({ todos }: { todos: Todo[] }) {
  const sorted = todos.sort((a, b) => a.name.localeCompare(b.name)); // mutates!
  return <ul>{sorted.map(t => <li key={t.id}>{t.name}</li>)}</ul>;
}

// Good: create a new array
function TodoList({ todos }: { todos: Todo[] }) {
  const sorted = [...todos].sort((a, b) => a.name.localeCompare(b.name));
  return <ul>{sorted.map(t => <li key={t.id}>{t.name}</li>)}</ul>;
}

Array.prototype.sort sorting in place is the classic one. reverse and splice are the same trap wearing different hats.

Hooks stay at the top level. Unchanged, but now enforced by something that will act on the assumption.

The lint rule is the actual advice

This is the bit I would do first, and it does not require installing the compiler at all. The compiler’s static analysis has been packaged as a lint rule inside eslint-plugin-react-hooks:

npm install --save-dev eslint-plugin-react-hooks@rc

// eslint.config.js
import * as reactHooks from 'eslint-plugin-react-hooks';

export default [
  reactHooks.configs['recommended-latest'],
  {
    rules: {
      'react-hooks/react-compiler': 'error',
    },
  },
];

The rule is not on by default in the release candidate, so turn it on explicitly. What it reports is the set of places where the compiler would have to give up: state set during render, refs read where they should not be, mutation of something that came in as a prop. Every one of those is a latent bug today, compiler or no compiler. Fixing them is worth doing on its own terms, and it happens to leave your codebase ready.

Adopting it in pieces

Nobody sensible turns this on across a large repository in one commit. There are two supported ways to go slowly.

Scope it to directories with Babel’s overrides, then widen:

// babel.config.js
module.exports = {
  plugins: [
    // other plugins...
  ],
  overrides: [
    {
      test: ['./src/features/dashboard/**'],
      plugins: ['babel-plugin-react-compiler'],
    },
  ],
};

Or flip it around and compile nothing except what you have marked:

// babel.config.js
module.exports = {
  plugins: [
    ['babel-plugin-react-compiler', { compilationMode: 'annotation' }],
  ],
};

function OptimizedComponent() {
  "use memo";
  // This component WILL be compiled
  return <ExpensiveTree />;
}

function RegularComponent() {
  // This component will NOT be compiled
  return <SimpleTree />;
}

Annotation mode is the one I would start with on an older codebase, because it inverts the risk. Directory mode compiles whatever happens to live in that folder, including the two components someone wrote in a hurry in 2022.

Going the other way, "use no memo" excludes a single component:

function ProblematicComponent() {
  "use no memo";
  // This component will be skipped by the compiler
  return <div>...</div>;
}

Treat that directive as a comment with a deadline attached, not a fix. It marks a component that is breaking a rule, and the rule was already being broken before the compiler noticed.

What Meta got out of it

The published figures come from the Quest Store: up to 12% faster initial loads and cross-page navigations, some interactions more than 2.5x faster, and memory usage flat. Flat memory is the number I find most reassuring, since caching everything is exactly the sort of change that trades one resource for another without telling you.

The number that says more about adoption risk is a different one: Meta’s monorepo holds over 100,000 React components, and onboarding them needed few code changes. That is the strongest available evidence that ordinary, non-heroic React code is already compatible.

When it goes wrong

React DevTools puts a sparkle badge next to compiled components in the tree, so you can at least see what you are dealing with. From there the loop is:

Add "use no memo" to the component you suspect.
See whether the symptom disappears.
If it does, the component is breaking a rule somewhere.
Run the lint rule and read what it says about that file.
Fix the cause, then delete the directive.

Nearly every real report I have seen traces back to code that depended on referential inequality for correctness. An effect that re-runs because a callback is a new object every render, for instance, works entirely by accident. Give it a stable reference and the effect stops firing, and the bug looks like the compiler broke your component when it only revealed what was holding it together.

You do not have to strip out existing useMemo and useCallback calls first. They keep working. The team’s advice is to leave them where they are, because removing them changes what the compiler chooses to cache, and doing that at the same time as adopting the compiler leaves you with two variables and one bug.

Where this leaves things

Some caveats, none of them dealbreakers. The plugin has to run on original source, which means adding Babel to a build that may have been happily Babel-free. Library code has to be compiled by the library’s own author, so nothing you install benefits until they ship it. The SWC path is still experimental. And a release candidate is a release candidate, however many Meta apps are running it.

Still, my honest reaction is relief rather than excitement. This is not a new capability, it is the removal of a chore, and specifically a chore that produced more bad advice than good code. Whole blog posts, mine included, exist to explain when useCallback is worth it. I would rather they stopped needing to exist.

I will not be putting this near the site you are reading, because Gatsby 4 and React 17 have their own opinions. The lint rule, though, goes on everything I touch from here 🧹

‘Till next time!

Tailwind CSS v4

Sat, 21 Jun 2025 00:00:00 GMT

Configuration that lives in the stylesheet

Tailwind v4 landed in January 2025 and it is a rewrite, not a refresh. The JavaScript config file is gone from the happy path, the engine underneath is new, and a pile of utilities have been renamed. If you have a v3 project, none of that is optional reading: the upgrade is mechanical in places and genuinely disruptive in others, and the disruptive parts are the ones nobody mentions in the release notes summary.

I have a soft spot for the direction here. Moving the design tokens into CSS custom properties is the same argument I made in CSS custom properties back in 2022, only now the framework agrees with me. What I am less thrilled about is what v4 asks you to give up to get there. More on that at the end.

The engine

The headline is the new engine, written from scratch. The numbers from the Tailwind team’s own benchmarks:

Build type	v3.4	v4.0	Improvement
Full build	378ms	100ms	3.78x
Incremental rebuild, new CSS	44ms	5ms	8.8x
Incremental rebuild, no new CSS	35ms	192µs	182x

The bottom row is the one that changes how the tool feels. 192 microseconds is below the threshold where you notice anything at all, so an incremental rebuild that produces no new classes is effectively free. Most keystrokes in most sessions fall into that row, because you spend far more time nudging existing classes around than inventing new ones.

I would treat the first row with the usual scepticism you apply to any benchmark shipped by the people who wrote the thing being benchmarked. It is a real improvement, and your project is not their project.

Configuration moved into CSS

Here is v3, where the theme lives in a JavaScript module that the build tool has to load:

// tailwind.config.js
module.exports = {
  content: ['./src/**/*.{html,js,tsx}'],
  theme: {
    extend: {
      colors: {
        brand: '#ff5733',
        accent: '#3498db',
      },
      fontFamily: {
        display: ['Satoshi', 'sans-serif'],
      },
      spacing: {
        18: '4.5rem',
      },
    },
  },
};

And here is v4, where it lives in the stylesheet:

@import "tailwindcss";

@theme {
  --font-display: "Satoshi", "sans-serif";
  --color-brand: #ff5733;
  --color-accent: #3498db;
  --spacing-18: 4.5rem;
  --breakpoint-3xl: 1920px;
  --ease-fluid: cubic-bezier(0.3, 0, 0, 1);
}

The naming is not arbitrary. The prefix of each variable tells Tailwind which family of utilities to generate: --color-* produces bg-brand, text-brand, border-brand and the rest; --breakpoint-* produces 3xl: variants; --ease-* produces ease-fluid. Learn the prefixes and the config stops needing documentation.

Everything in @theme is also emitted as a real custom property on :root. That means a value you defined for Tailwind is available to hand-written CSS, to inline styles, to a charting library that wants a hex string, and to getComputedStyle in a test. In v3, getting a theme value into JavaScript meant importing the config and running it through resolveConfig, which pulled the whole config into your bundle if you were careless. Now it is var(--color-brand) and there is no build step involved.

If you still need JavaScript for something, the config file is supported. It is no longer discovered automatically, so you point at it:

@import "tailwindcss";
@config "../../tailwind.config.js";

Plugins load the same way, with @plugin "@tailwindcss/typography"; in the stylesheet.

Setting it up

Two dependencies:

npm install tailwindcss @tailwindcss/postcss

One PostCSS plugin:

// postcss.config.mjs
export default {
  plugins: {
    "@tailwindcss/postcss": {},
  },
};

One import:

/* main.css */
@import "tailwindcss";

That is the whole setup. The three @tailwind base/components/utilities directives are gone, and so are postcss-import and autoprefixer as separate installs, since the Tailwind plugin now handles imports and prefixing itself.

Content detection is automatic. Tailwind walks your project, skips anything in .gitignore, and ignores binary files, so the content array has no successor. When the heuristic guesses wrong, which it does for template files that live outside the project root or inside a linked package, you tell it explicitly:

@import "tailwindcss";
@source "../../packages/ui/src";

On Vite, skip PostCSS entirely and use the first-party plugin, which is measurably quicker because it hooks into Vite’s own module graph instead of running as a post-processing step:

npm install tailwindcss @tailwindcss/vite

// vite.config.js
import { defineConfig } from 'vite';
import tailwindcss from '@tailwindcss/vite';

export default defineConfig({
  plugins: [tailwindcss()],
});

What v4 takes from modern CSS

Tailwind v3 was written for a browser landscape that no longer exists. v4 assumes Safari 16.4, Chrome 111 and Firefox 128 as its floor and spends that budget freely.

Cascade layers

Tailwind’s own output now sits inside real @layer rules. In v3 the ordering was emulated, and the emulation leaked whenever a third-party stylesheet had opinions about specificity. Now the browser does the ordering, which mostly means you stop losing arguments with a component library’s reset.

Container queries, no plugin

The container query plugin is now part of the core:

<div class="@container">
  <div class="grid grid-cols-1 @sm:grid-cols-3 @lg:grid-cols-4">
    <!-- Responsive based on container width -->
  </div>
</div>

There are @max-* variants too, and you can stack them into a range with @min-md:@max-xl:hidden. I wrote about why this matters for component-driven layouts in CSS container queries; the short version is that a card should respond to the column it is in, not to the window.

OKLCH and color-mix

The default palette moved from sRGB to OKLCH, which gives more saturated colours on wide-gamut displays and, more usefully, makes lightness steps behave consistently across hues. Opacity modifiers are implemented with color-mix(), so they now work on any colour value at all, including custom properties and currentColor:

<div class="bg-brand/50">
  <!-- 50% opacity of your custom brand color -->
</div>

<div class="bg-(--my-custom-color)/75">
  <!-- Works with any CSS variable too -->
</div>

Registered custom properties

Tailwind registers its animatable properties with @property, which tells the browser their type. A browser that knows --tw-gradient-from is a colour can interpolate it; a browser that thinks it is an arbitrary string cannot. This is what makes transitions on gradients and individual transform components work.

New utilities worth knowing

3D transforms are built in now:

<div class="perspective-500">
  <div class="rotate-x-12 rotate-y-6 transform-3d">
    <!-- 3D transformed element -->
  </div>
</div>

The set includes rotate-x-*, rotate-y-*, scale-z-*, translate-z-*, perspective-* and perspective-origin-*.

Gradients gained angles, radial and conic variants, and interpolation control:

<!-- Arbitrary angle -->
<div class="bg-linear-45 from-blue-500 to-purple-500"></div>

<!-- Radial gradient (new) -->
<div class="bg-radial from-yellow-300 to-orange-500"></div>

<!-- Conic gradient (new) -->
<div class="bg-conic from-red-500 via-green-500 to-blue-500"></div>

<!-- More vivid gradients via OKLCH interpolation -->
<div class="bg-linear-to-r/oklch from-blue-500 to-green-500"></div>

bg-gradient-* is now bg-linear-*, which the upgrade tool renames for you.

Numeric utilities take any value without arbitrary-value brackets, which quietly removes a category of ugly class names:

<!-- v3: needed arbitrary values -->
<div class="grid-cols-[15] mt-[4.25rem]"></div>

<!-- v4: just works -->
<div class="grid-cols-15 mt-17 w-29"></div>

The not-* variant negates a pseudo-class, a media query or a @supports condition:

<button class="opacity-100 not-hover:opacity-75">
  Subtle opacity when not hovered
</button>

<div class="not-supports-[display:grid]:flex">
  Fallback for browsers without grid
</div>

And starting: exposes @starting-style, which is how you do enter transitions without a JavaScript library holding a class on for one frame:

<div popover id="update-panel" class="transition-discrete starting:open:opacity-0">
  <!-- Fades in when opened, no JavaScript needed -->
</div>

A handful of smaller additions that I keep forgetting exist and then rediscovering: inset-shadow-* and inset-ring-* stack up to four shadow layers on one element, field-sizing makes a textarea grow with its content without a resize observer, color-scheme fixes the white scrollbar on a dark page, font-stretch drives variable-font width axes, and the nth-*, inert and in-* variants cover cases that used to need a custom selector.

The breaking changes that will actually bite

The migration tool handles the renames. These are the ones that survive it.

The scale shift. Shadows, radii and blurs all moved down a step, so a bare shadow is now shadow-sm and the old shadow-sm is shadow-xs:

v3	v4
`shadow-sm`	`shadow-xs`
`shadow`	`shadow-sm`
`rounded-sm`	`rounded-xs`
`rounded`	`rounded-sm`
`blur-sm`	`blur-xs`
`blur`	`blur-sm`

The same shift applies to drop-shadow-* and backdrop-blur-*. Codemods catch these in class attributes. They do not catch class names assembled at runtime from a variable, which is exactly where they will hide.

Defaults that changed. Default border colour is currentColor instead of gray-200, so every border with no colour beside it now inherits the text colour, which on a dark heading looks like someone drew on your card with a marker. Default ring width dropped from 3px to 1px and its colour from blue-500 to currentColor. outline-none was renamed to outline-hidden and the name outline-none was reused for outline-style: none, so the old class still compiles and no longer does what it did.

Hover. The hover: variant only applies where the primary input actually supports hovering. Sticky hover states on touch devices are gone, which is a fix, but if your design relied on a hover style being visible after a tap you will notice.

Variant stacking reversed. Stacked variants read left to right now, so first:*:pt-0 becomes *:first:pt-0.

Opacity utilities removed. bg-opacity-*, text-opacity-* and friends are gone in favour of the slash modifier:

<!-- v3 -->
<div class="bg-black bg-opacity-50"></div>

<!-- v4 -->
<div class="bg-black/50"></div>

Custom utilities. The @layer utilities pattern is replaced by @utility:

/* v3 */
@layer utilities {
  .custom-scrollbar {
    scrollbar-width: thin;
  }
}

/* v4 */
@utility custom-scrollbar {
  scrollbar-width: thin;
}

Variables in arbitrary values. bg-[--brand-color] became bg-(--brand-color), because square brackets now unambiguously mean “a literal value” and parentheses mean “a variable”.

And the big one: no preprocessors. Tailwind v4 does not run alongside Sass, Less or Stylus. Tailwind is the preprocessor now, handling imports, nesting and prefixing itself. If your project has a .scss file with mixins in it, you rewrite those styles before you upgrade, not after.

Migrating

Do it on a branch, because the diff is large and you will want to bisect it:

git checkout -b tailwind-v4-migration
npx @tailwindcss/upgrade

The tool needs Node 20 or later. It converts tailwind.config.js into @theme, renames deprecated utilities in your templates, and rewrites your PostCSS or Vite config. Then read the diff properly, and pay attention to four things.

Anything that builds class names at runtime. Template literals, lookup objects mapping a prop to a class, clsx calls with conditional fragments. The codemod sees strings, not intent.
Dark mode. The darkMode: 'class' option has no direct equivalent; you declare it as a variant: @custom-variant dark (&:where(.dark, .dark *));
The container utility. Its center and padding options are gone. Redefine it with @utility container if you relied on them.
Spacing between siblings. space-y-* changed its selector from > :not([hidden]) ~ :not([hidden]) to > :not(:last-child), and the margin moved from the top of the following element to the bottom of the preceding one. Layouts with a hidden first child will shift.

What the tool cannot do at all: rewrite your Sass, port a plugin that reaches into internals, or update snapshot tests that assert on generated CSS.

Who should wait

New project, no legacy: use v4, there is nothing to weigh up.

An existing project splits along one line, and it is not size. It is whether you have Sass. A large v4 migration without Sass is a long afternoon of renames and a careful look at the diff. A small project built on Sass mixins is a rewrite of your styling layer wearing a migration’s clothes, and the upgrade tool will not help with any of it.

The browser floor matters if you have users you cannot choose. Safari 16.4 is March 2023, so this is not aggressive, but “not aggressive” and “acceptable to your client’s compliance team” are different standards.

Component libraries are mostly fine. shadcn/ui, Headless UI and Radix all support v4. Smaller plugins are the risk, though it is worth correcting a claim I have seen repeated in a few places: the JavaScript plugin API was not removed. @plugin still loads a v3-style plugin. Plugins that dig into Tailwind’s internal config resolution are the ones that break, and that is a much smaller set.

What I’m doing about this site

Nothing, for now, and for a boring reason. This blog runs on Bootstrap 5 with a pile of hand-written SCSS, compiled by gatsby-plugin-sass. Tailwind v4 will not sit next to that, and I am not rewriting a working stylesheet to change which utility framework I am not using.

That is the part of this release I find genuinely awkward. The CSS-first configuration is good. The engine is fast. But “we are your preprocessor now” is a much bigger claim than “we renamed some shadows”, and it is the one that decides whether you can adopt v4 at all. Projects that started on Tailwind will not notice. Projects that grew a Sass layer over five years get to choose between two rewrites, and I suspect a lot of them will simply stay on v3 until something forces the issue 🎨

‘Till next time!

Next.js Server Actions

Sat, 24 May 2025 00:00:00 GMT

The endpoint you never write

Most React codebases eventually grow a folder of tiny HTTP endpoints whose only job in life is to let one form write one row to a database. Each of them costs a route file, a fetch call, a JSON round trip, and the same data shape declared twice in TypeScript. Server Actions delete that folder. You write the function, you hand it to a form, and Next.js does the plumbing.

They landed as an alpha in Next.js 13.4 and were marked stable in 14, so at this point they are the default answer for mutations in the App Router rather than a thing you try out on a side project. If you read my notes on React 19, the hooks further down will look familiar. The form hooks and Server Actions were designed together and they only really make sense as a pair.

What a Server Action is

A Server Action is an async function marked with the 'use server' directive. It runs on the server, only on the server, and it can be called from a Server Component or a Client Component without you writing any transport code:

'use server';

export async function createPost(formData: FormData) {
  const title = formData.get('title') as string;
  const content = formData.get('content') as string;

  await db.post.create({
    data: { title, content },
  });
}

Nothing in that file ever reaches the browser. What the client gets instead is a reference: an opaque identifier for the function, plus a bit of generated code that POSTs to it. Next.js serialises the arguments, sends them, runs the function, and serialises whatever comes back.

The second thing worth knowing is that Server Actions are wired into React’s transition machinery. Called from a form’s action prop, or wrapped in startTransition, they participate in concurrent rendering. That is what makes pending states and optimistic updates work without any bookkeeping on your side.

Compared with an API route

The usual way to write a create-post form before all this looked like the following. First the endpoint:

// app/api/posts/route.ts
import { NextResponse } from 'next/server';
import { db } from '@/lib/db';

export async function POST(request: Request) {
  const body = await request.json();
  const { title, content } = body;

  const post = await db.post.create({
    data: { title, content },
  });

  return NextResponse.json(post);
}

Then the form that talks to it, which has to become a Client Component because it needs an onSubmit handler:

// app/posts/new/page.tsx
'use client';

export default function NewPostPage() {
  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault();
    const formData = new FormData(e.target as HTMLFormElement);

    await fetch('/api/posts', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        title: formData.get('title'),
        content: formData.get('content'),
      }),
    });
  };

  return (
    <form onSubmit={handleSubmit}>
      <input name="title" type="text" required />
      <textarea name="content" required />
      <button type="submit">Create Post</button>
    </form>
  );
}

The same feature with a Server Action is two files that barely say anything:

// app/posts/actions.ts
'use server';

import { db } from '@/lib/db';
import { revalidatePath } from 'next/cache';

export async function createPost(formData: FormData) {
  const title = formData.get('title') as string;
  const content = formData.get('content') as string;

  await db.post.create({
    data: { title, content },
  });

  revalidatePath('/posts');
}

// app/posts/new/page.tsx
import { createPost } from './actions';

export default function NewPostPage() {
  return (
    <form action={createPost}>
      <input name="title" type="text" required />
      <textarea name="content" required />
      <button type="submit">Create Post</button>
    </form>
  );
}

No fetch, no headers, no JSON, no 'use client'. The page stays a Server Component and the form keeps working with JavaScript disabled, because a <form action={fn}> degrades to an ordinary browser form post. That last property is the one I keep coming back to. Progressive enhancement used to be something you argued for in a planning meeting and then quietly dropped; here it is the default and you have to work to lose it.

Where the directive goes

There are two placements, and they are not interchangeable in practice.

The module-level version puts 'use server' at the top of a file and turns every export into an action:

// app/actions.ts
'use server';

import { db } from '@/lib/db';
import { revalidatePath } from 'next/cache';

export async function createPost(formData: FormData) {
  const title = formData.get('title') as string;
  const content = formData.get('content') as string;

  await db.post.create({
    data: { title, content },
  });

  revalidatePath('/posts');
}

export async function deletePost(id: string) {
  await db.post.delete({ where: { id } });

  revalidatePath('/posts');
}

The inline version declares the action inside a Server Component:

// app/posts/page.tsx
export default function PostsPage() {
  async function handleDelete(formData: FormData) {
    'use server';
    const id = formData.get('id') as string;
    await db.post.delete({ where: { id } });
    revalidatePath('/posts');
  }

  return (
    <form action={handleDelete}>
      <input type="hidden" name="id" value="123" />
      <button type="submit">Delete</button>
    </form>
  );
}

I default to the file. It is easier to test, easier to grep for, and it keeps components readable. The inline form has one genuine advantage, though, which is that it closes over the render scope: if you need a snapshot of a value as it was when the page rendered, an inline action captures it for free. That convenience has a cost attached, and I come back to it in the security section.

Pending states with useFormStatus

A submit button that does nothing visible for two seconds is a bug report waiting to happen. useFormStatus from react-dom reads the state of the enclosing form:

// components/SubmitButton.tsx
'use client';

import { useFormStatus } from 'react-dom';

export function SubmitButton({ label = 'Save' }: { label?: string }) {
  const { pending } = useFormStatus();

  return (
    <button type="submit" disabled={pending}>
      {pending ? 'Processing...' : label}
    </button>
  );
}

import { createPost } from './actions';
import { SubmitButton } from '@/components/SubmitButton';

export default function NewPostPage() {
  return (
    <form action={createPost}>
      <input name="title" type="text" required />
      <textarea name="content" required />
      <SubmitButton label="Create Post" />
    </form>
  );
}

The hook has one rule that trips everybody up exactly once: it has to live in a component rendered inside the <form>, not in the component that renders the form. It reads from the nearest form context above it, so calling it alongside the <form> element gives you pending: false forever, with no warning and no error. That is why the button gets extracted into its own file even when it does nothing else.

Tracking the result with useActionState

useFormStatus tells you that something is happening. It does not tell you how it went. For that there is useActionState, which lives in react rather than react-dom and replaced the older useFormState:

// app/contact/page.tsx
'use client';

import { useActionState } from 'react';
import { sendMessage } from './actions';

const initialState = {
  success: false,
  message: '',
};

export default function ContactPage() {
  const [state, formAction, isPending] = useActionState(
    sendMessage,
    initialState
  );

  return (
    <form action={formAction}>
      <input name="email" type="email" required />
      <textarea name="message" required />
      <button type="submit" disabled={isPending}>
        {isPending ? 'Sending...' : 'Send Message'}
      </button>
      {state.message && (
        <p className={state.success ? 'text-green' : 'text-red'}>
          {state.message}
        </p>
      )}
    </form>
  );
}

The action on the other side gains an extra first argument, the previous state:

// app/contact/actions.ts
'use server';

export async function sendMessage(prevState: any, formData: FormData) {
  const email = formData.get('email') as string;
  const message = formData.get('message') as string;

  try {
    await sendEmail({ to: email, body: message });

    return { success: true, message: 'Message sent successfully!' };
  } catch (error) {
    return { success: false, message: 'Failed to send message.' };
  }
}

You get back the current state, a wrapped action to hand to the form, and an isPending flag. Two small notes. The prevState: any in the signature above is what most examples show and it is worth replacing with a real type, because the return type of the action and the initial state have to agree and TypeScript will happily let them drift. And whatever you return crosses the network, so it has to be serialisable. A Date survives. A Prisma model instance with methods on it does not.

Optimistic updates

For interactions where waiting for the server feels absurd, useOptimistic lets you render the outcome you expect and roll it back if the server disagrees:

'use client';

import { useOptimistic, useTransition } from 'react';
import { toggleLike } from './actions';

type Post = {
  id: string;
  title: string;
  liked: boolean;
};

export function PostItem({ post }: { post: Post }) {
  const [isPending, startTransition] = useTransition();
  const [optimisticPost, setOptimisticPost] = useOptimistic(
    post,
    (currentPost, newLiked: boolean) => ({
      ...currentPost,
      liked: newLiked,
    })
  );

  const handleToggle = () => {
    startTransition(async () => {
      setOptimisticPost(!optimisticPost.liked);
      await toggleLike(post.id);
    });
  };

  return (
    <div>
      <h3>{optimisticPost.title}</h3>
      <button onClick={handleToggle} disabled={isPending}>
        {optimisticPost.liked ? 'Unlike' : 'Like'}
      </button>
    </div>
  );
}

The startTransition wrapper is not decoration. Outside a form’s action prop, it is the thing that ties the optimistic value to the lifetime of the server round trip. Drop it and the optimistic state reverts the moment the render finishes, which produces a button that flickers back to its old label for a few hundred milliseconds and looks broken. If the action throws, React discards the optimistic value and you are back at post.liked.

Revalidation and redirects

A mutation that does not invalidate its cache is a mutation that appears not to have happened. Next.js gives you two levers.

revalidatePath throws away everything cached for a URL. revalidateTag throws away every fetch that was tagged with a given string:

'use server';

import { revalidatePath, revalidateTag } from 'next/cache';

export async function updatePost(formData: FormData) {
  const id = formData.get('id') as string;
  const title = formData.get('title') as string;

  await db.post.update({
    where: { id },
    data: { title },
  });

  // Blow away everything rendered for this path
  revalidatePath('/posts');

  // Or, more surgically, everything tagged 'posts'
  revalidateTag('posts');
}

Tags are the better instrument once an app has more than a handful of pages, since the same data usually appears in several places. To use them you tag the read side as well, with next: { tags: ['posts'] } on the fetch call. Paths are the blunt instrument you reach for when you are not yet sure which pages are affected, which in my experience is most of the time on a new project.

Redirecting after a mutation uses redirect from next/navigation:

'use server';

import { redirect } from 'next/navigation';

export async function createPost(formData: FormData) {
  const title = formData.get('title') as string;

  const post = await db.post.create({
    data: { title },
  });

  redirect(`/posts/${post.id}`);
}

redirect works by throwing an internal NEXT_REDIRECT error, which means a try/catch around it will swallow the navigation and leave the user staring at the form they just submitted. Call it after the try/catch, never inside. If you need both, store the target in a variable in the try block and redirect once you are out.

Validating what comes in

Client-side validation is a courtesy to the user. It is not a check. Anyone can POST to the action directly, so the action validates or nothing does. Zod is the usual pairing:

// app/actions.ts
'use server';

import { z } from 'zod';
import { revalidatePath } from 'next/cache';

const PostSchema = z.object({
  title: z.string().min(1, 'Title is required').max(200),
  content: z.string().min(10, 'Content must be at least 10 characters'),
  category: z.enum(['tutorial', 'guide', 'opinion']),
});

export async function createPost(prevState: any, formData: FormData) {
  const rawData = {
    title: formData.get('title'),
    content: formData.get('content'),
    category: formData.get('category'),
  };

  const validated = PostSchema.safeParse(rawData);

  if (!validated.success) {
    return {
      success: false,
      errors: validated.error.flatten().fieldErrors,
    };
  }

  await db.post.create({ data: validated.data });

  revalidatePath('/posts');

  return { success: true, errors: null };
}

flatten().fieldErrors gives you an object keyed by field name with an array of messages under each, which maps straight onto the inputs. Combined with useActionState, that is a complete server-validated form with per-field errors and no client validation library at all.

One wrinkle: formData.get() returns null for a missing field, and null is not a string. If a checkbox is unchecked or a select never gets touched, your schema sees null and produces an error message about the wrong thing. Either give the schema explicit handling for it or normalise before parsing.

Errors

Returning errors as data beats throwing them, because a thrown error in an action becomes a generic message in production. Next.js deliberately hides the details, which is the correct behaviour and also mildly infuriating the first time you meet it:

'use server';

import { revalidatePath } from 'next/cache';

export async function processPayment(formData: FormData) {
  const amount = Number(formData.get('amount'));

  try {
    const result = await paymentService.charge(amount);

    revalidatePath('/billing');

    return { success: true, transactionId: result.id };
  } catch (error) {
    if (error instanceof PaymentError) {
      return { success: false, error: error.message };
    }

    return { success: false, error: 'An unexpected error occurred.' };
  }
}

Anything you do throw lands in the nearest error.tsx boundary. Be deliberate about what you put in a returned error string: database driver messages and upstream API responses leak schema names, table names and occasionally credentials. Log the real thing on the server, return the sentence the user can act on. I wrote about a related version of this problem in the Stripe post, where the payment provider’s error codes and the message you want to show a customer are almost never the same text.

Calling actions outside a form

Actions are ordinary functions from the caller’s point of view, so an event handler or an effect can invoke them:

'use client';

import { incrementViewCount } from './actions';
import { useEffect, useTransition } from 'react';

export function PostView({ postId }: { postId: string }) {
  const [, startTransition] = useTransition();

  useEffect(() => {
    startTransition(() => {
      incrementViewCount(postId);
    });
  }, [postId]);

  return null;
}

// actions.ts
'use server';

export async function incrementViewCount(postId: string) {
  await db.post.update({
    where: { id: postId },
    data: { views: { increment: 1 } },
  });
}

Worth remembering that every call is a POST. Actions dispatched from the same page are handled one after another rather than in parallel, so a component that fires one per item in a list will queue them up behind each other. Batch on the server instead of looping on the client.

What Next.js protects you from, and what it doesn’t

This is the part I would put first if the post were shorter, because it is the part that gets skipped. An exported Server Action is a public POST endpoint. It does not become one when you attach it to a form; it already is one.

Next.js does put real protection in the way:

Mechanism	What it does
Encrypted action IDs	The client references actions by a non-deterministic ID, recalculated between builds, not by function name
Dead code elimination	An exported action that nothing in the app calls is stripped from the build and never gets a public ID
Origin/Host comparison	Requests whose `Origin` header does not match `Host` (or `X-Forwarded-Host`) are rejected, which handles most CSRF
Closure encryption	Variables an inline action closes over travel to the client and back, so Next.js encrypts them with a per-build key

That last row is the one to read twice. When you declare an action inside a component and it captures a variable, that variable is serialised into the payload the browser holds. It is encrypted, and the React team’s own advice is not to treat the encryption as a licence to close over secrets. Keep API keys and tokens out of the render scope of any component that declares an inline action.

If you self-host across several instances, each one generates its own encryption key at build time and actions created on one box fail on another. NEXT_SERVER_ACTIONS_ENCRYPTION_KEY pins it, base64-encoded, 32 bytes from openssl rand -base64 32. Behind a reverse proxy where the public domain differs from the internal host, the Origin check will also start rejecting legitimate traffic until you list the proxy in serverActions.allowedOrigins.

None of that is authorisation. The framework has no idea who is allowed to delete which post, so every action does its own check:

'use server';

import { auth } from '@/lib/auth';

export async function deletePost(id: string) {
  const session = await auth();

  if (!session?.user) {
    throw new Error('Unauthorized');
  }

  const post = await db.post.findUnique({ where: { id } });

  if (post?.authorId !== session.user.id) {
    throw new Error('Forbidden');
  }

  await db.post.delete({ where: { id } });

  revalidatePath('/posts');
}

Two checks, not one. Authentication asks whether there is a user; authorisation asks whether this user owns this row. A page-level guard covers neither, because the guard decides what renders and the action is a separate door into the same building. If you are picking a session library to hang off auth(), I went through the options in Better Auth.

Habits worth forming

Some of these I arrived at by reading the docs properly, others by getting them wrong first:

One action, one job. Actions that create and email and audit are impossible to reuse and unpleasant to test.
Keep the action thin and push the database work into a module marked import 'server-only'. The action becomes a validation and authorisation shell.
Return only what the UI renders. A returned Prisma record ships every column to the browser, including the ones you forgot were there.
Revalidate the narrowest thing that changed.
Rate limit anything that sends mail or costs money.

Where I’ve landed

I like Server Actions more than I expected to. The thing that won me over was not the reduced boilerplate, it was that a form works before the JavaScript arrives and keeps working after it does, and I did not have to argue with anyone to get that.

What I am still uneasy about is how invisible the endpoint is. An API route is a file you can see, curl and reason about. An action is a function that quietly acquires a public URL, and the only thing standing between it and the internet is whatever check you remembered to write inside it. The framework has done a lot to make that safe by default. It cannot do the authorisation for you, and the ease of writing an action makes it very easy to forget that you owe it one

‘Till next time!

Stripe Integration in Next.js

Sat, 12 Apr 2025 00:00:00 GMT

The part you only get to test in production once

Payments are the one feature where a bug does not produce a bad user experience, it produces a bank statement. Everything else in a web app can be fixed on Tuesday. A double charge cannot, and nor can an order that was paid for and never shipped because a request timed out at the wrong moment.

Stripe is the obvious choice and I’m not going to pretend otherwise. What I want to write about is the shape of a Next.js integration that survives contact with reality, which mostly means: the checkout session is the easy half, and the webhook is the actual integration.

Stripe shipped a new API version on 31 March, 2025-03-31.basil, and the Node SDK went to 18.0.0 the day after to match. There are two changes in there that will quietly break a subscription integration you copied from a tutorial written last year, so I’ll come back to those. For the database side I’m assuming something like the setup from Using Prisma with Next.js.

Two ways in, and only one of them is worth starting with

Stripe gives you Checkout, which is a page they host and maintain, and Elements plus the Payment Intents API, which is a set of components you assemble into your own form.

Start with Checkout. It handles Strong Customer Authentication, Apple Pay and Google Pay, local payment methods, address collection, tax and receipts, and every one of those is a thing you would otherwise have to build and then keep working. The usual objection is branding, and it’s a weak one: you can set a logo, colours and a domain, and the customers who abandon carts are not abandoning them over the shade of your button.

Move to Elements when you have a concrete reason, like a checkout flow with steps that have to happen between card entry and confirmation. Not before.

Keys, versions and the pin nobody sets

npm install stripe @stripe/stripe-js

stripe is the server SDK. @stripe/stripe-js is the browser loader, and for a plain hosted-Checkout redirect you do not actually need it at all, which surprised me. It earns its place once you use Elements or embedded Checkout.

# .env.local
STRIPE_SECRET_KEY=sk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_...

Only the NEXT_PUBLIC_ one reaches the browser, which is exactly right: the publishable key is meant to be public, the other two are not. Next.js will happily inline anything prefixed that way into the client bundle, so mis-prefixing the secret key ships it to every visitor. Worth a second look before you commit.

Pin the API version explicitly:

// lib/stripe.ts
import Stripe from 'stripe';

export const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!, {
  apiVersion: '2025-03-31.basil',
});

Without the pin you get whatever version your account defaults to, which means a Stripe dashboard setting can change your application’s behaviour without a deploy. Pin it, and upgrade deliberately when you have time to read the changelog.

Creating the session

One route handler, one job:

// app/api/checkout/route.ts
import { headers } from 'next/headers';
import { NextResponse } from 'next/server';
import { stripe } from '@/lib/stripe';
import { auth } from '@/lib/auth';

export async function POST(request: Request) {
  const session = await auth.api.getSession({ headers: await headers() });

  if (!session?.user) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
  }

  const { priceId, quantity = 1 } = await request.json();
  const origin = request.headers.get('origin') ?? 'http://localhost:3000';

  const checkout = await stripe.checkout.sessions.create(
    {
      mode: 'payment',
      line_items: [{ price: priceId, quantity }],
      client_reference_id: session.user.id,
      customer_email: session.user.email,
      success_url: `${origin}/order/success?session_id={CHECKOUT_SESSION_ID}`,
      cancel_url: `${origin}/cart`,
    },
    {
      idempotencyKey: `checkout:${session.user.id}:${priceId}:${quantity}`,
    }
  );

  return NextResponse.json({ url: checkout.url });
}

The session lookup is Better Auth, which I wrote about last month; swap in whatever you use. A few other things in there that the short examples leave out.

Use price with a Price ID created in the dashboard rather than inline price_data. Inline prices mean your catalogue lives in your source code, and the first time someone asks you to run a discount you’ll wish it didn’t. price_data is fine for genuinely dynamic amounts, and nothing else.

client_reference_id is the field that ties the Stripe session back to your user. It comes back on the webhook event. Without it you’re matching on email addresses, which works right up until someone checks out with a different one.

{CHECKOUT_SESSION_ID} is a literal template token that Stripe substitutes, not a JavaScript placeholder. It looks like a typo in every code sample and it isn’t.

The idempotencyKey means a double-clicked button creates one session rather than two. Stripe replays the original response for 24 hours. It costs nothing and removes a whole category of support ticket.

Redirecting

// components/checkout-button.tsx
'use client';

import { useState } from 'react';

export function CheckoutButton({ priceId }: { priceId: string }) {
  const [pending, setPending] = useState(false);

  async function handleClick() {
    setPending(true);

    const response = await fetch('/api/checkout', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ priceId }),
    });

    if (!response.ok) {
      setPending(false);
      return;
    }

    const { url } = await response.json();
    window.location.href = url;
  }

  return (
    <button onClick={handleClick} disabled={pending}>
      {pending ? 'One moment…' : 'Checkout'}
    </button>
  );
}

That’s it. No Stripe.js, no loadStripe, no promise held in module scope.

If you’ve seen stripe.redirectToCheckout({ sessionId }) in older posts, that is the legacy path. It still exists in @stripe/stripe-js, but Stripe’s own quickstarts have used session.url for years now, and the url approach is one fewer network round trip and one fewer script on your page.

The webhook is the integration

Here is the thing that separates a demo from something you can charge money with: do not fulfil the order on the success page. The customer’s browser can close, crash, lose signal or simply never follow the redirect, and the payment still went through. Stripe knows about the payment. Your database does not.

The success page is a receipt. The webhook is the truth.

// app/api/stripe/webhook/route.ts
import type Stripe from 'stripe';
import { stripe } from '@/lib/stripe';
import { fulfillOrder } from '@/lib/orders';

export const runtime = 'nodejs';

export async function POST(request: Request) {
  const body = await request.text();
  const signature = request.headers.get('stripe-signature');

  if (!signature) {
    return new Response('Missing signature', { status: 400 });
  }

  let event: Stripe.Event;

  try {
    event = stripe.webhooks.constructEvent(
      body,
      signature,
      process.env.STRIPE_WEBHOOK_SECRET!
    );
  } catch (error) {
    return new Response(`Invalid signature: ${(error as Error).message}`, {
      status: 400,
    });
  }

  switch (event.type) {
    case 'checkout.session.completed':
      await fulfillOrder(event.id, event.data.object);
      break;

    case 'checkout.session.async_payment_failed':
    case 'payment_intent.payment_failed':
      // notify, release stock, whatever your domain needs
      break;

    default:
      break;
  }

  return new Response(null, { status: 200 });
}

Four details worth spelling out.

Read the raw body. await request.text(), not request.json(). The signature is computed over the exact bytes Stripe sent, so parsing and re-serialising invalidates it. In the App Router this is easy; in the old Pages Router you had to disable the body parser with export const config = { api: { bodyParser: false } } and buffer the stream yourself. Not having to do that is one of the better reasons to be on the App Router.

Pin the runtime to Node. constructEvent uses Node’s crypto. On an edge runtime you want constructEventAsync instead, which goes through the Web Crypto API.

Verify before you trust. The endpoint is public. Anyone can POST to it. Without signature verification, “please mark order 42 as paid” is a curl command.

Handle duplicates. Stripe retries on any non-2xx response, and it will occasionally deliver the same event twice for reasons entirely on its side. event.id is stable, so store it and make fulfilment a no-op the second time:

export async function fulfillOrder(eventId: string, session: Stripe.Checkout.Session) {
  const created = await db.processedEvent.createMany({
    data: { id: eventId },
    skipDuplicates: true,
  });

  if (created.count === 0) return; // already handled

  await db.order.update({
    where: { id: session.client_reference_id! },
    data: { status: 'paid', paymentIntentId: session.payment_intent as string },
  });
}

And return quickly. Stripe’s delivery has a timeout, and a slow handler starts a retry storm. If fulfilment involves sending mail or talking to a warehouse, acknowledge first and queue the work.

Testing it locally

The webhook is unreachable from the internet on localhost, so the CLI tunnels for you:

stripe login
stripe listen --forward-to localhost:3000/api/stripe/webhook

stripe listen prints a whsec_... secret on startup. That is the one to put in .env.local while developing; it is not the same as the endpoint secret from the dashboard, which is the one you need in production. I have lost time to that twice.

To fire an event without going through the flow:

stripe trigger checkout.session.completed

The card numbers, once you’re in test mode:

Card	Expiry, CVC	Result
4242 4242 4242 4242	any future, any	Succeeds immediately
4000 0027 6000 3184	any future, any	Forces a 3D Secure challenge
4000 0000 0000 0002	any future, any	Declined, `generic_decline`
4000 0000 0000 9995	any future, any	Declined, `insufficient_funds`

Test the 3D Secure one. European cards increasingly require it, and a flow that dead-ends on the authentication modal looks fine on your machine and loses you every customer in the EU.

What Basil changed

Two breaking changes in 2025-03-31.basil are worth knowing before they find you.

Subscriptions are created after payment now. Previously, creating a subscription-mode Checkout Session created the subscription up front. Now Stripe waits until the customer has actually paid. This is a genuine improvement (it fixes the case where a failed first payment left a customer unable to update their billing details) but it means session.subscription and session.invoice are null until the session completes. Any code that reads those straight after sessions.create now reads nothing. Move it to the checkout.session.completed handler.

current_period_end moved. It used to live on the Subscription. In Basil it lives on each subscription item, because items in a single subscription can be on different billing cycles:

const subscription = await stripe.subscriptions.retrieve(subscriptionId);

// used to work, doesn't any more:
// const renewsAt = subscription.current_period_end;

const renewsAt = subscription.items.data[0].current_period_end;

Every “gate the feature until the period ends” implementation written before April 2025 reads undefined here, and undefined compared against a timestamp is false, so the failure mode is “everyone’s subscription looks expired”. Delightful.

Also gone: expanding total_count on list responses. If you were paginating with it, you now count differently.

The bit that still bothers me

Amounts are integers in the currency’s smallest unit. unit_amount: 2000 is twenty francs, not two thousand. Everyone knows this and everyone has still, at some point, written unit_amount: price where price was already in francs and charged someone one hundredth of what they owed. The API cannot catch it because both values are valid integers.

The only thing that has reliably worked for me is refusing to let a bare number cross the boundary. Wrap it, name it amountInCents, make the type system carry the unit, and validate the total against a server-side price lookup before creating the session. Never send an amount from the client. It is a form field, and form fields lie.

What I’d add next

Reconciliation. Everything above is correct as long as webhooks arrive, and mostly they do, but “mostly” is not a word I enjoy near an order table. A nightly job that lists Stripe’s completed sessions for the last day and compares them against local orders would close the gap, and I have not written it yet.

Also worth a look is embedded Checkout via ui_mode: 'embedded', which keeps the customer on your domain and returns a client_secret instead of a url. Slightly more work, noticeably less jarring.

‘Till next time!

Better Auth

Fri, 28 Mar 2025 00:00:00 GMT

Authentication you can read

Every project reaches the same fork. You either hand authentication to a service and accept that your users live in someone else's database, or you build it yourself and spend the next three weeks discovering what you forgot about session rotation. Neither option has ever felt right to me, which is why Better Auth caught my attention.

It is a TypeScript library, not a service. It writes its tables into your database, runs inside your app, and hands you a Response when a request comes in. There is no dashboard, no per-monthly-active-user pricing, and no third party holding your session table hostage.

I was already using Prisma on the project I tried it on, which I wrote about in Using Prisma with Next.js, so the adapter story mattered to me more than the feature list. Version 1.2.5 is current as I write this, and 1.2 landed a few weeks ago with a pile of new plugins, so this is a reasonable moment to look at it.

Where it actually sits

The comparison people reach for is Auth.js, and it’s the wrong one to start with. Auth.js is built around OAuth providers, with credentials support bolted on and deliberately discouraged. Better Auth treats email and password as a first-class citizen and hangs the social providers off the side.

The comparison that matters is Clerk or Auth0. Those give you hosted UI, a support contract and someone else’s uptime. Better Auth gives you none of that and, in exchange, your user table is a table in your database that you can join against. For a small product where the user record needs to be foreign-keyed to half the schema, that trade is worth making. For a team without anyone who wants to own auth, it is not, and I’d stop pretending otherwise.

Setting it up

npm install better-auth

Two environment variables. The secret is used for encryption and hashing, so generate a real one:

# .env
BETTER_AUTH_SECRET=  # openssl rand -base64 32
BETTER_AUTH_URL=http://localhost:3000

Then an auth.ts, exported as auth so the CLI can find it later:

// lib/auth.ts
import { betterAuth } from 'better-auth';
import { prismaAdapter } from 'better-auth/adapters/prisma';
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

export const auth = betterAuth({
  appName: 'My App',
  database: prismaAdapter(prisma, {
    provider: 'postgresql',
  }),
  emailAndPassword: {
    enabled: true,
  },
  socialProviders: {
    github: {
      clientId: process.env.GITHUB_CLIENT_ID as string,
      clientSecret: process.env.GITHUB_CLIENT_SECRET as string,
    },
  },
  trustedOrigins: ['http://localhost:3000'],
});

The as string casts are ugly and I resented typing them, but process.env.X is string | undefined and the option is not. Validate your environment with Zod at boot and the casts become honest.

If you’re not on an ORM, you can hand database a better-sqlite3 instance, a pg Pool, a mysql2 pool, or any Kysely dialect, and Better Auth runs the queries itself. The built-in adapters are Prisma, Drizzle and MongoDB. There is a memory adapter too, which is mostly useful for tests. That’s the whole list, and it’s worth knowing before you start, because a fair amount of what you’ll read online invents adapters that don’t exist.

The schema is yours, which is the point

Better Auth needs four core tables, and plugins add more:

user holds the profile plus an emailVerified flag
session holds live sessions, with IP address and user agent
account holds linked providers, OAuth tokens, and the password hash
verification holds short-lived tokens for email confirmation and password resets

The password living on account rather than on user threw me for a minute, and then made sense: email-and-password is just another provider, so it gets a row like GitHub does. Which also means a user can have several accounts and one of them happens to be credentials.

npx @better-auth/cli generate   # writes a Prisma/Drizzle schema or SQL file
npx @better-auth/cli migrate    # applies directly (built-in Kysely adapter only)

With Prisma you run generate and then your usual prisma migrate dev. The important habit is re-running generate every time you add a plugin or an additional field, because plugins ship schema. Forgetting that produces runtime errors about missing columns that read like library bugs and aren’t.

Additional fields go in the config rather than straight into the schema file, so the types flow through to useSession and signUp.email:

export const auth = betterAuth({
  // ...
  user: {
    additionalFields: {
      role: {
        type: 'string',
        defaultValue: 'user',
        input: false, // users don't get to set their own role
      },
    },
  },
});

That input: false is doing real work. Without it, the sign-up endpoint accepts a role from the request body and you have handed anyone with curl an admin account.

Mounting the handler

auth.handler takes a web Request and returns a Response. Every framework integration is a thin wrapper over that.

Next.js App Router:

// app/api/auth/[...all]/route.ts
import { auth } from '@/lib/auth';
import { toNextJsHandler } from 'better-auth/next-js';

export const { GET, POST } = toNextJsHandler(auth);

Express, where there is a trap:

// server.ts
import express from 'express';
import { toNodeHandler } from 'better-auth/node';
import { auth } from './auth';

const app = express();

app.all('/api/auth/*', toNodeHandler(auth));

// after, never before
app.use(express.json());

app.listen(3005);

Mount express.json() before the auth handler and the body gets consumed by the JSON parser, the handler waits for a stream that will never arrive, and every client call hangs on pending forever. There’s no error, just silence. On Express 5 the route pattern changes to /api/auth/*splat, and CommonJS is not supported at all, so "type": "module" in your package.json.

You can check the mount with a GET to /api/auth/ok.

The client

// lib/auth-client.ts
import { createAuthClient } from 'better-auth/react';

export const authClient = createAuthClient({
  baseURL: 'http://localhost:3000',
});

There are equivalents for Vue, Svelte, Solid and vanilla. The React one gives you useSession as a reactive hook, and everything else is a plain async call:

await authClient.signUp.email(
  { email, password, name },
  {
    onSuccess: () => router.push('/dashboard'),
    onError: (ctx) => setError(ctx.error.message),
  }
);

await authClient.signIn.social({ provider: 'github' });

The second argument is a set of lifecycle callbacks (onRequest, onSuccess, onError), which I ended up preferring to unwrapping { data, error } at every call site. Users are signed in automatically after sign-up unless you set autoSignIn: false.

On the server, the same endpoints are available under auth.api, taking body, headers and query instead of a merged object:

import { headers } from 'next/headers';
import { auth } from '@/lib/auth';

const session = await auth.api.getSession({
  headers: await headers(),
});

Failures throw APIError from better-auth/api, which carries a status, so mapping it to an HTTP response is straightforward.

Sending the mail is your job

Better Auth never sends an email. It calls a function you supply. I like that more than I expected to, because it means the verification mail lives with the rest of the templates instead of in someone’s dashboard. I’d been taking React Email apart the month before, so it dropped straight in:

// lib/auth.ts
import { sendPasswordResetMail, sendVerificationMail } from '@/lib/mailer';

export const auth = betterAuth({
  // ...
  emailAndPassword: {
    enabled: true,
    requireEmailVerification: true,
    sendResetPassword: async ({ user, url }) => {
      await sendPasswordResetMail(user.email, url);
    },
  },
  emailVerification: {
    sendOnSignUp: true,
    autoSignInAfterVerification: true,
    sendVerificationEmail: async ({ user, url }) => {
      await sendVerificationMail(user.email, user.name, url);
    },
  },
});

Keep the rendering behind that mailer module rather than inlining it here, because the templates are JSX and auth.ts would have to become auth.tsx to hold them. Small thing, annoying to discover halfway through.

With requireEmailVerification: true, an unverified user attempting to sign in triggers sendVerificationEmail again rather than getting a session, which is the behaviour you want and not the behaviour I assumed.

Sessions are worth a deliberate look too. They last seven days by default and get extended whenever they’re used and updateAge has passed:

session: {
  expiresIn: 60 * 60 * 24 * 7,
  updateAge: 60 * 60 * 24,
  cookieCache: {
    enabled: true,
    maxAge: 5 * 60,
  },
},

cookieCache keeps a signed copy of the session in the cookie so getSession can answer without a database round trip. With a five minute window, a revoked session stays usable for up to five minutes, so it’s a straight trade between latency and immediacy. There’s also freshAge, one day by default, controlling how recently a session must have been created before the sensitive endpoints will accept it. You can set it to 0 to disable the check. I would think about that one for a while first.

Plugins are where it gets interesting

The core is deliberately small. Everything else is a plugin, and most plugins come in pairs: a server half and a client half that adds the matching methods to authClient.

Two-factor authentication, as an example:

// lib/auth.ts
import { twoFactor } from 'better-auth/plugins';

export const auth = betterAuth({
  appName: 'My App', // used as the TOTP issuer
  // ...
  plugins: [twoFactor()],
});

// lib/auth-client.ts
import { twoFactorClient } from 'better-auth/client/plugins';

export const authClient = createAuthClient({
  plugins: [twoFactorClient()],
});

Then, after re-running the CLI:

const { data } = await authClient.twoFactor.enable({ password });
// data.totpURI -> render as a QR code
// data.backupCodes -> show once, never again

await authClient.twoFactor.verifyTotp({ code });

Worth knowing: twoFactorEnabled stays false until the user verifies a TOTP code. The flag reflects a working authenticator, not an intention, which is exactly right and caught me off guard the first time. After that, signIn.email for a 2FA user returns twoFactorRedirect: true instead of a session, and you handle the second step yourself.

The organisation plugin covers multi-tenancy: organisations, members, invitations, and since 1.2, teams inside an organisation. Role-based access control lives in better-auth/plugins/access:

import { createAccessControl } from 'better-auth/plugins/access';
import { defaultStatements, adminAc } from 'better-auth/plugins/organization/access';

const statement = {
  ...defaultStatements,
  invoice: ['read', 'issue', 'void'],
} as const;

const ac = createAccessControl(statement);

const admin = ac.newRole({
  ...adminAc.statements,
  invoice: ['read', 'issue', 'void'],
});

Spreading defaultStatements and adminAc.statements is not optional. Define a custom role without them and you silently overwrite the built-in permissions for that role, which is a fun afternoon.

The rest of 1.2 brought an API key plugin, a captcha plugin, a Stripe plugin for subscriptions, JWT encryption and a few more social providers. The passkey, magic link, email OTP, username, admin, multi-session and OIDC provider plugins were already there. It’s a lot of surface for a library that reached 1.0 four months ago, and I’d treat the newer ones with the caution that implies.

Testing

The nice consequence of auth being a library is that it is testable without a network. Point it at the memory adapter and call the server API directly:

// auth.test.ts
import { expect, test } from 'vitest';
import { betterAuth } from 'better-auth';
import { memoryAdapter } from 'better-auth/adapters/memory';
import { APIError } from 'better-auth/api';

const makeAuth = () =>
  betterAuth({
    baseURL: 'http://localhost:3000',
    secret: 'test-secret-not-for-production',
    database: memoryAdapter({
      user: [],
      session: [],
      account: [],
      verification: [],
    }),
    emailAndPassword: { enabled: true },
  });

test('signs a user up and returns a session', async () => {
  const auth = makeAuth();

  const result = await auth.api.signUpEmail({
    body: {
      email: 'user@example.com',
      password: 'correct-horse-battery',
      name: 'Test User',
    },
  });

  expect(result.user.email).toBe('user@example.com');
});

test('rejects a short password', async () => {
  const auth = makeAuth();

  await expect(
    auth.api.signUpEmail({
      body: { email: 'a@example.com', password: 'short', name: 'A' },
    })
  ).rejects.toBeInstanceOf(APIError);
});

A fresh instance per test keeps the in-memory store from leaking between them. The minimum password length is eight characters by default, which is what the second test is leaning on.

For anything involving cookies you’ll want a real adapter and asResponse: true, which gives you the full Response including Set-Cookie rather than just the data.

What I’d watch out for

trustedOrigins is not decoration. Get it wrong across subdomains and you’ll spend an evening staring at requests that succeed and sessions that never appear, because the cookie was set on the wrong host.

Plugin order in the array matters more than the docs suggest, particularly when several plugins hook the same endpoint.

And the schema is generated, not owned. Custom columns you add by hand outside additionalFields will not survive the next generate. Put them in the config or in a separate table.

Would I use it again

Yes, on a project where the user record is genuinely part of the domain model and I want it in the same transaction as everything else. I would not reach for it on something where auth is a checkbox and nobody on the team wants to think about session rotation. That’s what the hosted services are for, and they’re good at it.

I’m now curious about the Stripe plugin from 1.2, mostly because subscription state and user state ending up in two different databases is a problem I’ve watched several projects lose to. That might be the next thing I take apart.

‘Till next time!

React Email

Sun, 16 Feb 2025 00:00:00 GMT

Still tables, all the way down

Email markup never moved on. The HTML you write for an inbox in 2025 is roughly the HTML you would have written in 2003: nested tables, inline styles, and a quiet hope that Outlook does something reasonable. Every convenience I take for granted in the browser (components, types, a dev server that reloads) stops at the mailbox door.

React Email is an attempt to put those conveniences back. You write templates as React components, and it renders them down to the table soup that mail clients actually understand. The generated HTML is still horrible. The difference is that you no longer have to look at it.

I write React most days, so the pitch landed easily. The components now declare React 19 as a supported peer alongside 18, so if you’ve already moved to React 19 there’s no dependency wrestling involved. Below is what I found once I got past the readme.

What’s actually in the package

Two packages do the work, and it’s worth knowing which is which.

@react-email/components (0.0.33 at the time of writing) is the component library. It re-exports about twenty small packages: Html, Head, Body, Container, Section, Row, Column, Text, Heading, Button, Link, Img, Hr, Font, Preview, Markdown, CodeBlock, CodeInline and the Tailwind wrapper. It also re-exports render, so in practice one dependency covers you.

react-email (3.0.7) is the CLI and the preview app. It is a dev dependency, not something you ship.

The components are thin on purpose, and that’s the part I like. Unwrap the layout ones and there’s almost nothing there:

Component	What it renders
`Section`	`<table>` with a single `<tr>` and `<td>`
`Row`	`<table>` with a `<tr>`
`Column`	`<td>`
`Container`	`<table>` capped at `maxWidth: 37.5em`
`Preview`	hidden `<div>` with `maxHeight: 0`

That 37.5em is the 600px everyone has silently agreed on. Preview holds the snippet text that shows up in the inbox list, padded with invisible characters so the client doesn’t drag your footer into the preview line.

None of that is clever. It is just the table boilerplate you would have written by hand, given a name and a prop signature. Which is the correct amount of abstraction for a problem this dumb.

Getting it running

npm install @react-email/components
npm install --save-dev react-email

Templates go in an emails/ directory by default. Add a script:

{
  "scripts": {
    "email": "email dev --dir ./emails --port 3000"
  }
}

email dev starts a preview server that watches the directory and rebuilds on save. The other commands are email build and email start (build the preview app and serve it, useful if you want designers poking at templates on a staging URL) and email export, which writes the compiled templates to out/:

npx email export --outDir out --pretty
npx email export --outDir out --plainText

One thing that caught me out: --dir is resolved relative to the project root, not to wherever you run the command from. If the preview server shows an empty list, that’s usually why.

A template with some meat on it

The tutorial examples are all “Hello world in a <Text>”, which tells you nothing. Here’s something closer to a real transactional mail, with typed props and a preview fixture:

// emails/welcome.tsx
import {
  Body,
  Button,
  Container,
  Head,
  Hr,
  Html,
  Preview,
  Section,
  Text,
} from '@react-email/components';

type WelcomeEmailProps = {
  name: string;
  confirmUrl: string;
};

export const WelcomeEmail = ({ name, confirmUrl }: WelcomeEmailProps) => (
  <Html lang="en">
    <Head />
    <Preview>Confirm your address and you are done</Preview>
    <Body style={body}>
      <Container style={container}>
        <Text style={heading}>Hello {name}</Text>
        <Text>
          One more step. Confirm your address so we know the mailbox is real.
        </Text>
        <Section style={{ textAlign: 'center' }}>
          <Button href={confirmUrl} style={button}>
            Confirm my address
          </Button>
        </Section>
        <Hr style={rule} />
        <Text style={footnote}>
          If you did not sign up, ignore this mail. Nothing happens.
        </Text>
      </Container>
    </Body>
  </Html>
);

WelcomeEmail.PreviewProps = {
  name: 'Angelo',
  confirmUrl: 'https://example.com/confirm/abc123',
} satisfies WelcomeEmailProps;

export default WelcomeEmail;

const body = {
  backgroundColor: '#f4f4f5',
  fontFamily: 'Helvetica, Arial, sans-serif',
};
const container = { backgroundColor: '#ffffff', padding: '24px' };
const heading = { fontSize: '20px', fontWeight: 'bold' };
const button = {
  backgroundColor: '#111827',
  color: '#ffffff',
  padding: '12px 20px',
  borderRadius: '4px',
  textDecoration: 'none',
};
const rule = { borderColor: '#e4e4e7' };
const footnote = { fontSize: '12px', color: '#71717a' };

PreviewProps is the bit worth stealing. The preview server reads that static property and renders the template with those values, so you get a realistic preview without a fake wrapper component. satisfies keeps it honest when you rename a prop.

Note that Button is just an <a> under the hood, so textDecoration: 'none' is on you. Older versions had pX/pY props for padding; those are gone, and normal padding in the style object is the way now.

Tailwind, and where it stops

There’s a Tailwind component that wraps your tree and rewrites utility classes into inline styles at render time:

import { Tailwind } from '@react-email/components';

<Tailwind
  config={{
    theme: {
      extend: {
        colors: { brand: '#111827' },
      },
    },
  }}
>
  <Body className="bg-zinc-100 font-sans">
    <Container className="bg-white p-6">
      <Text className="text-xl font-bold">Hello</Text>
    </Container>
  </Body>
</Tailwind>;

The config prop takes a subset of a Tailwind v3 config object (theme, plugins, darkMode and a few others), so a shared theme file from your app mostly transplants.

Here is the caveat nobody puts in the headline. Anything that cannot be expressed as an inline style, so media queries, hover:, dark:, gets hoisted into a <style> block in the <head> instead. That works in clients which keep head styles, and it silently does nothing in the ones that strip them. Gmail’s web client is fine with it; several others are not. So responsive breakpoints in an email are a progressive enhancement, not a layout strategy. Build the thing to look correct at 600px first.

render() returns a promise now

This is the change most old tutorials get wrong. Since @react-email/render 1.0, render() is asynchronous:

declare const render: (
  element: React.ReactElement,
  options?: Options
) => Promise<string>;

renderAsync still exists but is deprecated, and now just points at render. The reason is that React is moving away from renderToStaticMarkup, and an async renderer is what keeps Suspense and server components working.

The practical consequence: if you forget the await, you pass a Promise where a string is expected and send an email whose body reads [object Promise]. It fails quietly, which is the worst kind of failing.

The options are small:

const html = await render(<WelcomeEmail {...props} />, { pretty: true });
const text = await render(<WelcomeEmail {...props} />, { plainText: true });

pretty beautifies the output (nice for snapshots, pointless in production, and it costs you bytes you may not have). plainText runs the HTML through html-to-text, and you can tune that with htmlToTextOptions. There’s also a plainTextSelectors export with sensible defaults if you want to build your own conversion.

Testing the templates

Templates are components, so they test like components. I covered the general approach in Testing Strategies in React; nothing here is special except the await.

npm install --save-dev vitest @vitejs/plugin-react

You do need the React plugin, otherwise Vitest will not transform the JSX in your .tsx templates and you’ll get a parse error that looks like a Vitest bug and isn’t.

// vitest.config.ts
import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  test: { environment: 'node' },
});

// emails/welcome.test.tsx
import { expect, test } from 'vitest';
import { render } from '@react-email/components';
import { WelcomeEmail } from './welcome';

const props = { name: 'Angelo', confirmUrl: 'https://example.com/c/abc' };

test('renders the confirmation link', async () => {
  const html = await render(<WelcomeEmail {...props} />);

  expect(html).toContain('https://example.com/c/abc');
  expect(html).toContain('Hello Angelo');
});

test('has a usable plain text alternative', async () => {
  const text = await render(<WelcomeEmail {...props} />, { plainText: true });

  expect(text).toContain('Confirm my address');
  expect(text).not.toContain('<td');
});

I would not snapshot the full HTML. It changes every time a component package bumps a patch version, and you end up approving diffs you haven’t read. Assert on the things that would actually cause a support ticket: the link is correct, the name is interpolated, the plain text alternative is not empty.

Sending through AWS SES

React Email renders. It does not send. That part is yours, and SES is the cheap option if you already have an AWS account.

npm install @aws-sdk/client-ses

// lib/send-welcome.tsx
import { SESClient, SendEmailCommand } from '@aws-sdk/client-ses';
import { render } from '@react-email/components';
import { WelcomeEmail } from '../emails/welcome';

const ses = new SESClient({ region: 'eu-central-1' });

export async function sendWelcome(to: string, name: string, confirmUrl: string) {
  const element = <WelcomeEmail name={name} confirmUrl={confirmUrl} />;

  const [html, text] = await Promise.all([
    render(element),
    render(element, { plainText: true }),
  ]);

  await ses.send(
    new SendEmailCommand({
      Source: 'hello@example.com',
      Destination: { ToAddresses: [to] },
      Message: {
        Subject: { Data: 'Confirm your address', Charset: 'UTF-8' },
        Body: {
          Html: { Data: html, Charset: 'UTF-8' },
          Text: { Data: text, Charset: 'UTF-8' },
        },
      },
    })
  );
}

Two details that matter and are easy to miss.

The file has to be .tsx, not .ts. It contains JSX. I have seen that exact mistake copied around in half the blog posts on this subject.

Send both parts. A message with only an HTML body scores worse with spam filters and renders as nothing at all in a text-only client. Since render already gives you the plain text version, there is no excuse.

On the AWS side, a new account starts in the SES sandbox: 200 messages per 24 hours, one message per second, and every recipient address has to be verified first. You request production access through a support ticket, which takes a day or two. Before you do, set up DKIM signing on the sending domain and add SPF and DMARC records, because the request asks how you handle bounces and complaints and the honest answer needs to be “with a configuration set publishing to SNS”, not “I don’t”.

If you’d rather use the newer API, @aws-sdk/client-sesv2 has its own SendEmailCommand with a different parameter shape (FromEmailAddress, Content.Simple.…). Same idea, different nouns.

The parts that still hurt

React Email fixes the authoring experience. It does not fix email.

Classic Outlook on Windows still renders through the Word engine, so no flexbox, no grid, and background images need VML if you really want them. The new Outlook is on a web engine, which helps, but you cannot assume your recipients have it.

Gmail clips a message once the HTML passes about 102 KB and hides the rest behind a “view entire message” link. That is HTML only, images don’t count, but a Tailwind-heavy template with lots of repeated inline styles gets there faster than you’d think. Rendering without pretty helps.

Dark mode is a mess. Some clients invert your colours for you, some do it badly, some ignore prefers-color-scheme entirely. I have stopped trying to control it and just avoid pure white backgrounds with near-white text.

And the preview server is not a rendering test. It’s Chrome. It tells you your template compiles and roughly how it looks; it tells you nothing about Outlook. For that you still need Litmus, Email on Acid, or a folder of test accounts and some patience.

Where I’ve landed

I’d use it again. Not because emails became easy, but because the templates now live next to the rest of the code, get type-checked in CI, and can be reviewed in a pull request by someone who does not know what VML is.

Next thing on my list is wiring email export into the build so the compiled templates are artefacts rather than something rendered at request time. I have not decided yet whether that’s a real improvement or just me enjoying a build step

‘Till next time!

Looking back at 2024

Fri, 31 Jan 2025 00:00:00 GMT

Eleven posts later

2024 moved a lot. There was travelling, there were relationships that shifted around, and there was the usual amount of quietly rearranging my own head. On balance a good year.

I’m not going to write much about the personal side of that here. I’d rather keep going, pick up new things and see where they lead. What I can point at is the writing, which is the habit I managed to hold onto through all of it.

Eleven articles, plus last January’s recap. One a month, every month, same as the year before.

Reading the list back, it leans further into React than I intended when I started the year. Not a single homelab post either, which is a bit of a shame. The rack did not stop existing just because I stopped writing about it 🙃

What I’d like to change

Fewer “here is a new feature” posts and more “here is what broke”. The ones I enjoy writing are the ones where something went wrong first.

See you all in 2025 🎉

‘Till next time!

React 19

Sat, 21 Dec 2024 00:00:00 GMT

The one that took two years

React 19 went stable on 5 December 2024, and if you have been running the Canary channel or a recent Next.js, most of it will already be familiar. That is the interesting part of this release: almost nothing in it is new, it is just finally allowed to be in a version number.

When I wrote about React 18 I said it was a set of small changes laying foundations for later releases. This is the release those foundations were for. The difference is that the patterns arrived first, in frameworks, and the library caught up afterwards. Nearly everything below spent a year or more in Canary before it got a version number.

So this is less an announcement than an inventory. Here is what you can now rely on.

Actions

An Action is an async function passed straight to a form’s action prop. React calls it with the FormData, and handles the pending state, the errors and the reset for you.

// server-side action
'use server';

export async function createUser(formData) {
  const name = formData.get('name');
  // perform server-side operations
}

// client-side component
'use client';

import { createUser } from './actions';

export default function UserForm() {
  return (
    <form action={createUser}>
      <input name="name" />
      <button type="submit">Create User</button>
    </form>
  );
}

The function can be a plain client-side async function too. 'use server' only matters when you want it to run on the server, and it is a common misreading to think the directive is what makes something an Action.

`useActionState`

When you need the result of an Action rather than just firing it, useActionState wraps it and gives you back the state, a wrapped action to hand to the form, and a pending flag.

The catch is the signature. An action used with this hook takes the previous state as its first argument:

'use server';

export async function createUser(previousState, formData) {
  const name = formData.get('name');

  if (!name) {
    return { message: 'Name is required' };
  }

  // create the user
  return { message: `Created ${name}` };
}

'use client';
import { useActionState } from 'react';
import { createUser } from './actions';

export default function UserForm() {
  const [state, formAction, isPending] = useActionState(createUser, {
    message: '',
  });

  return (
    <form action={formAction}>
      <input name="name" />
      <button type="submit" disabled={isPending}>
        Create User
      </button>
      {state.message && <p>{state.message}</p>}
    </form>
  );
}

If you have used this in Canary or in Next.js and remember it as useFormState from react-dom, that is the same hook. It moved to react and was renamed on the way.

`useFormStatus`

useFormStatus reads the state of the nearest parent form from inside a child component. This is the one I reach for most, because it means a submit button can know it is submitting without the form passing anything down to it.

'use client';

import { useFormStatus } from 'react-dom';

function SubmitButton() {
  const { pending } = useFormStatus();

  return (
    <button type="submit" disabled={pending}>
      Submit
    </button>
  );
}

Note the import path. It comes from react-dom, not react, which will catch you at least once. It also has to be in a component rendered inside the form; calling it in the component that renders the <form> itself returns nothing useful.

`useOptimistic`

useOptimistic shows the result before the server confirms it, and rolls back automatically if the action fails.

'use client';
import { useOptimistic } from 'react';

function CommentList({ comments, addComment }) {
  const [optimisticComments, addOptimisticComment] = useOptimistic(
    comments,
    (state, newComment) => [...state, newComment]
  );

  const handleSubmit = async formData => {
    const comment = formData.get('comment');

    addOptimisticComment(comment);
    await addComment(comment);
  };

  return (
    <form action={handleSubmit}>
      <input name="comment" />
      <button type="submit">Add Comment</button>
      <ul>
        {optimisticComments.map((c, i) => (
          <li key={i}>{c}</li>
        ))}
      </ul>
    </form>
  );
}

The first argument has to be the real state, not an empty array. Pass [] and you get an optimistic list that forgets everything the moment the action settles, which looks like a rendering bug and is not one.

The `use` API

use reads a resource during render. Given a promise it suspends until the promise resolves, and given a context it reads it.

import { use } from 'react';

function Comments({ commentsPromise }) {
  const comments = use(commentsPromise);

  return comments.map(comment => <p key={comment.id}>{comment.text}</p>);
}

It breaks the rules of hooks on purpose. You can call it inside a conditional or a loop, which is the entire reason it exists and reads wrong the first few times you see it. What you must not do is create the promise inside the component that consumes it, because a new promise on every render means suspending forever. Pass it in from a server component or a cache.

Static rendering APIs

react-dom/static gains prerender and prerenderToNodeStream. Unlike renderToString, these wait for all the data to load before producing output, which is what you want when you are generating static HTML rather than streaming it to a browser.

import { prerender } from 'react-dom/static';

async function handler(request) {
  const { prelude } = await prerender(<App />, {
    bootstrapScripts: ['/main.js'],
  });

  return new Response(prelude, {
    headers: { 'content-type': 'text/html' },
  });
}

Worth noticing that prerender resolves to an object, not an HTML string. prelude is a Web Stream. If you want a string you have to read the stream yourself, and if you are on Node without Web Streams, prerenderToNodeStream is the one you want.

Server Components, and what “stable” means here

React 19 includes the Server Components work that has been sitting in Canary for two years. Stable, in this case, means the feature will not break between minor versions.

It does not mean you can turn it on. Server Components need a bundler and a router that understand them, so in practice you get them through a framework. What React ships is the contract, not the implementation.

There is a second bit of small print worth knowing. The APIs a framework uses to implement Server Components do not follow semver in 19.x and may change between minors. If you maintain a framework or a bundler, pin your React version. If you are building an application on top of Next.js, this is somebody else’s problem.

The other thing to get straight: there is no directive for Server Components. Components are server components by default in an RSC setup, 'use client' opts out, and 'use server' marks a Server Action, which is a different thing entirely. Nearly everyone gets this backwards once.

// server action
'use server';

export async function createUser(formData) {
  // server-side logic
}

// client component
'use client';
import { createUser } from './actions';

export default function UserForm() {
  return (
    <form action={createUser}>
      <input name="name" />
      <button type="submit">Create User</button>
    </form>
  );
}

The smaller changes that will save you the most time

The headline features get the blog posts. These are the ones that will show up in your day-to-day work first:

ref is now a regular prop on function components. forwardRef is no longer needed, and a good chunk of wrapper boilerplate disappears.
<Context> renders as a provider directly. <Context.Provider> still works, but is on its way out.
<title>, <meta> and <link> rendered anywhere in a component get hoisted into the document head. No more helmet library for simple cases.
Errors from hydration mismatches finally tell you what did not match instead of printing two walls of HTML.

Upgrading

The codemods handle most of it and the official upgrade guide is worth the twenty minutes. Budget the time for everything that is not React, though: type packages, testing libraries, and any dependency with an opinion about forwardRef or Context.Provider.

I would not rush a large application over the line this month. The library is ready. The ecosystem around it is still catching up, and there is no prize for being early.

‘Till next time!

Using Prisma with Next.js

Fri, 29 Nov 2024 00:00:00 GMT

Coming to Prisma from the Django ORM

My roots are in Django, and the Django ORM spoils you. Models are the schema, migrations are generated from them, and the query API knows what your tables look like. Moving to a Node stack, the thing I missed most was not a framework. It was that.

Prisma is the closest the JavaScript world has come. You write a schema, it generates a typed client from it, and your editor knows the shape of every row before you run anything. Paired with Next.js, where server components and server actions have made database access on the server the normal thing again, it fits well enough that I stopped looking for alternatives.

Here is how I set it up, and the parts I get wrong when I am not paying attention.

Revised in September 2026. The original was written against Prisma 5, and Prisma 7 changed the setup underneath it: a different generator, a config file, a database driver you install yourself, and a generate step that no longer runs on its own. Everything below is what works with 7. If you have the older setup, the section near the end covers moving it across, and it is shorter than you would expect.

Why it works with Next.js in particular

The schema is a single file, and everything derives from it. Change a model, run one command, and the generated client, the types and the migration all follow. There is no second place where the shape of your data is written down, which is the failure mode of every hand-rolled query builder I have used.

The rest of the fit comes from Next.js rather than Prisma:

Server components and server actions run on the server, which is the only place a database client belongs.
API route handlers are still there when you want a real HTTP endpoint.
The generated types flow straight into your components, so a renamed column becomes a type error rather than an undefined at runtime.

The one thing worth saying out loud: none of this makes Prisma safe to import in a client component. More on that below, because it is the mistake everyone makes once.

Setting it up

Install and initialise

npm install prisma --save-dev
npm install @prisma/client @prisma/adapter-pg dotenv
npx prisma init

That gives you a prisma/ folder with schema.prisma, a prisma.config.ts at the project root, and a .env file for the connection string. The config file is new in 7 and it is where the CLI now looks for the database, so the schema no longer knows the connection string at all.

The adapter package is the other newcomer. Prisma 7 talks to Postgres through the same pg driver you would use without Prisma, and @prisma/adapter-pg is the wrapper that hands it over. There is one per database; SQLite gets @prisma/adapter-better-sqlite3, and MongoDB is not supported in 7 at the time of writing, so a Mongo project stays on 6.

Configure the database

In .env:

DATABASE_URL="postgresql://user:password@localhost:5432/mydb"

Then tell the CLI where things are, in prisma.config.ts:

// prisma.config.ts
import 'dotenv/config';
import { defineConfig, env } from 'prisma/config';

export default defineConfig({
  schema: 'prisma/schema.prisma',
  migrations: {
    path: 'prisma/migrations',
  },
  datasource: {
    url: env('DATABASE_URL'),
  },
});

The dotenv import is not decoration. Prisma 7 stopped loading .env on its own, so without that line env('DATABASE_URL') comes back empty and the error talks about a missing URL rather than a missing file. Next.js loads .env for the application itself; this import only serves the CLI.

Now describe your data in prisma/schema.prisma:

generator client {
  provider = "prisma-client"
  output   = "../generated/prisma"
}

datasource db {
  provider = "postgresql"
}

model Post {
  id        Int      @id @default(autoincrement())
  title     String
  content   String?
  published Boolean  @default(false)
  createdAt DateTime @default(now())
}

Two things changed here compared with the versions people copy from older tutorials. The generator is prisma-client, the one without a Rust engine, and output is required: the client is written into your project instead of into node_modules, which is why the import in the next section points at a folder rather than a package. The path is relative to the schema file, so ../generated/prisma lands next to prisma/ in the project root. Add generated/ to .gitignore; it is build output.

And the url line is gone from the datasource block, because the config file owns it now. If you leave it in, the CLI tells you it is deprecated and ignores it, which is polite but not helpful when the two disagree.

The ? on content is the nullable marker, and it propagates all the way into the generated TypeScript as string | null. Small thing, but it is the reason the types are worth having.

Migrate

npx prisma migrate dev --name init
npx prisma generate

Two commands where there used to be one. migrate dev writes a migration file and applies it, but since 7 it no longer regenerates the client afterwards. You run generate yourself, you will forget, and the type error that follows is the reminder.

The migration lands in prisma/migrations/ as plain SQL, which means you can read it before it touches anything you care about. I do read it. Prisma is good at working out what changed, but it cannot know that the column you renamed had data in it.

The client singleton, first

Before writing a single query, create this file. Every example after it depends on it.

// lib/prisma.ts
import { PrismaPg } from '@prisma/adapter-pg';
import { PrismaClient } from '../generated/prisma/client';

const globalForPrisma = globalThis as unknown as { prisma?: PrismaClient };

const createClient = () =>
  new PrismaClient({
    adapter: new PrismaPg({ connectionString: process.env.DATABASE_URL }),
  });

export const prisma = globalForPrisma.prisma ?? createClient();

if (process.env.NODE_ENV !== 'production') globalForPrisma.prisma = prisma;

The adapter in the constructor is the Prisma 7 change with the most reasoning behind it. Earlier versions carried a query engine written in Rust, downloaded as a binary for your platform at install time, and it did the talking to the database. Seven drops it. The client is TypeScript all the way down, the database connection is a plain pg pool, and you hand that pool to Prisma. No more binary to fetch during npm install, which is one deploy-time failure I do not miss.

The rest of the file is about hot reloading. In development, Next.js re-evaluates modules on every save, and a bare new PrismaClient() at module scope means a fresh client and a fresh connection pool each time. After twenty saves your database starts refusing connections and the error message points at nothing useful. Stashing the instance on globalThis in development only survives the reload; in production the module is evaluated once and the guard does nothing.

Plenty of tutorials instantiate the client inline in each example because it reads more clearly. It also reproduces the bug.

Querying

Server actions are where most of my database code lives now.

// app/actions.ts
'use server';

import { prisma } from '@/lib/prisma';

export async function getPublishedPosts() {
  return prisma.post.findMany({
    where: { published: true },
    orderBy: { createdAt: 'desc' },
  });
}

Then call it from a server component and render the result during the server render:

// app/posts/page.tsx
import { getPublishedPosts } from '../actions';

export default async function PostsPage() {
  const posts = await getPublishedPosts();

  return (
    <main>
      <h1>Published Posts</h1>
      <ul>
        {posts.map(post => (
          <li key={post.id}>{post.title}</li>
        ))}
      </ul>
    </main>
  );
}

No fetch, no API route, no serialisation boundary in the middle. The query runs on the server, the HTML comes back with the data in it, and the connection string never leaves the machine.

Route handlers are still the right answer when something outside your application needs the data:

// app/api/posts/route.ts
import { NextResponse } from 'next/server';
import { prisma } from '@/lib/prisma';

export async function GET() {
  const posts = await prisma.post.findMany();

  return NextResponse.json(posts);
}

If you are on the Pages Router, the same client works inside getServerSideProps and pages/api/* handlers. Nothing about Prisma changes; only where you call it from.

Deploying

Prisma runs fine on Divio, Vercel and Railway. I have written before about hosting on Divio, and the same rules apply wherever you land:

Run prisma generate in the build. The generated client lives in a folder you have ignored, so the build has to create it. Put it in the build script or a postinstall. Prisma 7 makes this less forgiving than before, because nothing else runs it for you any more.
Run prisma migrate deploy in the pipeline, not migrate dev. The dev command is interactive and will happily offer to reset your database. deploy applies pending migrations and nothing else.
Watch the connection pool on serverless. Each function instance opens its own pool, and a managed Postgres box has a low ceiling. A pooler such as PgBouncer or your provider’s own is not optional once you have real traffic.
Keep the connection string in environment variables. The CLI reads it through prisma.config.ts, the application reads it in the singleton, and both need it set in every environment. Obvious, and still worth writing down, because the build and the runtime are often configured in two different places.

Moving an older project to 7

If you have the 2024 version of this setup, the change is mechanical and took me less time than reading the upgrade guide did. In order:

Upgrade the packages and add the two new ones: npm install prisma@7 @prisma/client@7 @prisma/adapter-pg dotenv.
In the schema, rename the generator to prisma-client, give it an output, and delete url from the datasource. Create prisma.config.ts with the URL in it.
Replace import { PrismaClient } from '@prisma/client' with the generated path, and pass the adapter to the constructor.
Run prisma generate, then the type checker. The renamed import surfaces every file that constructed its own client, which is the audit you should have done anyway.
If you used prisma.$use() middleware, it is gone in 7. Client extensions do the same job with a different shape.
Delete the generate step you never had, then add it to the build, because now you need it.

The generated client is ESM. In a Next.js project that costs nothing, since Next handles modules itself. In a plain Node script you either set "type": "module" in package.json or tell the generator you want CommonJS, and the guide covers both.

Habits worth keeping

Import prisma from lib/, never construct a client anywhere else.
Keep it out of client components. If a file has 'use client' at the top, Prisma has no business in it, and the error you get if you try is not a helpful one.
Run prisma generate after every migration, and put it in the build so production does not depend on your memory.
Use TypeScript. Prisma with plain JavaScript works, but you have given up the reason to use Prisma.
Read the generated migration before applying it to anything with data in it.

What still annoys me

prisma generate is a step you have to remember in every environment, and Prisma 7 made it a step you have to remember twice, once after each migration and once in the build. Coming from Django, where migrations and the ORM live inside the framework, having a separate CLI that has to run at the right moment feels like something that should not be my problem.

The move to a client written in TypeScript and a driver I already understood is a real improvement, and the config file is where the connection string always belonged. That is a small complaint against a schema that types itself. I will take the trade.

‘Till next time!

Monorepos

Thu, 31 Oct 2024 00:00:00 GMT

One repository, several projects

A monorepo is one repository holding several related projects. That is the whole idea. Everything else (the tooling, the caching, the arguments on the internet) is about making that arrangement survive contact with a real codebase.

The version of this problem I know best comes from django CMS, which lives across a lot of separate repositories. Core in one, the editor in another, half a dozen packages beside them. A change that touches more than one of those means a pull request in each, a release in each, and a version bump in the right order. That coordination cost is exactly what a monorepo removes, and it is why the pattern keeps coming back.

It also introduces problems of its own, which is the less fun half of the post.

The shape of one

Take a project with two applications and a shared library:

my-monorepo/
├── apps/
│   ├── web/      # the main application
│   ├── docs/     # documentation site
│   └── api/      # serves the API
├── packages/
│   └── ui/       # shared UI components
├── package.json  # workspaces live here
└── turbo.json    # task configuration

apps/ holds things you deploy, packages/ holds things you import. That split is a convention rather than a rule, but every tool in this space assumes it, so go along with it.

What you get

Changes land in one commit. Update a shared component and the applications that consume it get the new version immediately, in the same pull request, reviewed together. No publish step, no version bump, no waiting for the registry.

Dependencies stop drifting. One lockfile at the root means one version of React across everything. Version mismatches between projects simply stop being a category of bug you have to think about.

Builds get cached. This is the part that actually changes how the day feels. If a package has not changed, its build output is reused rather than recomputed. On a cold clone that saves nothing; on the tenth build of the afternoon it saves most of your time.

Refactoring across boundaries becomes possible. Renaming an exported function and fixing every call site in the same change is only realistic when the call sites are in front of you.

What it costs

Pull requests get noisy. With everything in one place, the repository is busier, review ownership blurs, and the person reviewing your change may not know the package you touched. Codeowners files help. They do not fix it.

Clones get slow. The repository only grows, and git history is the thing that grows fastest. git sparse-checkout and shallow clones take the edge off, but a five-year-old monorepo is not a cheap thing to clone on hotel wifi.

Blast radius goes up. A shared package is shared. Change it carelessly and you have broken three applications rather than one, and you find out in CI rather than in a dependency bump you could have deferred.

Without caching, it is worse than what you had. Running every test in the repository on every commit is the failure mode. If you adopt a monorepo and skip the task graph, you have taken all of the costs and none of the benefits.

Monorepo or single repos

Aspect	Monorepo	Single repos
Code reuse	Import directly from a workspace package	Publish to a registry, or copy and paste and regret it
Dependency management	One lockfile, one version of everything	Independent, and drifts apart quietly
Build efficiency	Fast with a task graph and cache, slow without one	Fast per repo, no way to share work across them
Release cadence	Everything moves together, which is good until it is not	Each project releases on its own schedule
Onboarding	One clone, one install, one command to run everything	Find the repos, clone them all, hope the readme is current

The honest summary is that single repos are the right default for projects that do not share code, and a monorepo starts paying the moment two of them do.

Setting up Turborepo

Turborepo handles the task graph and the caching, and leaves package management to your package manager’s workspaces. It does less than the alternatives, which I like.

Scaffold one and run it:

npx create-turbo@latest
cd my-turborepo
npm run dev

The generated repository gives you two applications and a shared package, which is enough to see the caching work. Build twice and watch the second run report FULL TURBO.

The tasks config

At the root you get a turbo.json:

{
  "$schema": "https://turbo.build/schema.json",
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": [".next/**", "!.next/cache/**", "dist/**"]
    },
    "lint": {},
    "test": {
      "dependsOn": ["build"]
    }
  }
}

Note the key is tasks. It was called pipeline until Turborepo 2.0 landed in June 2024, and a lot of tutorials still have not caught up. If you inherit an older repository, npx @turbo/codemod migrate renames it for you.

Two things earn their keep here. dependsOn: ["^build"] with the caret means “build my dependencies first”, so Turborepo works out the order rather than you maintaining it by hand. And outputs tells the cache what to save; get that list wrong and your cache hits will restore nothing useful, which is a confusing way to spend an afternoon.

Give the shared package a name in its package.json:

{
  "name": "@my-monorepo/ui",
  "main": "./index.js"
}

Export a component from it:

// packages/ui/Button.js
import React from 'react';

export const Button = ({ children }) => <button>{children}</button>;

And import it in an application as if it came from npm:

import { Button } from '@my-monorepo/ui';

const HomePage = () => <Button>Click me!</Button>;

No build step for the package, no publish, no version. The workspace resolution does the work.

The alternatives

Nx does considerably more: a dependency graph you can visualise, generators for new packages, and framework integrations for React, Angular and NestJS. If you want the tool to have opinions about your project layout, this is the one.
Lerna is the old guard and still the best answer if your actual problem is publishing versioned packages to npm. It runs on Nx internally now, which tells you something about how that story ended.
Bazel comes from Google and handles codebases in many languages at once. The caching is excellent and the learning curve is genuinely steep. Reach for it when JavaScript is not the only thing in the repository.
Rush comes from Microsoft, targets large TypeScript repositories, and takes dependency policy seriously in a way the others do not.

For a JavaScript or TypeScript repository under a few dozen packages, Turborepo. Beyond that, or with more than one language involved, look at Nx and Bazel in that order.

Things worth doing early

Enable remote caching before you need it. A cache that only lives on your laptop helps you. A shared one helps CI, which is where the build minutes actually go.
Get outputs right in every task. It is the most common configuration mistake and the least obvious one, because nothing fails, it just stays slow.
Write conventions down. Conventional commits help more in a monorepo than anywhere else, because a scope in the commit message tells you which package a change touched without opening it.
Automate dependency updates. Renovate or Dependabot grouped across the workspace, so one pull request covers the repository instead of twenty.
Assign ownership per package. Not to gatekeep, but so that reviews land with someone who knows the code.

Where I would draw the line

If you have one application, keep one repository. A monorepo is a solution to a coordination problem, and if you do not have the problem you are just adding a build tool.

The moment there are two deployables sharing real code, move. Waiting until it hurts means migrating a repository with history, CI, and everyone’s muscle memory attached to it, and that is a much worse afternoon than setting it up on day one.

‘Till next time!

Headless CMS

Sat, 28 Sep 2024 00:00:00 GMT

Everything except the page

I have been a core developer on django CMS since 2014, which means I have spent a long time inside a CMS that renders its own pages. So when someone tells me that going headless solves everything, my first question is what they plan to do about preview, URLs and redirects. Those parts rarely make it onto the landing page.

That is not an argument against headless. It is an argument for knowing what you trade away. The trade is usually worth it these days, and the tooling has got genuinely good. But it is a trade, and the comparison tables you find online are written by the people selling you the API.

So: what a headless CMS actually is, how the pieces fit together, and where four of the common ones differ once you get past the marketing.

What “headless” actually means

A headless CMS stores and edits content but does not render it. The “head” is the presentation layer, and it has been cut off. What is left is a content repository plus an API over it, usually REST or GraphQL, often both.

Compare that with WordPress or django CMS in their default setup. The same application that stores an article also decides what the page looks like, owns the URL it lives at, and produces the HTML. That coupling is where a traditional CMS gets its best features. It is also where it gets its worst constraints, because everything you want to do has to be expressed as a theme or a template or a plugin.

The practical definition is short. You get structured content over an API, and you build the front end yourself. Everything else follows from those two facts.

How the pieces fit

There are two moving parts.

The content repository holds your models and your data. You define content types (an article has a title, a body, a hero image, a handful of tags), and editors fill them in through an admin interface.

The API layer exposes that content. Your front end queries it and decides what to do with the response. That front end can be Next.js, Nuxt, an iOS app, or the screen in a shop window that nobody has updated since 2019.

The workflow that falls out of this:

Someone models the content types. This is the step teams rush through and then regret for two years.
Editors create and edit entries in the admin.
The front end fetches entries over REST or GraphQL, at build time, at request time, or some mix of the two.
The front end renders. Nobody in the CMS knows or cares what it looked like.

Step four is the entire difference. It is also the source of every complaint I have ever heard about a headless setup.

Traditional against headless

Feature	Headless CMS	Traditional CMS
Front-end flexibility	Decoupled from the CMS, use any framework you like	Coupled to the CMS, limited to themes and templates
Content delivery	Delivered over an API	Rendered by the CMS itself
Multi-channel support	One content source feeds web, apps and devices	Built primarily for websites
Editing experience	Form-based, preview has to be built	In-context editing and preview come for free
Customisation	Total control over presentation	Customisation means fighting the template layer

Read the first row again, though. “Use any framework you like” also means “you have to build it”. And total control over presentation means total responsibility for it. The table is honest, it just does not tell you who pays.

What you give up

Being specific about this, because the vendor comparison pages will not be:

Preview. In a coupled CMS, preview is free. The CMS renders the page, so it can render the draft too. Headless, preview is a feature you build. Storyblok and Contentful both ship preview APIs, and Storyblok’s visual editor is the main reason people choose it, but you still wire the thing up yourself.
In-context editing. Clicking a paragraph on the live page and typing into it is the single thing editors love most about django CMS. It is also the hardest behaviour to reproduce over an API, because the CMS has no idea where that paragraph ended up on screen.
Routing and redirects. Someone has to own the mapping from a slug to a page. If the CMS does not, your front end does, and that includes the redirect you owe the world every time marketing renames a page.
The “who broke it” question. With two systems there are two places for a deploy to go wrong, and two teams who are fairly sure it was not them.

None of these are dealbreakers. All of them are work that was not on the estimate.

The four you will be asked about

Strapi

Strapi is the open-source default. Node.js, self-hosted, and you can customise most of it including the admin UI. Content types are defined through a builder that writes schema files to disk, which means they are in version control where they belong.

REST and GraphQL out of the box.
Roles and permissions for content teams.
A plugin ecosystem, of varying quality.

Start one with:

npx create-strapi@latest my-strapi-project

Pick it when you want control and are happy to run the thing yourself.

Payload CMS

Payload is also Node.js and also self-hosted, but aimed squarely at developers rather than at content teams. Configuration is code, and the generated client is type-safe end to end, which is a real advantage if your front end is TypeScript.

Authentication and access control built in.
Content modelling defined in config files, not a UI.
Rich text editing with Markdown support.

npx create-payload-app@latest

Pick it when the developers outnumber the editors and the types matter more than the admin polish.

Storyblok

Storyblok is hosted, and its selling point is the visual editor. Editors click on the rendered page and edit the component they clicked, which is the closest anyone has got to bringing coupled-CMS editing to a decoupled stack. You pay for that with a component model you have to design around.

Pick it when the editors have opinions and will voice them.

Contentful

Contentful is the enterprise option. Hosted, API-first, REST and GraphQL, a CDN in front of everything, and asset management that holds up at scale. The interface is competent rather than lovable, and the pricing model rewards you for planning your content model before you start.

Pick it when the content is distributed globally and someone else’s uptime is the point.

Where django CMS sits in all this

I would be leaving something out if I did not mention it. There has been an active effort this year to expose django CMS content over a REST API so it can be run headless while keeping the editing model that makes it worth using. The request for comments went up in March and djangocms-rest has been moving since May. It is early, and I am watching it with an obvious personal bias.

That is the shape of the interesting problem, I think. Not “headless or not”, but whether you can keep the editing experience of a coupled CMS while serving the content over an API. Nobody has fully solved that yet.

What I would actually do

For a marketing site with a small team and a real editorial workload, I would still reach for a coupled CMS first, because preview and in-context editing are worth more than framework freedom to the people using it every day.

For anything feeding more than one front end, headless, without much hesitation. Start with Strapi if you want to host it and Storyblok if you do not, and spend an afternoon on the content model before you write a single component. That afternoon is the cheapest one in the project.

‘Till next time!

Modular Frontend Development

Fri, 30 Aug 2024 00:00:00 GMT

Components you own

Every component library I have adopted has eventually turned into a fight. You want a button that is 2px shorter, the library disagrees, and you end up with a stylesheet full of specificity hacks aimed at code you cannot read. shadcn/ui takes a different route: it hands you the source and gets out of the way.

That sounds like a small distinction. It changes almost everything about how the library behaves in a real project, and it is worth being precise about what it actually is, because a lot of write-ups (including one I nearly published) describe it wrongly.

What shadcn/ui actually is

shadcn/ui is not an npm dependency. There is no import { Button } from 'shadcn-ui'. You run a CLI, it copies TypeScript source files into your repository under components/ui/, and from that point on the code is yours. You import from your own project:

import { Button } from '@/components/ui/button';

It is also not headless, and this is the description I see most often and which is simply wrong. The headless layer is Radix UI, which provides the unstyled behaviour: focus trapping, keyboard navigation, ARIA wiring, portal management. shadcn/ui is a set of opinionated Tailwind styles sitting on top of those primitives, plus a distribution mechanism. The components arrive styled, themed with CSS variables, and looking like something you could ship.

So the pitch is not “unstyled components you can style”. It is “a good default implementation, in your repository, that you can edit”. Those are very different products.

Why modular in the first place

The argument for breaking an interface into small, single-purpose components is not new and I won’t belabour it. Reuse, obviously. Components you can debug in isolation. A consistent visual language, so the fourth developer to add a form doesn’t invent a fifth shade of grey.

The one benefit I’d underline is that decoupling behaviour from presentation lets you change one without touching the other. A dropdown’s keyboard handling and a dropdown’s padding have no business living in the same commit, and most of the time in a hand-rolled component they do.

Getting started

The CLI was rewritten this month and the package moved from shadcn-ui to plain shadcn. If you follow an older tutorial you will see npx shadcn-ui@latest, which still resolves but is the deprecated name. Use:

npx shadcn@latest init

Run it inside an existing project and it detects the framework. Next.js, Vite, Remix and Laravel all work out of the box now, which was the main point of the rewrite. Run it in an empty directory and it will offer to scaffold a new project first.

The prompts ask about your style preset, base colour and whether you want CSS variables for theming. Answers land in components.json at the root, which is the file the CLI reads on every subsequent command. It also records your import alias, so @/components/ui resolves correctly.

Then add a component:

npx shadcn@latest add button

That writes components/ui/button.tsx into your project and installs whatever it depends on, here @radix-ui/react-slot and class-variance-authority. It leans on the cn helper that init already put in lib/utils.ts. Open the file. It is about forty lines and you will understand all of them, which is not something I can say about most component libraries.

Extending the button

Here is where I disagree with most of the shadcn/ui examples floating around, including the ones I wrote first. The obvious move is to override the styles at the call site:

// don't do this
<Button className="bg-blue-500 hover:bg-blue-600 text-white py-2 px-4 rounded">
  Click me
</Button>

It works. It also scatters your design decisions across every file that renders a button, and in six months nobody can answer “what colour is a primary button” without grepping.

The generated button.tsx defines its styles with class-variance-authority. Add your variant there instead:

// components/ui/button.tsx
const buttonVariants = cva(
  'inline-flex items-center justify-center rounded-md text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-2 disabled:opacity-50',
  {
    variants: {
      variant: {
        default: 'bg-primary text-primary-foreground hover:bg-primary/90',
        destructive:
          'bg-destructive text-destructive-foreground hover:bg-destructive/90',
        outline: 'border border-input bg-background hover:bg-accent',
        brand: 'bg-brand-500 text-white hover:bg-brand-600',
      },
      size: {
        default: 'h-10 px-4 py-2',
        sm: 'h-9 px-3',
        lg: 'h-11 px-8',
      },
    },
    defaultVariants: {
      variant: 'default',
      size: 'default',
    },
  }
);

And then the call site says what it means:

import { Button } from '@/components/ui/button';

export default function MyButton() {
  return (
    <Button variant="brand" size="lg">
      Click me
    </Button>
  );
}

One vocabulary, one file, and a variant prop that is typed, so a typo is a build error rather than a button with no background.

The className prop still exists for the genuine one-offs, and it merges properly rather than fighting the base classes, because cn() runs tailwind-merge under the hood. Conflicting utilities get resolved instead of stacking.

Adding state

Nothing special is needed for behaviour. It’s your component:

'use client';

import { useState } from 'react';
import { Loader2 } from 'lucide-react';
import { Button } from '@/components/ui/button';

export default function LoadingButton() {
  const [loading, setLoading] = useState(false);

  return (
    <Button disabled={loading} onClick={() => setLoading(true)}>
      {loading && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
      {loading ? 'Loading...' : 'Submit'}
    </Button>
  );
}

lucide-react is the icon set shadcn/ui defaults to, and it comes along with the components that need it.

There is no Modal component in shadcn/ui, and npx shadcn@latest add modal will get you nothing. The component is called Dialog, and it is composed from several exports rather than being a single element with an isOpen prop.

npx shadcn@latest add dialog

'use client';

import {
  Dialog,
  DialogContent,
  DialogDescription,
  DialogHeader,
  DialogTitle,
  DialogTrigger,
} from '@/components/ui/dialog';
import { Button } from '@/components/ui/button';

export default function DeleteProjectDialog() {
  return (
    <Dialog>
      <DialogTrigger asChild>
        <Button variant="outline">Delete project</Button>
      </DialogTrigger>
      <DialogContent>
        <DialogHeader>
          <DialogTitle>Are you absolutely sure?</DialogTitle>
          <DialogDescription>
            This permanently deletes the project and everything in it.
          </DialogDescription>
        </DialogHeader>
        <Button variant="destructive">Delete</Button>
      </DialogContent>
    </Dialog>
  );
}

The open state lives inside Dialog unless you lift it out with open and onOpenChange. asChild is a Radix idea worth learning: it tells the trigger to render its child instead of its own element, so you get a <button> with the trigger’s behaviour rather than a button nested inside a button.

DialogTitle and DialogDescription aren’t decorative. Radix uses them to wire up aria-labelledby and aria-describedby. Leave the description out and the console tells you off, which is the correct behaviour and mildly startling the first time.

What this buys you, and what it costs

The good parts are real:

You own the code. The button that is 2px too tall is a file you can edit. No !important, no wrapper component whose only job is to undo a style.
Accessibility is handled by Radix. Focus management, escape-to-close, scroll locking, keyboard navigation. Getting a dialog right by hand takes longer than people expect, and most hand-rolled ones are subtly wrong.
It composes. Because they’re plain React components with no framework of their own, you can wrap, extend and combine them without the library having anticipated it.
It reads as Tailwind. If your project already uses Tailwind, nothing new arrives in the stylesheet.

The cost is the same fact from the other side. You own the code, so you own the maintenance. There is no npm update that brings you upstream fixes. When shadcn/ui improves a component, you go and look at the diff yourself, and if you have edited that file, you merge it by hand. For a design system you intend to keep for years, that is fine and arguably what you wanted. For a team that wants to install a dependency and forget about it, it is the wrong tool and will feel like one within a quarter.

Growing it into a design system

The natural progression is to stop treating components/ui/ as vendored code and start treating it as your library. Add your variants to the generated files. Add your own components alongside them using the same cva pattern so the shape stays consistent. Point your Tailwind theme at the CSS custom properties in globals.css and change the whole palette from one place.

At that point the question of whether shadcn/ui is a dependency stops making sense, which I think is the intent.

I have not yet used the custom registry feature that arrived with the CLI rewrite, where you serve your own components over a URL and add them the same way. That looks like the missing piece for sharing a design system across several repositories without publishing a package, and it is what I want to try next 🙂

‘Till next time!

CSS Container Queries

Wed, 31 Jul 2024 00:00:00 GMT

Asking the box, not the window

A card component does not care how wide your monitor is. It cares how much room it has been given. Media queries have never been able to express that, and every workaround I have written for it involved JavaScript measuring a div and putting a class on it. Container queries finally make it a stylesheet problem.

I ended my post on modern CSS techniques by complaining about exactly this gap, so it seems fair to spend a post on how it closes. What follows is the syntax, the units, the parts that behave unexpectedly, and where browser support actually stands.

The problem

Put a product card in a three-column grid on a desktop and it wants a horizontal layout: image left, text right. Put the same card in a narrow sidebar and it wants to stack. With media queries, the card cannot know which situation it is in. All it knows is the width of the window, which tells you nothing about the width of the column the card landed in.

So you end up with variant classes. .card--wide, .card--compact, .card--sidebar, each one a promise about where the component will be used, and each one broken the first time somebody uses it somewhere new. Container queries remove the need for the promise.

The syntax

Two steps. Declare an element as a query container, then write rules that respond to its size.

.container {
  container-type: inline-size;
}

@container (min-width: 300px) {
  .item {
    background-color: red;
  }
}

Declaring a container

container-type takes three values:

inline-size queries the inline dimension only, which in a horizontal writing mode means the width. This is what you want roughly nine times out of ten.
size queries both dimensions.
normal is the default and makes the element a container for style queries only, not size queries.

There is also container-name, and a container shorthand that sets both:

.card-wrapper {
  container: card / inline-size;
}

Querying it

Inside an @container block, the rules apply to descendants of the nearest ancestor container:

@container (min-width: 300px) {
  .item {
    background-color: red;
  }
}

@container (min-width: 500px) {
  .item {
    background-color: blue;
  }
}

If you gave the container a name, address it by name and the query stops depending on which container happens to be closest:

@container card (min-width: 500px) {
  .item {
    display: grid;
    grid-template-columns: 120px 1fr;
  }
}

I have come round to naming containers by default. Unnamed queries work fine until somebody wraps your component in another container, at which point the styles start resolving against a box you have never heard of.

Both dimensions

If you set container-type: size, you can query height as well:

.container {
  container-type: size;
  height: 400px;
}

@container (min-width: 300px) {
  .item {
    background-color: lightblue;
  }
}

@container (min-height: 200px) {
  .item {
    border: 2px solid lightcoral;
  }
}

Note the explicit height on the container. It is not decoration, and I explain why in the gotchas below.

Nesting

Containers nest, and each query resolves against the nearest matching ancestor:

.outer {
  container: layout / inline-size;
}

.inner {
  container: panel / inline-size;
}

@container panel (min-width: 300px) {
  .item {
    background-color: lightgreen;
  }
}

@container layout (min-width: 900px) {
  .item {
    background-color: lightcoral;
  }
}

With names, that reads unambiguously. Without them, both blocks would resolve against .inner for any .item inside it, and the second rule would never fire the way you expected.

Container queries against media queries

	Media queries	Container queries
Responds to	Viewport and device features	Size of a specific ancestor element
Scope	The whole document	Descendants of that container
Good for	Page-level layout shifts, print styles, `prefers-reduced-motion`	Component-level layout, design systems, anything reused in more than one slot
Component portability	The component must know where it lives	The component adapts wherever you put it

They are not competitors. I still use media queries for the page skeleton and for things that genuinely are properties of the device. Everything inside a component is a container query now.

The units

Container queries bring their own length units, all relative to the query container:

cqw is 1% of the container’s width
cqh is 1% of the container’s height
cqi is 1% of the container’s inline size
cqb is 1% of the container’s block size
cqmin is the smaller of cqi and cqb
cqmax is the larger of the two

Prefer cqi and cqb over cqw and cqh if you care about writing modes; they are the logical versions of the same idea.

Where these earn their place is in the declarations, not in the condition. Type that scales with the component is the obvious use:

.card {
  container: card / inline-size;
}

@container card (min-width: 400px) {
  .card__title {
    font-size: max(1.5rem, 1.2rem + 1.5cqi);
  }

  .card__body {
    padding: 4cqi;
  }
}

Using container units inside the @container condition itself is legal but confusing, because they resolve against the parent container rather than the one you are querying. I would avoid it. px and rem in the condition, cqi in the rules, is a boundary that has never surprised me.

Gotchas

Four things bit me while I was working this out.

An element cannot query itself. The rules inside an @container block apply to descendants of the container, never to the container element. If you want a card to restyle itself, you need a wrapper element to be the container, which is a slightly annoying extra div.

container-type: size collapses your element. Size containment in the block direction means the element’s height no longer depends on its content. Without an explicit height it computes to zero and everything inside vanishes. This looks catastrophic and takes ten minutes to work out the first time. Use inline-size unless you genuinely need to query height.

inline-size containment cuts the other way too. The container’s own width can no longer be determined by its contents, so a shrink-to-fit element such as a floated box or an inline-block will not size itself as it did before. In a normal block or grid layout you will never notice.

Query containers cost something. Every element with a container-type is an element the engine has to watch for size changes. Declaring one on a wrapper is nothing; declaring one on every list item in a table of two thousand rows is a decision you will feel.

Browser support

Size container queries have been in stable browsers for a while now:

Chrome and Edge since version 105, from August 2022
Safari since 16.0, from September 2022
Firefox since 110, in February 2023

That puts them in every evergreen browser, and has done for well over a year. In practice you can use them without a polyfill, provided your traffic isn’t heavy on Safari 15 or some old Chromium-based embedded browser you inherited.

Style queries, which test a custom property’s value rather than a size, are the newer half of the specification. Chrome and Firefox have shipped them for custom properties; Safari had not at the time of writing. I’d treat them as a progressive enhancement for now.

If you do need a fallback, @supports handles it cleanly:

@supports (container-type: inline-size) {
  .container {
    container-type: inline-size;
  }

  @container (min-width: 300px) {
    .item {
      background-color: lightblue;
    }
  }
}

@supports not (container-type: inline-size) {
  .item {
    background-color: lightgray;
  }
}

Write the unsupported case as your baseline and layer the container query on top, rather than the other way round. Old browsers get a single-column layout that works, new ones get the adaptive version.

Where I’ve got to

I have started removing size modifier classes from components as I touch them and replacing them with a named container plus two or three breakpoints. It is less code and the components stop lying about where they can be used.

The wrapper-element requirement still bothers me. Every component that wants to respond to its own width needs an extra div whose only job is to be a container, and in a deeply nested React tree that adds up. There’s talk of allowing an element to query itself in a future revision of the specification. I’ll believe it when I can ship it 🙂

‘Till next time!

Modern CSS Techniques

Sat, 22 Jun 2024 00:00:00 GMT

Grid, Flexbox and the things around them

CSS spent about a decade being the part of the stack people apologised for. Then Flexbox landed, then Grid, then everything else arrived so quickly that I keep meeting developers who are still writing 2016 CSS out of habit. This is a tour of what I actually use.

None of this is exotic. All of it works in every evergreen browser as I write this. The point of the post is less “here is a new feature” and more “here is the syntax that catches people out”, because most of these tools are one small misunderstanding away from feeling broken.

CSS Grid

Grid is a two-dimensional layout system. Rows and columns at the same time, which is the thing Flexbox deliberately doesn’t do.

.grid {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  grid-template-rows: auto;
  gap: 10px;
}

.grid__item {
  background: #f0f0f0;
  padding: 20px;
  border: 1px solid #ccc;
}

Three equal columns, each taking one fraction of the free space. repeat() saves you typing 1fr 1fr 1fr, and gap puts space between the tracks without the negative-margin nonsense we all used to write.

Once the tracks exist, you can place items across them by line number:

.grid {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  grid-template-rows: 100px 200px;
  gap: 10px;
}

.grid__item1 {
  grid-column: 1 / 3; /* span two columns */
  grid-row: 1 / 2;    /* occupy the first row */
}

.grid__item2 {
  grid-column: 3 / 4;
  grid-row: 1 / 3;    /* span both rows */
}

The numbers are grid lines, not tracks. A three-column grid has four lines. That off-by-one is responsible for a genuinely embarrassing share of the time I have spent debugging layouts.

Template areas

Named areas are the feature I’d show someone first, because they make a layout readable by people who don’t know CSS.

.container {
  display: grid;
  grid-template-areas:
    "header header"
    "sidebar content"
    "footer footer";
  grid-template-rows: 100px 1fr 50px;
  grid-template-columns: 200px 1fr;
}

.header {
  grid-area: header;
}

.sidebar {
  grid-area: sidebar;
}

.content {
  grid-area: content;
}

.footer {
  grid-area: footer;
}

You can see the page in the stylesheet. Rearranging it inside a media query is a matter of rewriting those three strings, which is about as close to pleasant as responsive layout gets.

Subgrid

Subgrid lets a nested grid use its parent’s tracks instead of defining its own. The classic use is a row of cards whose headings, bodies and footers should line up across the row even though the text lengths differ.

.cards {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  grid-template-rows: repeat(3, auto);
  gap: 20px;
}

.card {
  display: grid;
  grid-row: span 3;
  grid-template-rows: subgrid;
}

Two things to get right, because I got both of them wrong the first time. There is no display: subgrid; the child is a normal display: grid and subgrid is a value of grid-template-rows or grid-template-columns. And the child has to actually span the parent tracks it wants to adopt, which is what grid-row: span 3 is doing. Miss that and the subgrid silently behaves like an ordinary grid.

Firefox has had subgrid since 2019, Safari since 16, and Chrome and Edge since 117 in late 2023. So it is finally usable without a fallback.

Flexbox

Flexbox is one-dimensional. Items in a row, or items in a column, with control over how the leftover space is distributed.

.container {
  display: flex;
  justify-content: space-between;
  align-items: center;
}

.item {
  background: #f0f0f0;
  padding: 20px;
  border: 1px solid #ccc;
}

justify-content: space-between spreads the items along the main axis, align-items: center centres them on the cross axis. The old joke about vertical centring being impossible in CSS died right here.

For sizing, there are flex-grow, flex-shrink and flex-basis, and in practice you almost always want the shorthand:

.item {
  flex: 1;
  margin: 10px;
}

flex: 1 expands to flex-grow: 1; flex-shrink: 1; flex-basis: 0%. That last value is the one people miss. A basis of 0% means the item’s own content width is ignored when space is divided, so every item ends up the same size. If you want the content to influence the result, you want flex: auto, which uses flex-basis: auto instead. I have watched this exact distinction cause a two-hour argument about a navigation bar.

Scroll snap

Scroll snap turns a scrollable area into something that settles on defined points rather than wherever the momentum happened to stop.

.container {
  scroll-snap-type: x mandatory;
  overflow-x: scroll;
  display: flex;
}

.item {
  scroll-snap-align: start;
  flex: 0 0 100%;
}

That is a carousel, in seven declarations, with no JavaScript and with native keyboard and touch behaviour intact. Use proximity rather than mandatory if the content varies in size, because mandatory will fight the user on a long item.

Filters

Filters apply graphical effects without touching the source image.

.image {
  filter: blur(5px) contrast(1.2);
}

They compose left to right, and they are cheap enough to animate on the compositor in most cases. The one to be careful with is filter on a parent element, because it creates a containing block for descendants with position: fixed, which is a fun afternoon when you don’t know it.

Transitions and animations

Transitions

A transition interpolates a property when its value changes, typically on hover or focus.

.button {
  background-color: #3498db;
  transition: background-color 0.3s ease;
}

.button:hover {
  background-color: #2c3e50;
}

Name the properties you’re transitioning. transition: all is tempting and will one day animate something you did not intend, usually a layout property, usually on a page with a hundred of these elements.

Animations

Animations run from keyframes, on their own schedule, without needing a state change to trigger them.

@keyframes slideIn {
  from {
    transform: translateX(-100%);
  }
  to {
    transform: translateX(0);
  }
}

.element {
  animation: slideIn 0.5s forwards;
}

The element slides in from the left over half a second. forwards is an animation-fill-mode value and keeps the final keyframe applied after the animation ends; without it the element snaps back to its original position, which looks like a bug and is reported as one.

Wrap decorative animation in @media (prefers-reduced-motion: reduce) and give people a still version. It takes four lines.

Beyond the layout systems

Custom properties

I covered CSS variables properly in an earlier post, so here is the short version.

:root {
  --primary: #3498db;
  --padding: 20px;
}

.container {
  color: var(--primary);
  padding: var(--padding);
}

The thing that makes them more than a preprocessor variable is that they live in the cascade. You can redefine --primary inside a .dark class or a media query and everything downstream updates, at runtime, with no rebuild.

Shapes

shape-outside makes text wrap around a non-rectangular path.

.shape {
  float: left;
  shape-outside: circle(50%);
  width: 200px;
  height: 200px;
  background: #f0f0f0;
  clip-path: circle(50%);
}

shape-outside controls how surrounding content flows, clip-path controls what you can see of the element itself. You usually want both, and you always need the float, because shape-outside only applies to floated elements. That restriction is why this feature never quite caught on.

Combining them

Grid for the page, Flexbox for the components inside it. That is the division of labour I use almost every time.

.grid-container {
  display: grid;
  grid-template-columns: 1fr 2fr;
  gap: 20px;
}

.flex-item {
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: start;
}

Grid decides where the boxes go, Flexbox decides how the contents of each box arrange themselves. Trying to do the whole page in Flexbox is possible and produces stylesheets full of percentage widths that nobody wants to inherit.

What I still find annoying

Sizing an element based on its container rather than the viewport is still awkward, and every workaround I’ve used has involved either JavaScript or a resize observer pretending to be CSS. That is the gap container queries are supposed to close, and I want to spend some proper time with them before I write anything about them.

‘Till next time!

Testing Strategies in React

Sat, 25 May 2024 00:00:00 GMT

Four kinds of test, and when each earns its keep

Nobody sets out to write tests. You write them because the third time you break the same checkout flow, someone in the team stops finding it funny. The question is never really whether to test, it's which kind of test to reach for, because they cost wildly different amounts to write and to keep alive.

This post walks through the four layers I use in React projects: unit, integration, end-to-end and screenshot tests. Each section has a working example. I’ve kept the examples small on purpose, because the interesting part is not the syntax, it’s knowing which layer a given bug would have been caught by.

What tests actually buy you

Four things, roughly in order of how much I value them.

They catch bugs while the code is still in your head. Fixing something the same afternoon you wrote it is trivially cheap. Fixing it after a release involves a support ticket, a bisect, and someone’s evening.

They let you refactor. This is the one I’d defend hardest. A component with decent tests can be rewritten without ceremony. A component without them accretes workarounds because nobody dares touch the middle of it.

They document intent. A test named submits the trimmed value tells you something the implementation doesn’t.

And they force a small amount of design discipline, because code that’s painful to test is usually code with too many responsibilities.

Unit tests

A unit test exercises one component or one function on its own. In React that usually means rendering a single component and poking at it.

Two tools do the work. Jest is the test runner: assertions, mocks, spies, snapshots. It came out of Facebook and moved to the OpenJS Foundation in 2022, which is worth knowing mainly because the governance change made the release cadence a lot calmer. React Testing Library is the rendering layer, and its whole philosophy is that you should query the DOM the way a user would, by visible text and accessible role, rather than by class name or component internals.

Here’s a button:

// button.js
import React from 'react';

const Button = ({ onClick, children }) => {
  return <button onClick={onClick}>{children}</button>;
};

export default Button;

And its test:

// button.test.js
import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import Button from './button';

test('Button displays the correct text and handles click events', () => {
  const handleClick = jest.fn();
  const { getByText } = render(<Button onClick={handleClick}>Click me</Button>);

  const button = getByText('Click me');
  fireEvent.click(button);

  expect(button).toBeInTheDocument();
  expect(handleClick).toHaveBeenCalledTimes(1);
});

Note that toBeInTheDocument comes from @testing-library/jest-dom, not from Jest itself. It’s in the setup file of every project I’ve worked on, so it’s easy to forget it needs installing at all.

If your project is on Vite, use Vitest instead of Jest. The API is close enough that most test files port over untouched, and it reuses your existing Vite config rather than asking you to maintain a parallel Babel setup. I went through that migration in more about Vite.

Integration tests

Integration tests check that components work together. Same tools, wider scope, which is why the boundary between “unit” and “integration” in a React codebase is mostly a matter of how much you rendered.

// form.js
import React, { useState } from 'react';

const Form = ({ onSubmit }) => {
  const [inputValue, setInputValue] = useState('');

  const handleSubmit = event => {
    event.preventDefault();
    onSubmit(inputValue);
  };

  return (
    <form onSubmit={handleSubmit}>
      <label htmlFor="name">Your name</label>
      <input
        id="name"
        name="name"
        type="text"
        value={inputValue}
        onChange={e => setInputValue(e.target.value)}
      />
      <button type="submit">Submit</button>
    </form>
  );
};

export default Form;

// form.test.js
import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import Form from './form';

test('Form submits the correct value', () => {
  const handleSubmit = jest.fn();
  const { getByLabelText, getByText } = render(<Form onSubmit={handleSubmit} />);

  const input = getByLabelText('Your name');
  const button = getByText('Submit');

  fireEvent.change(input, { target: { value: 'test value' } });
  fireEvent.click(button);

  expect(handleSubmit).toHaveBeenCalledWith('test value');
});

The htmlFor/id pairing in the component isn’t decoration. getByLabelText resolves through the accessibility tree, so an input without an associated label simply cannot be found by that query. Which is a nice property: if the test can’t find your field, a screen reader probably can’t either.

End-to-end tests

End-to-end tests drive a real browser against a running application. They’re the slowest and the most brittle layer, and they’re also the only one that will ever catch a broken redirect or a misconfigured CSP header.

Playwright is what I use. It’s from Microsoft, runs against Chromium, Firefox and WebKit, and its auto-waiting is good enough that you can mostly stop writing explicit sleeps.

// form.spec.js
const { test, expect } = require('@playwright/test');

test('Form submits the correct value', async ({ page }) => {
  await page.goto('http://localhost:3000/form');

  await page.getByLabel('Your name').fill('test value');
  await page.getByRole('button', { name: 'Submit' }).click();

  await expect(page.getByRole('status')).toHaveText(
    'Submission successful: test value'
  );
});

One thing to watch: toHaveText is a locator assertion, not a page assertion. expect(page) only supports a handful of checks such as toHaveTitle and toHaveURL. Asserting text means pointing at an element first, as above. I’ve seen that mistake in more than one blog post, including an earlier draft of this one.

Keep the number of these tests small and deliberate. Two or three journeys that would embarrass you if they broke in production, not a mirror of your unit test suite.

Screenshot tests

Screenshot testing, or visual regression testing, renders the app, takes a picture, and compares it against a stored baseline. It catches the class of bug no assertion will: the stylesheet change that quietly shifted every card in the grid by four pixels.

Puppeteer is a Node library that drives headless Chrome, and it pairs with jest-image-snapshot for the comparison. Storybook is the other half of a sensible setup, because it lets you capture components in isolation and in specific states rather than fighting a full application into position.

// screenshot.test.js
const puppeteer = require('puppeteer');
const { toMatchImageSnapshot } = require('jest-image-snapshot');

expect.extend({ toMatchImageSnapshot });

describe('Visual Regression Testing', () => {
  let browser;
  let page;

  beforeAll(async () => {
    browser = await puppeteer.launch();
    page = await browser.newPage();
  });

  afterAll(async () => {
    await browser.close();
  });

  it('should match the previous screenshot', async () => {
    await page.goto('http://localhost:3000');
    const screenshot = await page.screenshot();

    expect(screenshot).toMatchImageSnapshot();
  });
});

The expect.extend line is not optional. toMatchImageSnapshot is not a Jest built-in, and without registering it you get a confusing “is not a function” error rather than anything helpful.

Be warned that this layer is where flakiness lives. Font rendering differs between macOS and Linux, animations land mid-frame, and a caret blinking in a text input will fail a test at random. The fix for all three is the same: disable animations, hide the caret, and run the whole thing in a container so the rendering environment is identical everywhere. I went through that setup properly in screenshot testing with React, including the Docker side of it.

What I actually do

Write tests that would fail for a reason you care about. A test asserting that a <div> has the class you just gave it is theatre.
Name them as sentences. renders the empty state when the list is empty beats test 3.
Keep them independent. A suite where test 4 relies on test 2 having run is a suite that will eventually fail in a random order and take a morning to diagnose.
Mock at the network boundary, not inside your components. Mocking a component’s internals bakes today’s implementation into the test.
Run everything in CI, and run the screenshot tests in the same container image locally that CI uses. Otherwise you spend your life regenerating baselines.

If I had to keep one layer, it would be the integration tests. They catch the most per line of test code, they survive refactoring, and they don’t need a browser farm.

Introduction to Docusaurus

Tue, 30 Apr 2024 00:00:00 GMT

Dinosaurs taking over documentation

Every project I work on eventually sprouts a docs folder, and every docs folder eventually raises the same question: where does this get published, and who has to maintain the publishing? Docusaurus is Meta's answer to that, and if your team already writes React, it is a comfortable one.

Docusaurus is a static site generator built on React, aimed squarely at project documentation rather than at blogs or marketing sites. You write Markdown, it gives you a sidebar, versioning, an internationalisation pipeline and a theme that already looks like documentation. The part I actually care about is MDX: a Markdown file can import a React component and render it inline, so a page can contain a working demo instead of a picture of one.

That sounds like a gimmick until you have maintained the docs for a component library. Screenshots go stale quietly and nobody notices for a year. A live component either renders or breaks the build.

Getting it running

You need Node.js and nothing else. Open a terminal and run:

npx create-docusaurus@latest my-docs classic

The “classic” template gives you a docs section, a blog, and the default theme. Somewhere in the prompts the installer asks whether you want JavaScript or TypeScript. Take TypeScript. It costs nothing at setup time and saves you an afternoon later, when you are squinting at docusaurus.config.ts trying to remember what shape the themeConfig object wants.

Then start the development server:

cd my-docs
npm start

That serves the site on http://localhost:3000/ with hot reload.

Things worth knowing on day one

Three details caught me out early enough that they are worth writing down.

The sidebar is either generated from your folder structure or declared by hand in sidebars.ts. Auto-generation is fine right up to the moment you care about the order of pages, and then it isn’t. Switch to an explicit sidebar early; retrofitting one across forty files is dull work.

Versioning is a CLI command rather than something clever with git:

npm run docusaurus docs:version 1.0

It copies the current docs/ directory into versioned_docs/version-1.0/ and freezes it. That’s it. Run it before your documentation has settled and you will find yourself fixing the same typo in three places forever.

Search isn’t included in the build. The classic template is wired for Algolia DocSearch, which is free for open source projects but requires an application and a crawl. For anything private, @easyops-cn/docusaurus-search-local builds a static index at build time and needs no external service at all.

Running it in Docker

I don’t enjoy installing a per-project Node toolchain on my laptop, and documentation sites are exactly the kind of thing you come back to eighteen months later on a machine with a different Node version. So I containerise it.

Create a Dockerfile:

FROM node:20.13.0 AS builder

ENV NODE_PATH=/app/node_modules

WORKDIR /app

COPY package*.json /app/
RUN npm install

COPY . /app
RUN npm run build

FROM nginx:latest

COPY --from=builder /app/build /usr/share/nginx/html
EXPOSE 80

Two stages: the first installs dependencies and produces the static build, the second serves that build through nginx and carries no Node runtime at all. If you pull packages from a private registry, add .npmrc to the COPY line, but only if the file genuinely exists in the repository. A COPY of a missing source fails the build with an error message that takes longer to understand than it should.

For local work, a docker-compose.yml:

services:
  web:
    build:
      context: .
      target: builder
    ports:
      - '8000:80'
    volumes:
      - '.:/app:rw'
      - 'node_modules:/app/node_modules'
    command: npm start -- --port 80 --host 0.0.0.0

volumes:
  node_modules:

The target: builder line is the important one. It stops the build at the first stage, so you get the development server with hot reload instead of the nginx image. The named node_modules volume stops your host directory from shadowing the modules that were installed inside the container, which is the single most common reason a mounted Node project refuses to start.

Then:

docker compose build
docker compose up

You can turn the same repository into a template and push it to Divio Cloud, which handles certificates and environments for you. I wrote about that setup in more detail when I first moved this site over to Divio Cloud.

The alternatives

Docusaurus is not the obvious answer for every project, and picking it for a Python codebase because you happen to like React is a decision your colleagues will resent.

MkDocs is the one I reach for most often. Markdown, one YAML config file, no build toolchain to speak of. Paired with the Material theme it produces something that looks better than most hand-rolled documentation sites and takes about ten minutes to stand up.

Sphinx is the Python world’s default, and it earns that position. reStructuredText is more fiddly than Markdown, but the cross-referencing, autodoc and extension ecosystem have no real equivalent elsewhere. The django CMS documentation runs on it, as does most of the Python standard library’s.

Jekyll is worth mentioning mainly because GitHub Pages builds it natively. If your requirement is “documentation, on GitHub, with no CI pipeline”, that native support is hard to argue with.

VuePress sits in roughly the same spot as Docusaurus for teams who write Vue rather than React. Same idea, same Markdown-centred structure, different component model.

What I would pick

If the project is Python, use Sphinx or MkDocs and don’t overthink it. If the project is a React library and the documentation needs live, interactive examples, Docusaurus pays for itself quickly. If it is neither of those, MkDocs with Material is almost always enough, and “almost always enough” is an underrated property in a documentation tool.

The thing I still don’t love about Docusaurus is how much of it is React all the way down. A broken MDX import fails the production build, which is correct behaviour and mildly infuriating at 17:45 on a Friday 🙃

‘Till next time!

Exploring useEffectEvent

Sun, 31 Mar 2024 00:00:00 GMT

A hook you can’t ship yet

Before anything else: useEffectEvent is experimental. It is not in React 18, it will not be in your next npm install react, and the documentation page for it carries a warning that the API may change or be dropped entirely. Everything below is worth reading anyway, because the problem it solves is one you already have.

The React docs cover it under separating events from effects, with the API reference living at experimental_useEffectEvent. To try it you need the canary or experimental release channel, and the import is deliberately ugly so you can’t forget what you signed up for:

import { experimental_useEffectEvent as useEffectEvent } from 'react';

The problem

Here’s the situation the hook exists for. A chat room connects to a server, and when the connection succeeds you show a notification styled with the current theme:

function ChatRoom({ roomId, theme }) {
  useEffect(() => {
    const connection = createConnection(serverUrl, roomId);

    connection.on('connected', () => {
      showNotification('Connected!', theme);
    });
    connection.connect();

    return () => connection.disconnect();
  }, [roomId, theme]);
}

The linter is right to demand theme in that dependency array; the Effect reads it. But the consequence is absurd. Toggle dark mode and the chat disconnects and reconnects, because as far as React is concerned the Effect’s inputs changed and it has to be torn down and set up again.

Remove theme from the array and the reconnection stops, but now you’ve lied to React and you’ll eventually show a notification in last week’s colours. Both options are wrong, and this is the trap I’ve watched people fall into over and over. The dependency array conflates two different questions: what should this Effect re-run for, and what values should it read when it does run.

What the hook does

useEffectEvent splits those questions apart. The non-reactive part moves out into an effect event, which always sees the latest props and state but never causes the Effect to re-synchronise:

const serverUrl = 'https://localhost:1234';

function ChatRoom({ roomId, theme }) {
  const onConnected = useEffectEvent(() => {
    showNotification('Connected!', theme);
  });

  useEffect(() => {
    const connection = createConnection(serverUrl, roomId);

    connection.on('connected', () => {
      onConnected();
    });
    connection.connect();

    return () => connection.disconnect();
  }, [roomId]);
}

theme is gone from the dependency array, and not by cheating. It’s read inside onConnected, which is not reactive, so it genuinely isn’t an input to the Effect. Only roomId is. Change the theme and the connection stays up; change the room and it reconnects. That’s the behaviour you wanted in the first place, finally expressible.

Notice what’s missing: onConnected itself does not go in the dependency array either. That’s not an oversight to be silenced with a lint comment, it’s the rule.

The rules

There are three, and they’re narrower than most hooks:

Only call effect events from inside Effects.
Never pass them to another component or hook. Not as a prop, not as an argument.
Never put them in a dependency array.

The second one catches people out, because passing a callback down is such a normal thing to do. The reasoning is that an effect event has no stable identity from the outside; it’s a hole in the reactivity graph, and letting it travel would make it impossible to reason about who triggers what. If you want to hand a callback to a child, that’s an ordinary prop and an ordinary event handler.

Which is the other half of the picture. useEffectEvent is not a replacement for onClick. Logic that runs because a user did something belongs in an event handler and was never an Effect’s business. The hook is for the narrower case where the Effect owns something external, a socket, a subscription, an observer, and part of the code reacting to it should not participate in the Effect’s lifecycle.

Living without it

Since you can’t ship it, what do you do today? The usual approximation is the latest-ref pattern:

import { useCallback, useLayoutEffect, useRef } from 'react';

function useEventCallback(fn) {
  const ref = useRef(fn);

  useLayoutEffect(() => {
    ref.current = fn;
  });

  return useCallback((...args) => ref.current(...args), []);
}

The returned function has a stable identity, so it’s safe to leave out of dependency arrays, and it always calls the most recent closure. I’ve used variations of this for years, and it’s closely related to the memoisation work I wrote about in optimizing React components.

It is not the same thing, though, and it’s worth understanding where it falls short. Nothing stops you calling it during render, which reads a value that may not be committed yet. Nothing stops you passing it to a child. It’s a convention held together by discipline, whereas the real hook can enforce its rules because React owns the call site. That gap is precisely why the React team never blessed the userland version, and why this has taken so long to land.

One practical annoyance while you wait: the stable eslint-plugin-react-hooks doesn’t know what an effect event is, so if you do experiment with the canary build, exhaustive-deps will cheerfully tell you to add onConnected to the array. Ignore it there. That’s the one place the rule is wrong.

Where this leaves me

I’m not putting a canary React into anything that matters, so useEventCallback stays in my utils folder for now. What has changed is how I read my own Effects. Whenever the dependency array has an entry that makes me wince, that entry is almost always a value the Effect reads rather than a value it should re-run for, and now I have a name for the distinction 🙂

Worth watching the React releases page for when this graduates. Until then it’s a useful way to think about Effects, and a hook you can’t have.

‘Till next time!

Integrating AI and Machine Learning

Thu, 29 Feb 2024 00:00:00 GMT

Running a model in the browser

Nearly every "AI feature" I've been asked about in the last year turned out to be a fetch call to somebody else's API. That's usually the right answer. But there is a smaller category of feature where the model has to run on the user's machine, and React has no opinion about how you do that.

I wrote a while back about how AI changes the way we write code. This is the other half of it: shipping a model as part of the frontend rather than using one to produce the frontend. The tooling is more mature than I expected and the trade-offs are sharper than the marketing suggests.

The two libraries

TensorFlow.js is the serious one. It runs Keras and TensorFlow models in the browser or in Node, either on WebGL or, with @tensorflow/tfjs-backend-wasm, on WebAssembly. You can train in it, though mostly you won’t; the realistic use is loading a model somebody else trained and running inference on it.

Brain.js is the small one. Neural networks in a few lines, trained in the browser on data you have to hand:

import { NeuralNetwork } from 'brain.js';

const net = new NeuralNetwork();

net.train([
  { input: { r: 0.03, g: 0.7, b: 0.5 }, output: { black: 1 } },
  { input: { r: 0.16, g: 0.09, b: 0.2 }, output: { white: 1 } },
]);

net.run({ r: 1, g: 0.4, b: 0 });

That’s the whole API surface, more or less. If your problem fits in a small feedforward network and you can describe the training data in JSON, Brain.js will get you there faster than anything else. If it doesn’t fit, it won’t stretch, and you’ll be back on TensorFlow.js within a week.

A sentiment component, properly

The usual worked example is sentiment analysis, so let’s do that one and be honest about the sharp edges. TensorFlow.js publishes a sentiment example built on a small CNN trained against the IMDB reviews set, which is a reasonable thing to point at.

npm install @tensorflow/tfjs

First, loading. Almost every tutorial I’ve read loads the model inside the prediction function, which means a fresh download and a fresh parse on every keystroke. Load it once and hold the promise:

import * as tf from '@tensorflow/tfjs';

let modelPromise;

const loadModel = () => {
  modelPromise ??= tf.loadLayersModel('/models/sentiment/model.json');

  return modelPromise;
};

Then the prediction. The thing that trips people up here is that model.predict() returns a tensor, not a number. Render it straight into JSX and React will throw at you. You have to pull the value out with data(), which is async, and dispose of the tensors afterwards because the WebGL backend does not garbage collect them for you:

const predictSentiment = async (text) => {
  const model = await loadModel();
  // Tokenise to word indices and pad to the length the model expects.
  const input = preprocess(text);
  const output = model.predict(input);
  const [score] = await output.data();

  tf.dispose([input, output]);

  return score;
};

preprocess is doing a lot of quiet work in that snippet. A text model doesn’t take text; it takes a fixed-length tensor of integers, mapped through the exact vocabulary the model was trained with, padded or truncated to the exact sequence length. Get the vocabulary wrong and nothing errors. You just get confident nonsense, which is a considerably worse failure mode than a stack trace.

Finally, the component:

import { useState } from 'react';

const SentimentAnalyzer = () => {
  const [text, setText] = useState('');
  const [score, setScore] = useState(null);
  const [pending, setPending] = useState(false);

  const handleAnalyze = async () => {
    setPending(true);
    setScore(await predictSentiment(text));
    setPending(false);
  };

  return (
    <div>
      <textarea value={text} onChange={(event) => setText(event.target.value)} />
      <button onClick={handleAnalyze} disabled={pending}>
        Analyze
      </button>
      {score !== null && (
        <p>
          {score > 0.5 ? 'Positive' : 'Negative'} ({score.toFixed(2)})
        </p>
      )}
    </div>
  );
};

Nothing about that is React-specific, which is rather the point. The model is a module-level singleton, the prediction is an async function, and the component is a form. There’s no hook to learn.

What it costs

Three things, in roughly the order they’ll annoy you.

The library is a heavy dependency, and the model weights are a separate download on top of it, fetched at runtime as model.json plus binary shards. Neither belongs in your initial bundle. Lazy-load the whole feature behind a dynamic import and don’t touch it until the user does something that needs it.

Inference on the main thread will drop frames on a mid-range phone. WebGL helps, but the first call after loading is always the slow one while shaders compile, so warm the model up with a dummy prediction while the user is still typing.

And tensors leak. tf.dispose() and tf.tidy() exist for a reason, and forgetting them in a component that re-renders is how you end up with a tab consuming a gigabyte.

When to do this at all

My honest position: run the model in the browser when the data must not leave the device, when the feature has to work offline, or when a round trip per keystroke would be absurd. Local search ranking, an offline classifier, live pose or gesture detection from a webcam. Those earn it.

Everything else is better served by a request to a server, where the model is bigger, the hardware is yours, you can swap it without shipping a release, and nobody downloads several megabytes of weights to find out their review was positive.

The nice thing about 2024 is that both options are genuinely easy now. The hard part was never the integration; it’s deciding which side of the wire the model belongs on, and that decision has almost nothing to do with React.

I want to try the same component with the WASM backend on an older phone and see how far off WebGL it lands. If the gap is small it changes the calculus quite a bit for anything that has to work on hardware people actually own 📱

‘Till next time!

Looking back at 2023

Wed, 31 Jan 2024 00:00:00 GMT

A good year, mostly

This year saw a plethora of changes, including a lot of travelling and ever-changing personal relationships. Overall, though, it was a good year, which is more than I could say for the last one.

There’s not much I’d like to share beyond that. I’m moving forward, learning new skills and developing myself further. What I can point at is the writing, which for once I kept up all year without a single missed month.

Rather more React than I intended. See you all in 2024 🎉

‘Till next time!

Demystifying Conventional Commits

Sat, 30 Dec 2023 00:00:00 GMT

Commit messages a machine can read

Most git histories I've inherited read like a diary written by five people who never spoke to each other. "fix stuff", "wip", "final fix 2". You can live with that. What you can't do is hand it to a tool and ask what changed between two releases.

Conventional Commits is a small specification that fixes exactly that one problem. It puts a machine-readable prefix on the first line of every commit and leaves the rest of the message alone. That’s it. There’s no framework to adopt and nothing to run; a team can start using it this afternoon and only later decide whether to automate anything on top.

I’d like to go through the format, the parts people usually get wrong, and where I think the convention stops paying for itself.

The shape of a message

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

The type is the only genuinely load-bearing part. feat means a new capability and maps to a MINOR bump under semantic versioning. fix means a bug fix and maps to a PATCH. Everything else is convention rather than specification, though most teams settle on roughly the same set: docs, refactor, perf, test, build, ci, chore and revert.

The scope in parentheses narrows it down, which is the difference between a changelog you can skim and one you can’t:

feat(auth): add support for hardware security keys
fix(parser): handle empty arrays in the query string
docs: describe the migration path from v1
refactor(api): collapse the two response builders into one

Keep the description in the imperative and short. Commitlint’s conventional config caps the header at 100 characters, which is generous enough that hitting it usually means you’re describing two commits.

Breaking changes, properly

This is the part I see done wrong most often, and it’s worth being precise about because a mistake here silently breaks your versioning.

A breaking change is marked with an exclamation mark after the type or scope:

feat(api)!: require Node 18 or newer

Or with a BREAKING CHANGE: footer, which gives you room to explain the migration:

feat(api): switch the client to native fetch

BREAKING CHANGE: Node 16 is no longer supported. Upgrade to
Node 18 or pin the previous major.

You can use both together, and for anything with a real migration path you should; the ! catches the reader’s eye and the footer is what ends up in the changelog.

What does not work is dressing a breaking change up as a chore, which is the pattern I keep running into:

chore: upgrade library to v2.0.0

Semantically that’s just wrong, but the practical consequence is worse. Run that through semantic-release and its default rules produce no release at all. chore isn’t in the list of types that trigger a version bump. Your dependency bump quietly ships to whoever installs your package next with no major, no minor, no changelog entry, and no warning. The commit that most needed to be visible is the one that disappears.

What the convention buys you

Once messages are structured, a few things become possible that weren’t before.

Versioning without a meeting. Tools read the commits since the last tag and work out the bump themselves. Nobody argues about whether the release is a 1.4.0 or a 2.0.0, because the argument already happened when the commit was written.

Changelogs that write themselves. semantic-release groups commits by type, drops the noise, and produces something you can put in front of users.

A history you can query. git log --oneline --grep '^fix' v1.2.0..HEAD is a perfectly good bug report for a release, and it costs nothing to run.

The subtler benefit is the one that shows up in review. Writing feat or fix at the start of a message forces you to decide what the change actually is, and if you can’t decide, the commit is usually doing two things and wants splitting.

Tooling

You need none of this to start. Once the habit is there, three tools do most of the work.

Commitizen replaces git commit with a prompt that walks you through type, scope and description. Good for onboarding, mildly irritating once you’ve internalised the format.

commitlint checks messages against the convention and rejects the ones that don’t fit. Paired with Husky it runs on the commit-msg hook, not pre-commit, since the message doesn’t exist yet when pre-commit fires:

npm install --save-dev @commitlint/cli @commitlint/config-conventional husky
npx husky install
npx husky add .husky/commit-msg 'npx --no -- commitlint --edit "$1"'

With a one-line config:

// commitlint.config.js
module.exports = {
  extends: ['@commitlint/config-conventional'],
};

And semantic-release reads the log on CI, decides the next version, writes the changelog, tags and publishes. Its default mapping is worth memorising, because it’s the thing that turns your commit messages into real consequences:

Commit	Release
`fix:`	patch
`perf:`	patch
`feat:`	minor
`!` or `BREAKING CHANGE:`	major
anything else	none

Where I’d skip it

The convention has a cost, and it isn’t the learning curve. People pick the format up in a day. The cost is that once a tool is publishing releases off your commit messages, a sloppy message becomes a wrong version number, and fixing a bad tag on a published package is a considerably worse afternoon than writing a careful commit would have been.

So: any library other people install, yes, without hesitation. A repository with more than three people committing to it, yes. My own weekend projects, no. Nothing downstream reads those commits and I’m not going to lint myself into writing chore(deps): on a Sunday.

Starting on an existing repository

You don’t need to rewrite history, and please don’t. Turn on commitlint, let the old messages stay as they are, and set the initial version by hand at the first release so the tooling has a baseline. Everything from that tag forward will be clean, which is all anyone actually needs.

‘Till next time!

Keeping Mac Settings in Sync with Mackup

Thu, 30 Nov 2023 00:00:00 GMT

The settings you can’t script

My dotfiles get me most of the way to a working machine. Packages installed, shell configured, editors set up, all from a couple of commands. What they don't cover is the pile of preferences that live inside applications, buried in plist files and Application Support folders nobody wants to copy by hand.

That’s the gap Mackup fills. It knows where a few hundred applications keep their configuration, moves those files into a folder you control, and symlinks them back to where the app expects to find them. Put that folder somewhere that syncs and a second machine picks up the same settings.

I’ve mentioned it in passing in my post on dotfiles, where it handles the parts I never managed to turn into shell scripts. Here’s the actual setup.

Installing it

Nothing surprising. On both the old and the new machine:

brew install mackup

That’s the whole install. The interesting part is the configuration file, and Mackup does not create that for you. You write ~/.mackup.cfg yourself.

Telling it where to put things

Mine goes to iCloud, so a dead machine doesn’t take the backup down with it:

[storage]
engine = icloud
directory = Mackup

The [storage] header matters. Drop it and the two keys below are silently ignored, which is a fun half hour to spend. engine also takes dropbox, google_drive and file_system; the last one needs an extra path key and is what you’d reach for if you want the backup to land inside a folder your dotfiles already sync. directory is only the folder name inside that storage, and Mackup is the default anyway, so you can leave it out.

Two more sections are worth knowing about:

[applications_to_ignore]
ssh
gnupg

[applications_to_sync]
git
vim

They don’t stack the way you’d expect. applications_to_sync is an allowlist, and the moment you use it Mackup stops looking at anything else. I run the ignore list instead and let the default set do its thing, mostly because I’d rather forget to exclude something than forget to include it. The names come from mackup list, not from the app’s display name.

Backup, transfer, restore

On the old machine, run mackup backup. Your files move into the storage folder and symlinks take their place.
Wait for iCloud to genuinely finish uploading. Check the folder in Finder; Mackup has no idea what your sync client is up to.
On the new machine, install Mackup, put the same .mackup.cfg in your home directory, then run mackup restore.

mackup uninstall reverses everything: the real files get copied back out of storage and the symlinks are removed. Run it before you wipe a machine, otherwise you leave applications pointing at links that resolve to nothing.

Applications Mackup doesn’t know

The supported list is long but it will not have your niche thing. Create a ~/.mackup directory and add one .cfg file per application:

[application]
name = My personal synced files

[configuration_files]
.gitignore
.config/someapp/config.toml

Paths are relative to your home directory, and directories work as well as files. Run mackup list afterwards and your entry should show up alongside the built-in ones. If you name it after an application Mackup already supports, your file replaces theirs rather than adding to it.

The part that bites

Symlinks. The entire model rests on them, and macOS 14 stopped playing along. Symlinked .plist preference files get reset or quietly rewritten by the system, so an application you were sure was synced drifts back to its defaults, and you only notice weeks later when a keyboard shortcut is gone. There are open issues about this (#1924 and #1950) and nothing Mackup can do about it from its side.

So my rule now is dull but it holds: anything that keeps its settings in a plist goes into [applications_to_ignore], and Mackup gets the dotfile-shaped configs it was always good at. That’s a narrower job than the README suggests, and it’s still the job I needed doing.

Before you trust it

Run mackup backup once on a machine you can afford to break, then go and look at what actually moved. This is a tool that rearranges your home directory, and it’s worth knowing exactly which files it took before you find out the hard way at three in the morning on a fresh install 😅

‘Till next time!

Linting and Testing in Vite

Sat, 28 Oct 2023 00:00:00 GMT

The parts react-scripts took with it

In my previous article I swapped react-scripts out for Vite and the app came back up, faster than before. What didn't come back up was everything react-scripts was quietly doing on the side.

Linting, formatting and tests were all bundled into that one dependency, with configuration you never saw and mostly never needed to. Uninstall it and they leave together. That’s the honest cost of the migration, and it’s the reason people abandon it halfway through.

The good news is it’s an afternoon of work, once, and afterwards the configuration is yours.

ESLint

Start with the Vite plugin, which runs the linter as part of the dev server so mistakes appear in the browser overlay rather than only in CI:

npm install vite-plugin-eslint --save-dev

import eslint from 'vite-plugin-eslint';

export default {
  // ...
  plugins: [react(), eslint()],
};

Then ESLint itself and its config generator:

npm install eslint --save-dev
npm init @eslint/config

The generator asks a handful of questions and writes an .eslintrc.js that does not quite work. Here’s mine after fixing it:

module.exports = {
  env: {
    browser: true,
    es2021: true,
  },
  extends: [
    'eslint:recommended',
    'standard-with-typescript',
    'plugin:react/recommended',
    'plugin:react/jsx-runtime',
  ],
  overrides: [
    {
      env: {
        node: true,
      },
      files: ['.eslintrc.{js,cjs}'],
      parserOptions: {
        sourceType: 'script',
      },
    },
  ],
  parserOptions: {
    ecmaVersion: 'latest',
    sourceType: 'module',
    project: './tsconfig.json',
  },
  plugins: ['react'],
  rules: {
    '@typescript-eslint/semi': 'off',
  },
  settings: {
    react: {
      version: 'detect',
    },
  },
};

Two of those lines are the ones worth pointing at, because without them you get errors that look like your code is broken when it isn’t.

parserOptions.project is required by standard-with-typescript. It turns on the rules that need type information, and the parser can’t do that without knowing which tsconfig.json you mean. Leave it out and every lint run fails with a wall of text about the rule requiring parser services, on files that are perfectly fine.

plugin:react/jsx-runtime switches off react/react-in-jsx-scope. Since React 17 the new JSX transform means you don’t import React just to write JSX, and without this line the linter demands an import on every component file. I’ve seen people add hundreds of pointless imports rather than one line of config.

Prettier

The npm init @eslint/config pattern doesn’t carry over to Prettier, there’s no init command. Install it and write the config file yourself:

npm install prettier eslint-config-prettier --save-dev
touch .prettierrc.yaml

printWidth: 80
tabWidth: 2
useTabs: false
semi: true
singleQuote: true
trailingComma: none
bracketSpacing: true
bracketSameLine: false
arrowParens: avoid

If you’re copying an older config from somewhere, note that jsxBracketSameLine became bracketSameLine back in Prettier 2.4 and no longer exists in Prettier 3. Prettier just ignores the key it doesn’t recognise, which is a pleasant way to spend twenty minutes wondering why your JSX is being formatted the way you asked it not to be.

eslint-config-prettier is the other half. It turns off the ESLint rules that disagree with Prettier about whitespace, so the two tools stop rewriting each other’s output. Put it last in extends:

extends: [
  'eslint:recommended',
  'standard-with-typescript',
  'plugin:react/recommended',
  'plugin:react/jsx-runtime',
  'prettier',
],

Order matters here. Last wins, and prettier needs to win.

Tests

Now the part I have opinions about.

The obvious choice for a Vite project is Vitest. It reads your existing vite.config.ts, so your aliases, your plugins and your SVG imports already work, and the API is close enough to Jest that most test files run unchanged. If you’re starting fresh, use it. I would.

Jest is still what I reach for when the project has an existing suite, custom matchers and a CI pipeline built around it, because “close enough to Jest” is doing real work in that sentence and I’d rather not find the gaps on a deadline. So, Jest:

npm install jest ts-jest @types/jest jest-environment-jsdom --save-dev

// jest.config.js
/** @type {import('ts-jest').JestConfigWithTsJest} */

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'jsdom',
  setupFilesAfterEnv: ['<rootDir>/src/setupTests.ts'],
  moduleDirectories: ['node_modules', 'src'],
  moduleNameMapper: {
    '\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$':
      '<rootDir>/__mocks__/file-mock.js',
    '\\.(css|less|scss)$': '<rootDir>/__mocks__/style-mock.js',
  },
};

npx ts-jest config:init generates a bare version of this if you’d rather start from a stub.

The moduleNameMapper entries exist because Jest has no idea what to do with a .png or a .css import. Vite handles those; Node does not. Both mappings need to point at a module that exports something harmless:

// __mocks__/file-mock.js
module.exports = 'test-file-stub';

// __mocks__/style-mock.js
module.exports = {};

I’ve seen these mapped at setupTests.ts instead, and it does technically work, in the sense that requiring a file that exports nothing gives you nothing. It also means every stylesheet import in your app pulls in your entire test setup, and when that setup grows a global mock or a timer, the reason your test suite got strange will not be anywhere near where you’re looking.

setupTests.ts has an actual job of its own:

// src/setupTests.ts
import '@testing-library/jest-dom';

Note that moduleDirectories includes src, which is how Jest resolves the same import Button from 'components/button' paths that vite-tsconfig-paths handles for the build. Two tools, two configurations, same aliases. Keeping them in step is genuinely annoying and is another point in Vitest’s favour.

Leftovers

A few things react-scripts left in the project that no longer belong:

web-vitals and reportWebVitals. CRA generated src/reportWebVitals.ts and called it from the entry file. Unless you were actually sending those numbers somewhere, uninstall the package, delete the file and remove the import.
Browser targets. Your package.json still has a browserslist field, but Vite’s build target lives in vite.config.ts and doesn’t look at it. Install browserslist-to-esbuild and it converts the list you already have into esbuild targets, so you keep one source of truth instead of two lists that drift.
Everything else. npx npm-check-updates will show you what’s out of date, ncu -u writes the new ranges. Worth a look every few months rather than once a year in a panic.

The finished files

// vite.config.ts
import { defineConfig } from 'vite';

import react from '@vitejs/plugin-react';
import viteTsconfigPaths from 'vite-tsconfig-paths';
import svgrPlugin from 'vite-plugin-svgr';
import eslint from 'vite-plugin-eslint';
import browserslistToEsbuild from 'browserslist-to-esbuild';

// see more at https://vitejs.dev/config/
export default defineConfig({
  plugins: [react(), viteTsconfigPaths(), svgrPlugin(), eslint()],
  build: {
    target: browserslistToEsbuild(),
  },
});

// package.json
{
  "name": "my-app",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "scripts": {
    "start": "vite",
    "dev": "vite",
    "build": "tsc && vite build",
    "serve": "vite preview",
    "lint": "eslint src/**/*.{ts,tsx}",
    "format": "prettier --write src",
    "test": "jest"
  },
  "browserslist": {
    "production": [">0.2%", "not dead", "not op_mini all"],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "@testing-library/jest-dom": "^6.1.4",
    "@testing-library/react": "^14.0.0",
    "@testing-library/user-event": "^14.5.1",
    "@types/jest": "^29.5.6",
    "@types/node": "^20.8.9",
    "@types/react": "^18.2.33",
    "@types/react-dom": "^18.2.14",
    "@typescript-eslint/eslint-plugin": "^6.9.0",
    "@vitejs/plugin-react": "^4.1.0",
    "browserslist-to-esbuild": "^1.2.0",
    "eslint": "^8.52.0",
    "eslint-config-prettier": "^9.0.0",
    "eslint-config-standard-with-typescript": "^39.1.1",
    "eslint-plugin-import": "^2.29.0",
    "eslint-plugin-n": "^16.2.0",
    "eslint-plugin-promise": "^6.1.1",
    "eslint-plugin-react": "^7.33.2",
    "jest": "^29.7.0",
    "jest-environment-jsdom": "^29.7.0",
    "prettier": "^3.0.3",
    "ts-jest": "^29.1.1",
    "typescript": "^5.2.2",
    "vite": "^4.5.0",
    "vite-plugin-eslint": "^1.8.1",
    "vite-plugin-svgr": "^4.1.0",
    "vite-tsconfig-paths": "^4.2.1"
  }
}

Only react and react-dom are real dependencies. Everything else builds, lints or tests the thing and belongs in devDependencies, which the CRA-generated file never got right either.

Where this leaves me

The whole setup is now four config files I can read in five minutes, instead of one dependency I couldn’t change. That was the point of the exercise.

What I’m less pleased about is having Jest and Vite each resolving modules their own way. It works today, and it’s exactly the sort of duplication that quietly rots. The next project I start goes on Vitest.

‘Till next time!

Switch to Vite

Sun, 24 Sep 2023 00:00:00 GMT

The starter that stopped moving

"Create React App" got a lot of people, me included, past the part of React that used to be genuinely miserable: deciding what a build looks like. It answered the question so you didn't have to. It has also barely moved in a year, and that's now the problem.

Every few months I run npm audit on an older project and get a list of transitive vulnerabilities I can’t fix, because they live under react-scripts and react-scripts isn’t being updated. Waiting for someone else to bump a dependency isn’t a plan.

Is it dead?

Not dead. Dying, quietly, in the way open source projects usually do. The TypeScript 5 upgrade sat open for over a year. The React team’s own documentation pull request moved the recommendation towards frameworks, and the mentions of CRA came out of the official docs entirely:

If you’re still not convinced, or your app has unusual constraints not served well by these frameworks and you’d like to roll your own custom setup, we can’t stop you—go for it! Grab react and react-dom from npm, set up your custom build process with a bundler like Vite or Parcel, and add other tools as you need them for routing, static generation or server-side rendering, and more.

React Website

For small projects with low complexity I still reach for Parcel, because zero config really does mean zero config there. This walkthrough uses Vite instead, because once a project has path aliases, SVG imports, a proxy and three environments, I want the config file to exist.

Start with a Create React App

So there’s something to migrate, generate a standard CRA TypeScript project:

npx create-react-app my-app --template typescript
cd my-app
npm run start

Now run npx npm-check-updates on that brand new project and look at how much of it is already out of date. That list is the whole argument for this article.

Swap the packages

Out with react-scripts, in with Vite and the two plugins I always end up wanting:

npm uninstall react-scripts
npm install vite @vitejs/plugin-react --save-dev
npm install vite-tsconfig-paths vite-plugin-svgr --save-dev

All of these belong in devDependencies. Nothing here ships to the browser, it only builds the things that do.

There are plenty more plugins available, but these two replace behaviour CRA gave you for free:

vite-tsconfig-paths makes Vite honour the baseUrl and paths you already have in tsconfig.json, so import Button from 'components/button' keeps resolving instead of turning into ../../components/button.
vite-plugin-svgr lets you import an SVG as a React component rather than a URL.

One thing that will bite you on the SVG plugin. Version 4 landed a few days ago and changed the import syntax. The CRA-style named export is gone:

// CRA, and vite-plugin-svgr v3
import { ReactComponent as Logo } from './logo.svg';

// vite-plugin-svgr v4
import Logo from './logo.svg?react';

If you’re migrating an app with a hundred icon imports, that’s a find-and-replace across the codebase, and the error message you get otherwise (“does not provide an export named ‘ReactComponent’”) does not point at the plugin version at all. Pin the major version deliberately rather than discovering this later.

Write the config

Create a config file with touch vite.config.ts:

import { defineConfig } from 'vite';

import react from '@vitejs/plugin-react';
import viteTsconfigPaths from 'vite-tsconfig-paths';
import svgrPlugin from 'vite-plugin-svgr';

// see more at https://vitejs.dev/config/
export default defineConfig({
  plugins: [react(), viteTsconfigPaths(), svgrPlugin()],
  server: {
    port: 3000,
    open: true,
  },
});

The server block is optional and I add it every time. Vite serves on 5173 by default, CRA used 3000, and if you have a backend with CORS rules or an OAuth callback registered against port 3000 you will notice immediately.

Move index.html

This is the part that feels wrong the first time. CRA keeps index.html in public/ and treats it as a template. Vite treats it as the entry point of the application, so it goes in the project root:

mv public/index.html index.html

Vite still serves everything else in public/ at the root of the site, so %PUBLIC_URL% has no job any more. Strip it out:

<!-- before -->
<link rel="icon" href="%PUBLIC_URL%/favicon.ico" />

<!-- after -->
<link rel="icon" href="/favicon.ico" />

Then point the page at your entry module. CRA injected this at build time, Vite expects it to be in the HTML:

<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!-- add this line below the root declaration -->
<script type="module" src="/src/index.tsx"></script>

If the page loads blank with no errors in the console, this line is what’s missing. I’ve made that mistake more than once.

Sort out TypeScript

Open tsconfig.json and update the compiler target, because there’s no point shipping ES5 through a tool built around native modules:

"target": "ESNext",
"lib": ["dom", "dom.iterable", "esnext"],

CRA leaves behind src/react-app-env.d.ts, which references react-scripts types you just uninstalled. Delete it and write the Vite equivalent in its place:

rm src/react-app-env.d.ts
touch src/vite-env.d.ts

/// <reference types="vite/client" />
/// <reference types="vite-plugin-svgr/client" />

The first line gives you types for import.meta.env, the second for the ?react SVG imports. Put the file in src/, not the project root, or it won’t be picked up by the default include.

Environment variables

The quiet one. CRA exposed anything prefixed with REACT_APP_ on process.env. Vite exposes anything prefixed with VITE_ on import.meta.env, and process doesn’t exist in the browser at all:

// CRA
const url = process.env.REACT_APP_API_URL;

// Vite
const url = import.meta.env.VITE_API_URL;

Both the prefix and the object change, so every variable needs renaming in .env and in the code. There’s no warning for this. You get undefined at runtime, usually in the one code path that isn’t covered by a test.

While you’re in there, import.meta.env.MODE, .DEV and .PROD replace the NODE_ENV checks.

Scripts

Finally, the entries in package.json:

"scripts": {
  "start": "vite",
  "dev": "vite",
  "build": "tsc && vite build",
  "serve": "vite preview"
},

I keep start as an alias for dev so that muscle memory and any CI job referring to npm start both keep working. tsc && vite build runs the type check before the bundle, because Vite strips types with esbuild and will happily build code that doesn’t type-check.

Then npm install && npm run start and you should be looking at the same app, starting considerably faster.

Note there’s no test script here. react-scripts test was Jest with a pile of configuration behind it, and that configuration left with the package. That’s the subject of the next article.

Starting fresh instead

If you’re not migrating anything and just want a new project, skip all of the above:

npm create vite@latest my-app -- --template react-ts
cd my-app
npm install
npm run dev

Note it’s npm run dev and not npm run start, which catches me out roughly every second time. Pick react-ts rather than react unless you have a reason to avoid TypeScript. The full list of templates lives in the create-vite repository, and the React one is a genuinely small amount of code, which after years of squinting at CRA’s ejected webpack config is a relief.

What’s still missing

The app runs. Linting, formatting and tests do not, because all three came bundled with react-scripts and left with it. That’s what I’ll set up in the next article, along with the leftovers in package.json that nobody needs any more.

‘Till next time!

Exploring React's useContext Hook

Sun, 20 Aug 2023 00:00:00 GMT

Four levels down and still passing props

There's a moment in every React app where you add a prop to a component that doesn't use it, purely so it can hand it to a child that hands it to a child. That's the point at which I go looking for context.

useContext lets a component read a value from an ancestor without every component in between having to carry it. Themes, the current user, a locale, an open/closed flag for something global. It’s a transport mechanism, not a state manager, and keeping that distinction straight saves a lot of grief later.

The version with prop drilling

Take a navigation bar that tracks which item is active. Three files, and two of them are only involved because of the wiring:

// app.js
import React, { useState } from 'react';
import Navbar from 'components/nav';

const App = () => {
  const [activeItem, setActiveItem] = useState('Home');

  return (
    <div className="app">
      <Navbar activeItem={activeItem} setActiveItem={setActiveItem} />
      {/* other stuff */}
    </div>
  );
};

export default App;

// nav.js
import React from 'react';
import NavItem from 'components/nav-item';

const Navbar = ({ activeItem, setActiveItem }) => {
  return (
    <nav>
      <NavItem
        title="Home"
        activeItem={activeItem}
        setActiveItem={setActiveItem}
      />
      <NavItem
        title="About"
        activeItem={activeItem}
        setActiveItem={setActiveItem}
      />
    </nav>
  );
};

export default Navbar;

// nav-item.js
import React from 'react';

const NavItem = ({ title, activeItem, setActiveItem }) => {
  const handleClick = () => setActiveItem(title);

  return (
    <div className={title === activeItem ? 'active' : ''} onClick={handleClick}>
      {title}
    </div>
  );
};

export default NavItem;

Navbar does nothing with activeItem or setActiveItem. It is a courier. With two levels that’s tolerable, and by the time there’s a NavGroup and a NavDropdown in the middle it isn’t. Every new piece of navigation state means touching every file in the chain.

The same thing with context

Put the state in a provider and let the components that care read it directly:

// nav-context.js
import React, { createContext, useContext, useState } from 'react';

const NavContext = createContext(null);

const NavContextProvider = ({ children }) => {
  const [activeItem, setActiveItem] = useState('Home');

  return (
    <NavContext.Provider value={{ activeItem, setActiveItem }}>
      {children}
    </NavContext.Provider>
  );
};

const useNavContext = () => {
  const context = useContext(NavContext);

  if (context === null) {
    throw new Error('useNavContext must be used inside a NavContextProvider');
  }

  return context;
};

export { NavContextProvider, useNavContext };

// app.js
import React from 'react';
import Navbar from 'components/nav';
import { NavContextProvider } from 'components/nav-context';

const App = () => {
  return (
    <NavContextProvider>
      <div className="app">
        <Navbar />
        {/* other stuff */}
      </div>
    </NavContextProvider>
  );
};

export default App;

// nav.js
import React from 'react';
import NavItem from 'components/nav-item';

const Navbar = () => {
  return (
    <nav>
      <NavItem title="Home" />
      <NavItem title="About" />
    </nav>
  );
};

export default Navbar;

// nav-item.js
import React from 'react';
import { useNavContext } from 'components/nav-context';

const NavItem = ({ title }) => {
  const { activeItem, setActiveItem } = useNavContext();
  const handleClick = () => setActiveItem(title);

  return (
    <div className={title === activeItem ? 'active' : ''} onClick={handleClick}>
      {title}
    </div>
  );
};

export default NavItem;

Navbar is back to describing navigation instead of relaying props. Adding a third level of nesting costs nothing.

Always export a hook, never the context

The thing I’d insist on from that file is useNavContext. Exporting the raw context and calling useContext(NavContext) at every call site works, but it gives you no place to put the guard.

Without a guard, a component rendered outside the provider gets the default value back and fails somewhere else entirely, usually as Cannot destructure property 'activeItem' of undefined. With the guard you get a sentence telling you exactly what went wrong. That’s the difference between a ten-second fix and twenty minutes with the component tree open.

It’s also the seam where you’d later add a selector, a console.count while debugging, or a TypeScript type, without touching a single consumer.

The catch nobody mentions in the tutorial

Look at the provider again:

<NavContext.Provider value={{ activeItem, setActiveItem }}>

That object literal is rebuilt on every single render of the provider. Context compares by reference, so every consumer re-renders every time, even the ones reading a field that didn’t change. React.memo on a consumer does not help, because the update comes through the context rather than through props.

For a navbar with four items, this genuinely does not matter and I would leave it alone. For a provider sitting near the root of a large tree, it does. The fix is to keep the value’s identity stable:

import React, { createContext, useMemo, useState } from 'react';

const NavContextProvider = ({ children }) => {
  const [activeItem, setActiveItem] = useState('Home');

  const value = useMemo(
    () => ({ activeItem, setActiveItem }),
    [activeItem, setActiveItem]
  );

  return <NavContext.Provider value={value}>{children}</NavContext.Provider>;
};

setActiveItem is stable across renders because React guarantees the identity of a useState setter, so in practice only activeItem moves the needle. I still list it in the dependency array to keep the lint rule quiet. If you put callbacks of your own in the value, wrap them in useCallback for the same reason, which I went into in Optimizing React Components.

Splitting state from setters

There’s a second move worth knowing, and it beats memoisation when the state changes often. Use two contexts:

const NavStateContext = createContext(null);
const NavDispatchContext = createContext(null);

const NavContextProvider = ({ children }) => {
  const [activeItem, setActiveItem] = useState('Home');

  return (
    <NavDispatchContext.Provider value={setActiveItem}>
      <NavStateContext.Provider value={activeItem}>
        {children}
      </NavStateContext.Provider>
    </NavDispatchContext.Provider>
  );
};

A component that only dispatches, like a button that resets the navigation, subscribes to NavDispatchContext and never re-renders when activeItem changes. The setter’s identity is stable, so that context’s value never changes at all. Two providers is slightly more typing for a real reduction in work.

When I don’t reach for it

Context is not free and it is not a store. Some cases where I leave it alone:

The tree is two levels deep. Passing a prop twice is clearer than a provider, an exported hook and an extra file.
The data changes on every keystroke. Form state in context makes the whole subtree re-render as you type. Keep it local, lift it only as far as it needs to go.
You want a store. If you need selectors, middleware, devtools or persistence, use something built for that. Context plus useReducer gets you a surprising distance, and then it stops.
Server data. Caching, revalidation and request deduplication are a different problem, and a query library solves it properly.

The failure mode I’ve watched happen is one giant AppContext holding everything, which re-renders the application whenever anything anywhere changes. That’s a worse position than the prop drilling it replaced, and it’s harder to unpick.

What I’d take away

Small, focused providers close to where they’re used. One hook per context, with a guard in it. And a useMemo on the value if the provider sits anywhere near the root.

That’s about it. useContext is one of the few React APIs that’s genuinely simple, and almost all the trouble people have with it comes from asking it to be a state manager.

‘Till next time!

Understanding TypeScript Unions

Tue, 18 Jul 2023 00:00:00 GMT

The pipe character does a lot of work

A union type is the smallest possible piece of TypeScript syntax and one of the easiest to get subtly wrong. You write a pipe between two types and the compiler starts asking questions you didn't expect.

I want to go through how I actually use unions in React work: on props, on reducer actions, and in the places where the compiler stops helping and you have to give it a hand. There is also a small pile of things I got wrong for longer than I’d like to admit, so those get their own section at the end.

What a union actually is

A union says a value is one of several types, but not which one. You write it with a pipe:

type MyUnion = number | string;

MyUnion holds either a number or a string. The important half of that sentence is the part people skip: until you prove which one it is, TypeScript will only let you use the members that are common to both. .toFixed() is off limits, .toUpperCase() is off limits, and .toString() is fine because both have it.

Proving which one it is called narrowing, and it is most of the work.

Function parameters

The classic case is a parameter that accepts one thing or a list of them:

const greet = (names: string | string[]) => {
  if (Array.isArray(names)) {
    names.forEach(name => console.log(`Hello, ${name}!`));
  } else {
    console.log(`Hello, ${names}!`);
  }
};

greet('Alice');
// output: Hello, Alice!

greet(['Bob', 'Charlie']);
// output: Hello, Bob! Hello, Charlie!

Inside the if, names is string[]. Inside the else, it’s string. Array.isArray is one of the checks TypeScript understands natively, along with typeof, instanceof, in, and a plain truthiness test.

Conditional rendering in React

Same idea, applied to props. A component that renders differently depending on what it was handed:

type DisplayProps = { value: number | string };

const Display: React.FC<DisplayProps> = ({ value }) => {
  return (
    <>
      {typeof value === 'number' ? (
        <p>Number: {value}</p>
      ) : (
        <p>String: {value}</p>
      )}
    </>
  );
};

const App = () => {
  return (
    <>
      <Display value={42} />
      {/* output: Number: 42 */}

      <Display value="Hello, TypeScript!" />
      {/* output: String: Hello, TypeScript! */}
    </>
  );
};

The typeof check does the narrowing, so both branches know what they’re holding.

Where unions really earn their place on props is string literals. variant: 'primary' | 'secondary' | 'danger' gives you autocomplete in the editor and a compile error on a typo, which is roughly a hundred times more useful than variant: string and costs nothing.

Reducer actions

This is where I use unions most, and it’s the shape that made them click for me. A reducer takes one of a fixed set of actions, and the action’s type field tells you which:

type CounterAction =
  | { type: 'increment'; payload: number }
  | { type: 'decrement'; payload: number }
  | { type: 'reset' };

type CounterState = { count: number };

Note that reset has no payload at all. That’s deliberate, and it’s the part a single flat interface with optional fields cannot express. With { type: string; payload?: number } you can dispatch { type: 'reset', payload: 42 } and nobody stops you.

const counterReducer = (
  state: CounterState,
  action: CounterAction
): CounterState => {
  switch (action.type) {
    case 'increment':
      return { ...state, count: state.count + action.payload };
    case 'decrement':
      return { ...state, count: state.count - action.payload };
    case 'reset':
      return { ...state, count: 0 };
  }
};

Inside the increment case, action.payload is a number and the compiler knows it. Inside reset, reaching for action.payload is an error, because that member has no such field.

Discriminated unions

What makes the reducer above work is that every member has a property with a literal type, and the literals are all different. TypeScript calls that a discriminant, and a union built this way is a discriminated union. Switch on it and each case narrows the whole object rather than only the field you tested.

The discriminant does not have to be called type and does not have to be a string. Booleans work fine, which is handy for a two-state shape:

type Guest = {
  id: string;
  isGuest: true;
};

type RegisteredUser = {
  id: string;
  isGuest: false;
  email: string;
};

type User = Guest | RegisteredUser;

const getContact = (user: User) => {
  if (user.isGuest) {
    return null;
  }

  return user.email;
};

Note isGuest: true, not isGuest: boolean. That distinction is the entire mechanism. A boolean field is not a discriminant, it’s just a field, and narrowing will not happen.

Exhaustiveness checking

Here’s the feature that pays for the rest of it. Assign the narrowed value to never in the default branch:

const counterReducer = (
  state: CounterState,
  action: CounterAction
): CounterState => {
  switch (action.type) {
    case 'increment':
      return { ...state, count: state.count + action.payload };
    case 'decrement':
      return { ...state, count: state.count - action.payload };
    case 'reset':
      return { ...state, count: 0 };
    default: {
      const unhandled: never = action;
      throw new Error(`Unhandled action: ${unhandled}`);
    }
  }
};

If every case is covered, action in the default branch has type never and the assignment compiles. The moment somebody adds { type: 'set'; payload: number } to CounterAction and forgets the reducer, that line stops compiling and names the type it can’t assign. I’ve caught more bugs with this five-line pattern than with any amount of unit testing around reducers.

Combining unions with intersections

Unions and intersections are often introduced as a pair, which is a bit misleading because they do opposite things. A union is “one of these”. An intersection, written with &, is “all of these at once”.

They compose usefully. If several members share fields, factor the shared part out and intersect it with the union of the variable parts:

type WithId = { id: string };

type Guest = { kind: 'guest' };
type Registered = { kind: 'registered'; email: string };

type User = WithId & (Guest | Registered);

User is equivalent to (WithId & Guest) | (WithId & Registered), so id is available everywhere and email only appears once you’ve narrowed on kind. The intersection distributes over the union. For two members this is more ceremony than it’s worth, but at five or six shared fields it stops the shape from drifting apart.

Distribution, and why `Exclude` works

Most of the utility types you already use are built on the fact that conditional types distribute over unions. Exclude<'a' | 'b' | 'c', 'a'> gives 'b' | 'c' because the condition is applied to each member separately and the results are unioned back together.

That’s worth knowing mostly so you’re not surprised by it. It’s how you take an existing prop union and derive a narrower one, rather than writing the list out twice and letting the two copies drift.

Pitfalls

A union of object types without a discriminant

This is the mistake I made for a long time:

type Success = { data: string };
type Failure = { error: string };

type Result = Success | Failure;

It reads perfectly well and it is annoying to use. You cannot write result.data, because Failure has no data. There is nothing to switch on. You end up with 'data' in result checks scattered through the code, which works but reads like an apology.

Add a discriminant and the problem disappears:

type Result =
  | { status: 'success'; data: string }
  | { status: 'error'; error: string };

If you’re modelling API responses this way, it pairs nicely with the shape normalisation I wrote about in Using Normalizr: decide the shape once, at the boundary, and the rest of the app stops guessing.

Widening swallows your literals

type Size = 'sm' | 'md' | string;

That looks like “one of these two, or any other string”. It is not. 'sm' and 'md' are both assignable to string, so the whole thing collapses to string and your autocomplete quietly disappears. There’s no error, which is what makes it nasty.

If you genuinely want the suggestions plus an escape hatch, the workaround is to stop the collapse:

type Size = 'sm' | 'md' | (string & {});

It’s ugly. It works, the editor keeps suggesting sm and md, and I use it perhaps twice a year.

Too many members

A union with fifteen members is a signal, not an achievement. Somewhere in there is a second concept trying to get out. Split it, or find the field the members actually vary on and make that the discriminant.

The same thing applies to a component whose props are a union of four completely different shapes. That’s usually four components sharing a name because nobody wanted to pick three more. Some of this is just idiomatic programming applied to types: the clearest version is the one the next person can read without a map.

Where I’d start

If you take unions no further than string literal props and one discriminated reducer action type with a never check in the default branch, you have most of the value. Everything above that is refinement.

The one thing I’d add on top is the habit of asking, whenever a field is optional, whether it’s actually optional or whether there are two shapes hiding behind one interface. It usually is two shapes.

‘Till next time!

AI and Development

Tue, 13 Jun 2023 00:00:00 GMT

The autocomplete grew a personality

Half the internet spent this spring announcing that programming is finished, and the other half announcing that the whole thing is a parlour trick. I don't think either group has had the tools switched on in a real project for very long.

So instead of guessing, I want to walk through the four places this stuff actually touches my day: writing code, reviewing it, testing it and rewriting it. Some of it has earned a keyboard shortcut. A surprising amount of it is a demo with a pricing page attached.

One thing up front, and it connects to what I wrote about idiomatic programming: a machine that produces code nobody on your team would write by hand has not saved you anything. It has moved the work from typing to reading, and reading is the expensive part.

Completion, and what it is good at

The current generation of tooling starts with Codex, the model OpenAI trained on public code and shipped in 2021. It is the engine that GitHub Copilot was built on. If you went looking for the Codex API this spring, you found a tombstone: OpenAI deprecated it in March with three days’ notice and pointed everyone at GPT-3.5 and GPT-4 instead. Three days. That says more about building on somebody else’s model endpoint than any benchmark could, and it has nothing to do with whether the model was any good.

What survived is the editor integration, and that part I genuinely like. Copilot is at its best on the code I resent writing: the fifth case in a reducer, a test fixture with twelve fields, the regular expression I would otherwise have gone looking for. It reads the open buffers, notices the pattern I set up two files ago, and finishes the shape.

It is much worse at anything it has not seen a thousand times before. It will invent a method on a library that has no such method, give it the right naming convention and a completely plausible signature, and offer it with total confidence. Wrong code that looks like your code is more expensive to review than wrong code that looks wrong. That is the actual tax, and nobody puts it on the landing page.

Worth having a look at:

GitHub Copilot suggests completions inline as you type, using the surrounding files as context. It’s the one I’d start with. Their site has the current plans.
Tabnine does the same job and will run models locally, which matters a lot if your employer has feelings about source code leaving the building. tabnine.com
A ChatGPT tab is a different tool entirely. No editor context, so no completions, but far better for “explain this stack trace” or “what is this build error actually telling me”.

Review is where half the category is a rebrand

Snyk Code is the interesting one here, and it has a Swiss origin story. DeepCode came out of the Secure, Reliable and Intelligent Systems lab at ETH Zurich, learned issue patterns from a very large pile of open source repositories, and got bought by Snyk in 2020. It now ships as Snyk Code inside their platform. Nice to see something out of Zurich end up in that many editors.

The rest of the category is thinner than the marketing suggests. Code Climate is a maintainability and static analysis platform. DeepSource is a static analyser with automatic fixes. Both are useful, and neither is doing anything a linter author would recognise as machine learning. Putting “AI” on the box sells seats.

Snyk Code (formerly DeepCode) flags security and reliability issues from learned patterns rather than hand-written rules. snyk.io
Code Climate grades maintainability and tracks churn against complexity. codeclimate.com
DeepSource analyses on every push and can open the fix as a pull request. deepsource.io

The honest pitch for all three is that they raise the floor. They catch the null dereference at two in the morning that a tired reviewer waves through. They will not tell you that the abstraction is wrong, and that is still the review comment that matters most.

Tests are where it surprised me

I expected test generation to be the weakest of the four and it is the one that changed my mind.

Diffblue Cover reads Java bytecode, uses reinforcement learning to choose inputs that reach each path through a method, then writes JUnit tests with the mocks and assertions in place. The important detail is that it executes what it writes, so the output compiles and runs. That is a very different bargain from asking a language model for a test file and discovering the problem three commits later.

Testim goes at the other end of the pipeline. The real cost of an end-to-end suite is not writing it, it is the selectors breaking every time somebody restructures a component, and Testim uses a model to pick locators that survive that. It’s a neighbour of the problem I looked at in screenshot testing, where the suite tells you something changed without telling you whether it matters.

One correction while I’m here, because I keep seeing it repeated. DeepMind’s AlphaCode, from early 2022, is not a bug-finding system. It solves competitive programming problems by sampling an enormous number of candidate programs and filtering them against the example tests, and it placed roughly in the middle of the Codeforces field. Genuinely impressive research. It is not something you can point at your repository, and citing it as automated QA is just wrong.

Refactoring is still hand work

Facebook published Aroma back in 2019 and it gets described badly almost everywhere, including by me until I went and read the paper. It indexes a large corpus of code, takes a partial snippet as input, finds method bodies that are structurally similar, and clusters them into a handful of recommendations. It is code-to-code search. It does not remove redundant operations or improve your memory management, whatever the summaries say.

Sourcery is the closest thing to a daily-use refactoring assistant I’ve found, and it’s Python only. It sits in the editor and suggests the tidier form of what you just wrote, which is pleasant and occasionally slightly humbling.

But refactoring worth the name is not about rewriting a loop as a comprehension. It’s about moving a boundary, splitting a module that grew two responsibilities, deciding that a prop should have been context all along. That work needs the argument you had in the planning meeting, and none of these tools were in the room.

What stays switched on

Copilot stays on, because the boilerplate tax is real and it pays that for me. Snyk Code and DeepSource stay in CI, because a second opinion that never gets tired is worth the money. Test generation I would reach for on a Java service, not on a React frontend.

What I don’t do is let a suggestion into a commit I haven’t read line by line. That rule has not changed and I can’t see it changing.

This article is in part written using AI technologies 🤣

‘Till next time!

Understanding React Transitions

Sat, 27 May 2023 00:00:00 GMT

Not every update is urgent

Type into a search box that filters a long list and you can usually feel the problem. The characters lag behind your fingers, because React is busy re-rendering a few thousand rows for a query you have already moved on from. startTransition is how you tell React which of those two jobs matters more.

I mentioned transitions in passing when I wrote about React 18, but a paragraph does not do the idea justice, and the API has one sharp edge that is easy to walk into.

What a transition is

Before React 18, every state update was equally important. React took them, rendered them, committed them, and the rendering could not be interrupted once it started. If a render took 300ms, your keystroke waited 300ms.

The concurrent renderer changed that by letting React work on an update, pause, handle something more important, and come back. startTransition is the label you attach to an update to say “this one can wait”:

import { startTransition } from 'react';

// urgent: the input must reflect the keystroke immediately
setQuery(input);

// non-urgent: the filtered results can catch up
startTransition(() => {
  setSearchQuery(input);
});

Two state updates, two priorities. React commits setQuery straight away so the input stays responsive. The transition update renders in the background, and if another keystroke arrives before it finishes, React throws that work away and starts again with the newer value. Nothing half-rendered ever reaches the screen.

That interruptibility is the whole feature. It is not batching, and it is worth being clear about that because the two get confused. Automatic batching, also new in React 18, groups multiple setState calls into one render. Transitions decide which renders get to go first. You get automatic batching whether or not you use transitions.

The callback has to be synchronous

Here is the sharp edge. React marks every state update that happens while the callback runs as a transition. Updates scheduled after an await or inside a .then are not part of it, because by the time they fire, the callback has long since returned.

So this does not do what it looks like it does:

// this does NOT mark setData as a transition
startTransition(() => {
  fetchData().then(result => {
    setData(result);
  });
});

fetchData() is called synchronously, the promise is handed back, and startTransition returns. Whenever the response eventually arrives, React has no idea it was supposed to be related to a transition. setData is treated as a perfectly ordinary urgent update.

The fix is to wrap the update itself, on the other side of the await:

fetchData().then(result => {
  startTransition(() => {
    setData(result);
  });
});

The React team calls this a known limitation rather than a design decision, so I expect it to change eventually. Until it does, the rule is simple enough: if you can see an await, a .then or a setTimeout inside your startTransition callback, the updates after it are not transitions.

A subscription callback is fine, because the update inside it is synchronous:

import { startTransition, useEffect, useState } from 'react';

const ChatApplication = () => {
  const [messages, setMessages] = useState([]);

  useEffect(() => {
    const subscription = subscribeToNewMessages(message => {
      startTransition(() => {
        setMessages(previous => [...previous, message]);
      });
    });

    return () => subscription.unsubscribe();
  }, []);

  return (
    <div>
      {messages.map(message => (
        <Message key={message.id} content={message.content} />
      ))}
    </div>
  );
};

In a busy channel this keeps the message list from monopolising the main thread. If someone is typing a reply while forty messages land, the input wins.

useTransition when you want a spinner

startTransition on its own gives you no way to know whether the work is still going. The hook version does:

import { useTransition, useState } from 'react';

const Tabs = () => {
  const [isPending, startTransition] = useTransition();
  const [tab, setTab] = useState('home');

  const selectTab = next => {
    startTransition(() => {
      setTab(next);
    });
  };

  return (
    <div style={{ opacity: isPending ? 0.6 : 1 }}>
      <TabButtons onSelect={selectTab} />
      <TabPanel tab={tab} />
    </div>
  );
};

The tab button responds instantly, the old panel stays on screen while the new one renders, and isPending lets you dim it or show an indicator instead of a blank gap. This is the pattern I use most. Dimming the outgoing content reads far better than swapping it for a skeleton, and it is a one-line change once the transition is in place.

There is also useDeferredValue, which comes at the same problem from the other end. Rather than marking the update, you take a value and get back a copy that lags behind:

const deferredQuery = useDeferredValue(query);

Use it when you do not own the setState call, typically because the value arrives as a prop from somewhere above you. Same underlying machinery, different handle on it.

Caveats worth knowing

You can only wrap updates whose setter you have direct access to. If the value comes in as a prop, useDeferredValue is the tool, not startTransition.
Transitions cannot control a controlled text input. The value has to update urgently or typing breaks, which is exactly why the search example splits into two state updates rather than one.
A transition will be interrupted by any urgent update, and the work done so far is discarded. That is the point, but it means a transition can render several times before it commits once. Keep the render function pure and free of side effects, which you should be doing anyway.
If several transitions are in flight, React currently batches them together rather than tracking them separately.
Errors thrown while rendering a transition are caught by the nearest error boundary, the same as any other render error. No special handling needed.

Where it does not help

Transitions reorder work. They do not make work smaller. If your list takes 300ms to render, wrapping the update in a transition keeps the input responsive, but the results still take 300ms to appear, and now they appear at an unpredictable moment.

So before reaching for this, check whether the render is expensive for a reason you can fix: virtualise the long list, memoise the row component, stop recomputing a derived array on every pass. Those give you an outright faster app. A transition redistributes the cost so the user notices it less, which is genuinely valuable, but it is a second step and not a substitute for the first.

I have found the honest test to be whether you can name the urgent update. If there is a keystroke, a click or a tab switch that must respond immediately, a transition has something to protect. If there is no urgent update in the picture, wrapping things in startTransition is just moving deck chairs.

‘Till next time!

Optimising React Components

Fri, 14 Apr 2023 00:00:00 GMT

The hook you probably do not need yet

useCallback is the hook people reach for first when a React app feels slow, and it is the one that does nothing at all in most of the places it gets used. It is worth understanding exactly what it buys you before sprinkling it over a component tree.

React re-renders a component when its state or props change, and re-renders every child underneath it while it is at it. That is the default, and it is usually fine. When it stops being fine, useCallback is one of the tools available, alongside useMemo and React.memo, all of which arrived properly into everyday use around the same time as the React 18 concurrent renderer.

What it actually does

useCallback returns the same function reference across renders, as long as the values in its dependency array have not changed:

import React, { useState, useCallback } from 'react';

const App = () => {
  const [count, setCount] = useState(0);

  const handleClick = useCallback(() => {
    setCount(count + 1);
  }, [count]);

  return (
    <div>
      <p>Count: {count}</p>
      <button onClick={handleClick}>Increment</button>
    </div>
  );
};

Without the hook, handleClick would be a brand new function object on every render. With it, React hands back the one it stored last time until count changes.

Now the awkward part. In the example above that is worth precisely nothing. The only consumer of handleClick is a DOM <button>, and the DOM does not care whether the handler is the same object as last time. React will attach it either way. The component re-renders exactly as often as it did before, the counter behaves identically, and you have added a hook, an array and a dependency to maintain.

I keep this example in because it is the shape you will find in most tutorials, and it is worth knowing that the shape is a no-op.

Making it do something

Function identity only matters when something is watching it. In practice that is one of two things: a child component wrapped in React.memo, or another hook’s dependency array.

Here is the React.memo case:

import React, { memo, useState, useCallback } from 'react';

const Child = memo(({ name, onClick }) => {
  console.log('rendering', name);

  return <button onClick={() => onClick(name)}>{name}</button>;
});

const Parent = () => {
  const [selectedName, setSelectedName] = useState('');

  const handleChildClick = useCallback(name => {
    setSelectedName(name);
  }, []);

  return (
    <div>
      <p>Selected name: {selectedName}</p>
      <Child name="Thor" onClick={handleChildClick} />
      <Child name="Loki" onClick={handleChildClick} />
      <Child name="Odin" onClick={handleChildClick} />
    </div>
  );
};

memo tells React to skip re-rendering Child when its props are shallowly equal to last time. Click one of the buttons, selectedName changes, Parent re-renders. Because handleChildClick came out of useCallback with an empty dependency array, all three children get the same onClick and the same name they had before, so none of them re-render. The console stays quiet.

Take memo off Child and all three log again on every click, useCallback or not. Take useCallback off and leave memo on, and you get the same result: a fresh function each render means the props are never shallowly equal, so memo never gets to skip anything. Both halves are required. This is the single most common way the optimisation gets applied and quietly does nothing.

The second case is a dependency array:

const load = useCallback(() => {
  fetch(`/api/articles?page=${page}`).then(/* ... */);
}, [page]);

useEffect(() => {
  load();
}, [load]);

Here the effect depends on load. If load were recreated on every render, the effect would run on every render, which is a fetch loop rather than an optimisation. useCallback is what makes that dependency stable.

The dependency array is the interesting bit

Look again at the first example. [count] means a new function on every single increment, which is every render that matters. The memoisation is technically working and practically pointless.

The functional update form fixes it:

const handleClick = useCallback(() => {
  setCount(current => current + 1);
}, []);

setCount from useState is guaranteed stable by React, so it does not need to be in the array, and passing a function to it means we no longer need to read count during render. Empty dependencies, one function for the lifetime of the component. Whenever a useCallback has a dependency that changes as often as the component re-renders, this is the first thing to try.

One more detail that surprises people: the inline arrow function is still created on every render. It has to be, because it is the argument you are passing to useCallback. The hook then throws it away and returns the cached one. You are not saving the allocation, you are stabilising the identity. If someone tells you useCallback avoids creating functions, they have it backwards.

When to skip it

The honest default is to skip it and add it when you have a measured reason. More specifically, leave it out when:

Nothing is observing the identity

If the function is only used inside the component, or handed to a plain DOM element, or passed to a child that is not memoised, useCallback changes nothing about how often anything renders:

const Button = () => {
  const handleClick = () => {
    console.log('Button clicked!');
  };

  return <button onClick={handleClick}>Click me</button>;
};

This is fine as it is. Wrapping handleClick would add a hook call, a closure held in memory for the life of the component, and no behaviour change.

The child is cheap to render

memo is not free. It runs a shallow comparison of every prop on every render of the parent. For a component that renders one button and some text, the comparison can cost more than the render it is avoiding. Reach for memo and useCallback around genuinely expensive subtrees: long lists, charts, anything doing real work in render.

You would have to memoise half the tree to make it work

Shallow equality means an inline object or array prop breaks memo just as effectively as an unstable function does:

<Child onClick={handleChildClick} style={{ margin: 8 }} />

That style object is new every render, so memo never skips, and the useCallback next to it is wasted. If making one child memoisable requires wrapping four other values in useMemo, the cure is worse than the disease. That is usually a signal to restructure, moving state closer to where it is used, so fewer components re-render in the first place.

What I actually do

I write components without any of this, and I add useCallback in two situations: when a function is a dependency of useEffect or useMemo, or when the React DevTools profiler shows me a subtree re-rendering that has no business re-rendering. The first is a correctness concern and the second is a measured one. Everything in between is guesswork that makes the code harder to read.

Which is my one real irritation with this hook. It is not that it is slow, it is that it looks like diligence. A file full of useCallback reads as careful, performance-minded code, and most of the time it is decoration with a dependency array attached.

‘Till next time!

Idiomatic Programming

Fri, 31 Mar 2023 00:00:00 GMT

Code that looks like the language it is written in

Idiomatic code is code that follows the conventions of the language it is written in, rather than the conventions of the language you learned first. It is the difference between TypeScript that reads like TypeScript and TypeScript that reads like Java with the type annotations moved around.

Nobody enforces this. The compiler does not care, the tests do not care, and the code ships either way. What you get from it is that the next person, quite possibly you in eight months, can read the file without stopping to work out what the author was doing. Conventions are worth knowing even when they are not enforced, which is roughly the point I was making about class name ordering a few months back.

Below are the patterns I lean on most in TypeScript. Most of them have equivalents in other languages, so the habit transfers even if the syntax does not.

Describe your shapes with interfaces

An interface gives a name to a shape, and that name then shows up in every error message and every autocomplete popup instead of a wall of inline braces:

interface User {
  name: string;
  age: number;
  email: string;
}

const sendEmail = (user: User, message: string): void => {
  // do stuff
};

const user: User = {
  name: 'John',
  age: 30,
  email: 'john@wick.com',
};

sendEmail(user, 'Hello!');

Note where the void goes. The return type belongs to the function, not to the constant holding it. Writing const sendEmail: void = ... is a type error, and it is one I see a lot in code that was ported over from somewhere else.

Let the compiler do the inferring

TypeScript is good at working out types on its own, and annotating things it already knows is noise:

// inferred as string
const name = 'John';

// inferred as (a: number, b: number) => number
const add = (a: number, b: number) => {
  return a + b;
};

const result = add(2, 3);

Annotate the boundaries, infer the middle. Parameters need annotations because there is nothing to infer from. Return types usually do not, unless the function is part of a public API and you want the compiler to shout at you when the shape drifts.

Promises, and the catch that quietly eats your errors

Async work is everywhere, and there is a specific mistake worth calling out:

// don't do this
const fetchData = (): Promise<unknown> => {
  return fetch('/api/data')
    .then(response => response.json())
    .catch(error => console.error(error));
};

That .catch swallows the rejection. The promise it returns resolves with undefined, so any .catch the caller attaches never fires and the caller happily carries on with nothing. If you are going to handle an error, handle it. If you are not, let it propagate:

const fetchData = async (): Promise<Article[]> => {
  const response = await fetch('/api/data');

  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }

  return response.json();
};

Two other things happen here. async/await flattens the chain, and the return type is Article[] rather than any. response.json() returns Promise<any>, which means every value that flows out of it is unchecked from that point on. Give it a type, or use unknown and validate.

readonly, where it earns its place

The readonly modifier stops a property from being reassigned after construction:

interface Person {
  readonly name: string;
  age: number;
}

const person: Person = { name: 'John', age: 30 };
person.name = 'Bob'; // ERROR: Cannot assign to 'name' because it is a read-only property

This is a compile-time check only. Nothing stops the value being mutated at runtime, and nothing stops a plain Person object being passed somewhere that expects a mutable one. It is a hint to readers backed by the compiler, which is still worth having. ReadonlyArray<T> (or readonly T[]) does the same job for arrays and is the more useful of the two in my experience, because accidental push into a prop is a real bug and accidental reassignment of a field usually is not.

Enums, and why I mostly avoid them

The usual advice is to use an enum for a fixed set of named constants:

enum Color {
  RED = 'red',
  GREEN = 'green',
  BLUE = 'blue',
}

const setBackgroundColor = (color: Color) => {
  document.body.style.backgroundColor = color;
};

setBackgroundColor(Color.BLUE);

This works, and if your codebase already uses enums everywhere then consistency beats my preference. But enums are the one part of TypeScript that emits actual JavaScript. Everything else erases to nothing; an enum leaves an object behind in your bundle. And const enum, which is the version that does not, is unsupported under isolatedModules, so it breaks the moment you build with esbuild, Babel or anything built on them.

A union of string literals gives you the same exhaustiveness checking with no runtime cost:

type Color = 'red' | 'green' | 'blue';

const setBackgroundColor = (color: Color) => {
  document.body.style.backgroundColor = color;
};

setBackgroundColor('blue');

You lose the Color.BLUE namespacing. If you want it back, an object with as const and a derived type gets you most of the way there. You might not agree, and I have worked in codebases where the enum version reads better. This is a preference, not a rule.

Generics for the things that genuinely vary

A generic lets one function serve many types without collapsing into any:

function getFirst<T>(array: T[]): T | undefined {
  return array[0];
}

const numbers = [1, 2, 3];
const firstNumber = getFirst(numbers); // number | undefined

const strings = ['hello', 'world'];
const firstString = getFirst(strings); // string | undefined

The | undefined in the return type is the interesting part. Indexing an array in TypeScript is not checked by default, so array[0] on an empty array is typed T while actually being undefined. Writing it out is honest. Turning on noUncheckedIndexedAccess makes the compiler do it for you everywhere, which is the sort of setting that produces a hundred errors on day one and saves you a production incident later.

Generics are also where idiomatic code goes wrong most often. Three type parameters with conditional types and a mapped type on top will typecheck beautifully and be unreadable. If you cannot explain the signature out loud, it is too clever.

A few more worth knowing

I will not list every pattern in the language, but these come up constantly:

Type guards and narrowing. A function returning value is Article teaches the compiler something it cannot work out alone. This is how you get from unknown to a real type safely.
unknown instead of any. any switches type checking off; unknown forces you to narrow first. Anything crossing the boundary into your app, JSON, form data, a query string, should arrive as unknown.
Nullish coalescing and optional chaining. a?.b ?? fallback handles null and undefined without also swallowing 0 and '' the way || does.
Destructuring. Cheap readability, especially for props and function options objects.
never for exhaustiveness. Assign to a never in the default branch of a switch and the compiler fails the build when someone adds a new case to the union. Pairs perfectly with string literal unions.

Tuples, decorators, namespaces and currying all have their moments, but reaching for them without a reason is how you end up with the clever generic signature nobody can read.

Where to actually start

Do not try to adopt all of this at once. Pick the two settings that pay off immediately, strict and noUncheckedIndexedAccess, and let the compiler tell you where the codebase disagrees with itself. Fix those, and most of the patterns above turn up on their own because they are the shortest path to making the errors go away.

The rest is reading. Whatever language you are in, the fastest way to learn its idioms is to read its standard library and a couple of well-regarded projects, and notice what they do that you would not have.

‘Till next time!

Understanding React Suspense

Thu, 23 Feb 2023 00:00:00 GMT

It all comes down to fallbacks

Every app I have worked on ends up with the same scruffy corner: a pile of isLoading booleans, each one owned by a different component, each one rendering a slightly different spinner. Suspense is React's attempt to make "this part isn't ready yet" a thing the framework knows about rather than something you hand-roll.

I already touched on it when I wrote about React 18, but only in passing. The concurrent renderer is what makes it interesting, so it deserves its own article.

What it actually does

You wrap a part of your tree in Suspense and give it a fallback:

<Suspense fallback={<LoadingArticles />}>
  <Articles />
</Suspense>

React renders <LoadingArticles /> until <Articles />, or anything below it, is ready. When the tree is ready, React swaps the fallback out for the real thing in a single commit. No intermediate state where half the list has appeared and half hasn’t.

The important bit is who decides “ready”. Suspense is not polling your component or inspecting your state. A child signals that it is not ready by throwing a promise during render, and Suspense catches it. You will almost never write that by hand. Something else does it for you.

Not every loading state counts

This is the part that trips people up. Suspense only reacts to a Suspense-enabled source. As of React 18 that means two things in practice:

Data fetching through a framework that has integrated with it, such as Relay or Next.js.
Lazy-loading component code with lazy.

A plain fetch inside a useEffect is not one of them. If you write this:

useEffect(() => {
  fetch('/api/articles')
    .then(res => res.json())
    .then(setArticles);
}, []);

…then no Suspense boundary anywhere above you will ever show its fallback. The effect runs after the component has already rendered, so as far as React is concerned that render succeeded. You are back to your own isLoading flag.

I find this genuinely annoying, and it is the single thing that keeps Suspense from being useful in the average non-framework app. So for the example below I’ll use lazy, which works everywhere and shows the same mechanics.

A worked example with lazy

A well-built React app splits its code into several smaller bundles and ships only what the user needs for the page in front of them. You get that from a builder like Webpack or Vite, and lazy is how you tell React about the split point.

Start with something to load. Nothing clever here on purpose:

// Articles.tsx
import React from 'react';

const Articles = () => {
  return <p>A list of articles.</p>;
};

export default Articles;

Then the app that lazily imports it. I’ve added an artificial two-second delay so the fallback is actually visible on a fast connection, otherwise you blink and miss it:

// App.tsx
import React, { lazy } from 'react';

const Articles = lazy(async () => {
  await new Promise(resolve => setTimeout(resolve, 2000));

  return import('./Articles');
});

export const DemoAppLoading = () => <p>🌀 Loading...</p>;

export const DemoApp = () => {
  return <Articles />;
};

Note that lazy is called at module scope, not inside the component. Calling it during render creates a brand new lazy component on every pass, which means the chunk is treated as a fresh import every time and the fallback never goes away. It is an easy mistake to make and a miserable one to debug, because nothing errors. You just get a spinner forever.

Finally, wire it up:

// index.tsx
import React, { Suspense } from "react";
import ReactDOM from "react-dom/client";

import { DemoApp, DemoAppLoading } from "./App";

const root = ReactDOM.createRoot(
  document.getElementById("root") as HTMLElement
);
root.render(
  <React.StrictMode>
    <Suspense fallback={<DemoAppLoading />}>
      <DemoApp />
    </Suspense>
  </React.StrictMode>
);

DemoAppLoading stays on screen until the ./Articles chunk has been fetched and evaluated, plus the two seconds I added. App.tsx itself is a normal static import, so it is already in the main bundle and never suspends. Only the lazy child does.

Where you put the boundary matters

The fallback replaces the entire subtree, so a boundary near the root means the whole page disappears while one small chunk downloads. That is rarely what you want.

Boundaries nest, and each one reveals independently:

<Suspense fallback={<PageSkeleton />}>
  <Header />
  <Suspense fallback={<SidebarSkeleton />}>
    <Sidebar />
  </Suspense>
  <Articles />
</Suspense>

Here the sidebar can still be loading while the header and the articles are on screen. The nearest boundary above a suspending component wins.

Worth pairing them with an error boundary as well. If the chunk fails to download, because you deployed while someone had the tab open and the old hashed file is gone, lazy throws and the nearest error boundary catches it. Without one you get a blank page.

Keeping content on screen

There is a second behaviour that took me a while to notice. If content is already visible and an update would cause it to suspend again, React will hide it and show the fallback, which reads as a flicker back to the spinner.

Marking that update as a transition avoids it:

const [isPending, startTransition] = useTransition();

startTransition(() => {
  setTab('articles');
});

React keeps the previous content on screen while the new tab loads, and gives you isPending so you can grey it out or show a small indicator instead of blowing the whole section away.

What I’d still like

Suspense on the server is the part I am most pleased with in React 18. Combined with streaming SSR, which I covered in the hydration article, the server can send HTML for the fast parts of the page and stream the slow parts in as they resolve.

On the client it is still half a feature unless you are on a framework that has done the integration for you. The documentation is honest about this, which I appreciate, but it does mean “use Suspense for data fetching” is advice you cannot act on in most codebases yet. I’ll keep using it for code splitting and wait.

‘Till next time!

Looking back at 2022

Tue, 31 Jan 2023 00:00:00 GMT

Not the year I ordered

The past year hasn't been kind, and I'm looking forward to a more enjoyable one. I'll stick to the positive here so I can keep moving steadily onward. Thanks to everyone who has been there for me 🤗.

I kept journaling work, school and personal life, the same as I did for 2021. For my sanity and everyone else’s, here is the short version.

Work

We steadily continued to improve our product. The most noteworthy piece was the overhaul of the entire routing for the Control Panel, which made it exceptionally fast. The addons and boilerplates sections, the project overview, mirrors and metrics were all reworked along the way. We also added dark mode 🤣.

Beyond that: a long list of smaller improvements across the board, a lot of resolved issues, and a handful of internal tools that make our own lives easier.

Study

2021 was a straining and busy year of studying. In 2022 I took a break, mostly because of everything else that was going on. I also expected more from my school, and corona did nothing to improve the experience.

That did not stop the learning. I worked through several classes on Udemy, MIT’s OpenCourseWare and Harvard’s CS50. I’m more convinced than ever that this is the future of learning rather than the classical route 🤷‍♂️

Personal

Privately, 2022 was horrendous. I had to deal with an immense loss and with existential doubts about my studies. What came out of it was travel, and a lot of it: Spain, Hungary, Germany, France, Sweden, Denmark, Portugal, Cuba and Egypt. Plenty of concerts and festivals with friends, too.

Looking forward

Yeah. Looking forward to 2023. That’s about the gist of it (╯°□°)╯︵ ┻━┻.

‘Till next time!

Self-Hosted Jitsi Server

Sun, 18 Dec 2022 00:00:00 GMT

A personal Jitsi Server on Ubuntu

Are you tired of Google Meets, Skype or Zoom? Want an open-source solution that doesn't spy on you in the background? Say welcome to Jitsi Meet.

I’m not a privacy advocate, but I wanted to play around with Jitsi Meet for a while now that SWITCH’s meet.switch.ch has closed its doors. This is a great excuse to see if I can host Jitsi on my own and allow access from the outside.

Setting up Jitsi Meet isn’t that difficult. The team provides a great quickstart guide. That doesn’t mean it is without its challenges, especially when you are not following the installation instructions 🫣. You can run into encryption issues when the domain isn’t set up correctly, or connection troubles when a package is missing.

Getting started

I will start from a standard Ubuntu server installation on a minted VM. Always ensure Ubuntu is up to date, SSH into the VM and run:

sudo apt update
sudo apt upgrade
sudo apt full-upgrade

Next, we must figure out how to reach the server before installing Jitsi Meet. My scenario revolves around https://meet.dini.dev/. The service should be reachable through that domain from the outside, encrypted. For that to happen:

I need to configure my DNS provider to point to my WAN IP and refresh it, as it is not a static IP.
I must ensure my firewall accepts and forwards incoming connections to the Ubuntu server.

Configuring the DNS

I use dnsimple to manage my domains. The first step is to go to the configuration settings and set up a new subdomain for https://meet.dini.dev/ that points to my current WAN IP address. I can get that quickly by running curl -s https://icanhazip.com in a terminal window, as ip a would only show me the local addresses.

The problem is that my WAN IP is dynamic and changes over time. To solve this, I could either get a static IP or let dnsimple know about any changes. dnsimple provides some tools to help us with this. Their ddns.sh script is what I’m looking for. SSH into the Ubuntu instance, run vim ~/ddns.sh and add the following content:

#!/bin/bash

TOKEN="{% token from user settings > user access tokens %}"
ACCOUNT_ID="{% account id shown under account > automation %}"
ZONE_ID="{% your primary domain name %}"
RECORD_ID="{% the record id from one of the DNS records %}"
IP=`curl --ipv4 -s http://icanhazip.com/`

curl -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -X "PATCH" \
     -i "https://api.dnsimple.com/v2/$ACCOUNT_ID/zones/$ZONE_ID/records/$RECORD_ID" \
     -d "{\"content\":\"$IP\"}"

The TOKEN, ACCOUNT_ID, ZONE_ID and RECORD_ID need to be replaced accordingly. I added hints in the script on where to get it from dnsimple.

Next, run sudo chmod +x ./ddns.sh to allow the script to be executed through crontab. A crontab can easily be set up by running crontab -e. I decided to set up a crontab that runs the script every 5 minutes, so add the following at the end of the file:

*/5 * * * * /path/to/ddns.sh &

This command will update dnsimple’s records every 5 minutes with my current WAN IP address 🥳. A simple way of creating a dynamic DNS. Test if the crontab is working through sudo service cron start and check the logs by running systemctl status cron.

Configuring the Firewall

I’m running Pi-hole as my primary local DNS Server. As such, I can trick my local network into forwarding an incoming request from https://meet.dini.dev/ to redirect to my Jitsi server.

Log in to Pi-hole and select “DNS records” from the menu. Choose to add a “new domain/IP combination” and add the hostname (meet.dini.dev) and IP address of the Ubuntu server accordingly.

Next, we need to ensure traffic through the ports is routed correctly. Go to your router’s firewall settings and make sure the following ports are forwarded from your WAN IP to the Jitsi server:

TCP: 80,443,5349
UDP: 3478,10000

To enable additional security, enable ufw by running sudo ufw enable on the Jitsi server followed by:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3478/udp
sudo ufw allow 5349/tcp
sudo ufw allow 10000/udp

Check the status using sudo ufw status. Next, we install Jitsi Meet!

Installing Jitsi Meet

Before starting the installation, ensure the steps above are working correctly. I made a mistake installing Jitsi first, which required setting up the encryption manually afterwards.

To get started, run the following commands in sequence:

# preparing the server
sudo apt update
sudo apt install apt-transport-https gnupg2 nginx-full socat
sudo apt-add-repository universe
sudo apt update

# install dependencies
echo deb http://packages.prosody.im/debian $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list
wget https://prosody.im/files/prosody-debian-packages.key -O- | sudo apt-key add -
sudo apt install lua5.2

# setup the repository
curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg'
echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null
sudo apt update

# start installation
sudo apt install jitsi-meet

You will be prompted to configure your hostname and SSL/TLS certificate. For the hostname, enter the domain you decided to go with (for me, meet.dini.dev). For the certificate, choose “Let’s Encrypt” if you do not want to upload your certificate.

The verification process will only be configured appropriately when the domain is set up correctly. However, you can create the certificates manually afterwards by running:

sudo apt install certbot
sudo /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

At this stage, the Jitsi Server is running and can be reached through the local IP address of the Ubuntu server or the hostname you have set up. That’s all there is to it 🫣.

Yatta

I now have my own personal Jitsi Meet server running on https://meet.dini.dev/ 🎉.

‘Till next time!

Red pill or blue pill?

Wed, 30 Nov 2022 00:00:00 GMT

Does class name ordering matter?

Adding class names to an HTML document is as natural as almost any other HTML attribute. However, I've often encountered that the ordering of the class names needs to be understood.

One of the questions I ask my students during the CAS Frontend Engineering class revolves around the following example:

.red {
  color: red;
}

.blue {
  color: blue;
}

<div class="red blue">First example</div>
<div class="blue red">Second example</div>

Have a closer look at the code above. Which colour is applied?

The first example is red, and the second blue.
The first example is blue, and the second red.
The first and second examples are both blue.
The first and second examples are both red.

The answer follows below.

Understanding the order of CSS classes

The order of CSS classes in HTML does not matter. What matters is the order in which styles are defined in the stylesheet. When two rules set the same property and carry the same specificity, the one declared last in the stylesheet always wins.

This concept can be tricky when dealing with multiple files and cause issues where developers misuse !important from time to time. That is why naming conventions, such as BEM, are essential. Additional tools and frameworks make encapsulation easier these days, Tailwind being a good example.

As such, the answer to the example above is the first and second examples are both blue.

This is a concise post with an important concept to understand, as more and more developers use CSS-in-JS solutions and seem to get this little fact wrong. 🫣

‘Till next time!

Screenshot Testing with React

Sat, 15 Oct 2022 00:00:00 GMT

Using Jest, Puppeteer, Storybook and Docker

Testing your React app using Puppeteer, Jest, and Storybook enhances existing unit tests and saves you headaches in the future, while Docker makes everything consistent.

In an ideal world, we write code without any issues at all. Sadly reality is often disappointing, and we need to write tests. In this article, I’ll focus on screenshot testing. The tutorial was tested using Node 16 and 18.

The basics

The goal is to take snapshots and compare them to a reference screenshot stored alongside the test or in a centralised folder. The test will fail if the two screenshots do not match. Either the change is unexpected, or the reference screenshot needs to be updated to the new version.

We will start with a generic React boilerplate through create-react-app using TypeScript. Open a terminal, navigate to your projects folder and run the following command to get started:

npx create-react-app react-screenshot \
  --template typescript && cd react-screenshot

The command will set up our React application together with some sample files. The following sections will guide us to:

create a simple screenshot-capturing test using jest-image-snapshot
integrate Storybook to isolate testing from a development server
run everything through Docker to get a consistent result

Screenshots using Jest and Puppeteer

Let’s install some dependencies into our project before we get started. Inside the terminal, run the following commands:

npm install jest-image-snapshot \
  puppeteer \
  puppeteer-core \
  @types/jest-image-snapshot \
  @types/puppeteer --save

Check out the documentation for jest-image-snapshot and puppeteer to learn more about their configuration options. I will keep this tutorial as simple as possible, with only minor tweaks.

We’re ready to create our first screenshot test with our dependencies installed. Open src/App.test.tsx and replace the content of the file with:

import React from 'react';
import puppeteer from 'puppeteer';
import { render, screen } from '@testing-library/react';
import { toMatchImageSnapshot } from 'jest-image-snapshot';
import App from './App';

expect.extend({ toMatchImageSnapshot });

describe('React App', () => {
  let browser: puppeteer.Browser;
  let page: puppeteer.Page;

  beforeAll(async () => {
    browser = await puppeteer.launch();
    page = await browser.newPage();
  });

  afterAll(async () => {
    await browser.close();
  });

  it('renders learn react link', () => {
    render(<App />);
    const linkElement = screen.getByText(/learn react/i);
    expect(linkElement).toBeInTheDocument();
  });

  it('renders correctly', async () => {
    await page.goto('http://localhost:3000');

    const image = await page.screenshot();
    expect(image).toMatchImageSnapshot();
  });
});

To run the test, you will have to open two terminal sessions:

# the first one will run the React development server
npm run start

# and the second one our screenshot test
# which will run against the React development server
npm run test

Jest image snapshot takes screenshots against our running development server at http://localhost:3000. The snapshots themselves are stored within src/__image_snapshots__/. An additional folder is shown when the test fails. That folder will contain a visual representation of the diff between the reference and the new screenshot.

The reference screenshots can be regenerated by removing the image snapshots folder. However, as this tests against our development server, you can encounter issues while capturing the reference screenshots due to timeouts or incorrect rendering. So the whole process is somewhat flaky. We will solve this later on.

You can also emulate other devices by extending the page. Underneath a short example emulating an iPhone:

it('renders correctly on iPhone 13', async () => {
  await page.emulate(puppeteer.devices['iPhone 13']);
  await page.goto('http://localhost:3000');

  const image = await page.screenshot();
  expect(image).toMatchImageSnapshot();
});

A full list of supported devices can be found in puppeteer’s documentation.

Handling animations

You might have noticed that the React welcome screen has a rotating logo. This animation can be an issue when taking screenshots, as a couple of milliseconds of delays can move the logo by a pixel or two, creating inconsistent results. To avoid this, we’ll use a little trick. Add the following code snippet at the end of the src/App.css file:

.no-animations * {
  /* disable all animations */
  animation: none !important;
  /* hide the cursor caret when in inputs */
  caret-color: transparent !important;
}

Next, edit src/App.test.tsx and add the class right after navigating to the page:

it('renders correctly', async () => {
  await page.goto('http://localhost:3000');
  // make sure to disable animations
  await page.evaluate(() => {
    document.body.classList.add('no-animations');
  });

  const image = await page.screenshot();
  expect(image).toMatchImageSnapshot();
});

This change will add the no-animations class to the document’s body, which prevents all animations and hides any cursor that might be placed within an input element. It has to happen after page.goto(), as the navigation would otherwise throw the class away again.

However, a development server is not ideal for taking screenshots due to moving parts. On top of that, it is harder to achieve certain states before taking screenshots. For that reason, let’s introduce Storybook!

Screenshots using Storybook

Storybook is a fantastic tool for documenting and testing our components and views. To install it, run the following commands:

npx storybook init
npm run storybook

The latter will open up Storybook with some story examples. To add our React view, create src/App.stories.tsx and place the following code inside:

import React from 'react';
import App from './App';

import './index.css';

export default {
  title: 'Views/App',
  component: App,
};

export const Overview = <App />;

Storybook will pick it up automatically and display the story under the “Views” category. You can find additional stories from Storybook inside the src/stories/ directory.

Next, we want to take snapshots from all of the components and views inside Storybook. A couple of preparations are required to achieve this. At this point, we do not need the Storybook development server. Shut it down and run the following commands:

# remove puppeteer due to peer conflicts
npm remove puppeteer puppeteer-core --save

# install the storybook addon
npm install @storybook/addon-storyshots-puppeteer --save

# clear our testing cache
npx jest --clearCache

The addon-storyshots-puppeteer package will take care of the correct puppeteer version. Otherwise, we get peer dependency conflicts.

We have learned that testing against a development server can raise issues. Luckily, Storybook offers a static build of itself without running a server at all. To get this working, we run npm run build-storybook before executing our tests and then chain it with npm run test. Our testing setup reads the static build straight off the local file system through file://, which doesn’t require any server to run.

To get started, replace the content of our src/App.test.tsx with the following code:

import * as path from 'path';
import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';

initStoryshots({
  suite: 'storyshots',
  test: imageSnapshot({
    storybookUrl: `file://${path.resolve(__dirname, '../storybook-static')}`,
  }),
});

You could also create a separate file like src/screenshots.test.tsx for this process. You may want to delete src/__image_snapshots__/ before moving onward.

To execute the tests, run the following commands:

npm run build-storybook
npm run test

This procedure will add the generated screenshots again into src/__image_snapshots__/ with the same behaviour as before. Instead of just one screenshot, we end up with additional ones from all the components inside Storybook. It is beneficial when testing variants of the same view.

Glueing it together with Docker

The above examples already work when you’re the only person contributing to the project and only run the tests on your system. This setup might break as soon as you need to run screenshot tests on different operating systems due to how fonts are rendered. One solution to this problem is to use Docker to run our tests so that the testing environment is always consistent regardless of where you run them, locally on your Mac, Linux, or in the cloud.

To get started, create a Dockerfile in the root of the project and populate it with following content:

FROM node:16.17-alpine

WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH

COPY package*.json ./
RUN npm install --silent

COPY . .

Next, create a docker-compose.yml file in the root directory containing:

version: '3.9'

services:
  screenshots:
    build: .
    command: npm run test
    user: ''
    depends_on:
      - chrome
    volumes:
      - '.:/app:rw'
      - 'storybook:/app/storybook-static'

  chrome:
    image: 'browserless/chrome:1.53-chrome-stable'
    environment:
      CONNECTION_TIMEOUT: 3600000
    expose:
      - '3000'
    volumes:
      - '.:/app:rw'
      - 'storybook:/app/storybook-static'

volumes:
  storybook:

Docker will create two containers, a chrome container our tests can connect to and a screenshots container based on our Dockerfile that runs the tests encapsulated from our system.

Next, open up your test file and replace it with the following code:

import * as path from 'path';
import puppeteer from 'puppeteer';
import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';

let browser: puppeteer.Browser;

const getCustomBrowser = async (): Promise<puppeteer.Browser> => {
  browser = await puppeteer.connect({
    browserURL: 'http://chrome:3000/',
  });

  return browser;
};

const beforeScreenshot = async (page: puppeteer.Page) => {
  await page.evaluate(() => {
    document.body.classList.add('no-animations');
  });
};

initStoryshots({
  suite: 'storyshots',
  test: imageSnapshot({
    getCustomBrowser,
    beforeScreenshot,
    storybookUrl: `file://${path.resolve(__dirname, '../storybook-static')}`,
  }),
});

afterAll(async () => {
  await browser.close();
});

To finally have consistent and reliable tests, delete the src/__image_snapshots__/ folder and run the following sequence:

# we still need to build storybook first
npm run build-storybook
# build the containers
docker-compose build
# and then run the tests inside docker
docker-compose run --rm screenshots

Another great advantage is that such a setup can easily be integrated into a CI such as GitHub Actions or Gitlab Pipelines.

Summary

Unit testing is a very efficient and relatively cheap process to perform. Integration tests, such as automated screenshot testing, can be expensive and time-consuming.

At Divio, we have hundreds of screenshot tests with various components and views in both light and dark modes. This tutorial is just a mere start to screenshot testing, and the hole goes deep. Have fun exploring 😊.

‘Till next time!

Working with CSS Custom Properties

Tue, 27 Sep 2022 00:00:00 GMT

Say hi to CSS variables

CSS custom properties, sometimes called CSS variables, are properties you can define in your CSS to reuse variables across your stylesheets.

Initially introduced in 2012, it took roughly five years to reach wide browser adoption, around late 2017 once Edge had caught up. What Less and Sass have offered since the very beginning is finally supported in native CSS too.

How to use it

A custom CSS property always needs to be defined within a selector:

.box {
  --color: #000;
  --spacing: 10px;
  color: var(--color);
  margin-bottom: var(--spacing);
}

/* not allowed */
--font-family: "Helvetica Now", Arial, sans-serif;

In the example above, we define the properties in the box selector. Properties cascade (including pseudo-elements), so if you have an element inside the box, they will inherit the variables from their parent. Let’s have a look at this:

.box {
  --spacing: 10px;
}

.box__inner {
  padding: var(--spacing);
}

As long as box__inner is within the box in the HTML document, box__inner will have access to --spacing. What if that is not the case? Well, we can make use of default values if a variable is not defined:

.box {
  --spacing: 10px;
}

.box__inner {
  color: var(--color, #000);
  /* or use another var as backup */
  padding: var(--custom-spacing, var(--spacing, 10px));
}

We can use this to create global styles. What would be the uppermost element in a document? Right, it is html or the :root pseudo-element:

:root {
  --spacing: 10px;
  --color: #000;
  --font-family: "Helvetica Now", Arial, sans-serif;
}

We can further override variables down the cascading tree. We can use this mechanic to, for example, introduce a dark mode:

:root {
  --color: #000;
  --background: #fff;
}

.dark-mode {
  --color: #fff;
  --background: #000;
}

/* or change it with media queries */
@media screen and (prefers-color-scheme: dark) {
  :root {
    --color: #fff;
    --background: #000;
  }
}

Attach the dark-mode class to the <body>, and our global values are overridden from there down, so every element inside inherits the dark set instead. Alternatively, we can let the media query do it for us, neat, right?

All other operations, such as calc, multiplication or addition, are also supported. Custom properties behave in that regard similarly to normal CSS property definitions.

:root {
  --spacing: 20px;
  --spacing-sm: calc(var(--spacing) / 2);
  --spacing-xl: calc(var(--spacing) * 2);
}

Using JavaScript

There is another advantage compared to Less or Sass. You can read and change variables from within JavaScript natively, at runtime.

// get a reference to the element
const root = document.querySelector(":root");

// set a variable
root.style.setProperty("--primary", "#f00");

// read it back, but only if it was set inline
root.style.getPropertyValue("--primary");

// read the value the browser actually resolved,
// including the ones declared in a stylesheet
getComputedStyle(root).getPropertyValue("--primary");

That distinction bites people. element.style.getPropertyValue() only ever sees inline declarations, so a variable you defined in your stylesheet comes back as an empty string. Use getComputedStyle() when you want the value that is actually in effect.

With that knowledge, we can do some interesting manipulations:

.container {
  position: absolute;
  left: var(--x);
  top: var(--y);
  width: 50px; height: 50px;
  background: green;
}

const root = document.documentElement;

root.addEventListener("mousemove", e => {
  root.style.setProperty('--x', e.clientX + "px");
  root.style.setProperty('--y', e.clientY + "px");
});

Courtesy of CSS Tricks, have a look at the demo.

Sass and CSS variables

You can also combine CSS custom properties with Sass, but be aware of the syntax:

$link-color: #333;

:root {
  --link-color: #{$link-color};
  --link-color-hover: #{darken($link-color, 25%)};
}

.link {
  color: var(--link-color);

  &:hover {
    color: var(--link-color-hover);
  }
}

It is good to combine different technologies to get the best out of them. We just recently updated the Control Panel to use CSS custom properties as our go-to for variable declaration, which made the dark mode integration a lot less painful 🥳

‘Till next time!

React 18

Thu, 11 Aug 2022 00:00:00 GMT

Welcome to React 18

React 18 introduces features powered by their new concurrent renderer. Version 18 provides gradual adoption strategies for existing applications with a simple upgrade path.

Their new concurrency system is a new behind-the-scenes mechanism that enables React to prepare multiple versions of the UI simultaneously, adding more speed and efficiency. Naturally, we want to profit from that and upgrade our existing applications to React 18.

How to Upgrade to React 18

The first warning you will see after upgrading to React 18 is that ReactDOM.render is no longer supported. You have to upgrade to ReactDOM.createRoot;

// Before React 18
import ReactDOM from 'react-dom';
import App from './App';

const root = document.getElementById('root');
ReactDOM.render(<App />, root);

// After React 18
import ReactDOM from 'react-dom/client';
import App from './App';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(<App />);

The same applies to ReactDOM.hydrate, which we covered in the previous article.

There are some smaller changes to unmountComponentAtNode as well:

// Before React 18
unmountComponentAtNode(container);

// After React 18
root.unmount();

You can find more information in the official blog post.

Automatic Batching

In React, we are used to updating states as follows:

setLoading(true);
setData(undefined);

React is smart and tries to batch those state changes automatically together to avoid unnecessary re-renders. However, the challenge is when React should do this. Before 18, batching only happened inside React event handlers, so not in promises, setTimeouts or native event callbacks.

fetch('/api/').then(({ data }) => {
  setLoader(true); // re-render
  setData(data);   // re-render
})

React 18 introduces automatic batching for promises, setTimeouts and event callbacks. You can opt out by using flushSync:

flushSync(() => setLoader(true));

Transitions

Sometimes you want certain updates to be treated more urgent than others. For example, clicking or pressing are immediate responses that require immediate feedback. You can still load the content after a click, but the UI should indicate this immediately. So at best, we trigger an urgent update and transition to a non-urgent one:

import { startTransition } from 'react';

// show the input change immediately
setInputValue(input);

// fetch the data in a non-urgent manner
startTransition(() => setSearchQuery(input));

Additionally, useTransition is available as a hook to track the pending state.

Suspense

Wouldn’t it be nice to make the “UI loading state” a first-class declarative concept in the React programming model? This issue is what the <Suspense /> component solves in React 18. It shows the respective loading component if the corresponding component tree is not yet ready to be displayed:

<Suspense fallback={<AvatarLoading />}>
  <Avatar />
</Suspense>

Earlier versions of React had a limited understanding of Suspense. The only supported use case was code splitting using React.lazy, which wasn’t supported when rendering on the server.

React 18 added support for Suspense on the server while expanding its capabilities using concurrent rendering features.

New Client and Server Rendering APIs

Most of the features have been mentioned in the previous article. They include changes to React Hydration and how to stream data efficiently between SSR and CSR.

Have a closer look at React 18’s documentation if you want to dive further into the details and examples.

New Strict Mode Behaviors

In the future, React will add a feature that allows adding and removing sections in the UI while preserving the state. For example, when a user tabs away from a screen and back, future React will be able to show the previous screen immediately.

This approach requires React to be resilient to effects being mounted and destroyed multiple times. To help surface these issues, React 18 introduced a new development-only check to Strict Mode. This new check will automatically unmount and remount every component whenever a component mounts for the first time, restoring the previous state on the second mount and looks like this:

* React mounts the component.
  * Layout effects are created.
  * Effects are created.
* React simulates unmounting the component.
  * Layout effects are destroyed.
  * Effects are destroyed.
* React simulates mounting the component with the previous state.
  * Layout effects are created.
  * Effects are created.

See further information about ensuring a reusable state.

New Hooks

Besides useTransition there have been new hooks introduced to make life easier:

useId

It generates a unique ID that is stable across the server and the client, which makes it handy for accessibility attributes. It is not intended for generating keys in a list. Keys need to be generated from your data.

const CustomForm = () => {
  const id = useId();

  return (
    <>
      <label htmlFor={id}>Accept</label>
      <input id={id} type="checkbox" name="react" />
    </>
  );
};

useDeferredValue

It accepts a value and returns a new copy of the value that will defer to more urgent updates.

const [query, setQuery] = useState('');
const deferredQuery = useDeferredValue(query);

// we wait for the deferredQuery to be ready
// to avoid unnecessary re-renders
const suggestions = useMemo(() =>
  <SearchSuggestions query={deferredQuery} />,
  [deferredQuery]
);

// but we still want the query input to be immediate
return (
  <>
    <SearchInput query={query} onChange={setQuery} />
    <Suspense fallback="Loading results...">
      {suggestions}
    </Suspense>
  </>
);

useSyncExternalStore

It is a new hook that allows external stores to support concurrent reads by forcing updates to the store to be synchronous. It removes the need for useEffect when implementing subscriptions to external data sources and is recommended for any library that integrates with state external to React.

// general use case
const state = useSyncExternalStore(
  store.subscribe, store.getSnapshot
);

// or subscribe to a specific field
const selectedField = useSyncExternalStore(
  store.subscribe,
  () => store.getSnapshot().selectedField,
);

useInsertionEffect

This new hook allows CSS-in-JS libraries to address performance issues of injecting styles in the render. You probably won’t use this unless you are building a CSS-in-JS library yourself, so I refer to the documentation instead.

Conclusion

React 18 introduced a lot of small changes that set the foundation for future releases. At the same time, it keeps a straightforward upgrading path for us developers. Check out their keynote to learn more about React 18 and its future.

‘Till next time!

React Hydration

Fri, 29 Jul 2022 00:00:00 GMT

Understanding React Hydration

Let's initially load a scaffolding HTML representing our React application, followed by hydrating it with functionality to improve user experience and SEO!

That is the gist behind React Hydration. First, instead of showing an empty page on load, we use ReactDOMServer to render our components into the DOM during server-side rendering (SSR). Then, during client-side rendering (CSR), React Hydration kicks in and gradually adds interactivity to our components.

That premise seems simple, but the approach can cause performance concerns and even serious issues. This article will deal with them and provides additional references to strategies.

How does it work?

Let’s have a look at a basic React app and how it is created:

<!-- index.html -->
<html>
  <head></head>
  <body>
    <div id="root"></div>
  </body>
</html>

// DemoApp.js (component)
import React from 'react';

function DemoApp(props) {
  return (<div>Hello {props.title}</div>)
}

export default DemoApp;

// index.js (CSR)
import React from 'react';
import ReactDOM from 'react-dom/client';
import DemoApp from './DemoApp';

const root = ReactDOM.createRoot(
  document.getElementById('root')
);
root.render(
  <React.StrictMode>
    <DemoApp title="React Hydration" />
  </React.StrictMode>
);

In this scenario, the website will display an empty page and fill it with our demo application’s “Hello React Hydration” text. This process can be a problem, as the content “flashes” into existence, and crawlers don’t necessarily know about your content.

To remedy this, we can use SSR to serve our components to the browser without any interactivity:

// server.js (SSR)
import React from "react";
import ReactDOMServer from "react-dom/server";
import DemoApp from './DemoApp';

app.use('*', (request, response) => {
  fs.readFile(
    path.resolve('./index.html'), 'utf-8', () => {
    return response.send(ReactDOMServer.renderToString(
      <DemoApp title="React Hydration" />
    ))
  });
});

React Hydration will then gradually add that interactivity by using hydrateRoot instead of createRoot during CSR. In React < 18, it would be render and hydrate instead of createRoot and hydrateRoot.

Watch out for one detail: hydrateRoot takes the container and the element in one go, so there is no separate render call afterwards.

// index.js (CSR)
import React from 'react';
import ReactDOM from 'react-dom/client';
import DemoApp from './DemoApp';

ReactDOM.hydrateRoot(
  document.getElementById('root'),
  <React.StrictMode>
    <DemoApp title="React Hydration" />
  </React.StrictMode>
);

This technique is how React Hydration is applied to an application. Render a pre-build version of your application on the server, and then add interactivity through the hydration process.

This approach is also used by many static site generators such as Gatsby or Next.js. The sites got pre-generated by SSR and served via their public/ folders. The browser can “pick up” where the server left off and add app-like features accordingly.

However, this approach requires the React tree to be in-sync with the DOM. Otherwise, we end up with issues.

Issues with Server-Side Rendering and Hydration

React Hydration requires our DOM to be in sync with the React tree. Through SSR we pre-fill the DOM with the markup of our application, and React Hydration picks it up from there. What happens if our SSR DemoApp serves different content depending on SSR/CSR generation? Let’s have a look at a typical example:

// DemoApp.js (component)
import React from 'react';

function DemoApp(props) {
  const greeting = typeof window !== 'undefined'
   ? 'Hello ' : '';

  return (<div>{greeting} {props.title}</div>)
}

export default DemoApp;

In this case, if the window is defined, our app returns “Hello React Hydration”, and if not, just “React Hydration”. Now we end up with a mismatch, and the following error is thrown:

While rendering your application, there was a difference between the React tree that was pre-rendered (SSR/CSR) and the React tree that was rendered during the first render in the browser.

This issue is widespread and must be dealt with when working on SSR. Not all features are available on a server compared to a browser. For Hydration to work, the tree needs to be identical.

The above example could be avoided using useEffect, as the method is only called during the hydration phase and not while pre-rendering the content. For example:

// DemoApp.js (component)
import React, { useEffect, useState } from 'react';

function DemoApp(props) {
  // be empty during pre-generation
  const [greeting, setGreeting] = useState('');
  // add "Hello" after hydration takes place
  useEffect(() => setGreeting('Hello '), []);
  return (<div>{greeting} {props.title}</div>)
}

export default DemoApp;

This change causes the content to have the “flashing” effect but can be softened through loaders or skeletons.

There are other issues to consider, for example, when pre-fetching APIs during the SSR process. They need to be appropriately cached or passed onward to avoid mismatches. It is a common source of trouble when working with Gatsby or Next.js.

Hydration strategies

The idea behind different strategies is to decide which components need to be hydrated. Your website becomes unresponsive during the hydration process, which can block your users from interacting with your product. It might not be a big concern for smaller websites, but when you load huge bundles to add interactivity, it can get frustrating to wait to click on a menu to move onwards.

Different approaches surfaced in the past years to solve these problems. The one shown in this article is the plainest: render the whole tree with ReactDOMServer.renderToString and hydrate all of it in one go. Streaming Server-Side Rendering sends the markup down in chunks instead and has been available since React 16 through ReactDOMServer.renderToNodeStream.

Another is Progressive Hydration or Hydration on demand where you attach specific criteria to your components. These can be on visibility, scrolling, idling, or with a simple delay.

And the last I want to cover is Selective Hydration, new in React 18, which uses renderToPipeableStream instead of renderToString. This method makes it possible to start streaming HTML without waiting for the heavier components to be ready. Meaning that you can lazy-load components when using SSR.

There is also a great library collection from Google Chrome Labs on the different approaches if you are interested in code examples.

Let’s hydrate some more 💧

React 18 fixes several issues people often encounter when using SSR with React and Hydration. We’ll look closely at React 18’s changes in the following article.

‘Till next time!

Handling backups

Thu, 23 Jun 2022 00:00:00 GMT

Restoring your private backups

We've all been told to create regular backups of our data. Be it through your friendly neighbourhood IT guy or ads on services like iCloud or Google Cloud.

What is a backup

Yeah, I’m starting with the basics here, as I’ve seen things 😱.

First, what is not a backup; A NAS with a RAID configuration that survives one, two or more disk failures is not a backup. For sure, it is nice to know that your data is not lost once a disk failure occurs, but losing a second disk during the rebuild is common enough, and then all your information is gone anyway.

Also, a service that promises to keep your data safe may not be the right choice, as the company behind it can go bankrupt, be bought or resells your information…

Well, what is a backup then?!?

So what do I consider a legitimate backup that can save your data and sometimes your neck? For me, the following three principles need to apply:

One backup is none; two backups are one.
A backup doesn’t work if you can’t restore it.
Offline is better than online.

One backup is none; two backups are one

I learned early on that having a backup doesn’t guarantee you won’t lose your data. A backup drive wears down as you regularly keep updates and may fail you in an emergency case.

For this reason, computer scientists speak of “one backup is none, two backups are one”, and so forth. You should ideally back up your regular data on one drive (as you probably already do) and then either back up the backup drive or store a copy on an online service to end up with one valid backup.

You do not need to create a backup of a backup drive as often as your computer or mobile devices. Once a month or so is a good start, depending on how important the data is to you. You can also easily automate that process 🥇.

A backup doesn’t work if you can’t restore it

Creating one or two backups is excellent, but have you ever tried to restore one in a non-emergency case? It doesn’t help if you have a superb backup strategy or disaster recovery plan if you’ve never run a test run.

Make sure to test that you can restore your backups regularly. I usually do this every half a year or yearly, depending on the backup type.

For example, I back up my Plex VM about every month or so to a physical drive outside the RAID. The backup itself is not super important to me, as I can manually restore playlists, custom covers and settings, but it’s still a hassle.

So I do yearly tests where I deactivate the Plex VM and spin it up again through a backup. After a while, I use the backup as my new replacement and delete the old one to ensure everything works as expected. If not, I will fix it to make sure it won’t happen again in the future.

Offline is better than online

You may have heard about the many security breaches in the past with various services and backup solutions. So be careful with online backup solutions.

Apple or Google provide simple solutions for everyone to restore their devices or photos, but you don’t necessarily know what they do with your data. Primarily, Google is known for analysing your images and using the data to improve its services. But we do not see what happens in the background ¯_(ツ)_/¯.

As such, always ensure your backups are encrypted and enable all the available protection. That also applies to offline backups such as your physical external hard drive. Ensure someone can’t easily steal it and then access your data.

It is also a good idea to keep backups in different locations. That guarantees a backup is still available even after a physical incident.

That is it

Backups are great, keep your data secure and safe, and you’ll spare yourself a lot of hassle. No one wants to lose their precious memories or documents 🤣.

‘Till next time!

All about the console

Fri, 13 May 2022 00:00:00 GMT

Discovering the console methods

I'm using console.log more than I'd like to admit. Though, many more console methods are available to help in certain situations.

console.assert

The console.assert() method writes an error message to the console if the assertion is false. If the assertion is true, nothing happens.

console.assert('hello' === 'world', 'hello is not the world');
// Assertion failed: hello is not the world

console.assert(typeof 'hello' === 'string', 'hello is not a string');
// will return "undefined" as the assertion does not fail

console.clear

The console.clear() method clears the console if the environment allows it.

console.clear();
// clears the console entries

console.context

The console.context() method lists all the available context methods.

console.context();

// output
Context {dir: ƒ, dirXml: ƒ, table: ƒ, groupEnd: ƒ, clear: ƒ, …}
  assert: ƒ assert()
  clear: ƒ clear()
  count: ƒ count()
  countReset: ƒ countReset()
  debug: ƒ debug()
  dir: ƒ dir()
  dirXml: ƒ dirXml()
  error: ƒ error()
  group: ƒ group()
  groupCollapsed: ƒ groupCollapsed()
  groupEnd: ƒ groupEnd()
  info: ƒ info()
  log: ƒ log()
  profile: ƒ profile()
  profileEnd: ƒ profileEnd()
  table: ƒ table()
  time: ƒ time()
  timeEnd: ƒ timeEnd()
  timeLog: ƒ timeLog()
  timeStamp: ƒ timeStamp()
  trace: ƒ trace()
  warn: ƒ warn()
  [[Prototype]]: Object

console.count & .countReset

The console.count() method logs the number of times that this particular call to count() has been called.

[0, 1, 2].forEach(entry => console.count('increment'));
// increment: 1
// increment: 2
// increment: 3

It will continue to count when running the loop again until console.countReset() is called:

[0, 1, 2].forEach(entry => console.count('increment'));
// increment: 1
// increment: 2
// increment: 3

[0, 1, 2].forEach(entry => console.count('increment'));
// increment: 4
// increment: 5
// increment: 6

console.countReset('increment');

[0, 1, 2].forEach(entry => console.count('increment'));
// increment: 1
// increment: 2
// increment: 3

console.debug

The console.debug() method outputs a message to the web console at the “debug” log level. The message is only displayed to the user if the console is configured to display debug output. In most cases, the log level is configured within the console UI. This log level might correspond to the Debug or Verbose log level.

console.debug({ hello: 'world' }, 'debugging my object');

It behaves similarly to console.log() and console.info().

console.dir & .dirxml

The method console.dir() displays an interactive list of the properties of the specified JavaScript object. The output is presented as a hierarchical listing with disclosure triangles that let you see the contents of child objects.

In other words, console.dir() is the way to see all the properties of a specified JavaScript object in console by which the developer can easily get the properties of the object.

console.dir(console);
// displays all the methods similar to console.context()

The console.dirxml() method displays an interactive tree of the descendant elements of the specified XML/HTML element. If it is impossible to display as an element, the JavaScript Object view is shown instead. The output is presented as a hierarchical listing of expandable nodes that let you see the contents of child nodes.

console.dirxml(document);
// use dirxml to inspect DOM nodes

console.error

The console.error() method outputs an error message to the Web console.

console.error('This is a custom error message');
// displays the message in red to the console

console.group, .groupCollapsed & .groupEnd

The console.group() method creates a new inline group in the Web console log, causing any subsequent console messages to be indented by an additional level, until console.groupEnd() is called.

console.log('A normal log message');
// lets group the next logs in an accordion style
console.group();
console.log('This is now one level deeper');
console.log('This is now one level deeper');
// even further nesting is possible
console.group();
console.log('And another level deeper');

The console.groupEnd() method exits the current inline group in the Web console. See Using groups in the console in the console documentation for details and examples.

console.log('A normal log message');
// lets group the next logs in an accordion style
console.group();
console.log('This is now one level deeper');
console.log('This is now one level deeper');
// clear the group
console.groupEnd();

console.log('A normal log message');

The console.groupCollapsed() method creates a new inline group in the Web Console. Unlike console.group(), however, the new group is created and collapsed. The user will need to use the disclosure button to expand it, revealing the entries created in the group.

console.log('A normal log message');
// creates the group but collapses it
console.groupCollapsed();
console.log('This is now one level deeper');
console.log('This is now one level deeper');
// you can still combine it with a regular group call
// though the parent is still collapsed
console.group();
console.log('And another level deeper');

console.info

The console.info() method outputs an informational message to the Web console. In Firefox, a small “i” icon is displayed next to these items in the Web console’s log.

console.info('This is an info message');
// well, displays the text :)

console.log

The console.log() method outputs a message to the web console. The message may be a single string (with optional substitution values), or it may be any one or more JavaScript objects.

console.log('Some', 'log', 'message');
// you know this one

console.memory

Related to performance.memory that displays additional memory information. Be aware that it is not a method call:

console.memory;

// displays additional memory information
MemoryInfo {totalJSHeapSize: 10000000, usedJSHeapSize: 10000000, jsHeapSizeLimit: 3760000000}
  jsHeapSizeLimit: 3760000000
  totalJSHeapSize: 10000000
  usedJSHeapSize: 10000000
  [[Prototype]]: MemoryInfo

console.profile & console.profileEnd

The console.profile() starts recording a performance profile (for example, the Firefox performance tool).

You can optionally supply an argument to name the profile, enabling you to stop only that profile if multiple profiles are recorded. See console.profileEnd() to see how this argument is interpreted.

Be aware that this feature is non-standard and is not on a standards track. Do not use it on production sites facing the Web: it will not work for every user. There may also be large incompatibilities between implementations and the behaviour may change in the future.

const profileTest = callback => {
  let counter = 0;
  let interval = setInterval(() => {
    if (counter >= 3) {
      callback();
      clearInterval(interval);
    } else {
      console.log('Working...');
      counter++;
    }
  }, 1000);
};

console.profile('profileTest');
profileTest(() => console.profileEnd('profileTest'));

To view the result in Chrome, select the three dots in the Developer Tools and choose More tools > JavaScript Profiler.

console.table

The console.table() method displays tabular data as a table.

This function takes one mandatory argument data, which must be an array or an object, and one additional optional parameter column.

It logs data as a table. Each element in the array (or enumerable property if data is an object) will be a row in the table.

The first column in the table will be labelled (index). If data is an array, then its values will be the array indices. If data is an object, then its values will be the property names. Note that (in Firefox) console.table is limited to displaying 1000 rows (first row is the labeled index).

function Person(firstName, lastName) {
  this.firstName = firstName;
  this.lastName = lastName;
}

console.table(new Person('John', 'Smith'));

Or with multiple entries using arrays:

var john = new Person('John', 'Smith');
var jane = new Person('Jane', 'Doe');
var emily = new Person('Emily', 'Jones');

console.table([john, jane, emily]);

console.time, console.timeEnd, console.timeLog & console.timeStamp

The console.time() method starts a timer you can use to track how long an operation takes. You give each timer a unique name, and may have up to 10,000 timers running on a given page. When you call console.timeEnd() with the same name, the browser will output the time, in milliseconds, that elapsed since the timer was started.

console.time();
// do something very long
Array.from({ length: 10000000 }).map(n => n * 2);
console.timeEnd();
// wait
// default: 4767.52490234375 ms

The console.timeLog() method logs the current value of a timer previously started by calling console.time() to the console.

The console.timeStamp() method adds a single marker to the browser’s Performance or Waterfall tool. This approach lets you correlate a point in your code with the other events recorded in the timeline, such as layout and paint events.

console.trace

The console.trace() method outputs a stack trace to the Web console.

function foo() {
  function bar() {
    console.trace();
  }
  bar();
}

foo();

// traces now by execution:
// bar
// foo
// <anonymous>

console.warn

The console.warn() method outputs a warning message to the Web console.

console.warn('This is your last warning');
// shows an orange marked warning message

That is it

There you have it, all the console methods in a short overview 😎.

‘Till next time!

Using Normalizr

Wed, 27 Apr 2022 00:00:00 GMT

Normalizing API responses

Normalization is required when interacting with multiple API endpoints that have references.

We are using normalizr for that particular job at Divio. Tools such as React Query or rtk-query are great alternatives. Today we would probably use React Query. Yet, the migration effort to switch outweighs the benefits, for now. ¯\_(ツ)_/¯

So, what is the problem?

Let’s have a look at the following two API responses:

// Categories API response
const categoriesApi = [
  { uuid: 'iu6h3wr74vc2bhugglgzmp3hhu', name: 'Movie' },
  { uuid: '4g35twfl3vhu3njuxgbtardjrm', name: 'TV Show' },
  { uuid: 'oiz6zif455h3hctzzhobak5l2y', name: 'Other' },
];

// Movies API response
const moviesApi = [
  {
    uuid: '3l5buhzt7rcbdmhntsrdlxd3h4',
    name: 'Memento',
    category: 'iu6h3wr74vc2bhugglgzmp3hhu',
  },
  {
    uuid: 'zxevaw7jkfbhdiflgfp5lxwrku',
    name: 'Pulp Fiction',
    category: 'iu6h3wr74vc2bhugglgzmp3hhu',
  },
  {
    uuid: 'uexridtrxnb3tls2az7x3eemvu',
    name: 'Spirited Away',
    category: 'oiz6zif455h3hctzzhobak5l2y',
  },
  {
    uuid: 'x3eemv2az7x3etsrdlxd3h4dda',
    name: 'Breaking Bad',
    category: '4g35twfl3vhu3njuxgbtardjrm',
  },
];

The categoriesApi provides a list of categories. It uses the uuid to mark each entry with a universally unique identifier.
The moviesApi follows a similar pattern but holds a reference to the category uuid. This approach allows us to separate the data and avoid repeating unnecessary information, making the API slim and fast.

This response is not what we desire when passing the data to a component. We would want to have the category inlined inside the movie like so:

{
  "uuid": "x3eemv2az7x3etsrdlxd3h4dda",
  "name": "Breaking Bad",
  "category": {
    "uuid": "4g35twfl3vhu3njuxgbtardjrm",
    "name": "TV Show",
  }
}

This step is where normalizr comes into play. Normalizr takes the API response and stores them as entities internally, and links them together through a schema. Let’s have a look at how that would work.

Setting everything up

We will be using our two API responses categoriesApi and moviesApi, as references moving forward. Now, create a JavaScript project through npm init and install normalizr through npm install normalizr --save.

We are going to import normalize, denormalize and schema from normalizr. Add the following line on the top:

import { normalize, denormalize, schema } from 'normalizr';

Next, we need to let normalizr know about our data structure. This is where our entities will be stored and how normalizr knows what objects are linked. To define our entities, we call schema.Entity:

const category = new schema.Entity(
  // here, we define the name of the entity
  'categories',
  {},
  // the idAttribute is used to identify references
  { idAttribute: 'uuid' }
);
const movie = new schema.Entity(
  'movies',
  {
    // here, we let normalizr know that movies
    // has a reference to another category
    category: category,
  },
  { idAttribute: 'uuid' }
);

These definitions allow us to normalize the data from our API endpoints when calling normalize(). Normalizr requires a schema that defines what the data should look like:

const dataSchema = {
  categories: [category],
  movies: [movie],
};

Now, we can normalize the data by using normalize():

const normalized = normalize(
  {
    categories: categoriesApi,
    movies: moviesApi,
  },
  dataSchema
);

The output of normalized should look something like:

{
  "entities": {
    "categories": {
      "iu6h3wr74vc2bhugglgzmp3hhu": {
        "uuid": "iu6h3wr74vc2bhugglgzmp3hhu",
        "name": "Movie",
      },
      ...
    },
    "movies": {
      "x3eemv2az7x3etsrdlxd3h4dda": {
        "uuid": "x3eemv2az7x3etsrdlxd3h4dda",
        "name": "Breaking Bad",
        "category": "4g35twfl3vhu3njuxgbtardjrm"
      }
    }
  },
  ...
}

We do not have the data in our desired format yet. To achieve this, we need to denormalize the data. This step is where the entities will be linked together in a more usable format. Let’s denormalize() our data:

const denormalized = denormalize(
  {
    categories: normalized.result.categories,
    movies: normalized.result.movies,
  },
  dataSchema,
  normalized.entities
);

Denormalizing is finally bringing the data together in a format where we can process the data:

{
  "categories": [
    {
      "uuid": "iu6h3wr74vc2bhugglgzmp3hhu",
      "name": "Movie",
    },
    ...
  ],
  "movies": [
    {
      "uuid": "3l5buhzt7rcbdmhntsrdlxd3h4",
      "name": "Memento",
      "category": {
        "uuid": "iu6h3wr74vc2bhugglgzmp3hhu",
        "name": "Movie",
      }
    },
    ...
  ]
}

We could also remove categories: [category] from the dataSchema if we do not care about them individually. The categories will be inlined inside the movie object itself. However, that depends on how you process your data.

Finally, this is how everything in the file should look like:

import { normalize, denormalize, schema } from 'normalizr';

const categoriesApi = [
  { uuid: 'iu6h3wr74vc2bhugglgzmp3hhu', name: 'Movie' },
  { uuid: '4g35twfl3vhu3njuxgbtardjrm', name: 'TV Show' },
  { uuid: 'oiz6zif455h3hctzzhobak5l2y', name: 'Other' },
];

const moviesApi = [
  {
    uuid: '3l5buhzt7rcbdmhntsrdlxd3h4',
    name: 'Memento',
    category: 'iu6h3wr74vc2bhugglgzmp3hhu',
  },
  {
    uuid: 'zxevaw7jkfbhdiflgfp5lxwrku',
    name: 'Pulp Fiction',
    category: 'iu6h3wr74vc2bhugglgzmp3hhu',
  },
  {
    uuid: 'uexridtrxnb3tls2az7x3eemvu',
    name: 'Spirited Away',
    category: 'oiz6zif455h3hctzzhobak5l2y',
  },
  {
    uuid: 'x3eemv2az7x3etsrdlxd3h4dda',
    name: 'Breaking Bad',
    category: '4g35twfl3vhu3njuxgbtardjrm',
  },
];

const category = new schema.Entity('categories', {}, { idAttribute: 'uuid' });
const movie = new schema.Entity(
  'movies',
  {
    category: category,
  },
  { idAttribute: 'uuid' }
);

const dataSchema = {
  movies: [movie],
};

const normalized = normalize(
  {
    movies: moviesApi,
  },
  dataSchema
);

console.log('entity store', normalized);

const denormalized = denormalize(
  {
    movies: normalized.result.movies,
  },
  dataSchema,
  normalized.entities
);

console.log('denormalized', denormalized);

This short tutorial should explain the basics of normalization when using different API endpoints that reference each other. Just be sure to avoid circular references 🤯.

‘Till next time!

Monitoring with Checkmk

Sat, 19 Mar 2022 00:00:00 GMT

Identify issues or failures real-time

I'm a data hoarder; that's nothing new to people who know me. This year around, I drove my passion a little bit further and streamlined my setup. The following articles will be a multi-part series introducing various areas such as:

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Why monitoring at home

It is good practice to monitor your infrastructure. You want to know when, where and why an issue occurred and ideally fix it quickly. At Divio we go even further and automate the process of resolving these issues as much as possible, so little human interaction is needed.

However, at home, that might be a bit overkill. Nonetheless, I still want to know if something is not working as it should. Most notably:

Diskspace: react before space runs out and avoid outages
Heat: ensure the hardware is appropriately cooled for longevity
Usage: how much CPU or memory is utilised
Data: is there any packet loss

Monitoring using Checkmk

Checkmk interface

There are several free, open-source monitoring services out there. Most notably Nagios, Zabbix and Prometheus/Grafana. However, I settled for another popular one, Checkmk. Checkmk has excellent docs, is easy to install, and the raw edition is entirely open-source. Connecting the infrastructure is also very straightforward by using Checkmk’s agents. To get started, make sure you have a separate Ubuntu VM instance running where we will install Checkmk on top of it.

The Checkmk documentation has a pretty good Beginner’s guide, so I recommend reading through it to understand the basic principles. I will focus on the installation and configuration only in this article.

Installation

To start the installation, download the correct package for your system from the product selection. I will use the Checkmk Free Edition to get additional features to play with that the raw edition doesn’t support. The free edition is limited to one instance and a maximum of 25 hosts, which is not a problem for me. However, if you want to monitor more hosts, progress by installing the raw edition.

Download the package and install it on your server by moving it via scp or directly:

wget https://download.checkmk.com/checkmk/2.0.0p21/check-mk-free-2.0.0p21_0.focal_amd64.deb

apt install openssh-server
apt install dpkg-sig
wget https://download.checkmk.com/checkmk/Check_MK-pubkey.gpg
gpg --keyserver keys.openpgp.org --recv-keys 434DAC48C4503261
gpg --armor --export 434DAC48C4503261 > Check_MK-pubkey.gpg
gpg --import Check_MK-pubkey.gpg

dpkg-sig --verify check-mk-free-2.0.0p21_0.focal_amd64.deb

sudo apt install ./check-mk-free-2.0.0p21_0.focal_amd64.deb

You can test if the installation was successful by running omd version. It should show something like OMD - Open Monitoring Distribution Version 2.0.0p21.cre.

Next, you want to create a site. Follow the subsequent step-by-step instruction to set up a site. In short:

omd create homelab
omd status

Mind the instructions shown to you, and then start or stop your site by running omd start or omd stop, respectively.

Finally, open the web interface http://x.x.x.x/homelab and login with the username cmkadmin and the password you set up through the site creation process.

Agents

Now, we need to add hosts to monitor. To do this, head to Setup > Agents (Windows, Linux, Solaris, AIX). The easiest is always to register the monitoring server itself. This step is also recommended by Checkmk’s documentation. In our case, we download the “Linux / Deb” package. Rename that file on your computer to “agent.deb” and copy it over via scp. Then install it:

scp ~/Downloads/agent.deb user@x.x.x.x:/home/organizr/agent.deb
sudo dpkg -i ./agent.deb

That is all you have to do on any system you want to add. Some systems do not require an agent if they have built-in SNMP support. Also, have a look at the available agents. You can register ESXi and vSphere by using VMware ESX via vSphere and track the VMs altogether.

Configuration

To add a new host, head to Setup > Hosts and click on Add host. The only option to configure is the IPv4 address. Alternatively, you can also set up a hostname and use it instead. Click on Save & go to service configuration and add all discovered services by clicking on Fix all. Next, on the top right, click on x changes followed by Activate on selected sites. This action will activate your first monitored host.

Continue this with all other hosts until all of them are added. And then start figuring out what goes wrong and why by googling 🤣.

So what’s next

Well, that’s it 🤩. This is the last article I intended to add to this series. Though, there might be independent articles related to my homelab in the future.

‘Till next time!

Configuring Calibre, Organizr and more

Sat, 26 Feb 2022 00:00:00 GMT

Additional services to complement any homelab

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

More services for our toolbox

With Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar we cover a wide area of DVR systems to manage our media. However, many additional tools are available to make your life even more convenient. This section will cover some I find very helpful:

Calibre Web and Ubooquity to manage eBooks and PDFs
Tautulli and Ombi to gather stats and manage requests
Organizr to have a friendly landing page for all the services

Remember that we are still using a Ubuntu VM, as with the previous services.

Managing eBooks and PDF

Calibre web interface

The state of eBook and PDF management isn’t there yet. It would be great if Plex takes the “cord-cutting” to the next level by adding support, as it has been one of the most requested features on the platform.

Calibre is still the best option to go to until Readarr reaches maturity. The app itself works on any platform. Install it and designate a folder on your NAS to store all the eBooks or PDFs. Calibre creates a metadata.db and metadata_pre_restore.db file that hosts all the data necessary for your library. This process makes it easy to use the same library on different systems. It also has excellent plugin support to extend its functionality to, for example, also manage your Manga/Comic library.

In addition to the app itself you can also install Calibre Web. It provides an online interface where you can manage your library from anywhere you want. The installation is also straightforward:

sudo apt install git python3.7 python3-setuptools
sudo git clone https://github.com/janeczku/calibre-web.git /opt/CalibreWeb
cd /opt/CalibreWeb
python3 cps.py

Next, open up http://x.x.x.x:8083/config to check that everything is working. Complete the installation instructions and follow up by ensuring that the service starts automatically. Run sudo vim /etc/systemd/system/calibreweb.service and add the following content:

[Unit]
Description=Calibre-Web

[Service]
Type=simple
User=ddl
ExecStart=/usr/bin/python3 /opt/CalibreWeb/cps.py
WorkingDirectory=/opt/CalibreWeb

[Install]
WantedBy=multi-user.target

Adapt the User entry to your system’s user. Finally, enable the service to start after a restart:

sudo systemctl daemon-reload
sudo systemctl enable calibreweb
sudo systemctl start calibreweb

Streaming eBooks and PDF

Ubooquity interface

Audiobooks technically work on Plex, but better tools are around, such as Audiobookshelf or Booksonic for streaming. I use Plex to browse and load them manually into Apple’s books app for synchronisation.

Calibre isn’t the best choice either to stream Comics, eBooks and PDFs. Here is where Ubooquity shines. It allows establishing OPDS feeds to stream its content to apps such as Chunky or tiReader.

The installation is more involving, as you need to have Java installed on your VM. The setup process looks something like this:

sudo apt install default-jre
sudo mkdir /opt/Ubooquity
cd /opt/Ubooquity
sudo wget http://vaemendis.net/ubooquity/service/download.php
sudo mv download.php Ubooquity-2.1.2.zip
unzip Ubooquity-2.1.2.zip

Next, configure the service using sudo vim /etc/systemd/system/ubooquity.service and add the following content:

[Unit]
Description=Ubooquity Daemon
After=network.target

[Service]
User=ddl
Group=ddl

Type=simple
ExecStart=/usr/bin/java -jar /opt/Ubooquity/Ubooquity.jar --workdir /opt/Ubooquity --libraryport 8097 --adminport 8098 --remoteadmin --headless
TimeoutStopSec=20
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Again ensure that you use the correct User and Group, and optionally change the libraryport and adminport to your liking. Then fix the permissions and enable the service:

sudo chmod 770 /opt/Ubooquity -R
sudo chown ddl:ddl /opt/Ubooquity -R

sudo systemctl daemon-reload
sudo systemctl enable ubooquity
sudo systemctl start ubooquity

The admin is simple to understand and setup. Make sure to “Enable OPDS feed” from the advanced section.

Bonus: I’ve also created a nice Plex based theme that you can install by running sudo git clone https://github.com/FinalAngel/plextheme.git -b master /opt/Ubooquity/themes/plextheme. Once installed, go to the admin and select it from the “Generals” tab.

Gather statistics and manage requests

Ombi interface

Plex added its Dashboard showing basic statistics about usage, play history, or top played media a while ago. However, Tautulli offers deeper analytics and more graphs and filters. You can install it by executing:

sudo apt install git python3.7 python3-setuptools
cd /opt
sudo git clone https://github.com/Tautulli/Tautulli.git
sudo addgroup tautulli
sudo adduser --system --no-create-home tautulli --ingroup tautulli
sudo chown -R tautulli:tautulli /opt/Tautulli
sudo cp /opt/Tautulli/init-scripts/init.systemd /lib/systemd/system/tautulli.service
sudo systemctl daemon-reload
sudo systemctl enable tautulli.service
sudo systemctl start tautulli.service

Next, open up http://x.x.x.x:8181 and configure Tautulli to connect to your Plex server. The Tautulli login will synchronise with your Plex login.

Finally, if you want to manage requests from friends or family, use WhatsApp or talk to them 🤣. Alternatively, install Ombi and manage all requests centrally in one place. It can also be easily installed by running:

echo "deb https://apt.ombi.app/master jessie main" | sudo tee /etc/apt/sources.list.d/ombi.list
curl -sSL https://apt.ombi.app/pub.key | sudo apt-key add -
sudo apt update && sudo apt install ombi

Ombi has been installed successfully and will automatically start during boot if no errors are shown. It should be reachable by visiting http://x.x.x.x:5000. Open the page and connect it to your Plex server as well. Finally, begin inviting users to allow them to make requests.

Managing your bookmarks

Organizr interface

Last but not least, we need a place to reach all our tools and services easily. Organizr is great for this. It adds a nice interface to reach all of them. Host it on a separate Ubuntu VM instance; it interfered too much with my existing VMs. To install, run the following commands:

sudo git clone https://github.com/elmerfdz/OrganizrInstaller /opt/OrganizrInstaller
cd /opt/OrganizrInstaller/ubuntu/oui
sudo bash ou_installer.sh

Once installed, open http://x.x.x.x using the IP from your VM and start adding your services 🍺.

Honourable mentions

HakuNeko is fun to manage mangas
Prowlarr to manage indexers
The Servarr wiki for a lot more services

So what’s next

We’re ready for the final chapter. We will set up monitoring for all of our “critical infrastructures” there.

‘Till next time!

Looking back at 2021

Thu, 20 Jan 2022 00:00:00 GMT

Looking back at 2021

2021 hasn't been short on events either. We continued to improve the user experience at work and added new functionality. Studies progressed into the subsequent semesters, and life is treating me well.

I continued journaling my experiences at work, school and personal life. As already in 2020, for my sanity and everyone else’s; as bullet points:

Work

We made several improvements with our documentation presentation and organisation.
The continued effort moving away from Django templates to React delivers remarkable speed, efficiency and user experience.
We are doing more and more pair coding and pair reviews, which is excellent for the product’s quality and knowledge exchange.
We also had the chance to clean up many loose ends and improve on the overall build and quality of life stuff™.
Several improvements to the marketing sites have been made, which are lovely.
There are also new features available, like shortcuts on the control panel; check it out.

Study

Completed two semesters and continued with the third.
Improved on my Anki and Notion learning methods, adapted where needed. The Notion app is fantastic.
Though I’m overall not very happy about the study/school itself, more on that probably in 2022.
Again taught Modular Frontend Development at the Ostschweizer Fachhochschule (OST) 🥳

Personal

I read about 50 books again this year, though several have been non-fiction for school 😅.
I started playing Gloomhaven (Boardgame), which is very enjoyable.
I also played some computer games again, including Cyberpunk 2077, Final Fantasy VII Remake and FFXIV.
I gambled way too much with GME and the sorts with little gains 🤪.
I continued playing the guitar, though a bit less due to studies.
I installed a complete home lab in my cellar and started writing articles about it.

Looking forward

School consumed most of my time this year, and as such several other projects have been delayed. However, I’m sure 2022 will bring more changes, surprises and roadblocks as I go along.

‘Till next time!

Configuring Sonarr, Radarr and more

Wed, 29 Dec 2021 00:00:00 GMT

DVR services to manage your digital content

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Before we begin

Continuing from Download management, we’ll focus on setting up:

Sonarr to manage tv-shows
Radarr to manage movies
Lidarr to manage music
Readarr to manage books
Bazarr to manage subtitles
Mylar to manage comics

Certain services will use additional tools to organise the content. For example, with Radarr, we will manage movies through IMDB watchlists. Radarr will pull and synchronise the list and automatically download the requested movies in this scenario.

I recommend creating one Ubuntu VM with about 30 GB of disk space to host the services below. The downloads will be stored and managed on the NAS, so the VM doesn’t need much space.

There is also a separate section about Migrating from an existing instance to assist in these cases.

Installing Sonarr

Sonarr interface

Let’s get started by installing Sonarr first. Go to the Sonarr website and follow the instructions for Linux. As of this writing, you need to run the following sequence of commands:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 2009837CBFFD68F45BC180471F4F90DE2A9B4BF8
echo "deb https://apt.sonarr.tv/ubuntu focal main" | sudo tee /etc/apt/sources.list.d/sonarr.list
sudo apt update
sudo apt install sonarr

You may be asked to provide a username during the installation process. If so, choose the same as you have for the VM.

Once the installation finishes, check the status by running sudo systemctl status sonarr. If there are no errors, open a browser and enter http://x.x.x.x:8989 to begin the initial configuration. Replace the x.x.x.x with the IP address of your VM. If you are unsure, just run ip a to get the current IP address. If the service doesn’t start automatically, run the following commands:

sudo systemctl enable sonarr
sudo systemctl start sonarr

Make sure to clean up downloaded files from your server after the installation (usually inside ~/). This process will apply to all the services.

See the bonus section about Configuring the services, for further configuration details, as the configuration steps are very similar.

I do not use any additional tools to manage tv-shows. There are some available such as trakt.tv. However, I found it easy to manage shows via the Sonarr user interface.

Installing Radarr

Radarr interface

Next up is Radarr. Go to the Radarr website and follow the instructions for Linux. As of this writing, you need to run the following sequence of commands:

sudo apt update
sudo apt install curl mediainfo

curl -L -O $( curl -s https://api.github.com/repos/Radarr/Radarr/releases | grep linux.tar.gz | grep browser_download_url | head -1 | cut -d \" -f 4 )
tar -xvzf Radarr.master.*.linux.tar.gz
sudo mv Radarr /opt

sudo chmod -R 770 /opt/Radarr
sudo chown -R $user:$user /opt/Radarr

Alternatively, have a look at the community installation instructions.

You may need to adapt the User= and Group= entries inside /etc/systemd/system/radarr.service to match your VM’s username and group.

Finally, run the following commands to start Radarr:

sudo systemctl enable radarr
sudo systemctl start radarr

The last line should inform you that Radarr has been started and is running on http://x.x.x.x:7878. Replace x.x.x.x with the IP of your VM and open it up in a browser.

Managing movies through the Radarr user interface is also very simple. However, if you use IMDB often, you may be interested in the IMDB watchlist integration.

Whenever you see a trailer or an exciting movie, add it to the watchlist and Radarr will automatically add and download it.

Installing Lidarr

Lidarr interface

Let’s continue by installing Lidarr next. Go to the Lidarr website and follow the instructions for Linux. As of this writing, you need to run the following sequence of commands:

sudo apt update
sudo apt install curl mediainfo

curl -L -O $( curl -s https://api.github.com/repos/Lidarr/Lidarr/releases | grep linux.tar.gz | grep browser_download_url | head -1 | cut -d \" -f 4 )
tar -xvzf Lidarr.master.*.linux.tar.gz
sudo mv Lidarr /opt

sudo chmod -R 770 /opt/Lidarr
sudo chown -R $user:$user /opt/Lidarr

Alternatively, have a look at the community installation instructions.

You may need to adapt the User= and Group= entries inside /etc/systemd/system/lidarr.service to match your VM’s username and group.

Finally, run the following commands to start Lidarr:

sudo systemctl enable lidarr
sudo systemctl start lidarr

The last line should inform you that Lidarr has been started and is running on http://x.x.x.x:8686. Replace x.x.x.x with the IP of your VM and open it up in a browser.

Lidarr can import existing artists and albums very easily. The initial setup requires some work, though. I do not use any additional services to manage my library. I buy the album and then load it into Lidarr.

Installing Readarr

Readarr interface

Up next is Readarr. There is no packaged install, so I followed this write-up on r/DataHoarder and built it from source.

Be warned, though, Readarr is still in its early stages, and it is more difficult to install than any of the other services. So here’s what you need to do:

curl -L -O https://github.com/Readarr/Readarr/archive/develop.tar.gz
tar -xvzf develop.tar.gz
cd Readarr-develop
# install https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu
# install https://classic.yarnpkg.com/en/docs/install/#mac-stable

bash build.sh
yarn build

sudo mkdir /opt/Readarr
cd ~/Readarr-develop/_output/net462/linux-x64
sudo rsync -av --progress * /opt/Readarr/

cd ~/Readarr-develop/_output
sudo rsync -av --progress UI /opt/Readarr/

sudo chmod -R 770 /opt/Readarr
sudo chown -R $user:$user /opt/Readarr -R

Once all commands have run successfully, start the service by running sudo /usr/bin/mono --debug /opt/Readarr/Readarr.exe -nobrowser.

This process should start Readarr, and you should be able to open the web interface through http://x.x.x.x:8787. Replace x.x.x.x with the IP of your VM and open it up in a browser.

To create a startup file run sudo vim /etc/systemd/system/readarr.service, and add the following contents to the file:

[Unit]
Description=Readarr Daemon
After=network.target

[Service]
User=ddl
Group=ddl

Type=simple
ExecStart=/usr/bin/mono --debug /opt/Readarr/Readarr.exe -nobrowser
TimeoutStopSec=20
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Make sure to adapt the User= and Group= entries accordingly.

Alternatively, have a look at the community installation instructions.

Finally, run the following commands to start Readarr:

sudo systemctl enable readarr
sudo systemctl start readarr

I manage my books through Calibre as Readarr is still in its infancy. So for now, let’s sit back and watch its development 😅.

Installing Bazarr

Bazarr interface

Let’s tackle Bazarr now! Go to the Bazarr wiki and follow the instructions for Linux. As of this writing, you need to run the following sequence of commands:

apt-get install python3-dev python3-pip python3-distutils

curl -L -O https://github.com/morpheus65535/bazarr/releases/latest/download/bazarr.zip
unzip bazarr.zip -d Bazarr
sudo mv Bazarr /opt

sudo chmod -R 770 /opt/Bazarr
sudo chown -R $user:$user /opt/Bazarr

pip3 install -r /opt/Bazarr/requirements.txt
python3 bazarr.py

The last line should inform you that Bazarr has been started and is running on http://x.x.x.x:6767. Replace x.x.x.x with the IP of your VM and open it up in a browser.

There’s not much to manage here. Bazarr automatically downloads the requested subtitles, and it just works.

Installing Mylar

Mylar interface

Let’s wrap this up by installing Mylar! The installation process is also a bit more involving but still manageable. Run the following commands to get started:

sudo git clone https://github.com/mylar3/mylar3 -b master /opt/Mylar

sudo chmod 770 /opt/Mylar -R
sudo chown $user:$user /opt/Mylar -R

pip3 install -r /opt/Mylar/requirements.txt

Next, we want to create the systemd process, do this by running sudo cp /opt/Mylar/init-scripts/systemd/mylar.service /etc/systemd/system/mylar.service followed by sudo vim /etc/systemd/system/mylar.service. Make sure the content of the file looks something like this:

[Unit]
Description=Mylar3 Daemon
After=network-online.target

[Service]
User=ddl
Group=ddl

ExecStart=/usr/bin/python3 /opt/Mylar/Mylar.py -v --daemon --nolaunch --quiet
GuessMainPID=no
Type=forking
Restart=on-failure

[Install]
WantedBy=multi-user.target

Make sure to adapt the User= and Group= entries accordingly.

Finally, run the following commands to start Mylar:

sudo systemctl enable mylar
sudo systemctl start mylar

The last line should inform you that Mylar has been started and is running on http://x.x.x.x:8090. Replace x.x.x.x with the IP of your VM and open it up in a browser.

Mylar does a decent job to manage your favourite comics or manga. I use the Chunky Comic Reader and Ubooquity to read them on my devices. We’ll get into those extra services in the next blog post.

Bonus: Configuring the services

Changing the default port for each service once the setup is completed to avoid clashes is recommended. Choose a port that suits you and increment each time, e.g. 8091, 8092 and so on.

Additionally, choose a username and password to protect access to the services.

Finally, the most critical part is configuring the connection to SABnzbd under Download Clients. You should have no trouble adding all the necessary information. Make sure to add the appropriate Category that we defined in the previous article. Use the Test button to check if everything is correctly configured.

You also need to add your Indexers. There are plenty of presets to choose from. Pick the one that fits your indexers and adapt them accordingly or have a look at Prowlarr.

Other than that, I tend to leave the settings at their default values. You may want to choose different naming schemes, better qualities and profiles. I advise you to play around with the settings and then manually sync them across the services.

Bonus: Migrating from an existing instance

If you have an existing installation backup the <service>.db and copy it over to the new installation directory. The default directory is normally /var/lib/<service>/ or /opt/<service>/. My migration commands looked something like this with Sonarr:

# do this on both instances
sudo systemctl stop sonarr
# copy over the database
sudo tar cvfz sonarr.tar.gz nzbdrone/
scp sonarr.tar.gz username@x.x.x.x:~/
tar xvf sonarr.tar.gz
mv sonarr.db /var/lib/sonarr

Simply repeat the process for the individual services.

So what’s next

We’re ready for the next chapter with the services and download managers set up. We will set up additional services that can help manage our ever-growing digital library even more.

‘Till next time!

Setting up a VPN server using WireGuard

Sat, 27 Nov 2021 00:00:00 GMT

Understanding the VPN options

Accessing network resources through various open ports on a firewall isn't the most secure option. It's safer to create a VPN tunnel and block other traffic from your network.

Setting up a VPN server isn’t always that easy. If your router doesn’t support VPN out of the box, you may need to use third-party firmware or host a VPN server yourself.

Creating a VPN server, however, has its own set of problems. Both Windows and Apple offer VPN services by themselves. These aren’t the most secure options around and difficult to get working right.

OpenVPN, ProtonVPN and the likes offer you friendly services to get started, but I wanted to host my own server to have complete control without external influence.

Hello WireGuard

This is where WireGuard comes in. It works on Windows, macOS, Linux, iOS and Android, covering most use cases.

It is built on modern cryptography (Curve25519, ChaCha20, Poly1305), speaks UDP, and it is small enough that you can read the whole thing. It is also incredibly fast, which is specifically not the case with some of the alternatives.

WireGuard knows the concept of servers and clients. The server is where you set up your VPN to connect to, that forwards the traffic to your internal network. Clients connect to the server. For this to work, the server needs to know the public keys and IPs of the peers (clients) that want to connect, and each client needs the server’s location and public key. This approach allows for a very secure connection.

The following sections guide you through a step-by-step guide on setting up the server and a client example using macOS.

Configuring the server

Consult the official installation instructions for in-depth information and configuration options.

I set up my VPN server using a virtualised Ubuntu server 20.04 instance. Any other OS will do; just consult the setup instructions for your preferred OS.

After the initial installation and security updates, run the following commands to install WireGuard:

sudo -i
apt install wireguard

Check that there are no error messages or warnings. If everything ran smoothly, generate a public and private key by running:

cd /etc/wireguard/
umask 077; wg genkey | tee privatekey | wg pubkey > publickey
cat publickey
cat privatekey

The last two commands will output the public and private keys (in that order). You will need these for future references.

Next, create a WireGuard configuration file using sudo vim /etc/wireguard/wg0.conf. “wg0” will be used as a network interface name to establish the connection.

That’s the intelligent thing about WireGuard. It presents itself as an ordinary network interface, so all the usual routing and firewall tools keep working.

Add the following content to that file:

# VPN Server
[Interface]
PrivateKey = {the private key shown earlier}
Address = 192.168.x.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens160 -j MASQUERADE

# macOS
[Peer]
PublicKey = {to be added later}
AllowedIPs = {to be added later}

Important replace the following entries:

192.168.x.1/24 with your desired IP. That should be a different IP subnet from your internal network. It will be used to connect to and redirect traffic. E.g. if you use 10.0.0.0 for your internal network, choose something like 10.0.10.1.
ens160 replace this with your primary network interface. It is the interface that’s connected to your internal network.
51820 you can choose another port. When doing so, adapt it to other places as well. I will use the standard port in this example.

PostUp ensures that the VPN server automatically forwards the traffic through NAT into your network once the service is started. PostDown accordingly resets the configuration.

Next, if you have a firewall setup, run the following commands:

sudo ufw allow 22/tcp
sudo ufw allow 51820/udp
sudo ufw status

Let’s see if everything worked correctly, start the service and check that everything is working as expected by running:

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0
sudo systemctl status wg-quick@wg0
sudo wg
sudo ip a show wg0

Configuring a client

Once the server is set up, we need clients that connect to it. Clients have their WireGuard service configured with private and public keys. They can only link to the server if the server is aware that they exist. As such, we need to create a client and add the information to the server.

In our example, I’ll use a macOS computer to connect to my VPN server. Go to the WireGuard installation section and choose the software for your OS.

Open the application and click on the “+” to “Add an empty tunnel”. Choose a name and create the following configuration:

[Interface]
PrivateKey = {should already be there}
Address = 192.168.x.2/32
DNS = 192.168.1.1

[Peer]
PublicKey = {public key from the server}
AllowedIPs = 0.0.0.0/0
Endpoint = server-ip-address:51820
PersistentKeepalive = 25

Important replace the following entries:

192.168.x.2/32 use here the same IP as configured on the server. Just increment the host part 1. Further clients will keep increasing the IP host, e.g. 192.168.x.3/32.
DNS = make sure you add there the default DNS address of your router. This step ensures that the internet will continue to work when establishing a VPN connection.
server-ip-address replace this with the public IP address of your VPN server. DynDNS entries work here as well.

Once that configuration is done, go back to the servers configuration file and adapt the following information:

# macOS
[Peer]
PublicKey = {public key from the client}
AllowedIPs = 192.168.x.2/32

The allowed IP will be the same configured on the client. Also, copy and paste the client public key into the configuration.

Once done, configure port forwarding and activate the connection on the client.

Configure port forwarding

Check the manual for your router and enable port forwarding for 51820. This configuration will allow connections from the internet to your VPN server.

Save all configuration files and restart WireGuard on the server with sudo systemctl restart wg-quick@wg0.

Finally, go to your client and activate the connection. Congratulations, you have now access to your network through a subscription-free service you own yourself ;)

‘Till next time!

Configuring SABnzbd and Transmission

Sun, 17 Oct 2021 00:00:00 GMT

Manage your downloads

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

What we need

Continuing from Media streaming, we’ll focus on setting up SABnzbd and Transmission for our download management.

This step is needed before we install and configure Sonarr, Radarr, Lidarr and the likes.

Installing SABnzbd

Create an Ubuntu VM with about 30 GB of disk space. Our primary download directory will be on the NAS, so the VM doesn’t need much space.

A section is available that demonstrates how to set up the mounts on Ubuntu to access the NAS volumes.

Let’s get started! Inside your VM run the following command:

sudo apt install sabnzbdplus

Next, check the status running sudo systemctl status sabnzbdplus. If everything appears fine shut the service down by running sudo systemctl stop sabnzbdplus.

Then, edit the configuration file by executing sudo vim /etc/default/sabnzbdplus and change the content to:

USER={user} (replace {user} with the Ubuntu user)
HOST=0.0.0.0 (important to allow external access)
PORT=8089 (or to any port you like)

Save the file and restart the service by running sudo systemctl start sabnzbdplus.

Finally, open SABnzbd using http://x.x.x.x:PORT. Replace x.x.x.x with your IP and PORT with the configured port.

Configuring SABnzbd

Once you open the above URL you’ll be welcomed by the SABnzbd configuration wizard.

Choose a language for your system and then add the server details for your Usenet provider. I use Supernews, which is pretty good and cheap. Enable SSL if your provider supports it.

Server configuration can also be done later in the Servers section by choosing to Add Server. Make sure to test the connection.

Continue with the completed and temporary folders for now. These will be created in ~/downloads. We will clean this up later when configuring the remaining settings.

Head over to general and adapt the following settings:

Activate Advanced Settings
SABnzbd Username: Choose a username
SABnzbd Password: Choose a password
External internet access: Full Web interface - Only external access requires login

Hit Save Changes. Next, head over to User Folders. Create a downloads/ folder on your NAS and then set:

Temporary Download Folder: /path-to-nas/downloads/incomplete
Completed Download Folder: /path-to-nas/downloads/complete
Watched Folder: /path-to-nas/downloads/watching
Permissions for completed downloads: set to 777

Of course, you can also choose a different name instead of downloads; adapt the paths accordingly. However, make sure that the corresponding folder has full access set:

chmod 777 /path-to-nas/downloads/

It may also help to create the three folders (incomplete, complete, watching) in the downloads/ directory and to provide the same access rights. Leave the rest and hit Save Changes.

Next, browse to Categories. In categories, you want to add or modify the following rows:

The SABnzbd Categories settings with the rows to add or modify

Finally, head to the switches section and change the following settings:

Direct Unpack: enable
Replace dots in folder name: enable
Enable Indexer Integration: enable and add API key

That’s it, SABnzbd is configured 😊

Installing Transmission

Besides Usenet, you can also set up a torrent downloader of your choice. This section focuses on Transmission.

I recommend following the installation from the community. In summary:

sudo add-apt-repository ppa:transmissionbt/ppa
sudo apt-get update
sudo apt-get install transmission-cli transmission-common transmission-daemon
sudo service transmission-daemon stop

Next, follow the configuration steps for:

/var/lib/transmission-daemon/info/settings.json and
/home/{USER}/.config/transmission-daemon/

Similar to SABnzbd, you can create a torrents folder under downloads and set some access rights:

sudo usermod -a -G debian-transmission {YOUR_USER}
sudo chgrp -R debian-transmission /media/nas/downloads/torrents/
sudo chmod -R 777 /media/nas/downloads/torrents/

Additional permissions may need to be adapted:

sudo usermod -a -G debian-transmission {USER}

sudo systemctl stop transmission-daemon
sudo chown -R {USER}:{GROUP} /etc/transmission-daemon/
sudo chown -R {USER}:{GROUP} /var/lib/transmission-daemon/
sudo systemctl start transmission-daemon

netstat -an | grep LISTEN

That’s it, transmission should be up and running on:

http://x.x.x.x:PORT/transmission/web/

Extra: Setting up mounts

QNAP supports several protocols though I chose to use NFS as it worked best for my use case.

First, folders need to be created to mount the individual folders too. In my case, I used /media/ and added the subsequent sub-folders inside it.

Next, install the requirements into Ubuntu:

sudo apt install network-manager
sudo apt install nfs-common

Now, we can check if a connection can be established like:

sudo mount -t nfs x.x.x.x:/server/folder /media/photos

and replace x.x.x.x with the IP address of the storage server. If all works prepare the /etc/fstab file as follows:

x.x.x.x:/server/folder1 /media/folder1 nfs _netdev,auto,x-systemd.automount 0 0
x.x.x.x:/server/folder2 /media/folder2 nfs _netdev,auto,x-systemd.automount 0 0
x.x.x.x:/server/folder3 /media/folder3 nfs _netdev,auto,x-systemd.automount 0 0

Replace x.x.x.x again with the IP address of the storage server and map the respective folders to their counterpart on Ubuntu. The additional options ensure that the folders get automatically mounted after a restart and wait for the server to be available.

What’s next

We have our download managers set up. Next, we will configure the DVR recorders 🥳

‘Till next time!

An alcohol-free weekend

Thu, 30 Sep 2021 00:00:00 GMT

A review of different brands

I've been moving away from alcohol-based beverages to alcohol-free variants for a while now. The main reason wasn't based on health. I just had it with the after-effects from drinking. 🤣

You can enjoy drinking at a party without alcohol. This epiphany took me a while to realise. The cultural environment and peer pressure I lived in forces almost any young adult into a sip of beer, wine or stronger shi🤬. Of course, you can refuse and be abstinent, but that’s difficult when you are sixteen years old (the legal drinking age in Switzerland) and want to join the fray.

Sadly I started drinking at a much younger age, as it is often so in Wallis (a province known in Switzerland for its accent and drinking). I was around six years old and mistook a bottle of red wine with syrup. You can’t qualify this as a career starter, though. I had my first hangover when I was around thirteen and enjoyed my youth with plenty of binge drinking.

Discovering alternatives

That being said, I started to move away from alcoholic beverages to alcohol-free variants at around thirty. Back then, the market was still tiny, and there were only a handful of alcohol-free beers or wine available. The most famous of them was “Bilz”, which I loved to drink even when I was younger.

Today there are more alternatives available than ever before, and the market is growing fast. Supermarkets are picking up the pace and extend the choices even further, which is nice!

Luckily, I’m not alone with the notion of finding alternatives. This summer, I sat down with a friend and ordered almost every alcohol-free beer from Coop, a supermarket chain in Switzerland. The idea was to test and compare them during a beer tasting weekend 🍺.

It was pretty fun! Below you find a list of all beers we tried with the brand name, price and a small review:

The review

Brand	Description
	Appenzeller Leermond `Switzerland` ⭐⭐⭐⭐ 3.79/100cl – 0.0% – Brack Fruity and smooth taste. A great replacement for a typical lager beer.
	Appenzeller Sonnwendlig `Switzerland` ⭐⭐⭐ 3.79/100cl – 0.0% – Coop Has a watery, slightly fruity taste. Typical lager without any special surprises.
	Appenzeller Wheat `Switzerland` ⭐⭐⭐ 3.90/100cl – 0.0% – Coop Very nice and authentic wheat taste. Recommended as wheat beer alternative.
	Bilz Panaché `Switzerland` ⭐⭐⭐ 4.20/100cl – 0.0% – Coop It has to be here cause of childhood memories. In general, very sweet.
	Brewdog Punk AF `Scotland` ⭐⭐⭐ 9.70/100cl – 0.5% – Coop Fruity and malty taste with a slight bitterness. Gets close to a typical IPA feel.
	Chopfab Bleifrei `Switzerland` ⭐⭐⭐⭐ 5.91/100cl – 0.5% – Coop Slight fruity and mild taste. A very balanced lager and recommendable.
	Easy Rider Bulldog `Sweden` ⭐⭐ 10.91/100cl – 0.4% – Coop Has a velvety, watery and bitter taste. A mild taste for an IPA.
	Feldschlösschen `Switzerland` ⭐⭐⭐ 3.48/100cl – 0.5% – Coop Watery and neutral taste, good to drink very cold. Decent lager beer.
	Gran Alpin Senza `Switzerland` ⭐⭐⭐⭐⭐ 5.03/100cl – 0.0% – Coop Fruity and sweet taste with a hint of citrus. Has a slight bitter aftertaste.
	Grimbergen `France` ⭐⭐⭐⭐ 7.67/100cl – 0.0% – Coop Very sweet taste with variants of spices, and no bitter aftertaste.
	Heineken 0.0 `Netherlands` ⭐ 4.49/100cl – 0.0% – Coop Watery taste, misses any sign of flavour like the original.
	Lola `Germany` ⭐⭐ 11.52/100cl – 0.5% – Coop A very dry but slightly fruity IPA. A mild taste for an IPA.
	Lola Wit `Germany` ⭐⭐⭐ 11.52/100cl – 0.0% – Coop Very wheaty and soft beer. A good alternative for a good wheat beer.
	Lowlander IPA `Netherlands` ⭐⭐⭐ 9.60/100cl – 0.0% – UrbnDrnk Has a refreshing, spicy note. Very sparkly with a taste of cinnamon and orange.
	Lowlander Wit `Netherlands` ⭐⭐ 9.60/100cl – 0.0% – UrbnDrnk Very sweet and fruity taste with a hint of citrus.
	Riegele IPA Liberis `Germany` ⭐⭐ 10.30/100cl – 0.4% – Coop Very strong bitter and malty taste, stronger than a usual IPA.
	Schützengarten IPA `Switzerland` ⭐⭐ 5.05/100cl – 0.0% – Coop Very bitter taste with a whole lot of acidity, a bit stronger than usual IPAs.

All prices in CHF, based on a ⭐⭐⭐⭐⭐ star rating.

Thanks for a fantastic beer testing weekend V 😊.

‘Till next time!

Summer break

Tue, 31 Aug 2021 00:00:00 GMT

Summer break

It's been a busy year so far. Work and school continue from home as the Coronavirus ravages, and there is no end in sight.

So I’ve decided to use August to relax, go on some holidays and write another article again in September. In the meantime, have a look at the following articles:

‘Till next time!

Configuring Plex

Sat, 17 Jul 2021 00:00:00 GMT

“Plex marks the spot for free streaming”

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Before we begin

The following chapters will be focusing on tutorials with instructions on how to set up the respective services.

All services run on Virtual Machines using Ubuntu server 20.04 LTS. A section featuring the specific setup instructions is available at the end of this article.

Installing Plex

With your Ubuntu server ready, head over to the Plex website and choose the correct installation package from the downloads page. Copy the link and replace it with the {download-link} placeholder and run the commands in sequence:

wget {download-link}
sudo apt install beignet-opencl-icd
sudo apt install ocl-icd-libopencl1
sudo dpkg -i {download-package}
sudo dpkg -L plexmediaserver
rm {download-package}

Make sure to replace {download-package} with the respective package name 😎.

Monitor the prompt output for additional warnings or errors after running the commands. There may be subsequent packages you need to install.

Next, check the status of the Plex server and stop it for now by executing:

sudo systemctl status plexmediaserver
sudo systemctl stop plexmediaserver
sudo systemctl disable plexmediaserver

Following, enable the Plex repository by uncommenting the content in /etc/apt/sources.list.d/plexmediaserver.list. Save the file and run the following commands:

wget -q https://downloads.plex.tv/plex-keys/PlexSign.key -O - | sudo apt-key add -
sudo apt update
sudo apt upgrade
sudo systemctl start plexmediaserver

Restart the server, and you are ready to start configuring Plex. The administration interface is reachable through x.x.x.x:32400/web. You can get the full address by running netstat -lnpt | grep Plex.

Configuring Plex

If you want to migrate an existing Plex instance, stop here and read Migrating an existing Plex instance first. If this is not the case, continue with the configuration.

The configuration itself is pretty straightforward. Login to your account, give your server a name and add your shares. There is also a section available that demonstrates how to set up the mounts on Ubuntu.

Using a static IP address

This step is optional, but I prefer to set a static IP address for my VMs. Alternatively, add your instance hostnames to your DNS configuration to reach them locally.

To get started, figure out the network identifier that connects to your network by running ip a. In my case, it is ens224. Then create /etc/netplan/01-netcfg.yaml with the following content:

network:
  version: 2
  ethernets:
    ens224:
      dhcp4: no
      addresses:
        - 192.168.1.200/24
      gateway4: 192.168.1.1
      nameservers:
        addresses: [192.168.1.1, 8.8.8.8, 1.1.1.1]

You may need to replace ens224 with your network identifier and 192.168.1.200/24 with the desired IP address. You may also have a different gateway and nameservers. To apply the settings, run:

sudo netplan apply
ip addr show dev ens224

Enabling auto-updates for Ubuntu

I prefer Ubuntu to be updated at all times. If something breaks, I’d rather revert to a snapshot than sit on security issues. To get started, run:

sudo apt install unattended-upgrades
sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

In the editor uncomment the following lines:

"${distro_id}:${distro_codename}-updates";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";

Next edit /etc/apt/apt.conf.d/20auto-upgrades and replace the content with:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";

You can test the changes by running sudo unattended-upgrades --dry-run --debug and checking the logs in cat /var/log/unattended-upgrades/unattended-upgrades.log.

Enabling auto-updates for Plex

Plex needs some special care to update automatically. Luckily there is a great script available on github.com/mrworf/plexupdate. Follow the instructions in the README.md, and the subsequent terminal prompts.

Installing NVIDIA drivers

GPU passthrough is a bit of a hassle. Using a consumer GeForce GTX 1050 Ti doesn’t make it easier. However, I got it to work on my setup by using the following steps:

Make sure ESXi supports passthrough by adding:

hypervisor.cpuid.v0 = FALSE
pciPassthru0.msiEnabled = FALSE

to the VM configuration. Next, installing the drivers:

sudo apt install ubuntu-drivers-common
ubuntu-drivers devices
sudo ubuntu-drivers autoinstall

Next, we need to confirm that hevc_nvenc is working by running:

sudo apt remove ffmpeg
sudo snap install ffmpeg
ffmpeg -hwaccels

the last command should output something like:

Hardware acceleration methods:
vdpau
cuda     <- this is important
vaapi
drm
opencl

Finally, we need to make sure that the system doesn’t boot in UI mode by running:

sudo systemctl set-default multi-user.target
sudo shutdown -r now

Extra: Migrating an existing Plex instance

I had an existing Plex instance using a Mac Mini and needed to migrate it to the new Ubuntu server. This section is based on Moving an Install to Another System. In short:

On the old Plex instance, disable the Empty trash automatically after every scan preference for the server.
Do not configure anything on the new Plex instance yet. To be safe, sign out and stop the server by running sudo systemctl stop plexmediaserver.
Ideally, do the same for the old server. I had a running live system that I didn’t want to shut down during the process.

Now that everything is prepared copy the old instance’s data to the new server by using rsync. Rsync allows you to run the command multiple times even if there are network issues or running the old instance side-by-side.

Make sure both servers have rsync installed by executing sudo apt install rsync. Add the following script to the server of the new Plex instance:

echo "rsync -avz --delete --exclude 'Cache' user@x.x.x.x:/var/lib/plexmediaserver/ /var/lib/plexmediaserver/
sudo chown -R plex:plex /var/lib/plexmediaserver
sudo chmod -R 755 /var/lib/plexmediaserver" > ~/plexmigrate.sh
chmod +x ~/plexmigrate.sh

Replace user@x.x.x.x with the user/IP of your old Plex instance.

Finally, you can run the command using sudo ~/plexmigrate.sh and start the migration. Once finished, make sure to turn off the old Plex instance permanently and start the new one:

# on the old server
sudo systemctl stop plexmediaserver
sudo systemctl disable plexmediaserver

# on the new server
sudo systemctl start plexmediaserver
sudo systemctl enable plexmediaserver

Extra: Setting up mounts

QNAP supports several protocols though I chose to use NFS as it worked best for my use case.

First, local folders need to be created to mount the remote ones into. In my case, I used /media/ and added the subsequent sub-folders inside it.

Next, install the requirements into Ubuntu:

sudo apt install network-manager
sudo apt install nfs-common

Now, we can check if a connection can be established like:

sudo mount -t nfs x.x.x.x:/server/folder /media/photos

and replace x.x.x.x with the IP address of the storage server. If all works prepare the /etc/fstab file as follows:

x.x.x.x:/server/folder1 /media/folder1 nfs _netdev,auto,x-systemd.automount 0 0
x.x.x.x:/server/folder2 /media/folder2 nfs _netdev,auto,x-systemd.automount 0 0
x.x.x.x:/server/folder3 /media/folder3 nfs _netdev,auto,x-systemd.automount 0 0

Extra: Setting up Ubuntu

Get the latest version of Ubuntu server and follow the general installation steps. You may want to get the current IP through ip a and ssh into your Ubuntu instance. After that, run the following commands:

sudo apt update
sudo apt upgrade
sudo apt full-upgrade
sudo apt install net-tools

Apply these steps for any other service you want to install.

So what’s next

We now have instructions for setting up Ubuntu in general and our first VM running Plex. In the next chapter, I’ll be setting up the DVR to manage content.

‘Till next time!

Networking with Ubiquiti

Tue, 29 Jun 2021 00:00:00 GMT

Upgrading to a 10GbE Network

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Choosing the hardware

Who hasn’t struggled with their network setup at some point? From failed routing to dropped WiFi connections, networking done right is complex. I hope I finally learn how to set up everything correctly with proper monitoring to debug where the 🤬 packet drops come from at school…

I started with the typical vendor-specific routers with an included WiFi module. The standard setup everyone gets when ordering an internet product from UPC Cablecom, Swisscom and the sorts.

However, once fibre became available at my place, I immediately switched to Fiber7. It was not just cheaper but offered better uptime at higher bandwidth and excellent support. Though not everything was a 🌈. At the same time, my hate-love relationship started with the new Turris Omnia router. Open hardware is incredible, but that piece of 💩 had one of the worst WiFi modules installed I’ve ever come across. It just didn’t work most of the time. It did everything else flawlessly.

That was the moment where I decided to look for an alternative setup. I looked at AmpliFi, Linksys, TP-Link and the sorts, settling on Ubiquiti. Why? Cause it’s simple, affordable, provides a unified experience, an extendable mesh network, and it just works.

The stack

The first components I bought were a UniFi nanoHD together with a Cloud Key. The UniFi nanoHD and Cloud Key directly connected to my Turris Omnia. I also purchased a Unifi PoE 8-Port Switch to allow all my other LAN capable devices to connect to my router (as I ran out of ports). So far, I was pleasantly surprised by how smoothly everything worked.

I still got packet loss here and there due to my Turris Omnia. I decided to replace the remaining infrastructure entirely with Ubiquiti hardware once I started my server setup, ending up with:

1x UniFi nanoHD
1x UniFi Switch PoE 8 (150W)
1x UniFi Switch 48
1x UniFi Switch XG 16 (10-gigabit networking)
1x UniFi Security Gateway Pro 4 (new router)
1x UniFi Cloud Key Gen2 with Rack Mount (upgrade)

The nanoHD and 8-Port PoE switch were the only components I bought brand new. I acquired the remaining devices either through Tutti or Ricardo.

The 48-port switch is slightly overkilled but ended up cheaper than the 24-port candidates ¯\_(ツ)_/¯.

The setup

Now my fibre connection goes directly via SFP into the Security Gateway Pro. The second SFP connection is used as an uplink to the XG 16, which in turn has another SFP connection to the 48-port switch.

The XG 16 has two 10-gigabit connections to the storage server and another two 10-gigabit connections to the virtualisation server. That connectivity allows both to communicate super fast with each other. The additional NICs connect each via a 1-gigabit connection to the 48 port switch. The different links are mainly for custom VLAN configuration through VMware ESXi for some of the virtual machines.

I also plan to play around with custom VLANs from the Ubiquiti Administration Interface further to separate my internal LAN from the WAN where necessary.

There are also additional cables connected to the 48-port switch for general management, monitoring, power failure, etc.

The bottlenecks

One significant drawback in my whole setup is the connection from my server in the cellar to the network in the apartment.

I use two Zyxel PLA5456 Powerline connectors as there is no direct LAN cable. Even though they advertise as 1800MBit/s, the reality is around 1000MBit/s. Which is fine for my needs. Though I’m looking forward to proper cabling in the future 🥳

Checking the speeds

Performance tests were next in line. I tested the following scenarios, the “Client” being a computer inside the apartment:

Storage to virtualisation server and vice-versa
Client to the storage/virtualisation server and vice-versa
Client to another Client

I did that using iperf3 and speedtest.net. Here a rough outline of the procedure:

sudo apt install iperf3
sudo apt install speedtest-cli

Then start iperf3 on the target server and run the speed-tests on your source device:

# target server (e.g. 192.168.1.10)
sudo apt install iperf3
iperf3 -s -p 8088

# source device
speedtest-cli
iperf3 -c 192.168.1.10 -p 8088

So what’s next

With the network ideally set up, I’m ready to set up the software for managing my digital assets. We’ll cover this in the next section.

‘Till next time!

HomeOffice games

Mon, 31 May 2021 00:00:00 GMT

Let’s Play!

HomeOffice wasn't that uncommon in the IT industry before but became more apparent during the Corona pandemic. Distributed teams are growing, and so the lack of personal contact.

It has become difficult to grab a 🍺 with your co-workers. We are a distributed team at Divio spread across the world. Team events are costly and rare; getting everyone together in one place is difficult.

Of course, several team-building solutions sprang up in the last couple of months to provide a creative solution. The following consists of activities that have been tested through either Divio or with friends. All games are played inside a browser without installing any additional software; thus, it is convenient to start the game.

Drawing games

In the classical sense, they are charade-inspired word-guessing games. The objective is to draw a word that someone receives while the other participants try to guess it. There are multiple variations and arrangements.

In Skribbl, a group meets inside a drawing game where someone receives a word, draws it, and the others try to figure it out. The guess is submitted in a chat window, and points are distributed by how fast the drawing is guessed. The other participants won’t see the solution till the end of the round. The chat window shows if someone figured it out without displaying the solution.

skribbl.io

Sketchful and Gartic are similar but add some customisations, different lobbies and leaderboards.

Gartic Phone has a different but fun concept. It follows the idea of “telephone”, where a group of players form a line or circle. The first player comes up with a message and whispers it to the next person’s ear. That process continues until the word reaches the end of the line. The term is then compared to its original form.

In Gartic Phone everyone chooses a word and writes it down. That word is then passed to the next in line. The twist is that the word has to be drawn next. The drawing is then given to the next person, who has to guess the initial phrase. This process repeats until the initially defined word has reached the end of the cycle. Afterwards, the history is displayed in a chat-like tale.

garticphone.com

Escape rooms

An escape room is a game in which a team of players is locked in, searches for clues, solves puzzles, and accomplishes tasks to escape from the room. There are multiple versions available that follow the same flow but online:

There is a unique variation I would like to mention: CodinGame. It advertises itself as the very first escape game for coders and non-coders and is pretty fun for development teams and companies.

CodinGame

Another notable variation is Keep Talking and Nobody Explodes. Everyone meets somewhere online, and one starts the bomb. The others don’t see the bomb but have a manual to diffuse it. The one with the bomb has to describe the randomly generated components, while the others have to find the proper process to disarm them. If the time runs out, the team loses 💥.

Keep Talking and Nobody Explodes

These are games in which players attempt to uncover each other’s hidden role or allegiances. Commonly, they are played in teams with larger groups, while one side is considered “good” and the other “bad”.

One of those is One Night Ultimate Werewolf, the browser version of the tabletop classic. It is played with 3 to 10 people and, as the name promises, it lasts a single night. Each player gets a role with an agenda. The roles entail various effects, but in essence, there are two categories. As a villager, you try to find the werewolves among you, and as a werewolf, you hunt down the villagers while convincing the others that you are innocent. The concept is simple: everyone closes their eyes, the roles act in turn, and in the morning the table votes on who to accuse. Nobody gets eliminated, so a round takes about ten minutes and the whole team argues for all of them. That last part is the actual game.

Ultimate Werewolf

Another worthy mention is Among Us. An exception to my initially defined rule, you have to install an app. However, it is straightforward, available on all platforms and super easy to get started with a novice team. You are either a regular “crew member” doing tasks in a spaceship or the “impostor” who kills the crew members off one by one. The crew aims to figure out who the impostor is before they kill the entire team. The communication happens either after finding a dead body on the floor or by pressing an emergency button.

Among Us

Cooperative games

Cooperative games are designed to test the teams’ ability with memorisation/language or to explore their properties. They are typically designed to work together or in separate groups.

Codenames and Codewords are great examples of word guessing cooperative games. Rival Codebreakers race to identify which of the 25 Codewords/Codenames are their own. They do this by listening to their Codemasters, who take turns giving one-word clues. The Codebreakers try to guess which words their Codemaster meant, one at a time. If they guess correctly, they may continue guessing until they either run out of ideas for the given clue or get a Codeword wrong. Then it is the other team’s turn to give a hint and guess. The first team to reveal all their Codewords wins the game.

Another fantastic cooperative game is Hanabi. Instead of guessing words, team members work together to identify their cards. Like “Indian Poker”, players can see each other’s cards but cannot see their own. Information or fuse tokens provide information to the players on what colours or numbers they are holding. The goal is to create a “hanabi” (firework) by placing all cards in order and by colour on the table.

Hanabi

Other worthy mentions

There are other games on my list to try out I would like to share:

I hope you have a fantastic time playing some of these games with your co-workers. If you have any other recommendations, let me know.

‘Till next time!

Storage - QNAP vs. Synology

Fri, 30 Apr 2021 00:00:00 GMT

Getting over 70 TB of storage

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Choice of storage

I was pleased with my Synology DS1815+ as my primary storage solution for years. It handled the automatic download, categorisation and storage of all my digital assets. It was reliable, and I didn’t have any hardware or disk failure so far. So why did I switch to a QNAP?

The goal was to get a 2U 12-Bay NAS for under 1’000 CHF, including all upgrades, without the storage drives. Thus, it didn’t need to be the newest model. Additionally, I wanted upgradability for additional bays, expandable SSD cache, more memory or better CPUs for the future. Sadly, most vendors are restricted by one or the other technology. However, Synology lagged significantly behind when comparing their models to QNAP as the performance to price ratio seemed to be much better with QNAP’s hardware.

After some further research, I found a QNAP TVS-1271U-RP for a relatively affordable price. It had a stock Dual-core Intel® Core™ i3-4150 CPU with 8 GB memory. I promptly ordered a Quad-core Intel® Core™ i7-4790S CPU and upgraded the RAM to 32 GB. I also ordered two cheap GM512GB mSATA drives that will provide the SSD cache and an M.2 NVMe SSD Express card to add my spare 1TB Crucial SSD as an additional resource for virtual machines. On top of that, I also added a 10 Gbps NIC 😁, ending up with the following specs:

Intel® Core™ i7-4790S (4 Cores up to 4.00 GHz)
32 GB DDR3L 1600 MHz Memory (4x 8GB modules)
12 bay hard drive slots
2x 512 GB mSATA SSD
1x M.2 NVMe SSD express card with 1TB Crucial SSD
1 Gbps integrated quad-port NIC
10 Gbps X540 dual-port PCIe NIC

All that together kept me under the 1k range, by like a beer or two. Thanks to some cheap offerings from Tutti and the fantastic prices from AliExpress.

Choosing the drives

I used Western Digital Red for years, until the 2020 scandal around WD shipping SMR drives without labelling them as such. I got two 6 TB SMR drives to fill up my Synology from 6 to 8 drives just a couple of years ago, and ever since, the RAID consistency checks take forever to complete. Digitec support did a better job offering me replacements than the WD support, even if they didn’t have to. Kudos to their support! 🥳

These issues led me to switch to 12 x 8 TB Seagate IronWolf (CMR) NAS drives. I didn’t want to save costs there as the data was essential to me. So I waited for a reasonable offer to come along and bought them new at around 180 bucks (the original price was 260 bucks a piece back then).

Setting up the NAS

The QNAP interface was similar to Synology’s “DiskStation Manager” (DSM). QNAP calls theirs “Turbo NAS Operating System” (QTS). Both operating systems offer similar features. The notable difference is that DSM tries to simplify processes and provide fewer options, while QTS allows for basically any modification leading to a cluttered UI. If you are an IT specialist, QTS is for you. Otherwise, go with DSM (and Synology, for that matter).

The setup wasn’t that hard though, choose a user, tweak some settings and be done with it.

Resolving some issues

Though sadly, there was an issue. Every now and then, I had to restart the NAS multiple times to recognise the discs. Sometimes only 8 LEDs were displayed instead of all 12, meaning that four brand new drives went missing and the RAID failed.

QNAP’s support was excellent and responsive. They quickly discovered that I was running on a BIOS version with a disk problem known to the TVS-1271U-RP.

So I followed the provided information to upgrade the BIOS version to the latest, and everything was back in order.

Finally, I updated the firmware and operating system to the latest versions as well.

Setting up the storage

Now I had a working NAS without any issues and upgraded everything to the latest and greatest. Next was to set up the 12 drives, the cache, and the additional M.2 storage.

My storage pool consists of 12 NAS drives. I chose to go with RAID 6 and its two-disk failure tolerance. That gives me ample space and some breathing room. I have two volumes set up, one that holds my data and another one for surveillance recordings.

I also set up cache acceleration using the two 512 GB mSATA SSDs. It speeds up data transfer for the most requested files, which is nice. I never used cache acceleration, so I wanted to play around with it and see if it delivers real benefits. So far, it didn’t disappoint.

The M.2 drive is set up as an additional storage pool to host VMs or VM backups. I don’t use it yet, but it’s reserved for projects in a later stage.

Setting up backups

As most know, RAID is not a backup solution. If more than two disks fail, all data is gone. Even if just one disk fails and gets replaced, the rebuild can cause a cascading failure. To remedy that, I added two 4TB USB-Drives to the NAS for “not so important data but still important enough to be saved somewhere else” and a remote backup solution using Backblaze for my most treasured data.

QNAP offers an extensive interface to set up backups using “HBS 3 Hybrid Backup Sync”.

Accessing the volumes

After all that, the only thing left was tweaking some settings and figuring out how to connect to the drives.

Also, that was pretty simple. I headed over to “Control Panel” > “Network & File Services” and enabled “Apple Networking” and “NFS Services”. The latter will be used later for the VMs.

After that, I can connect to the drives through finder with specific users set up for network access.

I transferred the data on my Synology NAS via rsync to a designated folder and then rearranged everything to match my taste.

So what’s next

Now, I have a running QNAP TVS-1271U-RP with the latest firmware and OS. All my data is migrated to the new NAS, and everything was working, fast and reliable.

The following section will deal with my networking setup using Ubiquiti hardware to connect all the data. And yeah, there are some bottlenecks 😭.

‘Till next time!

Virtualisation with ESXi

Sat, 27 Mar 2021 00:00:00 GMT

Configure and setup ESXi and vSphere

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

Choice of server

I used a Mac mini as my media server and a Synology DS1815+ as my media manager previously. The goal was to consolidate both systems into a mighty server using virtualisation to host each service.

At first, I wanted to build the server myself using an Intertech 2U 20255 chassis. The components would have cost me over 1200.- CHF for a decent system that could host about 4-5 demanding virtual machines.

Taking that as a baseline, I scoured Ricardo and Tutti for some Dell PowerEdge or HP ProLiant systems, as they appeared to have a good price/performance ratio overall.

I ultimately settled on a baseline PowerEdge R820 for about 900 bucks and upgraded it through AliExpress for an additional 500.- CHF, ending up with the following specs:

4x Intel® Xeon® Processor E5-4650 (8 Cores up to 3.30 GHz)
768 GB DDR3 1333 MHz ECC Memory (32 GB modules)
4x 300 GB SATA and 4x 900 GB SATA drives
1 TB M.2 Kingston A2000 SSD with PCIe adapter card
PERC H710 raid card
1 Gbps integrated quad-port NIC
1 Gbps I350-T4V2 quad-port PCIe NIC
10 Gbps X540 dual-port PCIe NIC
2x Platinum+ 1100 Watt power supplies
MSI GeForce GTX 1050 Ti OC
Internal Dual SD Module (2x Sandisk Ultra U1 16GB)
iDRAC Enterprise

Not too bad for a total of around 1’500 bucks 🤪

Discovering the basics

I had no experience with server hardware before. So getting access to the management interface appeared to be challenging already. The server turned on when pressing the power button, which was good, yes? After getting a VGA cable, mouse, keyboard and monitor, I was ready to dig in.

The server took some time to boot up, finally showing a colourful VMware ESXi “ready” screen. “Nice, whatever this is”, were my first thoughts. There was a yellow-black screen with an IP shown; let’s open it! Ah! That’s a virtualisation UI, excellent.

The first hours of experimenting led me to some insights:

The booting sequence takes some time, as the server needs to “rediscover” all the hardware and regenerate its “Inventory”.
On boot, I could start the “Dell Lifecycle Controller” which is “an advanced embedded systems management technology… blah”. In short, it’s an advanced UEFI/BIOS interface that allows configuring or “deployment” of the server. We’ll get back to that one later.
VMware ESXi is an excellent virtualisation tool to manage virtual machines. The web interface looked decent, and naturally, I broke it after some time 😅
There is a dedicated iDRAC (Integrated Dell Remote Access Controller), which provides access to the server’s admin interface through the browser. The IP address is static and was visible after the boot. There was even a Virtual Console so I could ditch the extra monitor and continue from my laptop.

Resolving some issues

The server showed two issues when booting on the front display. The first indicated that the second power cable wasn’t connected, easy to solve.

The latter was a bit more tricky. A message about “The storage BP1 SAS A cable is not connected” indicated that the raid controller did not correctly connect to the hard drives, but they were, and even the hard drives showed up. I first tried disconnecting both SAS cables and reconnecting them again, with no success. I finally solved it by changing the order of the wires (wtf).

Always check that all connections are firm and the cables are in the proper order 🤓.

Updating the firmware

As the iDRAC provided access to almost any functionality the server provided, I booted into the Dell Lifecycle Controller and upgraded the firmware versions for its hardware and components. There were some issues, and I had to manually download some of the drivers and install them by hand. The web interface helped as it provided a simple upload form with good error feedback.

Setting up the storage

I used the iDRAC to recreate the RAID setup. I used raid six as the double-parity was more important to me than total space. I created two raid groups, “datastore_300” (name in ESXi) with the four 300 GB SAS drives and “datastore_900” with the four 900 GB SAS drives, plus “datastore_nvme” for the 1 TB M.2 SSD.

The purpose is to use “datastore_300” to use for ISO storage or when moving VMs around, “datastore_900” as an archive/backup drive and “datastore_nvme” as the primary host for the VMs.

Installing the Host OS

I looked at other virtualisation software such as Proxmox VE, Hyper-V or Citrix Hypervisor, amongst many others. Though they all seemed fine, the combination of VMware ESXi and vSphere seemed pretty good to me from what I read. Plus, I also got a free enterprise licence while at school 🥳.

It took me a while to figure out where the existing ESXi instance got installed. Dell provides a Dual SD Module where you can install the host system. Of course, you can also use the SAS drives, but ¯\_(ツ)_/¯, seemed to be good enough. I set up the system to mirror both SD cards. That way, it would be easy to restore a failure, plus I back up the contents regularly.

First, I installed a plain VMware ESXi, which showed some irregularities with missing drivers and weird behaviour. I later discovered that I need a DELL EMC Customized Image of VMware ESXi. After that was installed, everything seemed fine.

Virtualisation using ESXi

The default installation served me already pretty well. I tinkered a bit with the Networking defaults and promptly set up my datastores afterwards. Everything seemed fine. Creating VMs worked, but I wasn’t able to move VMs around freely, which was odd.

I discovered that I needed VMware vSphere for “advanced VM management”. The problem was that it was necessary to be installed as a separate VM on the ESXi host, which would later take over the ESXi host to manage it, weird or? 🤪

Well, I downloaded vSphere, got some licences via my school account and set it up. The installation procedure is pretty straightforward. vSphere creates a separate VM on the ESXi host to provide an improved administrative interface. Once vSphere completed the setup, it took over the host’s control, and I was able to move VMs around. vSphere even provided updated functionalities. However, I didn’t understand its inner working and broke the installation several times. I settled to use it for creating, editing and moving only for now.

So what’s next

Now, I have set up my Dell PowerEdge R820 with the latest firmware, ESXi as Host OS with vSphere as managing director, proper storage and resolved all significant issues. My virtualisation beast was ready, and I moved to the next step, setting up the storage monster to hold all the data. We’ll cover this in the next section.

‘Till next time!

My Homelab

Sun, 28 Feb 2021 00:00:00 GMT

Introduction

Introduction
Virtualisation using a PowerEdge R820
Storage with a QNAP TVS-1271U-RP
Networking with Ubiquiti
Media streaming with Plex
Download management with SABnzbd and Transmission
Service management with Sonarr, Radarr, Lidarr, Readarr, Bazarr and Mylar
Additional tools like Calibre, Ubooquity, Ombi or Organizr
Monitoring with Checkmk

A brief history

I started off storing data on CDs and later DVDs back in the days (you know, before the Y2K problems 🤦🏻‍♂️). Network Attached Storage (NAS) solutions were still in their early stages, and hard drives expensive. At some point, I switched to external USB drives and created mini raids for the most critical data like digital documents or photos. Though I still held onto the mountains of DVDs piling up.

About ten years ago, I decided to switch to a NAS setup and choose to build my own. It had about 1 TB of total storage, which was ample to hold all my data. I extended my hardware with a Mac mini in 2012 to create my first Homelab using Plex, Couch Potato, Sick Beard, Headphones and SABnzbd to manage my continuously growing library.

In 2015 my NAS and Mac mini failed me due to a power surge, and I decided to migrate all of my data to a Synology DS1815+ with 12 TB of total storage. I switched from Couch Potato to Sonarr and later from Sick Beard to Radarr. Headphones were replaced by Lidarr sometime later.

All new homelab

I’m continuously looking for challenges as a developer and decided, at the end of 2020, to overhaul my setup completely this year. I started by looking for a powerful server to host virtual environments for my services, an expandable NAS solution for my data and a good rack to store all my hardware.

I settled buying a Dell PowerEdge R820 with 4 Intel Xeon E5-4650 CPUs (32 cores, 64 threads), 768 GB DDR3 ECC Memory and a total of 3 TB SAS HDD space. Furthermore, I extended it with a 1TB NVMe and an NVIDIA GeForce GTX 1050 Ti. The VMs will be hosted on the NVMe drive, while the SATA drives serve as a backup. The graphics card will provide hardware-accelerated streaming for Plex.

I ended up buying a QNAP TVS-1271U-RP with 12 x 8 TB Seagate HDDs as my NAS solution. It was a good fit for the setup and extendable. I upgraded the hardware to include another 2TB of NVMe storage for cache acceleration and switched the shipped Intel® Core™ i3-4150 for an Intel® Core™ i7-4790S and upgraded the RAM from 8GB to 32GB. The upgrade provides me with ample power to manage QNAPs services and use it as an alternative for my VMs as a failsafe.

Both servers include a 4-port gigabit NIC as well as a 2-port 10 gigabit NIC each. All of that gets connected to my existing Ubiquiti network hardware (Unifi Security Gateway Pro, Unifi 48-port switch, CloudKey Gen 2). The hardware will be mounted in an APC NetShelter SX 24U that I got for nine bucks 💎.

The last piece in my setup is a Powerwalker VI 2200 for power failure management. The USV can handle up to 1320 watts (overkill), which is ample to secure the hardware. I still have a smaller APC left that provides up to 400 watts additionally.

I got most of the hardware or upgrades either through Ricardo, Tutti or AliExpress. It costs me less than 5000 CHF in total, which is a bargain (including everything, network hardware, cables etc.).

So what’s next

Everything is installed and running so far as expected. Next, I’d like to write up on how I installed and configured everything, providing tutorials on setting up ESXi, VSphere, QNAP, Plex and all the other services I use to make my digital life run on autopilot.

‘Till next time!

Python 3.9 - What's new

Thu, 14 Jan 2021 00:00:00 GMT

What’s new in Python 3.9.x

Python 3.9.0 has been released on the 5th of October, and Python 3.9.1 followed shortly after on the 7th of December.

They introduce a bunch of new functionality to Python’s syntax, build-in features, the standard library and the interpreter. For my learning objective, I’d like to go over some fundamental changes and provide examples and tips on using some of the new features.

Let’s start with how the language is maintained first, before moving onwards to the fancy part.

Moving towards a yearly release cycle

Previously, Python has been released on an eighteen-month cycle. 3.9 will change this to an annual release cycle. This change comes from the increasing popularity of the language in various areas such as data science or machine learning.

Naturally, releases will become smaller to land features and bug fixes sooner into the users’ hands. A shorter release cycle also decreases the rush before betas and creates further predictable planning for everyone involved.

Related Python Enhancement Proposals:

PEP 602

Performance improvements

Python has continuously improved its performance on every past release. However, 3.9 introduced two changes that don’t require any code adaptions at all. Without getting into details, the new PEG parser is more efficient, and many builtins now use the vectorcall convention to make their execution faster.

Related Python Enhancement Proposals:

PEP 590 – Vectorcall: a fast calling protocol for CPython
PEP 617 – New PEG parser for CPython

Internal changes

Cleaning up code, including improving and modernising the codebase, is an ongoing development task for every developer. This approach applies to Python’s core developers as well. Besides the removal of deprecated compatibility code, there are some noteworthy internal changes.

These include; module state access is now possible from C extension methods, garbage collection doesn’t block on resurrected objects, and several additional modules use multi-phase initialization and/or the stable ABI.

Related Python Enhancement Proposals:

PEP 384 – Defining a Stable ABI
PEP 489 – Multi-phase extension module initialization
PEP 573 – Module State Access from C Extension Methods

New features

There are three significant features I’d like to dig deeper into:

how prefixes and suffixes work in string operations
what the union operator allows us to do in dictionaries
the relaxation on decorators to improve maintainability

However, all changes are listed after them for completeness.

Prefixes and suffixes string operations

PEP 616 introduces string methods to remove prefixes and suffixes. Let’s have a look at an example:

if test_func_name.startswith("test_"):
    print(test_func_name[5:])
else:
    print(test_func_name)

There we check if test_func_name starts with “test_” (prefix). If this matches, we remove “test_” and return the new value or return test_func_name.

This can now be simplified by using removeprefix or removesuffix respectively:

# for the prefix case
print(test_func_name.removeprefix("test_"))

# for the suffix case
print(test_func_name.removesuffix("test_"))

Add union operators to dictionaries

PEP 584 introduces a union operator to dictionaries. That operator dramatically simplifies how two dictionaries can be merged. Let’s have a look at an example:

d1 = { "spam": 1 }
d2 = { "eggs": 3 }

print(dict(**d1, **d2))

You may also want to have a look at dict.update or collections.ChainMap, as these offer similar functionalities. Though the union operator is more readable and avoids some pitfalls when merging objects:

d1 = { "spam": 1 }
d2 = { "eggs": 3 }

print(d1 | d2)

More flexible decorators

PEP 614 relaxes the grammar restrictions on decorators. They were initially limited as a preference rather than a technical requirement. To demonstrate the benefits, let’s have a look at a code example:

button_0 = buttons[0]
button_1 = buttons[1]

@button_0.clicked.connect
def spam():
    ...

@button_1.clicked.connect
def eggs():
    ...

You can now simplify and make it more readable, idiomatic and maintainable:

@buttons[0].clicked.connect
def spam():
    ...

@buttons[1].clicked.connect
def eggs():
    ...

More type hinting

PEP 585 adds more type hinting capabilities which were first introduced in Python 3.5. More specifically, Python now enables support for the generics syntax in all standard collections currently available in the typing module.

PEP 593 adds a mechanism to extend the type annotations from PEP 484 with arbitrary metadata.

IANA time zone support

PEP 615 introduces support for the IANA time zone database in the standard library. This feature is a bit more abstract (and I don’t understand it well enough), so I recommend to dig into the PEP specification to learn more about it.

You can find more information about the IANA time zone database on their website.

Other changes

os.pidfd_open() has been added that allows process management without races and signals.
An implementation of a topological sort of a graph is now provided in the new graphlib module.

Summary

Python 3.9 adds a lot of excellent new features to improve the quality of life for any developer. A faster release cycle will allow us to upgrade faster with fewer compatibility issues and the new features are a welcome addition to the language.

We expect Python 3.10 to be released around the 4th of October. But don’t worry, all releases are supported for five years, so 3.9 will be active until 2025.

‘Till next time!

Looking back at 2020

Thu, 31 Dec 2020 00:00:00 GMT

Looking back at 2020

2020 hasn't been short on events. We added multiple environment support at Divio, with a bunch of fantastic UI improvements. My studies at FHNW finally started after a year of preparation, and of course, Corona disturbed all our lives.

To finish the year, I summarise what I’ve been doing at work, school and otherwise in this article. For my sanity and everyone else’s; as bullet points 😅.

Working hard

We added multiple environments, customisable services, new views, better logs, changelog, a lot of automation, and many more UI improvements to Divio Cloud.
We upgraded django CMS to support Django Version 3 and above plus released django CMS 3.8.0 along the way.
The django CMS Association was founded with support from many other companies to lead the future for django CMS and its ecosystem.
Corona didn’t spare our company, so we had to say goodbye to some great team members and friends.
I have been working at home for about two days a week in the past. Corona changed this to full-time home-office. It sounds great on paper, but I like to be at the office at least once or twice to get out of the house.
I also teach at HSR (now OST) and recreated most of the material for the lectures that year with success.

Learning again

I completed the preparation course to join FHNW’s bachelor programme successfully.
Now studying Computer Science and almost finished my first semester without any significant issues.
Passed the initial evaluation tests to join the English electives (highest grades) and got additional free ECTS points 😎.
Also, Corona isn’t much fun here. Even though we started the first days on campus, the latter half was remote.
Played Badminton at school at first, but again, Corona 🤮.
Grateful to all other students and new won friends, making it through this difficult time.

Personal achievements

I published my website and a blog post every month!
Started to record a journal about my days to create this summary ultimately 👍.
Began with new morning and evening routines that help me to be a bit more efficient.
A tropical sickness plagued me for weeks, that wasn’t fun.
Had some great financial success investing in companies, might start writing about that in the future.
Was able to read again about 50 books this year! Looking at less though in 2021 as studies will consume more time.
Neglected Anki and Udemy this year to focus on other things, let’s see if I can get that back on track 2021.

A big thank you

I want to thank everyone who helped me at work, at school or privately to have a good 2020. Especially Jasmin, my better half, who got me through some rough times. 2021 looks good so far, and I might have a ton of new updates by the end of it, looking forward to some big changes!

‘Till next time!

Host a Gatsby project on Divio Cloud

Sat, 28 Nov 2020 00:00:00 GMT

How I setup my website on Divio Cloud

Preparing a Gatsby project to be hosted on Divio Cloud is relatively simple. I'll guide you through the process of creating a Gatsby starter project, adding the Divio Cloud remote repository and deploying the project.

There have been a lot of improvements on Divio Cloud since I launched my website. These changes made the hosting setup even more straightforward. I decided to record my steps while upgrading my website. Start by heading over to control.divio.com and create an account.

First, add a new project and enter an appropriate name. Choose “Build your own” from the options, as we want to configure everything by ourselves. Hit “Continue” to proceed.

Select the developer subscription to get started. Make sure to upgrade your subscription as soon as you go live; otherwise, you may lack resources to handle the incoming traffic. Also, consider getting in touch with Divio’s excellent support team to assist in choosing the right subscription to your needs.

Next, create a project folder somewhere on your system and create a simple Gatsby project:

run mkdir my-gatsby
run cd my-gatsby
run npm install -g gatsby-cli
run gatsby new . https://github.com/gatsbyjs/gatsby-starter-hello-world

You can also use any other Gatsby starter with Divio Cloud, just make sure to set up the project in the root of the folder.

Next, connect the project to Divio Cloud:

run git init
run git remote add origin git@git.divio.com:gatsby-on-divio-cloud.git (make sure to use your git remote)
run git push origin master --force

Your local setup should now be in sync with Divio Cloud. Let’s configure your project to deploy it.

Create a Dockerfile with the following content:

FROM node:14.15.1-alpine

RUN apk add --no-cache \
  make g++ && \
  apk add vips-dev fftw-dev --update-cache \
  && rm -fR /var/cache/apk/*

# the cli is required to build the site
RUN npm install -g gatsby-cli

# switch to app dir to install node_packages into app/node_packages
# this allows them to be available on your host system through docker-compose
WORKDIR /app

# copy the package files first for them to get cached
COPY package*.json /app/
RUN npm install

COPY . /app

# build the project and serve it
RUN npm run build

EXPOSE 80

CMD ["npm", "run", "deploy"]

Next, create a .dockerignore file with the following content:

.cache/
node_modules/
public/

To finalise the setup, add the deploy script to the package.json file:

"scripts": {
  ...
  "deploy": "gatsby serve --port 80 --host 0.0.0.0"
},

Commit all your changes and push them to Divio Cloud. Go back to the web interface and hit the “Deploy” button on the test server. Wait a bit, and you are ready to go! Open up the test server by clicking on the “env url”. Make sure to check out the “Logs” section if you are experiencing any errors along the way.

Enjoy your new Gatsby playground.

‘Till next time!

Apps I can't live without

Sat, 31 Oct 2020 00:00:00 GMT

Top 10 tools I use every day

Everyone has their specific arsenal of tools to optimise workflows or to make life easier. As the saying goes: pick the right tool for the job.

This article focuses on ten or more tools I’m using almost every single day. These help me to stay organised so that I can store and retrieve knowledge on short notice and keep my information architecture structured.

Tools change over time; this list represents a snapshot of my current tool-chain. An overview of all the tools I’m using can be found at the end of the article.

Not every tool is indispensable, and some are used more often than others. The following tools are the most important to me in order of usage:

Notion
Unclutter
2Do
Habitica
Airmail
Calendar
In Your Face
VSCode & iTerm
Tower
Plex

1. Notion


Website	https://www.notion.so/
Purpose	Wiki, Organisation
Alternatives	Outline (self-hosted) Confluence

Notion serves as my second brain 🧠. It’s where I capture, organise and reclaim ideas, notes, summaries and any other long-term information that should be permanently stored.

The most notable features are:

Easily create pages, add content, link and personalise them.
Create powerful databases with different views to manage lists, notes, meetings, projects and much more.
Linked databases allow creating different views from existing data sets.
Available as an app for macOS, iOS, iPadOS and online with offline mode support.

Workflow: All short and long-term information is stored in Notion. The below image shows an outline of the structures I’ve created so far. Each category has further information and databases.

For example, I created an overview of all available modules at school and filter them by active modules (the ones I’m currently registered for). Each module has further entries with linked databases to store notes, due exams or other relevant information.

Example Notion landing page

2. Unclutter


Website	https://unclutterapp.com/
Purpose	Notes, Todos
Alternatives	Bear Evernote

I’ve been using Bear in the past to structure my information. The experience was never satisfying to me. Either some records got lost in the endless abyss of data storage, or I didn’t find what I needed. So I adapted my process when I switched from Bear to Notion.

This change introduced Unclutter as a new tool. It helps create quick notes and review them later on, to decide, if the information should go into long-term storage.

The most notable features are:

Superfast access via the scroll-wheel on the macOS menu bar.
The flyout shows the clipboard history, most used files (or folders) and notes.

Workflow: Notes or todo candidates are written directly into the flyout. The main criteria are that it is super fast accessible in any context, independent of the window that is currently active. The clipboard and files are just a bonus.

However, this only works on my computer. I do not have a reliable solution yet for iOS or iPadOS. There I use the standard Notes app alternately.

Example Unclutter panel

3. 2Do


Website	https://www.2doapp.com/
Purpose	Todos
Alternatives	Todoist Things

Even though Notion offers some todo functionality, it is far from usable. Instead, I use 2Do to manage my todos. Everyone should have a todo app/system in place to manage their tasks. Which one you use is mostly personal preference.

I choose 2Do because it had no subscription model compared to other tools available with free syncing to iCloud and no hidden costs.

The most notable features are:

Add, schedule and mark todos and categorise them.
Capable of adding recurring todos and checklists.
Synchronisation across all my devices.
Simple UI and fast creation of todos.

Workflow: I either open the app to create a todo or use the allocated shortcut, depending on what kind of todo that needs to be made. I try to keep the list as short as possible and time every todo. This process ensures that the todos will be done.

2Do is a holy place for me. They are promises to myself about my daily, weekly or monthly responsibilities and goals. As all of them are timed, having todos past their due date is a sin 😇. Anything else is just bad planning or evaluation from my end.

Example 2Do window

4. Habitica


Website	https://habitica.com/
Purpose	Habits, Todos
Alternatives	Habitify Strides

Habitica is fun. I have been working on improving my workflow and habits for a while now and came across this gem. It makes creating and tracking habits entertaining, and you can create or join challenges with other people. You can also invite friends into a group and work together.

I’ll send you an invite to our Party 🎉 if you send me your account information.

The most notable features are:

Add habits, dailies and todos.
A gamification system to keep you going with monsters, equipment, guilds and challenges.
Make habits with friends or other people.

Workflow: I’ve accumulated a set of habits that I keep track of since I started using Habitica. These include regular reading sessions, sharpening my skill set, eating healthy, playing the guitar and numerous more.

Personal challenges accompany habits. These challenges aim to gain new expertise or getting myself out of the comfort zone. For example, the creation of this website was a past challenge. My current challenge requires getting myself into the financial market.

Example Habitica overview

5. Airmail


Website	https://airmailapp.com/
Purpose	Messaging, Todos
Alternatives	GMail Spark

Everyone has an email client. The default Mail client on macOS has been a pain to me in the past. Apple either added weird updates or simplified the functionality to make it more accessible to anyone. So a more feature-rich email client was required.

Airmail covered all the functionality that Apple Mail lacked in the past. They have recently switched to a paid-subscription model, and so far I’m paying it.

The most notable features are:

Well, receive and send emails ✉️
Snooze and reschedule emails (todos, yay)
Multiple accounts with different providers
Apps for macOS, iOS and iPadOS

Workflow: Similar to 2Do, I tend to keep the inbox as clean as possible. Incoming mails that need no action get either archived or are moved into a specific folder.

Sometimes actions need to be taken from emails. Then I either create a todo and inform the sender, or I snooze the email to a point where I have time to deal with it. The general rule is to answer within 24 hours or less.

Example Airmail window

6. Calendar

This section will be short. I use the standard Apple Calendar and have not yet switched to another tool such as Fantastical.

The main reason is that I use the calendar for meetings only and not to schedule todos or deadlines. The only exemptions are birthdates or events.

Workflow: All calendars I have (school, work, private) have their feeds. These feeds are collected into Apple Calendar to generate an overview of all entries.

The calendar notifies me about upcoming meetings with a simple notification prompt. Though that’s not enough, so let’s have a look at “In Your Face” next.

7. In Your Face


Website	https://www.inyourface.app/
Purpose	Notification
Alternatives	Generally any calendar

If you ever had a member that missed a meeting, was late or didn’t notify you about their absence, mandate that they install the “In Your Face” app.

The notification prompt from a calendar entry is quickly closed and as we know: out of sight, out of mind. Suddenly you receive a message over Slack, that the meeting has started and you are missing. This experience is frustrating and embarrassing.

This app helps you by showing a colossal overlay blocking your entire screen, informing you that there will be a meeting in x minutes.

The most notable features are:

Shows an overlay blocking you from working
You can set the pre-notification time

Workflow: I set the notification to appear 2 minutes before the meeting. This timeframe gives me enough time to stop whatever I was doing and join the event.

Alternatively, if another task still occupies me, I can quickly inform the organiser about my absence.

Example In Your Face overlay

8. VSCode & iTerm


Website	https://code.visualstudio.com/
Purpose	Productivity
Alternatives	Alacritty IntelliJ IDEA

VSCode is my primary code editor. But I still use iTerm and Vim as companion tools for quick editing or running commands. Tmux has also become a great automation tool to streamline project initialisations. I’ve used Hyper in the past, but moved back to iTerm because of stability issues.

Other than that: ws + cd angelo.dini + code = win.

The most notable features are:

Fast startup and excellent stability.
Numerous extensions and exceptional community support.
Easy to get used to, challenging to master.
Automation capability via CLI

Workflow: I won’t go into much detail about my development workflow as it consists mostly of coding. However, I recommend having a look at my dotfiles, as they show the setup quite thoroughly. I’ve also created a blog post writing about my dotfiles setup there.

Example VSCode window

9. Tower


Website	https://www.git-tower.com/
Purpose	Productivity
Alternatives	Sourcetree GitHub Desktop

Git is a blessing and can easily be handled from the terminal, but visualisation of code changes is underwhelming. Most of my Git workflow involves the command line interface via iTerm, but for some, I prefer a GUI over additional Vim plugins.

Particularly for reviewing diffs and staging files or chunks, the command line lacks these visualisation tools. It’s very advantageous to scroll and navigate fast between two files from an interactive window and ensure the correct data gets committed. The rest is perfectly handled via the command line.

The most notable features are:

View visualised diffs
Push, pull, stage or stash files
Manage repositories, remotes, tags and submodules

Workflow: I use Tower only for its diff view. Occasionally I also fire up Kaleidoscope to compare files or folders in more depth.

Tower is excellent to assess the diff between commits as well as stage chunks or files. I’m used to the command line tools in general, but that workflow is still preferred to me.

Example Tower window

10. Plex


Website	https://www.plex.tv/
Purpose	Procrastination
Alternatives	Emby Kodi

Plex in itself is a hobby of mine. I’m a data hoarder and own a small server with my media files that I tinker around from time to time. I host all sorts of media files, not just TV Shows, Movies and Pictures.

The most notable features are:

Store and manage all your media files
View your media files anywhere 📺
Apps for macOS, iOS and iPadOS

Workflow: Plex is not the only tool used in that media-verse. It’s accompanied by Sonarr, Radarr, Lidarr, Mylar, Calibre, SABnzbd, Transmission and Ubooquity with lists stored on IMDb, Goodreads and Trakt.

That toolbox allows me to automate and streamline the process of downloading, archiving and viewing media files on all my major devices. It might be worth to create a blog post about it once 😎.

Example Plex window

Summary

This concludes the list of the top ten tools I use every day. Of course, this list is not complete nor final, and there are various other tools I’m using every day that I didn’t mention in more depth here. For example:

1Password, Grammarly, Chrome
Slack, Microsoft Teams, Discord
GoodNotes, MarginNote, MindNode, Anki
Affinity Photo, Affinity Designer

to name a few. You can find a complete list of all my applications in my dotfiles. I keep them up to date as I install new software.

Now it’s your turn. I’d love to hear from you about interesting tools that optimise your workflow. Add a comment or reach out to me via Twitter.

‘Till next time!

Improve your profile on GitHub

Sun, 20 Sep 2020 00:00:00 GMT

Create an engaging profile

GitHub is a fantastic platform to contribute and share your code. But it's not just that! Your profile page can provide a first impression on fellow members or companies interested in your work.

Like a resume, it should present your most significant work as well as personal interests and affiliations. Many members neglect the importance of an attractive digital appearance. To improve this, make sure to:

add a short bio about your interests, goals or aspirations
add information about where to reach you, like Twitter or a personal website
create a showcase of repositories you have been considerably involved in, so your skills are presented favourably

I’m not particularly interested in how many followers I have or whom I follow so far. So I’m leaving this off the list for now.

Express yourself by adding a GitHub README

The information provided above is static; it looks the same on almost every GitHub profile. It would be great to express yourself further by adding some custom content to stand out.

I recently discovered that you can do just that by adding a customisable README to your profile:

My customised GitHub profile

GitHub even has documentation available, showing you step-by-step how to do that.

In short, you have to create a new repository with a name that matches your username. So for me, it was github.com/finalangel/finalangel. Next, you create a README.md file and add the content you want to appear. Once ready, commit it, and your profile page will automatically show the content.

There are many resources available on how to use markdown to create a README. I recommend to familiarise yourself with the syntax to get the most out of it for your profile. Alternatively, you can use HTML right in the README.md file to structure the content instead.

Finally, I would like to mention some handy tools and inspirations I came along while creating my README.

Make the most out of your profile

Adding a README to your profile is a simple improvement with many possibilities. You can even add dynamic elements, charts, blog posts, tweets, gifs and much more. A great resource is awesome-github-profile-readme that references many creative examples of other users’ profiles.

There are also great dynamically generated GitHub widgets available on github-readme-stats which are simple to implement. They can enrich the personality or message you want to express.

In addition to that, you also get a free analytics tool for your GitHub profile. Go to the created repository and open up “Insights” followed by “Traffic”. These statistics offer some great insights into the number of visitors your profile receives.

Conclusion

Overall I recommend to keep it simple and don’t over clutter your profile page. The same goes for your resume. Less is more, keep it simple and focus on what you want to communicate with your visitors.

‘Till next time!

Getting into Gatsby

Mon, 31 Aug 2020 00:00:00 GMT

Creating Gatsby powered websites

It seems like we are going back in time, building static websites without server-side code or database: introducing Gatsby.

Gatsby is a framework that builds upon the modern React library. Similar to Next.js, it aims to provide accessible tools for developing static progressive web apps (PWAs).

It’s easy to get into it, if you know your React, and helps you to build fast and reliable websites that consume little resources.

Gatsby is a free and open-source framework based on React that helps developers build blazing-fast websites and apps. gatsbyjs.org

This article summarises my experience building the following three websites using Gatsby:

flavours.dev
divio.com
and my website
(I’m excusing myself from adding a link here 🤪)

I also do not intend this to be a step-by-step tutorial on how to get started with Gatsby. For that, have a look at Gatsby’s excellent quick-start documentation.

Getting started with a starter

Yeah… I just promised that this is not going to be a step-by-step tutorial, though I have to mention the fantastic starter projects Gatsby provides.

There you find over a hundred different opinionated Gatsby setups using best practices and plugins. You can quickly bootstrap these from the Gatsby CLI.

We at Divio, used gatsby-universal as an initial starter project to create flavours.dev. It is still a great starter. However, today I would recommend gatsby-advanced-starter, as it ships with more integrations out of the box and is actively maintained.

Know your React

Gatsby is based on React; as such, it is imperative to know React before trying out Gatsby. So head over to reactjs.org, go through the tutorial and create a few sample React apps. You don’t have to be a React developer with ten years of experience to handle Gatsby, but at least familiarise yourself with React before giving Gatsby a spin.

If you have already ten years of experience, let me know where you got those, considering React was released in 2013. Alternatively, if you are fluent in React, try setting up a Gatsby project from scratch and integrate TypeScript. It is a great combination. We did that while developing divio.com, and it was fun 🥳.

Taking the first steps

So why did we choose Gatsby over a traditional content management system? Well, because flavours.dev seemed to be a straightforward one-pager project with some content/assets and we wanted to have some fun with new technologies on Divio Cloud.

We also needed the website to be super fast and responsive (the fast one), which is hard to achieve with dynamic content.

Though our journey wasn’t without problems, obviously, best practices still need to be applied when dealing with the front-end code. Also, SEO needs to be spot on, by knowing which code to optimise where. That is where the starter helped us at the beginning but turned sideways once we needed to add further customisation.

Some originated from not understanding how GraphQL worked in Gatsby’s context. Gatsby uses Page Queries and Static Queries through either the <StaticQuery /> component or the useStaticQuery hook. Just the sentence alone stirs up headaches. You have to know when to use what to avoid side-effects. Find out more on the Gatsby documentation if you want to dig into the differences.

I also had to get used of the concept of <Helmet> at the beginning. It does a lot, and you can extend your pages easily with additional markup such as meta tags to enhance SEO. Many starters use Helmet, but you have to wrap your head around the implementation of each.

Another issue we faced often was the cache. While gatsby develop offers an excellent development environment, it’s far from perfect. Even gatsby serve has its short-comings, and you always need to re-run the command to get to a testable state. Nothing beat testing on the production server at the end.

We achieved a pretty remarkable result whilst keeping best practices in check, all thanks to Gatsby’s wonderful tools, plugins and community:

Speed score from 31st of August 2020 using Lighthouse

Taking the next step

Once we finished our first project, we eagerly wanted to convert divio.com as well, with the following goals in mind:

We wanted TypeScript with Gatsby to share code between the marketing website and the Control Panel.
Add further linting and automation tools to cohere with the standards from the Divio Cloud.
Improve the deployment process, to automatically deploy to Test/Live on Divio Cloud when a Pull Request is merged to master/develop.
Find a way to integrate the Blog system into Gatsby.
Use Storybook to share stories.

divio.com was undoubtedly a larger project. We had to migrate a substantial amount of content from the existing website into the new one.

For this, we converted most content into Markdown files and loaded them through GraphQL. We did little experimentations already on flavours.dev but now we had to build a more robust system. We could have also integrated other tools such as Contentful. However, we chose a more minimalistic approach.

We practised the same approach for the Blog as well, basically migrating them to single Markdown files with an appropriate folder structure to source them through GraphQL.

The integration of TypeScript, additional linting and deployment tools were pretty straightforward. The deployment bit was a bit more complicated, so I’d like to share the code for future references:

# .gitlab-ci.yml
image: node:latest

stages:
  - Build
  - Deploy

build:
  stage: Build
  script:
    - npm install
    - npm run build

deploy:
  stage: Deploy
  before_script:
    - npm install -g @divio/cli
  script:
    - echo ${DIVIO_DEPLOY_KEY} | divio login
    - divio project:deploy live --remote-id ${DIVIO_APP_ID}
  rules:
    - if: $CI_COMMIT_BRANCH == "master"

Diving deeper

I still followed the same principle when creating my websites. Most of the code was shared back and forth between the Divio marketing website and my own.

We added nice animations through Framer Motion, loaded data dynamically via APIs, worked with form validation and submissions, added page transitions, played around with gatsby-image, dug into Gatsby’s Node API and even added automated tests.

You can even subscribe to my Blog using an RSS Feed. All hosted neatly on Divio Cloud with fast responses and fantastic support (cough 🤷).

It has been an incredible journey so far, and I’ll keep using Gatsby, for now. If you’d like to hear more about implementation details or have some questions, let me know.

‘Till next time!

Automation using Dotfiles

Mon, 27 Jul 2020 00:00:00 GMT

Start hacking in just a few hours

It's a beautiful, warm and sunny day outside. You are longing for your after-work beer to enjoy the rest of the day. But there is this super important project that's due by the end of the week and it's just 10 in the morning. Suddenly a wild hardware failure appears and you are out of pokeballs to solve the problem and safely continue.

Your computer doesn’t boot up anymore. You are hurrying to the next hardware store to get a replacement. Sitting through the setup or restore process takes a lot of time and you still have to set up the project as well to continue your work. It’s the end of the day. 😠

In fairness, such a scenario does not happen that often, ideally. Though setting up a computer can be time-consuming. On top of that, all the little tweaks and settings that have been made over the years cannot easily be restored.

Dotfiles to the rescue

I first encountered the concept of dotfiles about 8 years ago. I was automating the installation of multiple macOS workstations through scripts that were executed on the host system. osx-bootstrap, a collection of simple bash scripts, was my attempt to solving that problem.

In essence, dotfiles are no different from the solution way back. They are a collection of scripts that run on certain conditions. Their power is only limited by your know-how of the environment/system. They can configure your entire system automatically, update apps and run maintenance scripts.

This inspired me to create my dotfiles. Their purpose is to shift my entire systems configuration into little neat shell scripts. Now, I can update and configure my machine with ease and without hassle.

So if my computer decides to go to Valhalla, I simply get a new one, run the scripts and I’m ready to go. Sadly, the scripts do not automate the actual work yet ¯\_(ツ)_/¯.

How to get started

I highly recommend to fork and review the code first, before giving my dotfiles a try. In theory, they can be installed on an existing system as well, but a fresh installation is recommended.

To get started, follow the installation instruction in the readme. After a successful installation you should be able to run the following command in a terminal:

dot --help

This lists all currently available commands that can be run. The options currently are:

dot --install runs the installation process again. It’s intentional that this can be run again, as you make changes to the dotfiles yourself, certain installation actions need to run again.
dot --update runs the update process. This should ideally be run regularly as it runs version updates on several packages.
dot --edit opens the dotfiles directory in your favourite editor so you can start hacking and adapting as fast as possible.

More functionality might be added in the future or add yours inside bin/dot. It’s also helpful to run alias in the terminal to see all available shorthand commands. Dotfiles normally add a bunch of them to your system.

Running my dotfiles in a shell through "dot --update"

What’s the structure like

I decided to follow a modular pattern in my dotfiles. I didn’t want to spend much time figuring out where to make changes or add new features. Especially if I just discovered a new configuration/feature to integrate.

As such, each folder contains its own independent logic/topic without much cross-linking. There are other patterns around. If you are interested, check out my credits section on GitHub with links to other repositories for inspiration.

There are exceptions I’d like to highlight though:

bin/ stores binary files that are added to the system $PATH automatically. If you want to create your scripts, simply add them there.
macos/ and linux/ contain system-specific configurations.
scripts/ holds the main install and update scripts that include other modules.
utils/ contain utility variables and functions that are used throughout the dotfiles.

In addition to that, most folders implement an install and update file that handle the respective install/update tasks. Even though most operations are almost identical (and some could be merged), I still prefer the separation of context for better organisation 📁.

What’s included

The dotfiles contain my flavour. As such, the tools used are tailored to my workflow. You might not agree and want to use alternative options. That being said, let me highlight some of the features included:

The zsh/ module automatically loads every aliases.zsh, paths.zsh and functions.zsh from other modules. This breaks the independence of that module but is a necessary design decision to keep maintainability.
The system/ module also breaks that chain by creating symlinks of every *.symlink file into the users home directory ~/.... This is very helpful to keep configuration scripts, such as .gitignore or zshrc, inside the dotfiles.
The fonts/ module installs the most used fonts into your system. Gone are the days of looking for your CI/CD fonts.
The git/ module automatically configures your git settings and even helps set up a GPG key.
The homebrew/ module installs common packages and applications without downloading them manually from every website.
The node/ and python/ modules configure my respective node and python environments to start hacking.
The tmux/, vim/ and vscode/ modules setup my development environment and editors.

There is also a hidden private/ folder that holds scripts, that should not be public. I backup these settings through “Mackup” to my iCloud drive. The module holds scripts for my morning routine, our standup (opening google meet, load standup board, share screen, set do not disturb mode etc.) and other work-related automation.

Simply automate your monotonous tasks and let the computer do the heavy lifting 🏋️.

Conclusion

Dotfiles are very powerful. With the right know-how, you can increase your productivity very efficiently. Though, it also requires an initial time investment setting everything up and then maintaining them. Your dotfiles evolve, as such, they are never finished. You gain knowledge and expertise that change your flow or tools. You may switch jobs and your expectations from the dotfiles are suddenly different.

My dotfiles are still very much focused on macOS. I’m playing around using Linux/Manjaro on a separate partition. The support for Mac hardware is very limited and several manual configurations are required. So it feels natural to incorporate them into my dotfiles.

I’d like to write a separate blog post about that adventure in the future 😉

‘Till next time!

Finally, a website

Sat, 06 Jun 2020 00:00:00 GMT

Look! It’s moving. It’s alive. 🧟

Creating this website was hard. The final result seems "simple" and "clean" but you know, it wasn't.

Development for this website started in December 2019 with the first mockups and drafts. Back then, the website looked quite different. It took roughly two separate designs and 6 months to get to this point. I’ve been able to overcome many obstacles to deliver a website I’m finally happy and proud about. It might still have glitches here or there but hey, nobody’s perfect. 🤷

It’s not just the visuals that mattered to me but also what’s under the hood. I tried to make the website as fast, intuitive and elegant as possible while still looking good. The devil lies in the details… The website was created using Gatsby, a very awesome React-based framework, and deployed using Divio Cloud to automate most of the boring stuff.

Design is not just what it looks like and feels like. Design is how it works. Steve Jobs

But I didn’t want to create just another website to wither away, it actually should serve a purpose 🥳. Yeah, really! I’m going to start a new chapter in my life by accomplishing a life-long dream of studying computer science. There are many opinions around on how to optimise studying. Ultimately everyone has to find their path. I try to find mine by summarising topics such as development, studying and teaching into interesting blog articles. By doing so I’ll be forced to do additional research, be precise and simplify as much as possible.

If you can’t explain it simply, you don’t understand it well enough. Albert Einstein

My goal is to publish at least 1 blog post per month (to my future self: if that doesn’t work out, travel back in time and beat the s**t out of me…). The next blog post will be about my recently completed dotfiles setup, so stay tuned and subscribe to my newsletter. I promise there will be no spam!

Thanks to everyone who helped testing while providing valuable feedback along the way. Don’t hesitate to contact me if something is not working as intended.

So long, and thanks for all the 🐟.

‘Till next time!

Getting your habits in order

Thu, 04 Jun 2020 00:00:00 GMT

How our habits work

Habits shape us, our lives are affected by them every day. Most people don't even recognise the intricate balance they hold over our daily routines. Have you ever considered that your morning coffee is actually not a choice, but rather a habit that enslaves you to pick up a cup in the morning and pour in hot, black, liquid fluid?

These behaviours are encoded inside our basal ganglia, an area which is found deep within the cerebral hemispheres of our brain. This is where our habits, good or bad, are formed and retrieved.

Change is hard

A lot of actions that we do intuitively are automatic, habitual behaviours. Our brain switches to autopilot and doesn’t need to think. This preserves several finite mental resources such as willpower. Willpower plays a key role in our daily driver of getting things done.

Formed habits do not draw from that pool. This is why it is sometimes so hard to change bad behaviours. You lack the will (power) thereof.

In our example, this vicious cycle starts with a cue triggered from your morning routine rewarding you with the first accomplishment of the day: a coffee.

Cue, routine and reward

To grasp the basic concept of this circle, we need to analyse the underlying psychological patterns to be able to influence and ultimately, assert control over our habits.

Cue: The cue is the first step in this process. You can identify the cue through 5 signals:

Where you were
What time it was
The emotional state you were in
The social connection around you
What preceded the cue before the habit emerged

Let’s apply our coffee example to these signals; The cue occurred in the morning, at the time you woke up. You were tired, your partner still in bed and eager to start your day.

Routine: The routine is normally easily identifiable. This is the process you fall into day by day. In our above example, it’s the process of waking up, getting your motors running.

Reward: That’s the craving that needs to be satisfied. It is imperative to identify the reward before you are able to change your habit. In relation to our example, the reward can be the fresh taste of coffee in the morning or the effect it has throughout your day.

Improve, one step at a time

Identifying the cue is the most difficult puzzle to solve. Besides the mentioned 5 signals, you can analyse the routine you are trapped in or figure out what the reward is.

‘Till next time!