Part of “Letters from the Lab”, a series of informal essays on my research written for patrons. Originally published October 2021; made publicly available January 2022.

It’s board game night in a post-COVID world. You and a few friends gather around a table to try out a new game. The thing is: there are a lot of these little cardboard tokens and playing cards and—I’m not even sure… totems?

You pick up the instructions and begin to read aloud to the group, but after a few minutes, everyone becomes restless. So you figure you’ll just start and figure it out as you go. Now the first turn is taking forever because you’re returning to the instructions every ten seconds, but you keep losing your place and needing to reacquire it each time; and after the second turn’s over, you realize you need to unwind the first turn because you made an invalid move which will screw up the game; and you’re constantly straining to remember what you’re “supposed to do”; and it’s hard to emotionally commit to the game when you know you’ll only be able to play a few moments before returning to the manual.

Maybe let’s just watch a movie instead?

I’ve written plenty here and elsewhere about one problem with books: that we tend to rapidly forget all but the gist. So here’s another problem with books, and specifically with books meant to help you learn a skill: the medium makes it difficult to collapse the distance between prose and action. Books rarely involve doing what they’re about. Most books, even books intended for skill-building, are only about what they’re about. Reading about the board game, not playing the board game. Reading about counterpoint, not composing counterpoint.

So what? I’ll elide formal learning science here in favor of an appeal to experience.

If you ask people about the highest-growth periods of their life, you’ll notice that the most enabling environments tend to involve doing. A summer spent preparing intensely with your team for an upcoming competition; a startup which failed but taught many valuable lessons; a challenge accepted to write a new song every day for a month; a multiweek meditation retreat; an overwhelming apprenticeship; etc. Books are sometimes a source of knowledge in these stories, but they’re often secondary to a great mentor, teammates, contextual motives, etc—and crucially, to doing.

The analogue here for our board game problem is familiar: consider playing a board game for the first time, but with someone who’s played before. This is a completely different experience from the “cold start” I illustrated earlier. In this situation, the experienced player might give you a brief introduction—nothing which would strain your memory—and then you’d simply begin playing. They’d handle setting up the board, either by themselves or by telling others to shuffle these cards, distributed those tokens, etc. They might narrate: “I’ll go first, to demonstrate. So I start by drawing two cards, then I can choose to either move here or play this card. I’ll move here, which will block John from moving into this open area. Now you’re up. Your goal is to XYZ, and you might start by moving this way. Now if you drew an action card, you can play that immediately if you want; otherwise, read what it says on the card…” As events occur throughout the game, they might continue, narrating what you need to know, just-in-time, explaining the options you might consider, offering feedback. So long as your experienced friend is graceful enough to avoid veering into overbearing Clippy territory, this is a much more pleasurable—and effective—way to learn to play the game.

This might be a consistently great way to pick up a board game, but skill-building books have many practical advantages. Consider information density. If you want to learn to program a quantum computer, there’s a huge amount of material you need to absorb before you can “play” much yourself. This might involve tens of hours of explanation from your experienced companion, which would quickly become burdensome for most people. Explanatory prose might lack personalization and interpersonal connection, but it can be more carefully honed; it does not tire and is ready whenever you are; it can embed figures and abstract notation; it can be consumed non-linearly; it can be read more quickly than speech can be heard; and so on. Perhaps most importantly, it’s a mass medium. The deepest experts and sharpest communicators in the world can craft a book on this topic, and millions of people can hold it in their hands for effectively zero marginal cost.

So: how might we create a mass medium which possesses the book’s advantages, but which is situated in doing? How might we create an explanatory mass medium which feels more like playing a board game with an experienced friend than playing a board game while juggling its instruction manual?

The role of the dynamic medium

One reason it’s hard to create books which are situated in doing: the book is static, fixed. You, the reader, must ferry its words to an environment where you can “do”. And even then, there’s little opportunity for interplay between the doing and the words on the page. Authors can advise how to reflect on an exercise and generate your own feedback, but these are scripts you must execute in your own mind. Video doesn’t appreciably change this situation. But the promise of computers, and the dynamic mediums they enable, is representations which behave and respond.This framing comes from Bret Victor’s “Stop Drawing Dead Fish”. Here, immutable figures (whether printed or animated) are “dead fish.”

People often propose we leverage this property to integrate simulated environments for doing. Maybe the biology textbook can embed a little simulated petri dish, which you can use to “do” certain kinds of cell biology, reducing the distance between text and action.

But at least where it’s possible, I’m much more excited about what happens when a computational environment becomes the authentic—not simulated, not “educational”—environment for doing. A non-linear video editing interface isn’t a “toy” way to edit a film; they’re how professional filmmakers actually edit films. Mathematica isn’t a “toy” way to manipulate symbolic expressions; it’s how certain kinds of mathematical work are authentically best done. And so a dynamic “book” about video editing need not involve a “toy” environment for doing, a simulated petri dish. Rather, it can situate itself in the same sort of environment used to edit the best films on Earth.

But what does it mean for an explanatory medium to “situate itself” in an authentic environment like this? How might the explanatory content interact with the contents of the environment?

In the last decade, authors and programmers have created dozens of interactive articles which might inform our answer (see Communicating with Interactive Articles for a good overview). I find the work in this space very inspiring, personally speaking. But I don’t know of any which quite matches the aspirations we’ve discussed so far. These articles may be interactive—may involve some doing—but the doing is situated in little purpose-built sandboxes, not the actual environments you would use to deploy the skill being built. They’re integrated with simulated petri dishes, not an actual lab bench.

An article on a topic in programming, for example, might be structured as a long text interspersed with small interactive code editors you can use to explore a concept. This is certainly an improvement on the typical paper! But relative to our aspirations, “doing” is very much the secondary activity here. These editors don’t much resemble an actual programming environment; you’d have to jump through hoops to take anything you’d done and apply it to a real program. This is a bit like reading a board game instruction manual with interactive pictures depicting simplified parts of the board game. Or, somewhat more unfairly: it’s a bit like an elaborate pop-up book. You’re still not really doing the thing.

If these interactive articles are often structured as wide seas of prose which contain islands of interactivity, one path to integrating authentic environments might be to invert this structure. How might we create “articles” which are primarily interactive environments, but with embedded islands of prose? Taking programming as an exampleWhat about computational notebooks like Mathematica and Jupyter? I don’t think these qualify; I don’t know of any strong examples which are “doing-centric”, where the code listings interspersed throughout the text are really about doing the topic in question. As Pavel Panchekha pointed out to me about his Web Browser Engineering book: this format seems focused on helping readers understand the code the author has written; it’s not so well suited to supporting the reader as they write the code themselves.: rather than a textual document with embedded source listings, could we move the whole experience into your IDE of choice, while somehow still presenting the explanatory text? Could we move the YouTube lesson on 3D modeling with Blender into Blender?

Video games excel at this. Sometimes tutorials appear in non-interactive cutscenes, sharply delimited from ordinary play, but better examples (e.g. Portal) present instruction and narrative as a seamless element of the interactive environment, never “stealing” the camera or the controls away from the player. The result is rich immersion in the game environment—a stark contrast with board game instruction manuals.Of course, some of the very best examples (including Braid and The Witness) use no overt explanation at all. This is an extremely powerful approach I’ll have to discuss at another time.

Video games also improve upon another problem of interactive articles: the challenge of separating prose and dynamic representation. These articles close the distance between text and interactive elements, for instance by linking a number in the prose to a parameter you can directly manipulate in a figure. But in most cases, they’re still physically separated, not visually integrated. The reader’s eye bounces back and forth between the text and the interactive elements, churning working memory to attach referents to objects. This isn’t just a problem for dynamic elements. Static figures in traditional texts have the same problems. But the solutions described in Edward Tufte’s books are rarely applied in the dynamic domain, perhaps because the authoring tools are more complex and isolated. Almost everyone, almost always, is still “separating by mode of production”.

Video games take advantage of audio to layer instruction onto what the player’s seeing, but even when using only text, they can position that text right next to the relevant part of the action. This arrangement allows games to avoid disruptive ping-ponging between instruction and interaction, as we experience with board game manuals. And because the narrative communication is integrated into a dynamic environment, it can behave and respond just like the rest of the environment’s elements. In good games, the authored narrative feels like a continuous response to players’ actions. This collapses the sense of distance one feels when reading a text separated from the “doing” it’s about.

Enter the Figma document

Oddly enough, the proximal cause of this letter was an extremely unusual Figma document.

(If you’re not familiar: Figma is, roughly, a collaborative tool for designing the visual representations of software interfaces.)

I’ve been assembling notes on this article’s topic over the past two years. At some point, I’d like to build some prototypes around these ideas and publish a much deeper treatment. I don’t feel I’ve gathered strong enough ideas for that yet. But you’re reading this now because a Figma document demanded that I write a preliminary “IOU” of sorts.

Figma changed the way copy and paste works in their interface. They made a document to explain the change to their users. I know this sounds awfully mundane. Stay with me. I encourage you to take a look yourself before continuing: click the “Duplicate” button to get started, then zoom in on the upper-left frame. You can explore the document with a free account from your web browser.

The document initially reads like a hypertext slide deck. It shows visually how the new clipboard functions behave. It’s cute that it’s a document about using Figma, both created and consumed in Figma, but that’s not so special. Interleaved with explanatory preamble, the document turns control over to you:

This is where we break down the wall between authored material and authentic “doing.” You’re invited to manipulate the objects which the author has created. The objects aren’t special; they’re the same “kind” of objects which you could create yourself elsewhere. You can copy and paste them into a new document. And when you manipulate these objects, you use the same tools the author used to create the document. More importantly: you use the same tools which you yourself would use to do authentic work in this space.

This could have been a blog post, with little interactive “demo” areas interspersed between paragraphs. But instead, you’re interacting with this document in the full-blown Figma environment. In the design world, this is a lab bench, not a simulated petri dish. Other than the scaffolding text, there’s no distance between the “doing” here and the “doing” in your own creative work. Reading this document, I found myself curious how the paste behavior would work if the groups were structured differently, so I simply used the tools I already understood to set that up, and answered the question for myself. Then later that day, I found myself—without even really thinking about it—using one of the new paste behaviors in a layout I was designing.

It’s important to recognize that there’s a lot of text in this document. That’s part of what makes it so interesting. “Worksheets” aren’t so unusual—there are lots of Figma documents like this which give you an exercise and some context to play in.

By comparison, the copy/paste Figma document is unusual because it’s roughly a thousand words long. There’s a lot of explanatory material here, and there could be even more. The possibility of expansive, in-depth documents opens the door to canonical works in the medium. Separately, the interaction between the authored material and the activities is much finer-grained in the copy/paste document. This creates shorter, more precise feedback loops, closer to the experience of a video game tutorial or to playing a board game with an experienced friend.

This is a Figma document about using Figma. That’s useful, insofar as an elaborate instruction manual can be. But it’s not a stretch to imagine a much more significant variation: a “textbook” about interface design, written in FigmaSuch a primer would be particularly valuable since, oddly, there is no “standard text” for user interface design.. In this primer, you wouldn’t just be reading about how to design. You’d actually be doing design, embedded within the environment you’d use as a professional. Because there would be no artificial boundaries between explanation and action, I believe such a book could come much closer to the feeling one gets playing a well-designed video game tutorial.

Elaborations on the Figma document concept

Of course, the Figma “meta-document” medium can itself be pushed much further.

The explanatory text in the copy/paste document does not behave and respond. It’s not really a dynamic medium for authors. The fine-grained exchange between explanation and action makes it easier to help readers generate their own feedback, but the medium could go further by reacting to the reader’s actions. Earth, a Primer demonstrates one simple approach, “checking off” suggestions as readers complete them:

We can also imagine topic-specific computational elements. A chapter on color theory could use linked representations to visualize secondary and complementary colors in response to your choice of “main” colors. A chapter on grid systems could help you visualize how different choices of type hierarchy ratios influence the baseline rhythms in your design. A chapter on accessibility might embed contrast ratio meters into your design canvas. Scaffolding elements can fade according to your comfort level with the material. And so on.

Another interesting direction—and one which fits well with Figma’s multiplayer primitives—would be to integrate opportunities for collaborative learning into the text. A standard “move” in collaborative learning is: introduce a problem which can be solved with several different approaches; juxtapose several contrasting students’ solutions and help students learn from each others’ ideas. A Figma design “textbook” could incorporate both solo and “shared” artboards, and potentially asynchronous orchestration features, to help students exchange ideas. The approach could even involve a facilitator and include tools like Desmos’s to separately support their work.

More broadly, the interactive explanatory medium I’m suggesting wouldn’t be limited to “educational” scenarios. They could also be quite useful in the course of working on some meaningful project. For instance, if we’re working on a design for a new operating system, and I invent some new control, I might present it to the team by creating not just a static Figma document, but a document which, through “doing,” helps you see how to use it in a design of your own. Such documents would also be useful as just-in-time professional reference: for example, something like this would come in handy if I’m designing for expanded-color-gamut displays for the first time. “Portals” in these documents might allow you to bring your own design projects “into” the explanatory document, so that you can understand the concept in the context of some real work you’re doing.

Extensions to other environments

Figma is certainly not the only environment which could support a format like this. Where else could we instantiate something similar? What qualities must a system have to enable this kind of document? I can outline at least a few.

Documents must offer some way for authors to communicate explanatory content, potentially at length. Figma has text elements; programming environments have comments. In other environments—like an audio production tool—we may have to add such affordances to the document model. Or maybe the explanatory content can be delivered through audio (or video) channels, though it would be important to ensure that readers could manipulate objects in the document without disrupting playback of the author’s material.

Authors must be able to establish relationships between passages of their content and corresponding elements in the reader’s interactive environment. In Figma, we can position text immediately above each element you’re meant to manipulate. In Finale (a music composition environment), text can be interleaved between musical staves or positioned above specific phrases. In a Roam-based medium, the outliner’s hierarchy can be used to relate author text and reader text.

Authors need ways to linearize and structure their explanation; readers need affordances for “navigation.” In Figma, artboards are arranged in a sequence which can be easily navigated by scrolling or through keyboard shortcuts. Hyperlinks and a “table of contents” artboard support navigation. And the hierarchical list of layers offers a persistent table of contents of sorts. In a programming environment, a sequence of tabs might approximate Figma’s artboards. Or perhaps, as in Natto’s tutorial, we can improve on these with special affordances.

In what other environments might we easily imagine creating such documents?

• A Hypercard document about writing good choose-your-own-adventure games (surely people have already used Hypercard in this way?)
• A primer on game development presented as an Unreal Engine document.
• A workshop on harmonic analysis presented as an extremely elaborate Finale score.
• A hands-on adaptation of How to Take Smart Notes as a Roam graph.

Here, I’ve limited myself to documents one could create in environments as they exist today. But rather than constraining ourselves to existing affordances, perhaps we’ll one day design systems like Figma so that they better support documents of this kind.

Thanks to Molly Mielke for discussion on the background of these Figma documents. Thanks also to Michael Nielsen and Jonathan Blow for past discussions which have helped shape these ideas.

My work is made possible by a crowd-funded research-grant from my Patreon community. If you find my work interesting, you can become a member to help make more of it happen, and to get more essays like this one.

Special thanks to my sponsor-level patrons as of January 2022: Adam Marblestone, Adam Wiggins, Andrew Sutherland, Ben Springwater, Bert Muthalaly, Boris Verbitsky, Calvin French-Owen, Dan Romero, Dwight Crow, Eugene Soltes, fnnch, James Hill-Khurana, Jesse Andrews, Kevin Lynagh, Lambda AI Hardware, Ludwig Petersson, Matt Knox, Mickey McManus, Mintter, Patrick Collison, Paul Sutter, Peter Hartree, Russel Simmons, Sana Labs, Tim O’Reilly, Todor Markov, Tom Berry, Tooz Wu, William Laitinen, Yaniv Tal.

Next unlocked “Letter from the Lab”: Armories for tool-maker/tool-user collaborations