Part of “Letters from the Lab”, a series of informal essays on my research written for patrons. Originally published December 2021; made publicly available April 2022. You can also listen to this essay (16 minutes).

It is the destiny of computers to become interactive intellectual amplifiers for all people pervasively networked worldwide.

— J.C.R. Licklider

If you want to help make good on this destiny—to invent human-computer interfaces which radically expand human cognition and creativity—then what do you actually need to do? How does progress happen?

Moreover, how can we make collective progress? What potential is there for shared knowledge, frameworks, methods, values, and traditions which mutually accelerate many separate lines of exploration?

Can—or, should—“tools for thought” become a field of science? A design discipline? A “scene” in the arts? A practice of craft?

It’s not an abstract question for me: it’s a very real question of how to shape my work throughout the day! And I believe this same confusion handicaps collective progress in this space.

For me, at least, no one of those labels seems quite right. Day to day, the process feels more like following my nose than following a playbook. But it can be awfully hard to tell what I’m actually sniffing for at any given moment, much less what I “should” be sniffing for. Is this trail about understanding something more deeply? About a sense of what could be? About a yearning for how things ought to be? About manifesting some kind of perfection?

My instincts draw on all four perspectives. Each mindset helps with the process of invention in its own way. But I think they can be synthesized into a clearer picture of what it means to contribute collectively to this project, to the proliferation and refinement of tools which augment human capacity.

Tools for thought as scientific field

The prize is the pleasure of finding a thing out, the kick in the discovery, the observation that other people use it — those are the real things.

—Richard Feynman, on the Nobel Prize

Tools for thought are not “discovered” in the same sense that DNA was discovered. Tools for thought are not governed by natural laws in the sense that strong nuclear forces can be understood. And so tools for thought are not a field of natural science.

That said, the drive to understand is a central feeling in my work, and an essential ingredient for most of my contributions. Likewise, some of the most valuable outputs I produce are a kind of knowledge. I don’t think of myself as a scientist, but I do feel a kinship with scientists through these traits.

What kinds of understanding and knowledge production are most useful to progress in tools for thought? Where are they a less helpful framing?

Herb Simon introduced a concept I’ve found helpful: some fields can be understood as “sciences of the artificial”. Physics and chemistry strive to understand natural phenomena, but certain kinds of progress in architecture and business management comes from studying artificial phenomena, “artifacts” which we ourselves have constructed.

This framing suggests a few powerful spaces where those pursuing tools for thought can create and share knowledge. One lies in deeply understanding the natural phenomena operating at the meeting point—the interface!—between the artificial environment and the natural environment in which it operates.

For software interfaces, that usually means cognitive psychology and domain details. Vannevar Bush’s arguments for the memex (giving rise to present-day hyperlinks, among many other concepts) rested on the associative nature of human memory. If you’d understood symbolic logic deeply enough, you too might have been able to invent Mathematica. Powerful interfaces often reify deep ideas about the reality they represent, so there’s great potential in cultivating such ideas.

SuperMemo, the first computerized spaced repetition system, owes its existence to ideas about human forgetting first established by Ebbinghaus in the late nineteenth century. Critically, though, author Piotr Wozniak’s real-world experiences with the system led him to form his own theories of forgetting, which in turn allowed him to improve his artifact. It seems paradoxical, but studying an artificial phenomenon unlocked novel insights about a natural phenomenon. This loop is what Michael Nielsen and I have called “insight through making.”

We can also study—perhaps even quantitatively and experimentally—the artificial constructs we’ve created, and their materials.

When Wozniak works to improve the algorithm of his memory system, some part of that work depends on the natural phenomenon of human memory, but much of the opportunity has also come from understanding the “artificial” dynamics of the scheduling system he created. Years later, I benefit from that knowledge when I build my own memory system, even though I have different goals. That’s the march of science, in its own way!

Usability studies are often about understanding the “materials” we build with. Fitt’s law led us to put menus and key actions at edges and corners. Nielsen Norman Group’s usability trials have honed standard controls like type-ahead search fields. My impression is that this sort of knowledge is helpful on the margin, but is rarely transformative.

For me, the most powerful kinds of understanding about tools for thought are often insights about what kinds of artifacts can be made in the first place.

The graphical user interface created at PARC—the one which established most of the primitives we still use today—rested on the deep idea that “doing with images makes symbols.” This idea produced a universal paradigm: we use a mouse to manipulate icons, which in turn represent abstract operations.

What kind of knowledge is this? What kind of knowledge is embodied in the creation—or is it discovery?—of abstractions like a “file” or an “app” or a “retweet”?

These sorts of knowledge expand the space of artifacts which can be produced. They suggest whole new categories of artifacts. When I’m on the trail of something like this, chasing a new abstraction, part of what I’m doing comes from a deep impulse to understand. But most of it comes from a strong sense of the possible, some tantalizing form calling to me through the haze.

When my attention shifts away from trying to understand what is, and toward chasing what could be, that’s a way of being I associate with design.

Tools for thought as a design discipline

The natural sciences are concerned with how things are. … Design, on the other hand, is concerned with how things ought to be, with devising artifacts to attain goals.

—Herb Simon, The Sciences of the Artificial

What an astonishing thing the computerized spreadsheet is. The central innovation—that a cell can contain not just a value but an expression which references other cells’ values—is embodied in a new primitive abstraction: the “formula cell”. Critically, apart from their values being automatically calculated, those formula cells behave just like other cells. Formula cells compose neatly into a larger whole, one which is already exquisitely tuned by years of use.

What does it feel like to invent something like this? Obviously, I’m not Dan Bricklin; I haven’t invented something on the order of VisiCalc. But I’ve had lesser tastes of the experience.

For me, it feels like tracing my finger over the seams of reality, the edges surrounding a problem space. In my other hand, I’m fiddling with a bag of puzzle pieces, some of which have shapes which partially match those seams in reality. I’ll start collecting pieces which might fit, rotating them this way and that, combining and sometimes reshaping their edges, until all of a sudden there’s a sort of ker-chunk. Some group of puzzle pieces click into place against reality’s seams. Now I see a new whole, where there had once only been parts. Now I see those seams of reality more clearly, and I can create a better fit. Now I glimpse my puzzle pieces in a new way, and I see how to use them to reach this larger seam over here. Now I watch the seams resonating against the edges of the artifact, and I see how to play with the modes of oscillation. I start to harmonize.

In Bricklin’s case, I can imagine some of the puzzle pieces that might have been at play. I can imagine turning them in my hand. Homoiconicity. Symbolic reference. Linked representations. Array programming. Late binding. And so on. It must have felt extraordinary.

When I’m working on primitives like this, the main drive I feel is a sense of creative possibility. Understanding is involved, and it’s important; big steps often come from understanding more deeply. But the urge to understand feels secondary in these moments to some broader process of creation, of making good on what could be. Sometimes it feels like I’m not creating, but discovering. I’m seeing an image suddenly pop out of a Magic Eye rendering. It was hiding in plain sight all along!

Progress in design comes from inventing new primitives, finding new ways to combine old ones, spotting new places to apply them, and so on. Growth feels like an accumulation of patterns, principles, and methods. We may find unifying principles and frameworks now and then, but they’re forever contingent. We’ll never “hit bottom” because problems, and their solutions, are endless.

Collective progress, then, looks like effectively sharing this accumulation. Some new patterns (ideas about coordinating Mechanical Turks?) will only apply to a few practitioners. But many will be at least partially relevant across many domains. Direct manipulation, linked representations, copy and paste, and menus are good examples. More recent wide-reaching primitives include “multiplayer” editing affordances, increasingly reliable voice-to-text interactions, and, yes, contextual backlinks. At the periphery, more tenuous ideas—the pervasive financialization of computational primitives; ML-generated media; etc—may end up becoming important puzzle pieces.

Can this sort of design become a science, a design science? Simon suggests this will require “a body of intellectually tough, analytic, partly formalizable, partly empirical, teachable doctrine about the design process.” If we had such a thing, walking the search space might feel more like following GPS directions and less like stumbling through fog. Certainly many people have tried to formalize the design process in this way. I’ve not read that literature deeply, but I’m pessimistic about this. At least for the near future, I think innovative design ideas in tools for thought will continue to require a fuzzy process of ingenuity, luck, and fortitude.

For me, at least, progress also requires a kind of expressive yearning.

Tools for thought as an artistic scene

Every portrait that is painted with feeling is a portrait of the artist, not of the sitter. The sitter is merely the accident, the occasion. It is not he who is revealed by the painter; it is rather the painter who, on the coloured canvas, reveals himself.

—Oscar Wilde, The Picture of Dorian Gray

The dominant methodology in design these days is “human-centered design”. Reductively, in this framework, a designer begins by immersing themselves in the worlds of potential users, trying to understand their values, goals, challenges, barriers. Then, often with the direct participation of the users, designers iteratively create artifacts which will solve the users’ problems.

This is a remarkably effective method for creating products and solving problems. But, among other limitations, I think it’s missing something that’s been central to many of the most transformative tools for thought: a strong perspective on how the world should be, what’s beautiful, what’s worth amplifying.

“Dream Machines” is a telling title. Ted Nelson had a dream of what computers could mean for personal creativity and freedom. This is not “design thinking”. Much of Alan Kay’s work was driven by an almost spiritual belief in the wasted creative potential of young children. Consider his metrics: “Where some people measure progress in answers-right/test or tests-passed/year, we are more interested in Sistine-Chapel-Ceilings/lifetime.”

Bret Victor is adamant that we must break computation free of “little black rectangles.” There are problem-solving explanations for this, but it seems clear to me that they’re not behind this drive. Bret doesn’t want to physicalize computing because screens are too small and hurt your eyes, or because embodied cognition is higher bandwidth. It’s because he thinks being in the world, with our bodies, with each other, is humane and beautiful and the way things should be. He’s described this drive as “a yearning.”

Yearnings like this are sometimes the driving force for my work, too. Orbit could be framed as a tool for retaining what you learn. That’s what every other memory system does. But that’s not how I think about it. What gets me excited is the feeling of imbibing ideas more deeply, of being supported in forming an ongoing communion. In some very real sense, this project is an expression of how I want to relate to knowledge.

The impulse I just described feels quite different from the sense of possibility I feel when designing, or the hunger to understand I feel when engaging analytically. It’s more like a desire to manifest that which I think is beautiful. It’s personal, idiosyncratic. It’s a reflection of my telos.

When this drive dominates, I feel like I’m making a kind of art.

Much of the most interesting work going on now in the vicinity of “tools for thought” is driven at least in part by this kind of artistic expression. Omar Rizwan’s work consistently reflects his strong and unusual aesthetics. The Ink and Switch projects around local-first computing feel to me as much about artistic expression as about problem-solving. Sprout reflects how its creators believe collaboration should feel. In Gentle I see Rob’s consistent reverence for orality. In Cuttle I see Toby making a statement about the contemporary role of craft.

Collectively, both this modern group and the computing pioneers express a consistent yearning: the radical power of computation can be wrested from a beige-world of centralized tabulation, and instead invested in human-scale forms, where it will transform what we can think and do, individually and together. That’s not a design statement. It’s certainly not a scientific statement. It’s more like a manifesto.

If we think of tools for thought as an artistic “scene”, I think collective progress comes from transcendent work which inspires, instructs, and redefines. The ten year anniversary of Inventing on Principle is coming up. Perhaps we’re due at last for another contribution so revelatory?

Tools for thought as a practice of craftsmanship

Vi Hart and Bret Victor have this way of being nuts part of the time, and the other part of the time having exquisite attention to detail at the level of an obsessive. It’s like Michelangelo: first he had to imagine putting something on the ceiling of the Sistine Chapel, but he also personally spent four years lying on his back with candle wax dripping into his eyes, painting the goddamn thing. That’s the simplest recipe: find Michelangelos.

—Alan Kay

One more impulse often shapes my practice, and I’m less certain of its proper place.

To set this up, I’d like to tell a story I haven’t shared publicly, about my first week at Apple. I was fresh out of college, joining UIKit, the technical team responsible for the iPhone and iPad user interface and “app” structure. At the time (iOS 3), it was a team of around eight. The youngest member was at least ten years older than me. I felt more than a little out of place, and therefore somewhat sheepish when I had to ask about the little black objects everyone had on their desk.

“Oh. That’s a jeweler’s loupe. We use it to see the pixels.”

That knocked me on my back. It was an absurd answer. I’d spent my entire young life with software people, and I’d never met one with a jeweler’s loupe. Yet it was also obviously right. It was a potent symbol of what people there valued. And those loupes were, in fact, quite practically useful! In my first weeks I was given a subtle bug to fix: the edges of a particular kind of button were “fuzzy.” I looked. I couldn’t see what they were talking about. I looked with the loupe, and then I saw. A straight edge seemed to be antialiased into the neighboring pixels. I took the loupe away, and I couldn’t un-see it.

The loupe is synecdoche for my whole experience at Apple. This same story happened again and again over the years—with touch interactions, with animation, with conceptual models. Each time the burr in my colleague’s eye was at first invisible to me; each time I was helped to see; and each time I could no longer un-see thereafter. I left Apple, but the indoctrinated obsession for craftsmanship has not left me.

This impulse feels different from a drive to understand, or a sense of opportunity, or an expressive yearning. It feels like a consuming desire to personally manifest a kind of perfection. It feels like a ritualistic respect for things finely honed. And, because I can rarely achieve the sublimity this mindset demands, it’s often frustrating.

I don’t know how to feel about this impulse.

On the margin, it’s obvious to me that academic human-computer interaction would benefit from an infusion of craftsmanship. It’s absurd to run elaborate evaluative studies on some new interface when that interface is executed so sloppily that few people would voluntarily use it outside a study. Yet this is exactly what happens in most systems papers published at the field’s premier conference.

Sometimes craftsmanship is essential to expressing an idea at all in this space. The Mother of All Demos was a visionary work of design, but at the time, making that show happen in realtime on a stage in San Francisco required the absolute pinnacle of technical craft. The same is true of Dynamicland today.

Another factor here is that reality has a surprising amount of detail. The line between essential design elements and craftsmanship is often unclear. When the folks at Xerox PARC added proportional-width fonts to Bravo (the first WYSIWYG word processor), was that an inseparable part of the design solution, or was it an urge to make something limited more perfect? When I obsess about the perceived lightness of spaced repetition prompts, that can feel like sanding edges, but I suspect it’s actually a “load-bearing” design element.

Another reason the craftsmanship impulse is useful is that many insights are only reachable if users make a system a part of their lives, if they use it to do something that really matters to them. No one wants to use a piece of junk for anything important. So a finely honed system has the potential to involve more real people in real situations, and therefore to produce more insight.

More prosaically, craftsmanship helps draw attention. It makes one’s work more legible and more attractive. I don’t think this is a high-order bit, but it’s worth considering.

On the margin, I think this impulse is probably too strong in me. I’ll often spend an afternoon polishing the fine details of an interaction I’ll soon discard. Some amount of this is necessary, but I suspect I could often get away with much less.

All in all, my instinct is that craftsmanship is not usually the high-order bit or the bottleneck for progress in this space, past some moderate threshold. I think there’s value to cultivating a tradition of craftsmanship among practitioners in this space, for the reasons I’ve described above, but I’m not sure what it would mean to make collective progress through craft as a primary drive.

Negotiating these forces

I don’t think I could make do with any one of these impulses. They all seem essential, in their own way, to my practice.

At least for me, the most consistently powerful lens is that of design. Progress depends on bold, imaginative ideas about how to shape an abstraction so that it dances with the seams of reality. These new artifacts do usually depend on deep understanding, both for their creation and for their improvement. And the most interesting artifacts tend to express some personal yearning, beyond fitness for purpose. It may be that craftsmanship is best understood in this framework as one element of the iterative design process—I’m less sure about that.

Likewise, I think our best model for collective progress looks less like a scientific field and more like a design discipline. We already build on a substantial shared collection of patterns, principles, and methods. Sometimes we can also build on empirical knowledge and theories of the relevant domain. These accelerate our work, but each project is necessarily bespoke. There are no formulas for creating new abstractions. Progress will continue to require the lightning strike of creative ingenuity.

If you find my work interesting, you can become a member to help make more of it happen, and to get more essays like this one.

I’m grateful to Michael Nielsen for introducing me to the notion of design science and for shaping how I think about the role of science in interface invention; to Bret Victor for an email exchange which prodded me to think harder about this; and to Kanjun Qiu for helpful discussions about the distinction between the “science” and “design” drives here.

Special thanks to my sponsor-level patrons as of April 2022: Adam Marblestone, Adam Wiggins, Andrew Sutherland, Ben Springwater, Bert Muthalaly, Boris Verbitsky, Calvin French-Owen, Dan Romero, Dwight Crow, Eugene Soltes, fnnch, James Hill-Khurana, James Lindenbaum, Jesse Andrews, Kevin Lynagh, Lambda AI Hardware, Ludwig Petersson, Maksim Stepanenko, Matt Knox, Mickey McManus, Mintter, Nathan Lippi, Patrick Collison, Paul Sutter, Peter Hartree, Russel Simmons, Salem Al-MansooriSana Labs, Tim O’Reilly, Todor Markov, Tom Berry, Tooz Wu, William Clausen, William Laitinen, Yaniv Tal.

Next unlocked “Letter from the Lab”: Lessons from 2021