January 2024. Part of “Letters from the Lab”, a series of informal essays on my research written for patrons. You can also listen to this essay (17 minutes).

See also previous annual reflections: on 2022, 2021, 2020.

Systematic output; one-off methods

It’s so tempting to conflate aims with methods.

I’m trying to invent systems that work for lots of people in lots of contexts. My instinct, then, is to focus the process of invention on scalability and generality. If I’ve got an idea, I’ll usually try to make something linkable—something I can send to a wide range of people, for use in a wide range of contexts. I’ll send out a survey; I’ll pick some telemetry variables to measure and analyze; maybe I’ll run a randomized controlled trial; I’ll interview users about their experiences.

Possibly the most important thing I’ve learned in recent years is that these instincts are often a mistake. To invent a systematic solution, I’ll often need to synthesize insights from many non-systematic experiments. In other words, my aims—general solutions—will often require methods (and intermediate outputs) with very different properties.

Paul Graham has famously advised startups to “do things that don’t scale.” Things like talking to customers door-to-door instead of buying a national ad campaign. He mostly justifies that advice in terms of business, marketing, and brand reasons; I think that’s what most people remember. But he does briefly discuss non-scalability as a method of creation, and although he’s focused on startups, his arguments capture some of my experience as well. I’d like to elaborate with some observations on “doing things that don’t scale” in the context of discovery and invention.

Single-user evaluation

In late 2022, I felt that my experiments had somehow distanced me from my aim: enabling people to acquire rich understanding of complex ideas that really matter to them. I’d been building systems and running big experiments, and I could tell you plenty about forgetting curves and usage patterns—but very little about how those things connected to anything anyone cared about. I knew that I was making some progress, but I was mostly flying blind, without the feedback I needed to drive my iteration.

So in 2023, I switched gears to emphasize intimacy. Instead of statistical analysis and summative interviews, I sat next to individuals for hours, as they used one-off prototypes which I’d made just for them. And I got more insight in the first few weeks of this than I had in all of 2022.

Of course, I got a different kind of insight. I didn’t learn anything about what would generalize to a wide population. I couldn’t measure fine differences in memory performance as I changed some parameter. Instead I could see, in great detail, the texture of the interaction between my designs and the broader learning context—my real purpose, not some proxy. I could see things going as planned in my prototype, and then going decidedly off the rails when that knowledge was put into use. I could see the emotional beats of that experience as they were happening. I could see a hundred questions I hadn’t thought to ask in a survey or post-hoc interview.

Is this “rigorous” research? Not as that word is typically used. But working this way, I feel better-equipped to make progress on my ideas than I have in quite some time. Single-user experiments like this emphasize problem-finding and discovery, not precise evaluation. In fact, my work with one student uncovered a startling hypothesis that consumed much of my year: what seems like a problem of forgetting is sometimes a problem of reading comprehension—never having understood in the first place—and we can’t reliably tell the difference.

As far as evaluation goes, negative results in single-user experiments do seem quite instructive. If I make a bespoke solution for one particular student, and it doesn’t work well, I rarely feel the need to double-check that negative finding with a larger trial. Some failures might be personal or idiosyncratic, but my designer’s instincts can sort out much of that; I’m not too worried about false negatives. And, unlike a controlled experiment with formal measures, this kind of qualitative observation will usually tell me a great deal about why my solution’s not working well. Practically speaking, a good heuristic for evaluating my work seems to be: try designs 1-on-1 until they seem to be working well, and only then run more quantitative experiments to understand how well the effect generalizes.

Single-context designs

My aim is to invent augmented reading environments that apply to any kind of informational text—spanning subjects, formats, and audiences. The temptation, then, is to consider every design element in the most systematic, general form. But this again confuses aims with methods. So many of my best insights have come from hoarding and fermenting vivid observations about the particular—a specific design, in a specific situation. That one student’s frustration with that one specific exercise.

Fred Brooks keenly distills Christopher Alexander:

The only way to achieve good fit between any design and its requirements is to find misfits and remove them; there is no direct way to derive form from requirement. Good fit is the absence of all possible misfits.

It’s often hard to find “misfits” when I’m thinking about general forms. My connection to the problem becomes too diffuse. The object of my attention becomes the system itself, rather than its interactions with a specific context of use. This leads to a common failure mode among system designers: getting lost in towers of purity and abstraction, more and more disconnected from the system’s ostensible purpose in the world.

I experience an enormous difference between “trying to design an augmented reading environment” and “trying to design an augmented version of this specific linear algebra book”. When I think about the former, I mostly focus on primitives, abstractions, and processes. When I think about the latter, I focus on the needs of specific ideas, on specific pages. And then, once it’s in use, I think about specific problems, that specific students had, in specific places. These are the “misfits” I need to remove as a designer.

I think often of Ted Nelson’s quixotic efforts with Xanadu, originally motivated by his ambitions as a screenplay writer. Decades of abstraction and infrastructure aspiring to encompass all possible forms of writing, but (as far as I can tell) none of that involved him actually trying to write anything meaningful using those designs. And so, interesting as those systems’ ideas were, most remain unrealized even today. It’s not a matter of young technologists ignoring the ancients’ wisdom: many of these designs simply aren’t developed adequately to solve the problems they aspire to solve, or to suggest whether they can solve those problems. By contrast, consider Douglas Engelbart’s contemperaneous NLS, which was developed for his lab to use as a collaboration tool. Every key element of that design has long been realized in our tools today.

Of course, I do want my designs to generalize. That’s not just a practical consideration. It’s also spiritual: when I design a system well, it feels like I’ve limned hidden seams of reality; I’ve touched a kind of personal God. On most days, I actually care about this more than my designs’ utilitarian impact. The systems I want to build really do require abstraction and generalization. Transformative systems really do often depend on powerful new primitives. But more and more, my experience has been that the best creative fuel for these systematic solutions often comes from a process which focuses on particulars, at least for long periods at a time.

The feeling of the particular

Also? The particular is often a lot more emotionally engaging, day-to-day. That makes the work easier and more fun.

In a one-hour interview or observation session, I can build some emotional connection with a person. They’re more vivid than an email thread or a row in a spreadsheet. But then the session ends, and I generally move on, influenced but not transformed. By contrast, if I work with a particular student for many hours across many sessions, I develop quite a strong desire to help that particular person flourish, at whatever really matters to them. I find myself naturally thinking about them as I work, again and again. I feel vicarious joy when something goes especially well. When they struggle, I really want to solve those problems. (I can understand the appeal of being a teacher!)

All this moves my motivation from the spacious, timeless theoretical to the sharp, focused interpersonal. This helps with my understanding, as I’ve described, but it’s also creatively energizing and much more immediately motivating.

Throughout my career, I’ve struggled with a paradox in the feeling of my work. When I’ve found my work quite gratifying in the moment, day-to-day, I’ve found it hollow and unsatisfying retrospectively, over the long term. For example, when I was working at Apple, there was so much energy; I was surrounded by brilliant people; I felt very competent, it was clear what to do next; it was easy to see my progress each day. That all felt great. But then, looking back on my work at the end of each year, I felt deeply dissatisfied: I wasn’t making a personal creative contribution. If someone else had done the projects I’d done, the results would have been different, but not in a way that mattered. The work wasn’t reflective of ideas or values that mattered to me. I felt numbed, creatively and intellectually.

By contrast, when I’m doing work that I find gratifying and meaningful over the long term, the day-to-day experience is usually frustrating and unpleasant. The work is gratifying because it’s deep and personal and unique. Unfortunately, in my projects, those same attributes also mean that progress tends to be inconsistent and hard to discern; it’s rarely clear what to do next; there’s rarely anyone I can ask for help; I usually feel incapable.

Of course, these qualities don’t need to produce suffering, and I’m slowly getting better at handling them skillfully. But I’ve also noticed that when I focus my work on particular people in particular contexts, that more immediate emotional connection sometimes overpowers the day-to-day frustration that comes with being lost in the woods. For several long stretches this year, I found the work really gratifying, both in the moment, and retrospectively over the long term. Even if focusing on the particular didn’t help the creative process in the other ways I’ve described, this emotional effect would be well worth pursuing on its own.

Mignardises

Progress often doesn’t look like progress

It often feels like I’m not making any progress at all in my work. I’ll feel awfully frustrated. And then, suddenly, a tremendous insight will drive months of work. This last happened in the fall. Looking back at those journals now, I’m amused to read page after page of me getting so close to that central insight in the weeks leading up to it. I approach it again and again from different directions, getting nearer and nearer, but still one leap away—so it looks to me, at the time, like I’ve got nothing. Then, finally, when I had the idea, it felt like a bolt from the blue.

When the insight arrived, I didn’t notice the connection to the trail I’d laid on the preceding pages. My experience was of making no progress, and then, finally, making some. In hindsight, I can see that I had been making plenty of progress over those weeks; I just couldn’t tell at the time. I suspect this is pretty common in my work. So, “I feel like I’m not making progress” is probably not a good local heuristic for guiding my work. Alternately, the lesson might be that I need to become more sensitive to the many subtler flavors of progress in this kind of work.

Stillness is trainable

I’ve written that one of the hardest challenges for my work is: “how to cultivate deep, stable concentration in the face of complex, ill-structured creative problems?” I now have several years of data on self-reported focus and energy levels, and it’s comforting to see that this does get easier with practice.

I work in one big block with no interruptions; in 2022, this was usually around 7:30 AM–1:30 PM, and I’d get quite restless towards the end. By late 2023, I usually didn’t start feeling antsy until 2:30. Slowly, over 2023, I increased the amount of time I spent “highly focused” by 38%, while simultaneously lengthening my total working time by 12%.

Some of this is the accumulation of many small tweaks to my practices, but I think much of it is me slowly—still—becoming less reactive to the discomfort of sitting in stillness and confusion.

Help can come without domain expertise

This year in particular, I’ve benefited enormously from friends who are willing to talk in great detail about challenges in our respective creative lives. To my surprise, it often doesn’t matter if these friends are experts in my domain, or even that they have much context about my projects. I shouldn’t have been so surprised. In late 2021, when I sought out a coach, I wanted to find someone who had experience with research and invention. I imagined that we’d want to talk at great length about my projects and their problems. Two years later, my coach has helped me enormously, but he still has only a hazy idea of what I’m researching.

It turns out that my coach hasn’t needed a detailed understanding of my work, for roughly the same reason that my helpful friends haven’t. Much of the time, whether I’m aware of it or not, the conversation I most need to have about my creative work is about my emotional relationship to the work and to the process. I can get that from sensitive, creative people who care. They don’t need to deeply understand my work as a domain expert. Of course, help from domain experts has often been extremely helpful. But I’ve underrated help from highly creative people outside my field, to my detriment.

New patterns of collaboration

My collaborations have mostly fallen into two forms: 1) everyone’s driving the project creatively, full-time, together; 2) someone’s driving the project creatively, and someone else is mostly “taking tickets”—doing well-defined piecework. This year, I had two happy collaborations in different modes.

Since last February, I spent an afternoon each week with Matthew Siu, the winner of last year’s research fellowship. Our project isn’t an idea that either of us were thinking about when we met; it’s not on the “critical path” of any of my projects; instead, it’s something we discovered was fertile ground over many hours of conversations. Matthew is very much the primary driver, while I mostly advise, sketch, and synthesize during our weekly meetings. (More on this project soon! It turns out six months was a very unrealistic fellowship length…)

In parallel, Derrek Chow and I had a different sort of collaboration. I’d gotten to know him over many long walks, and I’d come to admire the artist’s temperament that he brings to human-computer interaction problems. Derrek was looking for raw material for his creative practice, and for a good excuse to collaborate, so he asked if I had any project ideas he might be able to work on. I suggested the prompt which became BookBridge, and we began to meet weekly to discuss an ongoing stream of prototypes. As in my collaboration with Matthew, Derrek gave the project much more time than I between our meetings. But this collaboration had a different texture because it needed me to do more “driving”: this was a prompt I’d suggested based on my experiences as a reader, not a need Derrek felt viscerally himself, so my role involved more actively steering iteration and synthesizing a whole from the parts.

Both of these collaborations were exciting for me because I felt newly able to scale myself creatively: with a few hours of participation each week (plus intermittent bursts of lots more), I could cause a new branch of idea space to be meaningfully explored. In Matthew’s case, it’s a problem I wouldn’t have considered alone; in Derrek’s case, it’s a problem I’d wanted to explore but for which I hadn’t made the time. One very rewarding element of my collaboration with Matthew has been investing in his growth as a researcher and designer. It would be good for my “field” if I could do that repeatedly. Meanwhile, in my collaboration with Derrek, I benefited from complementarity: his skills and mindset are very different from mine, in ways which are interesting and additive. I’d like to embark on more collaborations of both these types—and others!—in the future.

Crowdfunding continues more-or-less steadily

2023 was my fourth year of crowdfunding my research. Year over year, my member count increased by 18%, which is lovely. At the same time, my income actually declined 5%. The disparity comes from a shift in the distribution of contribution amounts, from the higher to the lower tiers. That’s okay, I think: it smooths out the effects of inevitable churn.

The fundamental dynamics remained the same as in past years: similar churn rates and conversion rates. The biggest lever available remains getting more people into the “top of the funnel”, and I don’t want to spend any real attention on that. I experimented with talking about crowdfunding and my Patreon more actively on Twitter this year, and that helped a bit, but reaching new audiences drives much more growth: my appearance on Dwarkesh’s podcast led to most of my new subscribers this year. All in all, my income is still somewhere between that of a postdoc and that of junior faculty. Not great, but OK, for now. If I’d like to sponsor more collaborations, I’ll need to seek more outside funding.

At the start of 2019, I wrote some five-year goals for myself, and then ignored them completely until a few weeks ago, when I exhumed them for a review. It’s amazing to me how fixated I was on hopes and concerns around funding. I suppose it shouldn’t be surprising: my fundraising experiences at Khan Academy were still raw at the time. My greatest wish was to have a source of funding steady enough that I could pursue whatever creative projects I found most interesting. My greatest concerns were that I’d be incredibly distracted by fundraising, or that I’d allow my work to be distorted by funders’ interests. It’s pretty ironic, then, that I actually hit these goals in 2021, and didn’t even realize that I was several years ahead of the ambitious five-year schedule I’d set for myself.

It’s an unbelievable privilege to wake up each day and fly by my own creative compass. A small group of people made that possible: all patrons, past and present, could fit in a typical Broadway theater. That’s a stirring image to visualize whenever I need a hit of gratitude. So, thank you, to all who have made this very unusual life possible.

My work is made possible by a crowd-funded research grant from my Patreon community. If you find my work interesting, you can become a member to help make more of it happen. You’ll get more essays like this one, previews of prototypes, and events like seminars and unconferences.

Finally, a special thanks to my sponsor-level patrons as of January 2024: Adam Marblestone, Adam Wiggins, Andrew Sutherland, Andy Schriner, Ben Springwater, Bert Muthalaly, Boris Verbitsky, Calvin French-Owen, Dan Romero, David Wilkinson, fnnch, Heptabase, James Hill-Khurana, James Lindenbaum, Jesse Andrews, Kevin Lynagh, Kinnu, Lambda AI Hardware, Ludwig Petersson, Maksim Stepanenko, Matt Knox, Michael Slade, Mickey McManus, Mintter, Peter Hartree, Ross Boucher, Russel Simmons, Salem Al-MansooriSana Labs, Thomas Honeyman, Todor Markov, Tooz Wu, William Clausen, William Laitinen