Part of “Letters from the Lab”, a series of informal essays on my research written for patrons. Originally published December 2022; made publicly available January 2024. You can also listen to this essay (16 minutes).

Reflecting on my work with the mnemonic medium, I notice that I’m making good progress on questions of breadth and scale, but not nearly enough along the much more important axes of depth and impact. That’s not for lack of trying—my attempts to explore the latter keep fizzling. But I think I understand better now why that is, and at least partially what to do about it. In the new year, I’d like to aggressively reorient my work towards traction on this central question: how to make a radically more powerful memory system, one which makes it easy to build a deep, flexible, and durable understanding of complex conceptual material?

In this letter, I’ll lay out the situation and describe my best thinking on what to do about it. Beyond helping myself understand, my aim here is to create a good context for conversations with others who can help me improve my plans. I’m still quite confused about many of the points I discuss here, so this should be read as a provisional, interim discussion—but hopefully a useful one on the way to a better path.

Why I’m excited about memory systems

Since this is a letter about my progress being off-target, it’s worth carefully re-articulating the aspiration that drives me here.

I’ve spent almost a decade exploring how we might use computers (and specifically dynamic mediums) to create exceptionally high-growth environments. I’ve done projects around discovery learning in microworlds, computer-supported collaborative learning, mastery learning, ML-driven personalized learning, and so on. But my personal experiences with memory systems have been so vivid that all the other mechanisms I spent years investigating seem laughably weak by comparison. Memory systems conjure a powerful sense of vertigo for me. I feel like a caveman who’s stumbled on a box of simple machines. I might not understand the principles of mechanical advantage, but even primitive fumbling produces startling results!

Today’s memory systems are niche tools, used mostly for memorizing vocabulary and simple facts. These applications wouldn’t be so compelling on their own, but a few expert users have discovered clever ways to apply them much more broadly, supporting many kinds of thought. To give a few unusual examples, I use memory systems to amplify creative insights, and to digest meaningful experiences. In this letter, though, we’ll focus on where the impact has felt most immediately transformative: understanding difficult conceptual material.

When I use a memory system alongside a book or a paper on an unfamiliar topic—and when everything’s going just right—I feel an alien lightness. I may still struggle with the ideas, but there’s an internalized certainty: I can carve away truly reliable footholds; I can keep accumulating until I’ve understood as much as I want; I’ll be repaid with durable access to that understanding months later. The feeling is absolute discontinuity with all my prior experiences studying new subject matter.

I’m reminded of the move from analog to digital computers. When components accumulate error, there’s a pervasive sense of fragility, a sharply rising complexity as the system grows. By rectifying errors within each component, you can assemble reliable systems which grow without strain. This is an imperfect metaphor. Understanding isn’t binary; I’m not claiming that all thought can or should be made “error-free.” But I have no doubt that this sort of “rectifier” radically expands the cognitive ground I can cover.

With that glowing description, one might imagine that memory systems just need to be popularized, “scaled”, rough edges removed. But there are good reasons for memory systems’ limited adoption. In my view, the most important problem is that it’s very difficult to do what I just described—to reinforce arbitrary material with them. One must develop a specialized skill to do this at all; I’ve spent years accumulating strategies for digesting different kinds of ideas into memory system prompts. And it’s not enough to just help readers develop those skills. Even for me, writing these prompts takes so much time and mental energy that I cover only a fraction of the knowledge I find meaningful. To make matters worse, it’s especially tough to use memory systems well with unfamiliar material—which is of course when they’re most useful. As a non-expert, my prompts often emphasize the wrong elements, or miss the forest for the trees, or fail to encode enough connective tissue to support rich understanding.

Why I’m excited about the mnemonic medium as an avenue for more powerful memory systems

Prompt-writing involves productive struggle, and I’m sure that contributes enormously to deep understanding. Sometimes there’s no way around this work: many of my most valuable prompts draw connections to personal experience. Yet I believe we can do much more with much less. Today’s memory systems “make memory a choice”—but it’s a stark choice between moderate-cost/high-benefit and no-cost/no-benefit. We can design systems which offer intermediate points. More importantly, I believe we’re nowhere near the efficient frontier. We can both lower the costs and also radically increase the benefits.

Ideally: as I learn new material, rich representations of that knowledge are continuously encoded into my memory system. This system not only helps me reliably remember what I learned, but it actually deepens my understanding of the material over time. The whole process integrates naturally into my intellectual life, with so little burden that it can be applied pervasively. This ideal may be beyond reach, but if we can make something which comes close, it’s tough to imagine that such a system would not “change the thought patterns of civilization”.

An obvious initial approach here is for experts to create and share memory system prompts about various topics. Plenty of people have tried. Shared prompts seem to work moderately well for vocabulary and simple facts, but not for more conceptual material. When I’ve tried these collections, I’ve found that I can parrot back responses to the prompts, but my understanding is quite brittle—for instance, I’d struggle to explain the material to a friend. In user interviews, other users consistently express similar problems.

By contrast, my understanding is much less brittle when I digest a well-written explanation into my own memory system prompts. I’m sure part of the reason is that prompt-writing forces me to think much more about the material. But I notice that when I practice those prompts, I’m often transported back to the relevant passages in the narrative. The isolated prompts are anchored (at least partially) in a broader, well-connected context. That richer context also keeps me more emotionally engaged during review.

That’s why I’m so excited about the mnemonic medium. If we integrate expert-written prompts into an explanatory narrative, perhaps they can take root in more fertile soil. This strategy could bring us much closer to the ideal I described earlier.

In practice, we might worry: how much understanding and connection do we sacrifice when an expert writes those prompts, rather than the reader? Interviews suggest that readers rarely experience the kind of brittleness and detachment that I hear so consistently from users of downloadable memory system decks. But I need to understand this trade-off much better, and to improve it where I can.

Apart from its practical effects, the mnemonic medium has potential to become a powerful laboratory for experiments exploring: how can we make much better memory systems?

Here’s a simple example. What if, in the weeks after you read a book, you’re asked not only to recall what you’ve learned, but also to apply that material in simple practice exercises—ones you haven’t seen before? Might that help you deepen your understanding over time? It would be tough for you to set this up for yourself. Even if you could write good exercises, the idea is to apply the material to a novel situation. The mnemonic medium offers a natural way for one person to author material like this, and for another person to use it with fresh eyes.

The mnemonic medium also makes it much easier to run iterative experiments. For example, a designer could include these new practice exercises only for certain topics, and then compare how the reader’s understanding grows for those concepts versus others. If the designer learns something new, they can iterate on the system, observing how their changes affect new readers.

There are dozens of improvement ideas I’d like to explore, and questions I’d like to answer—some about cognition and efficacy, others about emotion and feel. Ideally, the mnemonic medium would support rapid cycles of iteration towards a much better memory system. But I’m writing this letter because I haven’t yet managed to create the right environment to iterate on the most important questions. In fact, Quantum Country’s fourth essay implements the “practice exercise” idea I just described—and I haven’t managed to understand its impact, at all.

Struggling to push for depth

Over the past two years, most of my progress has been in solving problems specific to the mnemonic medium’s conceptual design. That is: problems with using expert-written memory system prompts, and with embedding memory systems into different kinds of texts in different kinds of situations. This is good and necessary! Such problems are central to the idea and must be solved to make the medium work.

But as I’ve made more progress here, I’ve slipped into solving increasingly peripheral problems—making the medium work for non-experts, in increasingly casual scenarios. I’m straying towards issues of scale, and away from my central aim: how to make a radically more powerful memory system, one which makes it easy to build a deep, flexible, and durable understanding of complex conceptual material?

To make matters worse, I can’t tell you very much about how well today’s mnemonic medium already accomplishes this. I’ve run plenty of experiments on readers’ ability to recall the embedded prompts, but I know little about how the medium impacts what actually matters—the depth of their understanding; their capacity to make meaningful use of the knowledge.

Through collaborations with university professors and course instructors, I’ve tried several times to create a context where I can observe the downstream impacts of my designs. I failed to learn much from any of these experiments. Few students seemed interested in deeply understanding the material; not just the memory system but the courses in general suffered from low participation.

Even without these problems, though, I now believe that my approach with these experiments was wrong. I was thinking in terms of statistical tests, trying to measure causal connections between properties of the memory system and downstream performance. But what I really need now is not statistical power but intimacy—a detailed, vivid picture of readers’ understanding as it develops in concert with the memory system. I’d rather establish a close relationship with a handful of serious test readers than run a high-powered experiment which can only report numeric change in test performance. The latter will be useful later, to optimize and validate, but it’s not the right data to drive early, open-ended design decisions—to identify the right fundamental primitives.

I’d thought the trouble with my course collaborations was that attrition made my sample sizes too small. But the real problem was that these contexts weren’t setting up the right kind of relationship between me and my test readers. The few students who participated actively in these courses were rarely willing to offer more than occasional perfunctory feedback on the memory system. This is reasonable: it’s not what they signed up for! It’s an extra burden. Even if they were more willing to help me, these students are generally just trying to “get through” the class. I can’t effectively learn how a memory system affects depth of understanding from someone who isn’t trying to understand the material deeply.

By contrast, my dream test readers would be quite serious about deeply understanding the material. They’d be extremely demanding of the system, ideally behaving like collaborators in its creation. They’d push at the edges of the system and creatively co-opt it for their own purposes. They’d happily tackle difficult challenges to proactively probe for weak spots in their understanding.

There’s an important distinction to recognize here—one that’s quite alien to typical Silicon Valley practices. Yes, of course, an ideal future memory system would also help less demanding people in less demanding contexts. But that doesn’t mean those are the right people to help invent the system. Feedback from that audience is the right force to broadly scale a system that’s already astoundingly transformative in the best case. It’s the wrong force to push a nascent-but-promising system to become astoundingly transformative, unless the system’s power pivotally depends on its social embedding.

To find the form of a system with the highest ceiling on its power, I want to design for (and with) demanding experts, in difficult situations. I want a close, collaborative relationship, not an arms-length “vendor” relationship.

Sketching a path forward

Now that I’ve articulated this aspiration—this shift to depth over breadth—how should I go about actually making it happen? My leading idea is to collaborate closely with a small group of people as they try to deeply understand a topic of great interest, with the help of a memory system. I’ll choose people who are already expert practitioners with today’s memory systems, and who are excited to collaborate on much better memory systems for the future. My aim is to form an intimate, nuanced picture of readers’ cognitive and emotional experiences—then to use that to drive an insight-through-making loop.

A few concrete practices I’d like to try:

• Personal “memory Wizard-of-Oz”: I follow my collaborator around, adapting whatever’s next on their reading list to the mnemonic medium—partially simulating a future ideal in which these affordances are available pervasively and effortlessly. Each adaptation can encode one or more new ideas to test (either at a system level or at a content level).
• Small-group course: I teach a small-group course on a difficult subject I know well. I create or adapt materials, using the mnemonic medium and whatever augmentations other seem most useful. I repeat the course several times, trying new ideas each time, and improving old ones through experience from the previous iteration.
• Memory coach: I help my collaborator become a much better practitioner of existing memory systems. We’ll meet regularly to review the material they’ve been writing, and to discuss how it’s working for them. I’ll help them contend with obstacles and suggest ideas for improvement. My aim here is to put pressure on my principles as an expert practitioner; this should help me test and distill my methods, and ideally develop improved ones. Moreover, this practice would help me focus on insights around object-level use of memory systems, rather than always systematizing for creation of new systems.

I’m not trying to set up sensitive statistical evaluations here, or to make rigorous claims which would generalize to large populations. I’m looking for qualitative surprises and glaring quantitative discontinuities—hints of transformative power I can exploit, clues about obstructions which might be cleared away. The theme here is small-N, bespoke, detail over generalization, exploration over validation.

To that end, I’ll focus on discussion and diaries, which I’ll ask my collaborators to keep while they read and practice. We want to notice moments when: understanding feels shaky, brittle, or parrot-like; thoughts seem sluggish; readers are looking things up; readers are bored, dutiful, or emotionally disconnected. But also moments of: unexpected ease, velocity, confidence, connection, or excitement. How does it feel to use the newfound knowledge—in creative work, in social settings, in personal reflection? How does it feel to explain the ideas to others, to learn downstream material, to solve problems?

As an adjunct, I’d also like to use the mnemonic medium much more myself. So much of my understanding of memory systems comes from years of experimentation to support my own creative and intellectual work. Those experiences have helped me develop a nuanced understanding for many aspects of memory systems. But I have no intimate personal experience of learning difficult material using the mnemonic medium. The trouble is that it’s surprisingly difficult to do that, since I’m usually in the position of creating mnemonic texts!

The closest I’ve gotten was during the development of Quantum Country, but I was too distracted by fixing early issues with the system to pay enough attention to my own learning process. More fundamentally, I didn’t have an intense personal drive to deeply understand quantum computing. No problem sets, projects, or social contexts put pressure on my knowledge, so I don’t have much sense of its solidity.

I stopped reviewing Quantum Country’s material long ago, mostly because it wasn’t integrated into my regular memory system. Enough time has elapsed that I could probably learn a lot about the mnemonic medium by repeating Quantum Country afresh, perhaps with a project in hand that I felt excited about. But this is a bit backwards. It would be much better to pursue this sort of experience in a domain where I already have a live need for deeper understanding.

I’d like to experiment with adapting a textbook of interest for my own use by hiring someone with expertise in both memory systems and in the relevant domain. It would be tough to iterate on structural changes to the memory system itself with this approach alone: I’d need to make new functionality available, then rely on someone else to use it well. Still, it’s hard to imagine that I wouldn’t benefit enormously from greater personal intimacy with the medium, particularly alongside tight collaborations with other readers.

So: What better approaches am I missing? What key considerations am I ignoring? Where are my premises weak?

I’ve left the concrete details of my experimental design quite unspecified. What likely traps await me there? Why might I find myself unable to see what I need to see?

This letter grew directly from a series of conversations with Michael Nielsen, and much of its thinking is due to him. I’m particularly grateful to Michael for the central observation that I’ve been slipping into issues of scale, the emphasis on small-N methods, and the idea of memory coaching.

I’d also like to thank Rob Ochshorn, Joe Edelman, and Joël Franusic for thoughtful comments.

My work is made possible by a crowd-funded research grant from my Patreon community. If you find my work interesting, you can become a member to help make more of it happen. You’ll get more essays like this one, previews of prototypes, and events like seminars and unconferences.

Finally, a special thanks to my sponsor-level patrons as of January 2024: Adam Marblestone, Adam Wiggins, Andrew Sutherland, Andy Schriner, Ben Springwater, Bert Muthalaly, Boris Verbitsky, Calvin French-Owen, Dan Romero, David Wilkinson, fnnch, Heptabase, James Hill-Khurana, James Lindenbaum, Jesse Andrews, Kevin Lynagh, Kinnu, Lambda AI Hardware, Ludwig Petersson, Maksim Stepanenko, Matt Knox, Michael Slade, Mickey McManus, Mintter, Peter Hartree, Ross Boucher, Russel Simmons, Salem Al-MansooriSana Labs, Thomas Honeyman, Todor Markov, Tooz Wu, William Clausen, William Laitinen