We live in a world of typed text. What place does the Penman have?

My friend, the Penman

I have a bizarre friend who sends me, every few months, earnest if long-winded handwritten letters. I’ll call him the ‘Penman’, as he occasionally signs himself, and because a common theme in his letters is the lamentation of declining penmanship in our society.

“That ‘the pen is mightier than the keyboard’ is a latent truth among crafters of ideas” he wrote recently. “Writing is thinking; handwriting, in my experience, elicits the highest quality of thinking. Where the keyboard stutters, the pen flows – its unhurried pace giving a profundity to ideas that frenetic typers, however fast they type, will never capture. The best writing is handwritten.”

The Penman is given to hyperbole, but I’m at least sympathetic to his claims. As are some scientists: studies have repeatedly found that students who take handwritten notes remember more than their frenetically-typing classmates. And for anyone who works with equations, or even frequently draws pictures, the blank sheet of paper is an unparalleled tool for thought. Richard Feynman, asked about the scraps of paper where he scribbled physics equations, was very particular about this: “the paper is the work.”

Yet, unlike the Penman, I’m not about to renounce modern technology, retreat to rural Washington, and spend my days writing letters about how more people should be writing letters. I have taken to prefer handwriting to typing for ‘acts of creation’, as he calls them – first drafts, ideation sketches, physics derivations – but believe that modern knowledge management software is unparalleled at refining those first drafts. I’m a Zettelkasten-devotee, and an Obsidian die-hard. Vannevar Bush and Ted Nelson’s dreams of computers as tools-for-thought are now (at last) being realized (for the power users, at least) – with the slight caveat that they only apply to typed text.

Thus my topic: can we marry the Penman and modern PKM? Can we integrate handwritten notes into Obsidian as more than an afterthought? Can handwriting really become a seamless, integral part of knowledge work?

I believe it can. Indeed, for the past year, I’ve been refining an automation that endeavors to bridge these two mediums by syncing and automatically transcribing handwritten notes into my Obsidian vault. The result: one can enjoy all of the unparalleled creative energy of the pen and blank page, and afterwards apply to it all of the organizational and ideational wizardry of modern knowledge-management tools.

Are you an Obsidian-curious handwriting acolyte? Or a handwriting obsessed Obsidian neophyte? An ardent ‘Mac Power User’ wanting to automate note-taking? A lover of E-ink tablets wanting more flexibility in note management? Read on.

The only requirements of this automation are:

1. An E-ink Tablet (or some equivalent way of digitizing notes into PDFs), preferably with the ability to automatically export notes. I’m most familiar with the Onyx Boox line. (See the appendix for notes on doing this with a reMarkable.)
2. An automation-ready notes app – I’ll describe steps for Obsidian, but it can easily be adapted to Notion, Craft, etc.
3. A Mac. (The automation tools used below are, sadly, Mac exclusive.)
4. A file-automation utility on the Mac. I’ll use Hazel, one of the two essential Mac automation utilities.

What it does

Whenever occasion arises to put pen to paper, I create a new ‘note’ on my E-ink tablet, and write something – for example, an early draft of this article. I’ll go on for a number of pages. When finished, I save the note, and within a few seconds, this appears appended to my Obsidian daily note:

Note the form: a markdown heading with the notepad’s name, the transcription of its contents, and an embedding of the note itself. The transcription isn’t perfect, but it’s most always good enough that you (or an LLM) can easily polish it up. The gist of the content is there, and if it’s not, you can refer to the embedded PDF.

The transcription does its most impressive work with equation-strewn notes, like this guy. It even extracts drawings as embedded images.

Ain’t that something?

So far, you might note that plenty of handwritten notes apps offer comparable features. In Goodnotes, for example, handwriting is searchable, and can be exported to text with a selection and long tap (or by dragging the selection into a text field). The transcription quality is even a little better, since Goodnotes has access to the strokes that produced handwriting. But there’s no good way to automatically export notebooks, much less automatically transcribe them – nor even to transcribe multiple pages at once. Goodnotes and its ilk are little walled gardens of note-taking. They have a system, inspired and constrained by the analog: notes within notebooks within folders. Notes begin and end their little lives as pages within this system.

This is where the premise and promise of Zettelkasten enters: notes, in its view, are the seedlings of ideas. Those ideas grow and blossom only with careful tending – reengagement with them over time. If we believe the Penman, handwritten notes are an especially fertile source of ideas. But too often those ideas then languish, half formed, trapped within the pages of some notebook long since stashed away on a physical (or digital) shelf. When this happens, as Sonke Ahrens puts it, in “How to Take Smart Notes”:

It is not surprising that my friend has a bookshelf filled with notebooks full of wonderful ideas, but not a single publication to show.

But under this automation, handwritten notes automatically burst out of those journal pages, transcribed into a form suitable for ‘zettelkasting’ or ‘digital gardening’. You can do the ideation in the medium best suited to it: pen and (e-)paper. And you can do the hard, sustained work of developing those ideas with the aid of modern PKM tools.

How it works

Here’s a high-level overview:

1. On exiting a notepad on my e-ink tablet (an Onyx Boox), it’s automatically exported to a PDF, which is synced to a folder on my Mac using Syncthing – as described in Some E-ink Workflows for Mac Users.
2. Hazel, a file automation utility, notices this new PDF, and passes it as input to a shortcut.
3. The shortcut makes an API request to MathPix, a company with best-in-class handwriting to text conversion. This API request returns a transcription of the PDF to markdown text (complete with embedded images & equations).
4. The shortcut then adds this markdown text to my Daily Note in Obsidian. For good measure, it also saves the PDF file into my Obsidian vault and embeds it in the Daily Note, right below the markdown transcription.

1. Syncing Handwritten Notepads to PDF files

This is the most idiosyncratic step, depending on your method of digitizing your handwritten notes.

If you have an E-ink tablet, particularly one which runs Android, like the Onyx Boox line, see Some E-ink Workflows for Mac Users. Follow the instructions here for setting up automatic note export and Syncthing, and you’re all set.

Other E-ink tablets may require more care. The reMarkable (and, I believe, Kindle Scribe) let you email notes to yourself as a PDF. If you save (or have your Mail program automatically save) those PDFs to a folder, you can hook that folder into the rest of the automation.¹

2. Setting up Mathpix

Mathpix is the proverbial ‘AI behind the curtain’ in this automation. It was initially developed (by, I believe, Harvard mathematicians) with a very niche use case: taking pictures of handwritten equations, and converting them into LaTeX. Over time, perhaps realizing that this is not a large market, they’ve expanded the core product to take pictures of entire documents (e.g. our handwritten PDFs) and convert them into any format – along the way developing one of the best handwriting OCR methods on the market.

Mathpix offers a consumer-facing wrapper around their technology – a web-based collaborative notes app with “Search AI!” Fortunately, for lock-in conscious knowledge workers like ourselves, they make all of this functionality available through what seems to be their main product: an API. This is what we’ll use.

So,

1. Head to mathpix.com, hit “Try for Free” and make an account. Don’t be put off by their monthly subscription – that’s just for the web-based notes app, which we don’t need. We’ll do everything through the API, which costs pennies to the page.
2. Follow the steps in Mathpix OCR User Guide: Creating an API key to, yes, create an API key! Make a note of it, along with your account id (app id), e.g. in a password manager. We’ll be using it soon.

3. The Penman’s Shortcut - turning PDFs into Markdown & TeX

Here’s the meat of the automation: a fairly large shortcut we’ll assemble in three steps.

(If you’ve joined Riddle’s membership program, here’s a thank you: you can scroll down to the bottom of this post and download a pre-made version of this shortcut.)

Shortcut outline

On a high level, the shortcut does the following:

1. Accepts a PDF input,
2. Stores the account id and API key from above in variables,
3. Runs a python script using these variables,
4. Saves the output to the daily notes.

3.1 Create a new shortcut, accept PDF input, extract the PDF’s path and name

Open up ‘Shortcuts’, create a new ‘Shortcut’, and check the boxes in the right-hand pane shown below.

You can also refine the input the shortcut takes – set it to ‘Images and PDFs’, and specify that if there’s no input, it should ask for files.

Next, drag in a ‘Set Variable’ block, and set it to the file path of the Shortcut’s input.

Next, we extract the file’s name, and (for my own convenience) replace any dates in that name with blank text.

3.2 A ‘No OCR’ failsafe

There are some handwritten documents you don’t want flying through someone else’s servers, even though Mathpix says it deletes the files after processing them. Meeting notes, essay drafts, and little math ideations are fine enough; personal journal entries, perhaps not. Yet, we probably want all of those files – including the journal entries – saved into our PKM.

We’ll do this by having the shortcut check whether the text ‘no ocr’ is in the file’s name.

3.3 Looping through the PDF

Unfortunately, Mathpix sometimes fails to transcribe PDFs – especially large ones, with complicated handwriting. I’ve found it works much more reliably if, instead of asking it to transcribe the entire PDF, we feed it individual pages. We can do that by nesting the shell script within this block of actions:

First, we create an empty variable to append each page’s transcription to. Then we loop through the pages, creating a new variable with the path of each individual page. We pass this through the shell script, and append the result to OCR'd text.

Let’s fill in the actual code now.

3.3 Making API requests

Shortcuts doesn’t provide the necessary functionality here, so we’ll offload it to python. If you’ve never used python before, you’re in luck: it comes preinstalled in MacOS.

First, run

/usr/bin/python3 -m pip install requests

to install a needed library to the system python. (Unfortunately, it does have to be the system python, so no fancy package management, pythonistas.)

You should also install openssl to your system, if you haven't already, e.g. with brew install openssl (this is with the open source Mac package manager homebrew, which you can install here: Homebrew — The Missing Package Manager for macOS (or Linux)).

Next, fill the shell script action we made above with this code.

This python script uses Python’s requests library to make an API call to Mathpix. It sends the binary contents of the PDF across the internet, then checks every few seconds until Mathpix says the conversion is complete.

And, finally, replace the strings ‘split page path (File path)’, ‘mathpix_id’, and ‘mathpix_api_key’ with the Shortcuts variables of the same names, by right-clicking within the text field and selecting ‘Insert Variable’, to make the bottom of the script look like this:

Be sure that when you insert the ‘Filepath’ variable, you select its type as ‘File Path’, and not PDF.

3.4 Saving the result to an Obsidian note

If you use Carlo Zottman’s excellent Actions For Obsidian, you can do this with three blocks.

Zottman’s app is a one-time purchase with a unique ‘pay whatever you want’ model. There’s a two-week free trial. I recommend starting with this, to see how well the automation works for you.

You can also fairly easily save the PDF to your obsidian folder with Shortcut’s built-in actions; it just takes a little more fiddling, and is somewhat idiosyncratic to your Obsidian setup. Here’s a starting point:

3.5 Testing the shortcut

The shortcut is complete! But before we run blithely forward, we’d better test that it works. So hit the play button at the top right, select a PDF and watch it run.

If you haven't previously, you'll need to allow Shortcuts to run shell scripts, by opening the shortcut settings, and checking this box:

If it does, hurrah! If not, take a look at the troubleshooting section. Be aware that the transcription may take a few minutes, during which time the shortcut will appear frozen on the shell script action.

4. Automating these PDFs with Hazel

So your handwritten PDFs now automatically appear in, say, a folder on your desktop. From here, you could manually drag them into a digital notes app, or manually execute the following shortcuts. But where’s the fun in that? As my cyborg-curious friend Otto the Automator has put it, in defense of his own automation addiction,

“People laugh at the Rube-Goldbergian contraptions I construct to save myself a few seconds here and there. Why not just do it manually? You’ll spend more time building the automation than you’ll ever save. But seconds matter. More importantly, having things happen automatically matters. Each of those seconds of manual effort increases the odds that I won’t keep doing the thing – the thing my past self, with these automations, is helping my future self do.”

In this case, the ‘thing’ is a smarter way of working with your notes over time. It’s a practice, and nothing motivates you to keep practicing than having your latest scribbles automatically appear in your note system.

Enter Hazel, the premier file-automation utility on Mac. Hazel provides a GUI that lets you perform actions on files that meet criteria, for example ‘If a file is in my Downloads folder for 60 days, move it to the trash’. Hazel is moderately expensive (a license costs $42 as of writing), but (along with Keyboard Maestro) it is one of the two Mac automation tools. Those curious enough to try this automation will likely find many other uses for it (such as those suggested in this episode of the Automators podcast).

Once you have a copy of Hazel, here’s what you can do:

1. Within Hazel, add your handwritten notes folder to the left sidebar,
2. Create a rule to rename the file to include the current date. This helps prevent conflicts between notes with the same name.
3. Create another rule that runs the shortcut we made above.
4. Create a final rule that disposes of the original PDF. Not to worry; the shortcut will save its copy to Obsidian.

In summary, this automation scans your Notes folder for new PDFs, and first renames them, before passing them into the shortcut we just created. That shortcut will add its own copy of the file to your note system, so Hazel moves the original copy to the trash. This will all happen within a few seconds of the PDF’s appearance.

5. Tying it all Together

Intrepid automator that you are, you likely have ideas to further tweak this automation to your liking. Maybe each PDF should get its own note in Obsidian, instead of just being appended to the daily note. Maybe you want to run the transcription text through Ollama’s local LLMs to clean it up – or even to summarize it, or extract TODOs and add them to your task manager, or even have Perplexity browse the web for you based on a query in the handwritten note, and save the results back to your E-ink Tablet as a well-formatted PDF!

Let us know what you get up to, and stay tuned for more automations in this vein.

The Penman would be proud. I hope he and Obsidian get to know each other better. They have a lot in common.

Troubleshooting

Some of my PDF pages are just transcribed to ‘stdout’.

This happens when Mathpix fails to process a PDF page, usually due to unusually complex diagrams or illustrations. Sadly, there’s nothing we can do about that – but it does help that the original PDF is right below the (failed) transcription for reference.

Whenever I open and close a note on my e-ink tablet, it gets re-added to Obsidian. How can I add it just when I’m done with the note?

With the Boox note-exporting automation described above, the note is re-exported whenever it’s modified. By default, this will result in duplicates of the note in your Obsidian daily notes. These don’t actually bother me – I view ’em as snapshots of work in progress, and leave them be. When I’ve finished with a note, then I’ll go through the PDF’s transcription and turn it into its own Obsidian note.

You could, however, add an if clause to the beginning of the shortcut that cancels itself if the PDF doesn’t have some key phrase, like ‘finished’.

Hey, why do members get a pre-made shortcut? I want one!

Our Riddle Membership Philosophy is this: we’ll publicly share the substantial ideas behind our work – e.g. the big picture outline of tying together Hazel, Shortcuts, Mathpix, and Obsidian. A MacStories-style shortcut download is sort of a product. Our interest in this post is not pitching (much less providing support for) a technical product. We just want to share a set of ideas for seasoned automators to play with — with the expectation that these seasoned automators will tweak and troubleshoot the automation to their own purposes.

That said, if you support us with a membership, we’ll support you with the pre-made shortcut. And you have our assurances that we’ll continue to update this automation for years to come.