Rewriting my website in Pollen

August 16, 2021

After a very long time procrastinating I’ve finally rewritten my website into Pollen, a wonderful tool for creating digital books. I’m creating a website but it’s like a book in a way — it has text, some images, and should be presentable, maybe even beautiful. In this post I’ll explain what Pollen is and my rationale for choosing it, and the overall setup process.

#What is Pollen, and why?

As mentioned above Pollen is obstensibly a tool for building digital books. However, it’s pretty flexible and isn’t limited to making “books” — in fact, ignoring PDFs and non-HTML for a moment, if you consider that a digital book is in fact a website with certain requirements (such as a table of contents and a chapter-based navigation setup) you can really consider Pollen to be a library for creating and generating websites. It’s written in Racket and has the philosophy, “the book is a program,” and as such supports creating and embedding Racket directly in the document that’s being written. This manifests in both simple ways, such as embedding metadata like the title and description of this post, to more complex ways like literally embedding code into the document like this:

; The syntax for the next two blocks
(define syx `(for/list ([i (in-range 1 10)]) (* i 2)))

; (format "~a" syx)
(for/list ((i (in-range 1 10))) (* i 2))

; (format "~a" (eval syx))
(2 4 6 8 10 12 14 16 18)

To summarize, Pollen is a system for creating digital documents, such as websites, where those documents are themselves programmable.

As for why, I won’t bury the lede: a huge aspect of my desire to use Pollen is because it’s written in Racket. I really enjoy using Racket — I think I would go as far as to say it’s my favorite programming language — and having a tool in that language that is written in a well thought out way by someone who primarily writes and thus has thought about certain little details? It’s like a double whammy of awesome.

Beyond this, there are a few more nuanced reasons, especially considering the tools around creating websites and blogs have really never been better. I have tons of choices for tools, like Next.js and Gatsby, and even just creating a simple Markdown→HTML script. So, with these in mind, why Pollen?

#At the beginning there was nothing

To give some back story, I had originally set out to have no build process on my website. I had planned to write everything from scratch — all elements, all HTML, everything — I was the build process. This meant that if I wanted to write something like this:

Howdy y’all 🤠 Welcome to my super cool website!

I would, by hand, write this:

<p>Howdy y’all 🤠 <em>Welcome to my <strong>super cool</strong> website!</em></p>

My goal was to hearken back to a simpler time of websites being written in just HTML and CSS, by doing everything by hand and avoiding things like build tools and using JavaScript to do “cool shit” on my website. To be fair I wasn’t completely unwavering in this philosophy, and I also wasn’t sadistic — I did use git as the source control of my website, and I did automatically sync my website whenever I made a push up to Github, where the website source is hosted. But I really wanted to keep that amount of automation the maximum I would introduce.

But I slowly came to the conclusion that this was both untenable and unrealistic. The initial version of the website was easy, I just had my homepage and my CV. As I expanded into writing that changed, and after after writing my first post I was exhausted and honestly frustrated at the number of times I needed to shift things around between paragraphs and having to move around tags, make new tags, and this frustration led to me seriously avoiding writing anything else. In addition, over time my philosophy on having “cool shit” changed and I realized that I’d like to explore some fun things on my website, like generative art, doing Advent of Code puzzles, my photography, and being on the “bare metal” just didn’t really make sense in that context.

Sometime after I had written that post (and having aspirations for writing others) I was looking to improve my Racket abilities and returned to Beautiful Racket (by the same author of Pollen, Matthew Butterick) and eventually stumbled upon Pollen. I was interested and wanted to try it out, but didn’t take the time to use it and rewrite my website into it until now.

#The more nuanced aspects

Naturally, one question that comes up is, if a pain point from your previous website was the raw HTML aspect and you’re focusing on writing with some markup like italicization or emboldening, why not use something like Markdown?

The answer is quite simple: I don’t want to only write plain posts. As mentioned above I do want to do other things that involve non-markup like generative and interactive art using JavaScript and Markdown just wouldn’t work. Of course there are now more advanced versions of Markdown that do allow you to embed HTML, but that’s still limiting in that I still don’t have an ability to extend either Markdown or add things like components (of course web components exist, but you’re still stuck in Markdownland if you primarily use Markdown).

Of course, there are tools that, like Pollen, allow you to utilize much more power and use a programming language to generate your document, such as Next.js and Gatsby. There are even tools that turn the Markdown-HTML embedding on its head by rendering Markdown as code, which is then rendered again by some program (in this case, React). These tools are conveniently all JS, my most used language, but a tool like Jekyll or Hugo could be substituted, and ultimately I feel all of these still miss the mark. I think Matthew sums up part of my core thoughts quite well in his case against Markdown:

An animating principle of Pollen, as explained in the Backstory, is that after 20 years, we ought to move beyond thinking of HTML as a source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that’s not good enough. I’m ready for the tools to expand to fit my ideas. I refuse to keep cutting down my ideas to fit the tools.

Although less applicable to more feature-rich tools, I think it still applies. With Markdown, I’m still writing HTML (with less flexibility of HTML since it only supports some ways of formatting). With some other tools, I’m stuck in the same boat of having to write something that is basically flavored HTML, and with other tools still, I’m almost writing an app rather than text and often the tool imposes certain restrictions because of its app-like nature, and combined with the fact that I already write web apps for a living I don’t necessarily want to write another one if I don’t need to (and a website doesn’t need to be).

Lastly, as mentioned in the beginning, the fact that Pollen is in Racket helps a lot, and even disregarding the points I just mentioned I would still be really inclined to choose Pollen for that reason. It also helps that in a year or two I won’t have to worry about upgrading my website to the “next version” of the tool. It will probably still work in 5, 10, maybe even 15 or 20 years without much hassle.

#Getting started

Okay, I rest my case and have said enough about why. How about how?

The documentation, as expected for a good Racket library, is really great. Installation was pretty straightforward:

raco pkg install pollen

After a moment of setup, I was ready to rock, literally:

> raco pollen start ./src
pollen: starting project server ...
pollen: welcome to Pollen 3.1.2883.469 (Racket 8.2)
pollen: project root is ...
pollen: project server is http://localhost:8080 (Ctrl+C to exit)
pollen: project dashboard is http://localhost:8080/index.ptree
pollen: project server permitting access to all clients
pollen: ready to rock

It actually worked even without rewriting anything yet; Pollen simply passed the files with .html through unprocessed, which is really nice. However, nothing was yet in Pollen language so I needed to rewrite the various pages of my website (thankfully there were only a few) and create a template.

Rewriting those pages and creating the template is pretty rote so I won’t go into much detail, since it’s explained in quite well in the docs. In short, I basically created an HTML template called template.html.p in my repo root, transformed each of the HTMl files I had into Pollen, renaming them to <pagename>.html.pm, translating the HTML tags to Pollen tags, creating little functions in my pollen.rkt file, and embedding the right metadata into my posts so the template renders slightly differently for them.

Okay, actually it wasn’t completely rote and it wasn’t completely without trouble. If you end up experimenting with Pollen too, I do have a few notes which might help:

  1. Templates don’t inherit from other templates, but the root template is applied in the absense of more specific templates in a directory. If your pages are like mine where there’s not much different, you can probably get by with just adding ◊when/splice conditionals in your root template combined with tagging certain pages with metadata.
  2. Remove all indentation from your template. It isn’t processed in the same way as the other files (it’s more or less passed as-is), and if there’s intentation in it, if you add something like a <pre> tag it will render strangely.
  3. I recommend making a few helpers inside of your pollen.rkt file for things like getting metadata, it’s very helpful if you do end up having a template that renders slightly differently based on that metadata. I do this and I’ll publish a helper library at some point.
  4. Do not rely on the automatic page tree, and instead create your own. This is probably the most annoying part of the process you’ll likely encounter as you’ll need to do it each time you add a page.
  5. The index.html page is included in the pagetree, it isn’t handled automatically.
  6. The stack traces are often nebulous and not very useful. I strongly recommend creating contracts for your functions as this helps stack traces quite a bit. It also helps to keep in mind that symbols are used in most cases and you can probably safely assume what you’re dealing with is a symbol, until proven otherwise.
  7. If you come from React, the ‘@ tag is akin to a <Fragment> tag. It lets you embed something in your document without needing to create an actual rendered tag like div.
  8. The lozenge character is a little hard to print, just like any special character in a language. I got around this by making a exported variable called loz and wrap it using |. For example, ◊|loz|some text renders into ◊some text.
  9. If you use the built-in Pygments helper highlight, you might need to set #:python-executable “python3” when invoking it. Speaking of, if you pass in an incorrect language to highlight it will render but might look strange, rather than throwing. The main one I ran into trouble with was ‘shell-session. It’s also worth pointing out that the symbols are PascalCase-ized, so ‘shell-session becomes ShellSession to Pygments.
  10. If you follow the tutorial from the docs and add #:string-proc (compose1 smart-quotes smart-dashes) to your root function, keep in mind that it doesn’t care where the dashes are rendered in, including in highlight. I recommend to not use it or to make a slightly more restricted version that looks for " -- " specifically.

#Getting it built and deployed

Okay. All my pages have been rewritten into Pollen, I’ve made my template, now I need to deploy it. I planned to use basically the same setup I have now — copying the website to my server using rsync inside of a Github Action.

But wait, stepping back, something’s missing. Since I’m using a tool rather than writing HTML, how can I reproducibly build my website? I still wanted to keep things light and not introduce too many new tools so I just went with Make and made a simple Makefile. This seemed to work locally, but once I pushed up to the CI it failed... and it turns out that I’ve run into one of the areas that Racket doesn’t do so well: package management.

#Lovely raco

I’m used to pretty good package managers like Bundler, npm, and pip, so it was a little surprising that Racket’s built in package manager raco was a little lacking. I had a really tough time getting the packages I wanted installed in a way that was simple and reproducible. That is, I could easily install a package using raco pkg install <package-name> but translating this into a way that I can set it up to automatically install on the CI (or to keep my local version of these dependencies up to date) was a struggle.

The approach of installing dependencies using raco pkg install works well enough and considering I only have a few dependencies so it would work to just put them inside of the makefile and call it a day, but I didn’t want to accept this. As the saying goes, why do something in 5 minutes what you can automate in 5 hours?

After some research I tried to just create an info.rkt file in my main website repo, add the dependencies, create a make command, and run it. This failed, with the following message:

> make install-deps
raco pkg install
Linking current directory as a package
raco pkg install: invalid package source;
inferred package name includes disallowed characters
given: .../joshuasmock.com/
make: [install-deps] Error 1

It turns out that the package name is (in my opinion strangely) inferred from the directory, even if one is set in the info.rkt file. I wasn’t going to rename my repo just to suit raco, so instead I made a fake directory called /deps and put info.rkt in there. After changing the commands in the makefile, things worked! However, things didn’t work after I ran it again — because raco treats that directory as a package now, and because the package version hasn’t been bumped, it thinks that everything is up to date and says as much:

> make install-deps
raco pkg install ./deps
raco pkg install: package is already installed
package: deps
make: [install-deps] Error 1

After even more research, trial and error, repeated, I came up with the following commands:

.PHONY: install-python-deps
install-python-deps: pip3 install -r requirements.txt .PHONY: install-deps
install-deps: install-python-deps
# We may or may not have installed it, ignore any error if so - raco pkg install ./deps raco pkg update -a ./deps .PHONY: install-deps-ci
install-deps-ci: install-python-deps raco pkg install --batch --auto --no-docs -i ./deps

These commands ensure that I have both Pygments and the various Racket dependencies installed and up to date whenever I install locally. On the CI I can just install since I know for sure ./deps hasn’t been installed.

#Building my website

Onto actually building the website. This was a lot simpler than the above, all I needed to do was just run the render and publish commands from Pollen:

.PHONY: build
build: raco pollen render -r ./src raco pollen publish ./src ./build

All this does is turn the .html.pm files into compiled .html files, and copy the non-.pm files from ./src to .build. This means that your assets, like your CSS and JS files, will be correctly copied, as well as any other non-Pollen HTML files.

With these commands, I updated my workflows to use the new commands, pushed up to a new commit to the PR, and... it worked! Merging the PR, thereby running the master workflow, also worked! My website is now fully built with Pollen.

#Conclusion

The whole process of rewriting my website into Pollen took a few hours of work, spread over a few days, with a large chunk of the work being rote translation from HTML like <em>this</em> to ◊em{this}. I also spent some time learning how syntax quoting and unquoting works, as it’s very useful when creating your own tags or even something akin to components and something I hadn’t used very much outside of a couple Beautiful Racket exercises. If you end up using Pollen I also recommend learning what symbols like `, ,, and ,@ do in the context of a function.

Anyway, I found the process itself quite interesting and overall felt like it was a great learning experience. I was already relatively comfortable with Racket when I picked it up, thanks to both my previous use of it during Advent of Code the past two years, a little bit of playing around here and there, and my current job (I am using Clojure there, but enough is transferable), but this project helped reinforce some things I needed practice with as well as help me apply other concepts I hadn’t outside of a learning setting, specifically quasiquoting and unquoting. I also learned a bit about some less positive parts of Racket, namely package management.

To end, I would definitely encourage you to try out Pollen, especially if you are looking for something different from the mainstream tools, want to focus more on writing and less on the tool, and want to flex (or begin to gain) your Racket muscles. Once you get a few things set up, which I hope this article helps demystify some of, it basically gets out of your way and you can get down to creating, and you won’t need to worry about keeping everything up to date with Dependabot, or migrate your blog to some new system in a year or two, when the version of your current framework or library bumps, or try to remember what custom system you set up and how to get it all working next time. In other words, it’ll just work and continue to work and you can continue to work in harmony with it.

Thanks for reading! ◾️