Flores, plugin architecture and dependency hell

A few months ago I started working on a static site generator called Flores, with the main use case in mind being my own site. I wanted to try to describe why I felt like making yet another static site generator (I promise it's not because I thought I could "do it better"!) and talk about some of the common problems I have with frameworks in general.

Why build a new generator anyway?

Before Flores, I was using Jekyll to generate my site. I actually think Jekyll gets a lot of things right and that's probably why it's popular; Flores is also very much inspired by Jekyll. There were, however, two big problems with Jekyll for me: Jekyll's scope and plugin-based architecture, and the resulting dependency hell.

To me, the whole purpose of a static site generator is simplicity: you use one because you don't want to handcraft every single page, inevitably repeating yourself, and also because you want to be able to deploy your site with minimal effort, ideally on any platform. As such, an SSG's job should be to take care of all of the "common sense" things that the vast majority of the users will want, and of course leave space for extra functionality for the "power users" that want to do more complex stuff.

Where frameworks like Jekyll fail — in my humble opinion — is that they actually do not take care of all of the basic, "common sense" stuff, and they instead choose to delegate pretty much everything to plugins (or otherwise extra config settings). Take code highlighting for example: when I was using Jekyll for my personal site, you needed to add a dependency on Rouge, specify it as the highlighter (although now I think it's the default?) and specify a Markdown flavor. I get the motivation behind this: you enable power users to potentially use their obscure, little-known highlighter and Markdown parser. And at the end of the day, sure, why not? Computers are a social construct anyway.

Well, I think this goes against the whole SSGs strive for simplicity thing. When designing an interface, engineers tend to want to expose as many buttons and switches as possible (probably because we ourselves would like more access "under the hood"), without necessarily questioning this choice. But if we think about it... is "I wanna use my own Markdown flavor/code highlighter" a "legitimate" use case, to the point where the framework has to bend over backwards to support it? I disagree with that. You throw a rock out here, you hit a different shade of bike shed paint. You can always argue that one flavor of Markdown is better than another, or that one highlighter plugin is better than another, but at the end of the day — again, in my humble opinion — any of them will do just fine, so long as they're decent.

To give a concrete example: Flores also depends on a code highlighter, Pygments. However, it does not allow the user to specify a different one, let alone roll their own one. This comes with two advantages:

1. You don't have to worry about maintaining compatible dependencies. It's Flores' job to guarantee that it will work on the platforms it claims to support, and it's simpler for Flores maintainers (yours truly) to have to deal with a single code highlighter, both in terms of how it interacts with the rest of the SSG and in terms of compatibility with different platforms.
2. You don't have to care. Why specify something in a config file? Why not have code highlighting work out of the box with literally no input from the user as far as the config is concerned? If you're using a SSG, that means you're tech savvy enough to write some HTML, CSS and maybe even some JavaScript, so it's likely that you'll have code snippets in your pages/blogposts.

To expand on (2), that's kind of what I meant before when I was talking about "common sense" features; an SSG will most likely power a site that has snippets, that has blogposts and so on. Another feature that I didn't find in Jekyll (and that I wrote down immediately when listing the things that I wanted Flores to support out of the box) is image optimization: it is extremely common to want to optimize image size, whether that concerns compressing more or producing multiple different sizes out of an original image to serve to devices with different screen sizes. With Jekyll I had to write my own bash script that did that; with Flores I can just specify it in a few lines of config. Same thing goes for Sass/SCSS support: why not have it out of the box in 2023? Flores supports it, and if you think it's horrible and don't want to use anything but pure CSS, well... you just can. You simply use .css files, and that's it, Flores doesn't process them in any way whatsoever. The only thing that's not supported out of the box is some sort of JavaScript transpiler, something like TypeScript or Babel, but that's because it's much "heavier" as a dependency and because static sites should probably have little or no JavaScript anyway (again, aren't they supposed to be simple?). I'm still thinking about supporting it though (optionally, just like Sass/SCSS).

The problem(s) with plugin architecture

More generally (i.e. not just regarding SSGs), I dislike the whole "plugin architecture" model, where basically everything's a plugin and the "core" of the framework only supports the absolute bare minimum.

If the only downside of plugins was that you might get overwhelmed by the choices, then maybe that would have been fine; the problem is that these choices do not come for free. Jekyll, for instance, has you managing the Gems (that's Ruby for libraries, I believe) that you use via plugins, along with the dependency hell that comes with that: you might have version conflicts, or some Gems might not work on some platforms etc. Again, I understand the design choice here: you push that responsibility onto the user, and if they construct a site with a gazillion dependencies, too bad, that's on them. I disagree with it because I believe that in most cases the user just wants to have a site, not deal with the internals of the SSG or the nuanced conflicts of different Gems.

Flores is not completely free of this either, by the way: for the moment, it's using Python-Markdown (or Markdown? folks please choose one name), which, uh... leaves a lot to be desired, to say the least. It's also plagued by plugin architecture, but its documentation (unlike Jekyll's) is sparse and confusing (looking at you, CodeHilite) and way too many things are considered "optional" (like tables, seriously?). At least that's not exposed to the user, but Flores has to deal with this mess internally.

Even if you still think that plugin architecture is a good idea, you first have to do it right: the core must still support all of the basic, "common sense" features, and leave the truly optional stuff to plugins. If we again take Python-Markdown as an example, given that it is a developer-oriented framework, then sure, you can argue that abbreviations are not an essential feature, but I really don't think you can say the same for tables or code blocks or code highlighting. Sure, maybe many of these problems come from Markdown itself and how it's defined (or rather, how it is not), but I don't agree that a Markdown parser should adopt the "not my problem" attitude here; at the end of the day, when we get a chance to fix an issue, we should, right?

The last problem I have with plugin architecture is that you're counting a lot on adoption. We can say that libraries of programming languages are plugins, sort of; the standard lifecycle of a programming language dictates that, if a library is used extensively by the majority of the community and/or becomes basically an essential part of the language, then the language devs usually merge that library into the standard library or the builtins of the language (mainly to allow devs to use it without having to depend on an "external" library for something essential). This is why I said that plugin architecture depends on adoption: you're counting on people developing that sort of stuff for you so that you can even have that functionality available to the users of the framework. But what if your framework is not very popular, or what if you don't have the time to develop a plugin that's actually an essential feature yourself? This is more or less the current status with code highlighting for Python-Markdown: the "main" plugin is CodeHilite — which again leaves a lot to be desired — and as far as I know the only alternative is Highlight, which has its own set of problems. So basically my choice as Flores' developer is to either roll my own highlighter, which defeats the whole purpose of delegating this task to the Markdown parser, or to just live with the problems of the (limited) existing choices. I truly think that CodeHilite would be better if it was the only choice and part of the core of Python-Markdown, because then you have to make something solid and deal with bug reports and feature requests, as there is no alternative. This pushes the burden on the developer, which is where it should be: the framework is the developer's job, not the user's!

Is Flores actually better?

Of course, I'm way too biased to answer this question in a conclusive way. I can, however, compare building my own site with Jekyll and with Flores. To be clear, I'm basing the comparison on the previous version of the site, simply because a lot of simplifications were done during the latest redesign that have nothing to do with the underlying SSG. By using Flores' built-in image optimization feature alone, we save almost 70 MB worth of data on the "source code" of the site (meaning the contents of the Git repo, not the actual site that's served). In terms of the actual site size, we save ~35 MB, although some of that is due to missing page issues that Flores has at the time of writing; a big part of it is nonetheless due to the fact that, because of the janky homebrew workaround I used to have in order to optimize images, Jekyll will also dump the original images on the site, while Flores will only dump the "produced" images (i.e. optimized images in the specified sizes). Of course that can be fixed in Jekyll, but in Flores you simply don't have to worry about it.

It's also just much neater. With Jekyll:

$ tree . -L 1
.
├── 404.md
├── about-me.md
├── archive.md
├── blog.md
├── compile_cvs.sh
├── compile_images.sh
├── _config.yml
├── css
├── cv_pdfs
├── _data
├── _drafts
├── favicon.ico
├── Gemfile
├── Gemfile.lock
├── images
├── _includes
├── index.md
├── js
├── _layouts
├── _posts
├── _projects
├── README.md
├── _sass
├── _site
├── sketches.md
└── thanks.md

12 directories, 14 files

And with Flores:

$ tree . -L 1
.
├── _assets
├── compile_cvs.sh
├── _css
├── cv_pdfs
├── _data
├── _drafts
├── _js
├── _pages
├── _posts
├── _projects
├── README.md
├── _site
└── _templates

11 directories, 2 files

Again, you can argue that this is "purely aesthetic" or irrelevant to the actual efficacy of the SSG or whatever, but I think it really counts. A more organized repo means that you can spot where things are much more easily; I truly do not understand why Jekyll went with the "just dump all the .md page files at the root of the repo" option.

The last thing I'd like to note here is that Flores' architecture isn't exactly the "opposite" of plugin architecture: you can still totally add extra steps or mechanisms to the site build. For example, the way I currently deal with my CV is that I have an extra step during the deployment of the site, where my CV repo is cloned, the CV PDF is built from source with TeXLive and finally it gets copied into the _assets directory, allowing Flores to pick it up and treat it just like any other asset. Similarly, you could choose to call a script to transpile TypeScript sources to JavaScript, and let Flores handle the resulting JavaScript files as "normal" JavaScript files. I think this way of dealing with "extra features" allows power users to do whatever they want and at the same time does not add extra cognitive load to the average users who just want the bare minimum. And besides all that: I don't claim to have designed a perfect SSG, far from it! If anyone comes around with a feature request that is legitimate and falls under the "common sense" features, I'm more than happy to add it to Flores as a core feature, given that it doesn't unnecessarily bloat the SSG or introduce scope creep.

Conclusion

To be clear, I don't mean for this post to be about "bashing Jekyll/Python-Markdown" or anything of the sort; I'm only using them as examples to show why I personally disagree with how (and how often) this "plugin architecture" is used. It ends up pushing almost every responsibility onto the user and having them deal with relatively "low level" stuff, digging into the framework or even having to deal with dependency conflicts, when all they wanted to do was to build their damn static site. I think there's a time and a place to expose all the bells and whistles of the underlying interface, but we do have to remind ourselves: not everyone cares about every nook and cranny of a framework we've built, and in fact they most likely never want to look "under the hood" if they don't have to.

More generally, and for QA's sake, software should strive to have as few dependencies as possible. I'm not saying that you should never have a plugin-based architecture, only that you don't always need it, and that we should think twice before implementing a design choice that adds a lot of extra load QA-wise without having a very good argument supporting its use.