Macros in Fennel

Automating the fun stuff · June 09, 2020

Thanks to the willingness of Technomancy & Jaawerth to answer my ever growing font of questions, I've gotten a wrap on Macros in Fennel. And I'm rather pleased to say that despite the quirks of how the system is built, it's feature rich and easy to work with once you've gotten your head around it.

Macros are at the heart of any sort of lisp, and if you've read Let Over Lambda you'll probably have a head swimming with fantastic dreams of DSLs. At least that's where I find myself. To put it in a more relatable manner, macro programming is the automation of programming. Anyone who's found themselves doing systems administration for any amount of time can drink to that mentality. Automation is the savior of the overworked.

Automating away repetitious code, functions, data structures. Emitting code, using code. We can use toAPK as an excellent example of macro programming in Fennel. Currently in toAPK we have a strip function that is defined per package type. That is to say, if I need to convert an Arch PKGBUILD then I have a PKGBUILDstrip function, for VOID templates I have a TEMPLATEstrip function. These functions are a mapping between the package specific variables & Alpine APKBUILD variables.

(fn PKGBUILDstrip [pkg]
  (each [line (io.lines pkg)]
    (do
      (var dat (split line "="))
      (if
       (= (. dat 1) "pkgname")
       (tset APKBUILDtbl "PKGN" (.. APKBUILDtbl.PKGN (. dat 2)))
       (= (. dat 1) "pkgver")
       (tset APKBUILDtbl "PKGV" (.. APKBUILDtbl.PKGV (. dat 2)))
       (= (. dat 1) "pkgdesc")
       (tset APKBUILDtbl "PKGD" (.. APKBUILDtbl.PKGD (. dat 2)))
       ...))))
 

Nothing inherently bad about that strip function, but what if we had two of them, or six? It would work just fine, but I would have to write out a new function, or extend an existing one with a new series of conditions. That would result in longer, less readable code, and wasted time. Lets just define the function once, and make it build our code for us.

Since our function does little more than checks for the existence of a variable, and if it exists captures the variables value, we can do something like the following.

(fn packagestrip [pkg check]
  (each [line# (io.lines ,pkg)]
    (do
      (var dat# (split line# "="))
      (each [k# v# (pairs ,checks)]
        (if
         (- (. dat# 1) k#)
         (tset APKBUILDtbl v# (.. (. APKBUILDtbl v#) (. dat# 2))))))))
 

At the surface package strip takes a package file, and a list of checks, and for each line in the file it creates an (if (= package_var apkbuild_var) (do x)) structure for us. Using our little macro looks something like this.

(packagestrip "~/PKGBUILD" {"pkgname" "PKGN"
                            "pkgver" "PKGV"
                            "pkgdesc" "PKGD"
                            "url" "URL"
                            "arch" "ARCH"
                            "license" "LIC"
                            "source" "SRC"
                            "checkdepends" "CDEP"
                            "depends" "DEP"
                            "options" "OPTS"})
 

Anyone comparing the invocations can see plainly why one might prefer the macro over the boilerplate. There's an eloquence about it. All of that said, it's also fair to say that this could in fact be done with a regular function, but where's the fun in that?

To explain the code a little better, in Fennel var# is an auto-gensym invocation. Inside of the macro scope (denoted by `()) variables can be trampled, to prevent this gensym. By defining check# val# what we're really doing is (let [val (gensym) check (gensym)]) and building the rest of the macro function inside of the generated lexical scope.

Further when access external values, such as the package path, and list of variable bindings, we access them with a back tick (,pkg & ,check). This works the same as it does in Common Lisp. That is to say the back tick brings us up one level inside of the lexical scope.

Really wonderfully useful stuff, but there's a small short sight in packagestrip. It relies on a function definition, split. Probably not a huge issue for toAPK, but what if packagestrip was to be used somewhere else? Monocle uses an almost identical system to process it's .monoclerc file for example. Fortunately in Fennel we can embed entire functions in our let bindings!

(fn packagestrip [pkg checks]
  `(let
       [split# (fn split# [val# check#]
                 (if (= check# nil)
                     (do
                       (local check# "%s")))
                 (local t# {})
                 (each [str# (string.gmatch val# (.. "{[^" check# "]+)"))]
                   (do
                     (table.insert t# str#)))
                 t#)]
     (each [line# (io.lines ,pkg)]
       (do
         (var dat# (split# line# "="))
         (each [k# v# (pairs ,checks)]
           (if
            (= (. dat# 1) k#)
            (tset APKBUILDtbl v# (.. (. APKBUILDtbl v#) (. dat# 2)))))))))
 

Fennel even supports &optional arguments through the use of (lambda). If instead of (fn) we used (lambda packagestrip [pkg checks ?optional ?optional2] ...). In fact the only "gotcha" I've run into thus far, is how the macro modules are loaded. The macros cannot exist inside of the same file as the rest of your code. In Common Lisp we can just intermingle (defmacro)'s alongside (defvars)'s and (defun)'s. In your macro file you need to include a mapping at the bottom.

And then inside of the files you mean to use the macro in you need to import them.

(import-macros {: packagestrip} :macros)
 

To further iterate just how cool this functionality is, we can inspect the compiled Lua code that Fennel produces. Below is the macro invocation inside the main body of toAPK.

(if (= typ "-a")
        (do
         (configure)
         (packagestrip pkg {"pkgname" "PKGN"
                            "pkgver" "PKGV"
                            "pkgdesc" "PKGD"
                            "url" "URL"
                            "arch" "ARCH"
                            "license" "LIC"
                            "source" "SRC"
                            "checkdepends" "CDEP"
                            "depends" "DEP"
                            "options" "OPTS"})
         (PKGBUILDrest pkg)
         (PKGBUILDclean)
         (printAPKBUILD typ)))
 

That little snippet expands into this monstrous Lua block. Complete with a local definition of the split function. We've all heard the expression, automating the boring stuff, but even automating the fun stuff is entertaining when your results are this cool!

if (typ == "-a") then
   configure()
   do
      local split_23_0_ = nil
      local function split_23_0_0(val_23_0_, check_23_0_)
		 if (check_23_0_ == nil) then
			local check_23_0_0 = "%s"
		 end
		 local t_23_0_ = {}
		 for str_23_0_ in string.gmatch(val_23_0_, ("([^" .. check_23_0_ .. "]+)")) do
			table.insert(t_23_0_, str_23_0_)
		 end
		 return t_23_0_
      end
      split_23_0_ = split_23_0_0
      for line_23_0_ in io.lines(pkg) do
		 local dat_23_0_ = split_23_0_(line_23_0_, "=")
		 for k_23_0_, v_23_0_ in pairs({arch = "ARCH", checkdepends = "CDEP", depends = "DEP", license = "LIC", options = "OPTS", pkgdesc = "PKGD", pkgname = "PKGN", pkgver = "PKGV", source = "SRC", url = "URL"}) do
			if (dat_23_0_[1] == k_23_0_) then
			   APKBUILDtbl[v_23_0_] = (APKBUILDtbl[v_23_0_] .. dat_23_0_[2])
			end
		 end
      end
   end
   PKGBUILDrest(pkg)
   PKGBUILDclean()
   printAPKBUILD(typ)
end
 

Fennel continues to surprise me. From the robustness of the language, to the similarities to Common Lisp, and even now the developers have managed to enable static compilation. I'm delighted to have this little lisp in my repertoire, a tiny cross platform lisp, with a powerful punch!