TL;DR

During the development of another library, a new lifecycle managment library for Clojure “revealed” itself. Without getting too entangled in Heidegger’s philosophy, this blog post discusses this new library.

Redelay is a Clojure library, offering first class State objects for state lifecycle management. You can think of State objects like Clojure’s Delay objects, but tracked and resettable. Realized State objects can be reset in reverse realization order by calling (stop). Because of its first class nature you get to decide how you like to create, pass around and use your state.

The long story

State lifecycle management in Clojure is something that highly interests me. I started with Component way back when. Heck, we even had something custom similar to Component when it didn’t exist yet at my first professional Clojure job. Later I discovered mount and I liked it very much. So much actually, that I created a spin-off called mount-lite. I also tried Integrant and I blogged about how a small change to Component’s Lifecycle protocol solves the problems I have with Component.

Of these solutions, I still like mount(-lite) the most. Adding a lifecycle around some stateful part of the application is incredibly easy and starting and stopping the system just works. It goes out of your way and it is hardly a framework.

Tradeoffs, always

However, in software development there are always trade-offs, and mount(-lite) is no exception. One of the advantages of the solutions that use maps with components - or “system maps” - over global components is their unit testing story. Inside a unit test, one can simply create a system map that contains just enough components and has “test” versions of them, and pass it to the unit under test. Both Component and Integrant work this way. For example, a unit test in Component is like this:

(deftest my-test
  (let [system {:api (mock-api)}]
    (is (= (unit-under-test system) :ok))))

With mount(-lite) however, you possibly have to “substitute” states and start the system, ensuring only the applicable states are started. And at the end of the test you have to ensure that the system is stopped again. It works fine, but it is a bit of a hassle. For example, in mount-lite it could look like this:

(deftest my-test
  (mount/with-substitutes [#'api (mock-api)]
    (mount/start)
    (is (= (unit-under-test) :ok))
    (mount/stop)))

Of course, this is only the case when the (unit-under-test) refers to the global state directly. In most cases however, you’d have designed your functions such that they receive the state as a parameter. This would make the unit test more like the Component example again.

A related downside of mount is that the state is always global. This implies that only one system can be started at any given time. Mount-lite 2.0 does allow multiple systems, but it’s API was not the most intuitive.

Introducing… mount-lite 3.0??

Now, I could continue this post by introducing mount-lite 3.0, because version 3.0 fixes the former two downsides. Actually, this new version of mount-lite is as good as finished (see 3.x branch). And while it may see its release not too far from now, it is not the reason for this blog post, as you’ve guessed from the title. This is because one issue was still nagging me.

Like mentioned before, state in mount is global. I obviously do not have a problem with that per se, as long as one does not refer to it from every corner of the application. However, there is something else global as well. It is its internal registry of which global states actually exist.

When you define a mount (defstate ...), the global state is added to this registry if it wasn’t already known. When you invoke mount’s (start), it uses this internal registry to determine what states to start and in what order. The same goes for (stop).

While this makes mount’s start/stop so easy to work with, it can cause trouble. For example when you introduce a new defstate while your application is already fully loaded. Because mount and mount-lite rely on the load order of the Clojure compiler, it cannot determine the correct order for this new defstate. The only solution now is fiddling with the internal registry, or restarting the application. The latter defeats the purpose of these kind of libraries.

Mount-lite 3.0 currently toys with a function called (forget!), which cleans the internal registry. Afterwards one can reload the application, fixing any inconsistencies in the registry. The name of the function or this entire functionality may change or vanish though. Because, I don’t really like having it. It complicates things. Can’t I fix this as well, just as I fixed the other two downsides? What is actually the underlying problem here? Back to the drawing board.

Don’t call mount/start

Turns out, the problem is not the fact that the states are global. I mean, take Clojure namespaces for example. Those can be seen as global registries with vars. Mount’s registry is not the problem either. Again, look at the clojure.spec library. That library has a global registry as well, yet does not need something like a (forget!).

The difference between those global registries and the one in mount(-lite) lies in the way they are queried.

A namespace does not need to keep track of some sort of order for its vars, or which ones are new or old, because it will get queried precisely at the moment a particular var is needed. Likewise, the clojure.spec registry doesn’t have this problem either. A spec definition is either needed at some point, or not. Put simply, it is usage-driven. One function needs another function, so it queries the namespace. One spec needs another spec, so it queries the specs registry. If a new function or spec has been added, it just works. If an old function or spec is not used anymore, that’s fine.

However, with mount(-lite) it is not usage-driven. Calling (start) starts all the states. Therefore it needs to know the order, it cannot recognize how a new state fits in this order, and it cannot recognize that a state is actually old and not used anymore (except with some tricks).

Now the question is, how can we design a mount-like solution to be usage-driven? Easy, just remove (start). Let me explain.

Introducing… redelay!

What if we design our state construct to only be activated when it is required, so it becomes usage-driven? In other words, only run the state’s start logic when requested. Clojure already has a construct that does this: a Delay. The expression you pass to delay is lazy; it is only realized once it is dereferenced using deref, force or @. However, once a Delay is realized, it stays that way.

What if we create a Delay that can be reset, possibly by running some stop logic? That’s exactly what the redelay library is about. Instead of Delay objects, it lets you create State objects. State objects take a start expression, but optionally also a stop expression. By dereferencing a State object the first time, it is realized by executing the start expression. The result is cached and returned when dereferenced again.

A State object also implements Java’s Closeable. When calling .close on it, the stop logic is run and its cache is cleared. This makes the State ready to be realized again.

Adding a lifecycle around a stateful part of the application using an object like this makes it first-class. In other words, we can create, keep and pass around State everywhere we like, however we like. This is also the reason that redelay does not have some sort of start or init function. It simply does not know where your state is, and it does not need to know. As long as your functions receive or can retrieve the State objects, the “system” starts itself.

But what about resetting the system? Next to offering a State object, the redelay library has one more trick up it’s sleave. It keeps track of State objects that have been realized and in what order. Calling the (status) function enumerates the realized State objects. And while a start function is not part of the library, a stop function is. Calling (stop) will close all the realized State in reverse order, wherever they may be. Even if the application or your REPL lost track of the reference to a State object, redelay can still reset it. After calling (stop) your system is ready to be started again. It may now start with redefined States and/or it may ignore some old ones. Anything goes, now that State has become first class.

The full API

Time for some source code! Let’s have a look at how the redelay API works. First, creating a State object is done using the state macro. The body of the macro consists of any number of forms. These forms can be qualified using a keyword. Valid qualifiers are :start, :stop, :name and :meta. If no qualifier is given, :start is assumed. For example:

;; A valid State object, but rather useless.
(state)

;; A State object with only an (implicit) :start expression.
(state "hi")

;; A State object with both a :start and a :stop expression.
(state "hi" :stop (println this))

;; The same, but with an explicitly qualified :start expression.
(state :start "hi" :stop (println this))

;; A State object with all options.
(state :start (println "starting greeter")
              "hi"
       :stop  (println this)
       :name  greeter
       :meta  {:dev true})

As you can see, the state macro is flexible. Both the :start and the :stop expression can consist of multiple forms. Also note that the :stop expression has an implicit this parameter to refer to its own value. A State object also supports metadata (meta and with-meta), which can also be set using the :meta expression.

The :name expression takes a (optionally namespaced) symbol. This makes recognizing State objects easier. Notice how the State objects are printed differently depending on whether a name is specified:

(state)
;=> #<State@59165[user/state--312] :not-delivered>

(state :name my.app/config)
;=> #<State@96725[my.app/config] :not-delivered>

If you want your State objects to be global, there is an extra macro called (defstate). The following expressions are all the same:

(def foo (state :start "bar" :name user/foo))

(defstate foo :start "bar")

(defstate foo "bar")

To use a state, it must be dereferenced. If it is dereferenced for the first time or after it has been reset, it is realized by running :start expression. The result of the expression is cached and the State is now tracked by redelay. The (status) function returns which States are realized.

(defstate alice
  :start "Alice"
  :stop  (println "Stopping alice..."))

(defstate bob
  :start (str @alice " and Bob")
  :stop  (println "Stopping bob..."))

(status)
;=> ()

@bob
;=> "Alice and Bob"

(status)
;=> (#<State@121352[user/alice] "Alice">
;=>  #<State@691526[user/bob] "Alice and Bob")

To stop the system, one can call (stop). All the tracked states are closed in reverse realization order. Continuing the last example, it would look like this:

(stop)
Stopping bob...
Stopping alice...

And that’s almost the full API: a state macro, a defstate convenience macro, a status and a stop function. There is also a state? function that returns true if the object given to it is a State object.

The last part of the API is the watchpoint var. While it simply points to itself, you can call Clojure’s add-watch on it. The watchers receive notifications of realized or closed State objects. This allows you to add logging for example, but also to create registries of your own. For example, this adds simple logging:

(defn logger [_ _ closed realized]
  (if realized
    (println "Started" realized)
    (println "Stopped" closed)))

(add-watch redelay/watchpoint :my-logger logger)

@bob
Started #<State@121352[user/alice] "Alice">
Started #<State@691526[user/bob] "Alice and Bob">

Actually, the status and stop functions are implemented using the watchpoint. In that sense, only the state macro and the watchpoint var make up the core of the library.

What this design means

Now that any stateful part of your application can be wrapped in a first class object, one is completely free where and how to create this state, where to keep it, when to realize it, how to pass it around or refer to it, et cetera.

If you like your state to be global - as it is with mount - you can use defstate as in the examples above. For unit tests, you can use with-redefs to substitute “production” state with test variants. No special “substitution” API is needed here. And if you like a (start) function, you can simply create such a function yourself. Either explicitly naming the states:

(def my-states
  [my.app/config my.app/db]

(defn start []
  (run! force my-states))

Or, more implicitly using the metadata that is set by the defstate macro:

(defn all-states []
  (->> (all-ns)                       ; all namespaces
       (mapcat ns-interns)            ; all symbol-var pairs
       (map val)                      ; all vars
       (filter #(:defstate (meta %))) ; all defstate vars
       (map deref)))                  ; all states

(defn start []
  (run! force (all-states)))

But maybe you like your components bundled together in a system map - as it is with Component. You can easily create such a map yourself. However, I recently blogged about the updated rmap library. In short, its rmap macro let’s you create recursive maps, where values can refer to other entries in the same map, using the ref function. The values are not evaluated yet at that point. Only after calling valuate! the values are evaluated and a realized/normalized map is returned. Let’s use this to create a system map:

;; Declare a blueprint of our system.
(def production
  (rmap/rmap {:config   @(redelay/state (load-config :prod))
              :users-db @(redelay/state (users-db (ref :config)))}))

(def system nil)

(defn start []
  (alter-var-root #'system
    (fn [current]
      (when current
        (redelay/stop))
      ;; Actually start the system from the blueprint.
      (rmap/valuate! production))))

(defn stop []
  (redelay/stop)
  (alter-var-root #'system (constantly nil)))

In the example above, the production map is not a started system yet. It is only when calling valuate! that the system map is really started. Creating a variant on this map - say for an integration test - is also easy. For example, we could update the config before creating the system map:

(deftest my-integration-test
  (let [integration (assoc production :config (load-config :int))
        system      (valuate! integration)]
    ...
    (stop)))

Again, these are just examples. The point is that redelay is hopefully flexible enough that it lets you deal with state and its lifecycle in your preferred way. Be it globally or in a system map, be it programmatically or using data, be it lazily or started explicitly, it’s your choice. There is no fixed approach, no rules to follow, no framework.

State management distilled to just a resettable Delay.

As always, have fun!

P.S. You can find more structured documentation at cljdoc and the source code at github.