Routing, why so difficult?

Every web facing application needs a request routing component. So did I for a web facing Clojure application I am working on. A popular choice is compojure. But compojure was out of the question for me. It is ruled out by the second goal of the following two goals I had for how I want to handle request routing:

1. Web related components should be able to register (and deregister) their routes at a single routes management component

2. The routes must be bidirectional, such that each component in the app can lookup a route by name from the routes management component.

One way to do this, would be to register two functions for each web component. One that does the routing/handling, and another that does the bidirectional lookup. This might work, but now you have to deal with ordering (which routing/handling function should the main ring handler try first?) and it yields a closed system: it is impossible (or at least hard) to get a full overview of all the routes, and detect conflicts for instance.

A better way would be to just use data. Web components register their routes datastructure, and that’s it. The routes management component combines the registered routes to a single routing table, which can be inspected and validated. And bidirectional lookups can be implemented centrally. Of course there are many more advantages to using data for this, but more on this later.

So I started to search for a data-driven, bidirectional routing library that could fulfill above goals. I found bidi. It looked powerful enough and supports looking up routes by name (using a “tag”). I set up the central routing component to work with bidi. Combining multiple routes datastructures seemed simple enough: just wrap the concatenation of them with [true ...], and call bidi.ring/make-handler on the result.

Yet, two issues arose with using bidi for this. First, the order in which the routes are defined matters in bidi. So web components had to specify some sort of priority value together with their routes structure. Second, the datastructures seem overly hard to write and read. The bidi documentation has a BNF specification which defines how routes can be created. The bidi API also supplies a schema for route validation. Having a BNF specification to describe the structure of your routes hints that, for me at least, it is harder than necessary. I also ran into schema validation problems on routes that were valid according to the BNF.

Maybe it is just me, but I found it too hard to use, just for the simple task of defining routes. That, together with complicating features I do not need in a routing library (like protocols that enable things such as tagging and disclosing resources), made me decide to look for other alternatives.

Alternatives to bidi

I found a couple, such as ronda-routing, tripod and clj-routing. The ronda-routing library shows what possibilities are gained by basing the routes on data and separate routing and handling. The downside of ronda-routing however is that it needs another library for the actual route matching.

The downside of tripod is that the interceptor model is baked in (instead of an option you can program yourself such as with ronda-routing), I did not like the nested routes structure and lacks support for the HTTP request method (verb).

I liked clj-routing the most. It has a simple datastructure for routes, and only a few functions that just do what a library like this should do in my opinion, and nothing more. And although I was inspired by above libraries, especially by ronda-routing and clj-routing, I was still not completely satisfied.

Enter sibiro

The downside of clj-routing is that the order of the routes still matters. Also only the first match is returned. To realize my goals, I created a new routing library that looks like clj-routing, but with a tad different behaviour, sibiro:

• simple, bidirectional routing library
• using datastructures for routes with clout-like URI patterns.
• for Clojure and ClojureScript
• compiles to fast matching structures
• the order of the routes does not matter
• returns multiple matches lazily using strict precedence rules
• and separates routing and handling.

The datastructures used for routing in sibiro look a lot like those of clj-routing. The small API of sibiro offers all the possibilities that ronda-routing and tripod offer, as we will see in an example shortly. Enough with the talking, let’s show some routes!

(def routes
  #{[:get  "/admin/user/"          user-list             ]
    [:get  "/admin/user/:id{\\d+}" user-get    :user-page]
    [:post "/admin/user/:id{\\d+}" user-update           ]
    [:any  "/:*"                   handle-404            ]})

A routes structure is a collection of sequences, with three or four elements each: the request-method, a clout-like URI, a handler (or any object really), and optionally a tag. Again, the order in which the routes are specified does not matter. Paths with more parts take precedence over shorter paths, exact URI parts take precedence over route parameters, the catch-all parameter (*) is tried last, and specific request methods take precedence over :any.

To use the routes with the other API functions, they need to be “compiled” using compile-routes. This transforms above datastructure into faster structures and functions. For example:

(require '[sibiro.core :refer (compile-routes match-uri uri-for)])

(def compiled (compile-routes routes))
 => #sibiro.core.CompiledRoutes@5728466

Matching an URI

Now that the routes are compiled, they can be used for matching and lookups. The function match-uri takes the compiled routes, an URI and the request method, and returns a map with :route-handler, :route-params and a lazy :alternatives, or nil if no match was found. For example:

(match-uri compiled "/admin/user/42" :post)
 => {:route-handler update-user,
     :route-params {:id "42"},
     :alternatives ({:route-handler handle-404, :route-params {:* "admin/user/42"}})}

The values in :route-params are URL decoded for you and the value of :alternatives is lazy, so it won’t search for alternatives if not requested.

Looking up an URI

To go the other way - looking up a route - the function uri-for can be used. It takes the compiled routes and a route handler (or tag), and optionally route parameters, and returns a map with :uri and :query-string, or nil if the route could not be found. For example.

(uri-for compiled update-user {:id 42 :name "alice bob"})
 => {:uri "/admin/user/42", :query-string "?name=alice%20bob"}

An exception is thrown if parameters for the URI are missing in the data map, or are not matching the inline regular expressions. The values in the data map are URL encoded for you. There is also a convenience function called path-for that concatenates the :uri and :query-string from uri-for.

That’s all there is to the core of sibiro.

Above is all you need for the most basic and for more intricate routing and handling. For some basic defaults, the namespace sibiro.extras contains a base handler and some supporting middleware. This will get you up to speed quickly.

Example use of sibiro basics

The README of sibiro contains a more intricate use of the sibiro basics by adding static conditional middleware and external regular expressions for the route parameters. That’s quite a mouth full, but it does show how the basics can be used. We finish this blog post with a smaller example of how sibiro can be used. Just as the README example, it shows the power of using data for routes, and that only Clojure knowlegde is required for leveraging this power.

Again, the route handler does not need to be a handler function per se. It can be any object. In the following example, we set it to be a chain of processors keywords. The last keyword should dispatch to the actual handler. The keywords before that should dispatch to something that acts like middleware. Follow along, please.

First, we use a multi-method for executing the processor chain:

(defmulti handle-route
  (fn [request] (first (:route-handler request))))

(defn handle-next [request]
  (handle-route (update request :route-handler rest)))

Now we define the routes. Some routes are public, some require admin rights, others require the user to be logged in. Some also require the body to be parsed to JSON, et cetera.

(def routes
  #{[:get  "/login"          [:public       :login-page     ]]
    [:post "/login"          [:public       :login-handler  ]]
    [:get  "/dashboard"      [              :dashboard-page ]]
    [:get  "/admin"          [:admin        :admin-page     ]]
    [:any  "/rest/*"         [        :json :liberator      ]]
    [:any  "/rest/admin/*"   [:admin  :json :liberator-admin]]})

An advantage of using data for routes, is that we can preprocess them. In this case we preprocess them to ensure that, before the last keyword is “called”, the request is authorized. This preprocessing must be done before compile-routes.

(def authorized-routes
  (map (fn [[m u h]]
         [m u (concat (butlast h) (cons :authorized? (last h)))])
       routes))

Now let’s define the processor chain methods. The final handler keywords are excluded from this example.

(defmethod handle-route :authorized?
  [request]
  (if (or (:authorized request) (session/get :user))
    (handle-next request)
    {:status 403 :body "unauthorized"}))

(defmethod handle-route :public
  [request]
  (handle-next (assoc request :authorized true)))

(defmethod handle-route :admin
  [request]
  (if (admin? (session/get :user))
    (handle-next (assoc request :authorized true))
    {:status 403 :body "not admin"}))

(defmethod handle-route :json
  [request]
  (handle-next (update request :body json/parse-string)))

And finally, we create the handler.

(def handler
  (let [compiled (compile-routes authorized-routes)]
    (fn [request]
      (let [match (match-uri compiled (:uri request) (:request-method request))]
        (handle-route (merge request match))))))

There you have it. Quite a simple way of adding behaviour to the routes using keywords and preprocessing of the datastructure. Note that this is just an example of how sibiro can be used. There are many other interesting ways to use it. For more info on the sibiro library, have a look at github, check out the API documentation or tell me what you think of it via twitter.