The RADAR Architecture: RESTful Application, Dumb-Ass Recipient

Yesterday, I posted a brief note about changes in the URL conventions in Rails 2.

Sam Aaron responded with a comment:

Maybe I just haven't yet grokked the whole REST thing yet (most probably the situation), but the seeming re-marriage of verb and noun (or at least the loss of ability to easily distinguish them) disturbs me slightly. So (just out of interest) my question is: why was a great Huzzah! heard throughout the lands?

Let's look at how we got to where we are. And then let's look at a different way of thinking about the problem that might help simplify things.

It Starts With Verbs and Nouns

Roy Fielding's REST work showed us that the design of the interaction between a client and a server could make a major difference to the reliability and scalability of that application. If you follow a set of conventions embodied in REST when designing your application, is means that (for example) the network will be able to cache responses to certain requests, offloading work from both upstream network components and from the application.

But REST was about more than performance. It offered a way of separating the actions performed by your applications (the verbs) from the things acted upon (the nouns, or in REST terms, the resources). What's more, REST said that the verbs that you use should apply consistently across all your resources.

When using HTTP, the HTTP verbs (such as GET, PUT, POST, and DELETE) are also the verbs that should be used with REST applications. This makes a lot of sense, because things like routers and proxies also understand these verbs.

One of the successes built using these principles is the Atom Publishing Protocol. Atom allows clients to fetch, edit, and publish web resources. Atom is probably most commonly applied to allow people to use local editors to create and maintain blog articles—client software on the user's computer uses Atom to fetch existing articles and to save back edited or new articles.

Let's look at how that happens with Atom.

I fire up my Atom-based local client. It needs to fetch the updated list of all the articles currently on the server, so it issues a GET request to a URL that represents the collection of articles: the verb is GET, and the resource is the collection.

   GET   /articles

What comes back is a list of resources. Included with each resource are one or two URIs. One of these URIs can be used to access that resource for editing. Let's say I want to edit a particular article. First, my client software will fetch it, using the URI given in the collection:

   GET   /article/123

As a user of the editing software, I then see a nicely formatted version of the article. If I click the [EDIT] button, that view changes: I now see text boxes, drop down lists, text areas, and so on. I can now change the article's contents. When I press [SAVE], the local application then packages up the changed article and sends it back to the server using a PUT request. The use of PUT says “update an existing resource with this new stuff.”

Browsers are Dumb

If you use a browser to interact with a server, you're using a half-duplex, forms-based device. The host sends down a form, you fill in the form, and send it back when you're ready. This turns out to be really convenient for people running networks and servers. Because HTTP is stateless, and because applications talk to browsers using a fire-and-forget model, your servers and networks can handle very large numbers of devices. This is exactly the architecture that IBM championed back in the 70s. The half-duplex, poll-select nature of 3270-based applications allowed mainframes with less processing power than a modern toaster to handle tens of thousands of simultaneous users. The browser is really not much better than a 3270 (except the resolution is better when displaying porn). (Recently, folks have been trying to circumvent this simplicity by making browser-based applications more interactive using technologies such as Ajax. To my mind, this is just a stop-gap until we throw the browser away altogether—Ajax is just lipstick on a pig.)

How well do browsers play in a RESTful world?

Well, for a start, they really only use two of the HTTP verbs: GET and POST. Is this a problem? Yes and no.

We can always hack around the lack of verbs by embedding the verb that you wanted to use somewhere in the request that you send to the server. It can be embedded in the URL that you use to access a resource or it can be embedded in the body of the request (for example as an element of the POST data). Both work, assuming the server understands the conventions. But both techniques also mean that you're not using REST and the HTTP verb set; you're simply using HTTP as a transport with your own protocol layered on top. And that means that the network is less able to do clever things to optimize your traffic.

But a browser talking HTML over HTTP is also lacking in another important area. It isn't running any application functionality locally. Remember our Atom-based blogging client? It had smarts, and so it knew how to display an article for editing. Browsers don't have that (without a whole lot of support in terms of bolted on Javascript or Flash). So, as a result, people using RESTful ideas to talk to browsers have to put the smarts back on the server. They invent new URLs which (for example) return a resource, but return it all wrapped up in the HTML needed to display it as a form for browser-based editing. This is how Rails does it. Using the Rails conventions, if I want to fetch an article for viewing on a browser, I can use

  GET   /article/1

If, instead, I want to allow the user to edit the resource, I issue

  GET  /article/1;edit

The application then sends down a form pre-populated with the data from that article. When I hit submit, the form contents are sent back to the server, along with a hidden field that tells the server to pretend that the HTTP POST request is actually a RESTful PUT request—the result is that the resource is updated on the server.

The ;xxx notation looks like it was tacked on. I think that in this case, looks aren't deceiving—these modifiers really were something of an afterthought, added when it became apparent that a pure Atom-like protocol actually wouldn't do everything needed in a real-world browser-based web application. That's because the client of an Atom server is assumed to be more that just a display device. An Atom client would not say "fetch me this resource and (oh, by the way) send it down as a form so that I can edit it." The client would just say GET, do what it had to do, then issue a PUT.

So, in pure REST, the client is a peer (or maybe even in charge) of the interchange. But when talking to a dumb browser, the client is no better than an intermediary between the human and the application. That's why it was necessary to add things such as ;edit.

Client-independent Applications

One of the other benefits of designing your applications against the limited verbs offered by HTTP REST is that you can (in theory) support many different clients with the same application code. You could write blogging server that (for example) looked at the HTTP Accepts header when deciding what to send back to a client. The response to a GET request to /articles could be a nicely formatted HTML page if the client wants an HTML response, an XML payload if the client asks for XML, or an Atom collection if the client asks for application/atomcoll+xml. In Rails terms, this is handled by the respond_to stanza that is duplicated in each controller action of a RESTful server.

But the benefits run deeper than simply being able to serve up different styles of response.

The HTTP verb set maps (with a little URL tweaking) pretty nicely onto the database verb set. POST, GET, PUT, and DELETE get mapped to the database CRUD verbs: Create, Read, Update, and Delete.

So, in theory at least, you should be able to write your application code as a fairly thin veneer over a set of resources. The application exports the verbs to manipulate the resources and the clients have fun accessing them. This is similar to the vision of Naked Objects.

However, I personally think it was optimistic to try to treat the two styles of interaction (smart and dumb clients) using the same protocol. The ugliness of the ;xxx appendages was a hint.

I think there's a lot of merit in following a CRUD-based model for interacting with your application's resources. I'm not convinced all the hassle of bending dumb browser interactions into a REST-based set of verbs, and then extending those verbs to make it work, is worth the extra hassle. To put it another way, I believe the discipline and conventions imposed by thinking of your application interface as a RESTful one are worthwhile. They lead to cleaner and easier-to-understand designs. The hoops you have to jump through to implement it with a browser as a client seem to suggest that a slavish adherence to the protocol might be trying to push it too far.

An Alternative Model

Does that mean I'm down on the RESTful, CRUD based approach to application development? Not at all. For some categories of application, I think it's a great way of structuring your code. But REST isn't designed for talking to people. So let's accept that fact when creating applications. And, while we're at it, let's take advantage of the fact that HTTP is such a flexible transport. Rather than trying to design one monolithic application than has both the CRUD functionality and the smarts to be able to talk HTML to end users, why not split it into two smaller and simpler applications?

Put the main application logic into a RESTful server. This is where all the CRUD-style access to resources takes place.

Then, write a second proxy server. This is an HTTP filter, sitting between dumb browsers and your core resources. When browser-based users need to interact with your resources, they actually connect to this proxy. It then talks REST to your main server, and interprets the RESTful responses back into a form that's useful on the browser. And this filter doesn't have to be a dumb presentation layer—there's nothing to say that it can't handle additional functionality as well. Things like menus, user preferences, and so on could all sit here.

Where does this filter sit? That depends. Sometimes it makes sense to have it running locally on the user's own computer. In this case you're back in a situation like the one we looked at with the blogging software. Sometimes it makes sense to have the proxy running on the network. In this case you gain some interesting architectural flexibility—there's no rule that says you must colocate your REST and presentation servers. Potentially this can lead to better response times, lower network loads, and increased security.

With this approach, we can lose all the hacks we need to make to our URLs and POST data to make them act RESTfully. And we can lose all the ugly respond_to stanzas in our Rails server code. We might even be able to get better reuse, sharing both resource servers and presentation servers between applications.

Let's call it RADAR. We have a RESTful Application talking to Dumb-Ass Recipients.

Does it work? I don't know. You tell me.

Comments

[linoj]

why not just article/1.form for the edit form (versus 1.html for show)?

[Oleg Andreev]

Hi, Dave!

Thanks for such a good picture of RADAR.
I have a better vision. With not so dumb-ass clients, as browsers happen to be right now.

Here I replace Ajax.Request with ActiveResource on JS:
http://novemberain.com/2007/4/20/js-goes-rest

Don't mind the cyrillics; see the code and (especially) redesigned picture.

It would be great if you drop a line about the idea.

[Fabio Papa]

It seems to me that using REST is worth it (to the programmer) just solely on the basis of the url helpers (:get => post_url(123) instead of :action => :show, :controller => :posts, :id => 123), and the resource nesting, even if you don't care about RESTful clients, etc.

And another thing . . . :) Browsers aren't so dumb. In fact, to use your example, Dave (who's books and views I adore), the atom client is dumber. The atom client can only handle pure xml resources. We're sending browsers all sorts of html intermingled with resources. I'm sure somebody could write an extension for stupid firefox in about fifteen minutes to fetch, edit and submit an xml resource. I'd like to see smarty-pants atom handle the css box-model. No, if anything is of below-average intelligence in this layer cake we call the web, it is html. Yes, browsers need to start implementing all of the REST verbs, but that's not the problem, we can fake it like we have been. The problem is that we (the programmers) aren't telling these innocent, hard-working browsers that we're sending them a resource intermixed in all that presentation code. We need a standard that allows us to tell the browser, "this is a resource that can be edited." Then the browser could easily create a form on the spot for the resource without GETting any evil semi-coloned URIs (without requesting anything at all, for that matter). Am I wrong?

[Dave Thomas]

Fabio:

Don't confuse the implementation with the architecture :)

The helpers are easily written independent of the underlying transport (RESTful or not).

And the fact that Atom is simple is actually the point. If browsers could support that kind of protocol, but provide a rich user experience, then we'd be in a lot happier place. But the reality is that even the cleverest client code is only going to provide an anodyne user interface if it relies solely of data-only payloads from the server. This kind of technology is tried every 8-10 years, and it never really works (so far).

So all I'm advocating is a separation of concerns; we have the application logic in a RESTful server, and presentation logic elsewhere, either on another server or on the client (where it's likely to require more than simple forms-bqased-on-data style of interface).

[Fabio Papa]

Thanks for your response, Dave.

You are bang-on about the helpers being independent from REST. Thanks.

As for the rest (no pun intended), I guess I am just idealistic: having a beautiful, RESTful app with a non-REST interface for browsers Just Smells Bad(tm). No more pesky semicolons and repetitive respond-to stanzas, but isn't there going to be repitition amongst your (ActiveRecord) models in the RESTful app and your (ActiveResource) models in your "browser-only" app (validations, associations, etc.)?

Data-only payloads??? No way! We've come so far with css and un-obtrusive javascript and, most of all, xhtml. That's not at all what I'm suggesting. All I proposed, rather poorly apparently, was to _extend_ xhtml to allow us programmers to tell the browsers that amongst all that presentation mark-up there is a resource (or resources). Then the browser could choose to display that resource in a view-only way or an editable way based on the user clicking an edit button, or something. Sure you could do that with javascript, but then you'd have to deal with users that don't have javascript or have it disabled. An addition to xhtml would mean that it would (eventually) work on all browsers. A fielding-esque dream, maybe.

To put it another way: when a browser requests a universal RESOURCE locator (or indicator) it gets back the resource, plus a whole bunch of other information on how to frame and display that resource in a way that human eyes can grasp it. The problem is that no distinction is made (indeed none can be made) between the the framing and displaying of the resource, and the resource itself. There is no separation of the human-readable and the machine-readable. I believe that extending our beloved extensible mark-up language to enable it to make such a distinction is the best way, long-term and independent of which framework you choose to use, to allow us to have truly RESTful apps.

Thanks for igniting this, Dave.

[bryanl]

Last night at our B'More on Rails[1] meeting, I brought this article up. This idea make lots of sense to me, and now you have inspired us to bake up a proof of concept.

Thanks :)

[1] http://ruby.meetup.com/85/

[Breton]

"But, what if the app didn't define "brief" and that's what I needed? I'd have to make 4 GETs, right? One for each part {title, author, abstract, conclusion}.

Is it better to do the REST thing and make 4 GETs, or is it better to build a custom RPC? I ask that in all sincerity, for it seems to me that most applications will need, over time, the flexibility to GET custom sets of items or attributes. And, if I have to build that flexibility, when do I get to use REST? "

Then you use the query syntax documents/5?title=show&author=show&abstract=show&conclusion=show and you build your application to respond to that.

People mix up REST with the concept of beautiful URL's, when in fact, REST is just adhering to the semantics of the http spec, nothing more. It's the idea that RPC is a similar abuse of semantics as tables for design. http defines the query syntax, for querying collections of documents using named parameters.

the http spec defines no semantics for the semicolon, but still reserves it for application authors. sir Tim Berners Lee wanted to use semicolons for representing dimensional spaces- such as maps, which are not suited to representation using a heirarchical notation. Different views of the same resource are very similar to different views of a 2d map. A map is a single thing, and TBL is using semicolons to specify different views for that same map resource.

http://www.w3.org/DesignIssues/Axioms.html#matrix

a year and a half, or some time ago, I was reading the http spec, discovering these things, and posting to a mailing list about REST, and getting rather excited about my discoveries. DHH, just by chance, happened to be reading the same mailing list. I got excited about semicolons directly at him in a campfire session. I explained everything I just did to you. He was intrigued, when I pointed out that ; and ? have different semantics. Next thing you know, semicolons are in rest on rails.

Which, may go some way to explaining why he says it was a "Neat idea, but not worth it" when removing semicolons. I hyped him up on the semicolon. It's rather dissappointing that it causes so many problems with proxies and safari- It means that safari is in violation of the http spec, which specifically tells clients to NOT screw up with semicolons.

[breton]

here's a better link to sir Tim Berners-Lee's explanation of Matrix URI's (which use the semicolon)

http://www.w3.org/DesignIssues/MatrixURIs.html

[nitendra]

Hi,
Its Nitendra Mahajan here, i would like to know how the actual diff will happen while working of application server and web server independently..please let me know diff between app and web server architecture

with regards
nitendra.

[jordan shoes]

I didn't know we were now moving away from RESTful URLs. In any case you could make them discoverable via a central page that listed all the root resources that the server provides, and so on.

[ugg boots]

Good topic. thanks.

[ryw]

Hi Dave, thanks for the article.

We're getting ready to rebuild our apps architecture (see http://www.rywalker.com/chaos-2010 for more details), we considered RADAR-like architecture a few years ago, but decided to no decouple the rails models from the views of the apps, it just didn't seem like the extra benefit was worth the pain of having to duplicate code potentially in places...

What's your take on RADAR architecture in 2010 - are you more a fan of the idea today?

We're considering using something like Sinatra/Riak @ the webservice layer by the way, rather than two rails apps.

[Bothb]

This isn't exactly RADAR, but it does decouple the model from the view: http://documentcloud.github.com/backbone/

It should let us write modular views in js that can be used in multiple web apps. For example, an "employee selector" for choosing an employee name, and an "employee checkbox list" for choosing multiple employees. Write once, use in all our apps, (or at least that's the hope.)