A First Erlang Program

One of the joys of playing at being a publisher is that I get to mess around with the technology in books as those books are getting written. Lately I've been having a blast with Joe Armstrong's new Erlang Book. At some point I'll blog about the really neat way the Erlang database unifies the set-based SQL query language and list comprehensions (it's obvious when you think about it, but it blew me away when I first read it). But I just wanted to start off with some simple stuff.

One of my standard Ruby examples is a program that fetches our book sales ranks from Amazon using their REST interface. I thought I'd try the exercise again in Erlang. In this first part we'll write a simple Erlang function that uses the Amazon Web Services API to fetch the data on a book (identified by its ISBN). This data is returned in XML, so we'll then use some simple XPath to extract the title and the sales rank.

My Erlang module is called ranks, and I store it in the file ranks.erl. Because I'm just playing for now, the interface to this module is simple: I'll define a function called fetch_title_and_rank. This returns a {title, rank} tuple. So, I started off with something like this:

  -module(ranks).
  -export([fetch_title_and_rank/1]).

  fetch_title_and_rank(ISBN) ->
    { "Agile Web Development with Rails", 1234 }.

The -module directive simply names the module. The next line tells Erlang that this module exports a function called fetch_title_and_rank. This method takes one parameter. (This idea of specifying the number of parameters seems strange until you realize that in Erlang the function signature is the function name and the parameter count. It is valid to export two functions with the same name if they take a different number of arguments—as far as Erlang is concerned they are different functions.)

Next we define the fetch_title_and_rank function. It takes a single parameter, the book's ISBN. In Erlang, variable names (including parameters) start with uppercase letters. This takes a little getting used to. Just to make sure thinks are wired up correctly, let's try running this code. In a command shell, I navigate to the directory containing ranks.erl and started the interactive Erlang shell. Once inside it, I compiled my module (using c(ranks).) and then invoke the fetch_title_and_rank method.

  dave[~/tmp 11:51:09] erl
  Erlang (BEAM) emulator version 5.5.4 [source] ...

  Eshell V5.5.4  (abort with ^G)
  1> c(ranks).
  ./ranks.erl:11: Warning: variable 'ISBN' is unused
  {ok,ranks}
  2> ranks:fetch_title_and_rank("0977616630").
  {"Agile Web Development with Rails",1234}

Let's start wiring this up to Amazon.

To fetch information from Amazon using REST, you need to issue an ItemLookup operation, specifying what you want Amazon to return. In our case, we want the sales rank information, plus something called the small response group. The latter includes the book's title. The URL you use to get this looks like this (except all on one line, and with no spaces):

http://webservices.amazon.com/onca/xml?
Service=AWSECommerceService
&SubscriptionId=<your ID goes here>
&Operation=ItemLookup
&ResponseGroup=SalesRank,Small
&ItemId=0977616630

The ItemID parameter is the ISBN of our book. Let's write an Erlang function that returns the URL to fetch the information for an ISBN.

  -define(BASE_URL,
        "http://webservices.amazon.com/onca/xml?" ++
        "Service=AWSECommerceService" ++
        "&SubscriptionId=<" ++
            "&Operation=ItemLookup" ++
            "&ResponseGroup=SalesRank,Small" ++
            "&ItemId=").

  amazon_url_for(ISBN) ->
    ?BASE_URL ++ ISBN.

This code defines an Erlang macro called BASE_URL which holds the static part of the URL. The function amazon_url_for builds the full URL by appending the ISBN to this. Note that both the macro and the function use ++, the operator that concatenates strings.

We now need to find a way of sending this request to Amazon and fetching back the XML response. Here we bump into one of Erlang's current problems—it can be hard to browse the library documentation. My solution is to download the documentation onto my local box and search it there. I tend to use both the HTML and the man pages. Figuring I wanted something HTTP related, I tried the following:

  dave[Downloads/lib 12:04:07] ls **/*http*
  inets-4.7.11/doc/html/http.html        
  inets-4.7.11/doc/html/http_base_64.html 
  inets-4.7.11/doc/html/http_client.html  
  inets-4.7.11/doc/html/http_server.html 
  inets-4.7.11/doc/html/httpd.html
  inets-4.7.11/doc/html/httpd_conf.html
  inets-4.7.11/doc/html/httpd_socket.html
  inets-4.7.11/doc/html/httpd_util.html

Opening up http.html revealed the http:request method.

request(Url) -> {ok, Result} | {error, Reason}

This says that we give the request method a URL string. It returns a tuple. If the first element of the tuple is the atom ok, then the second element is the result for the request. If the first element is error, the second element is the reason it failed. This technique of returning both status and data as a tuple is idiomatic in Erlang, and the language makes it easy to handle the different cases.

Let's look a little deeper into the successful case. The Result element is actually itself a tuple containing a status, the response headers, and the response body.

So let's take all this and develop our fetch_title_and_rank method a little more.

  fetch_title_and_rank(ISBN) ->
    URL = amazon_url_for(ISBN),
    { ok, {Status, Headers, Body }} = http:request(URL),
    Body.

We call our amazon_url_for method to create the URL to fetch the data, then invoke the http:request library method to fetch the result. Let's look at that second line in more detail.

The equals sign in Erlang looks like an assignment statement, but looks are deceptive. In Erlang, the equals sign is actually a pattern matching operation—an assertion that two things are equal. For example, it is perfectly valid in Erlang to say

  6> 1 = 1.
  1
  7> 9 * 7 = 60 + 3.
  63

So what happens when variables get involved? This is where it gets interesting. When I say

  A = 1.

I'm not assigning 1 to the variable A. Instead, I'm matching A against 1 (just like 9*7 = 60+3 asserts that the results of the two expressions are the same). So, does A match 1? That depends. If this is the first appearance of A on the left hand side of an equals sign, then A is said to be unbound. In this condition, Erlang says to itself “I can make this statement true if I give A the value 1,” which is does. From this point forward, A has the value 1 (in the current scope). If we say A = 1. again in the same scope in our Erlang program, A is no longer unbound, but this pattern match succeeds, because A = 1. is a simple assertion of truth. But what happens if we say A=2.?

  10> A = 1.
  1
  11> A = 1.
  1
  12> A = 2.

  =ERROR REPORT==== 16-Apr-2007::12:32:36 ===
  Error in process <0.55.0> with exit value: {{badmatch,2},[{erl_eval,expr,3}]}

  ** exited: {{badmatch,2},[{erl_eval,expr,3}]} **

We get an error: Erlang was unable to match the values on the left and right hand sides of the equals, so it raised a badmatch error. (Erlang error reporting is, at best, obscure.)

So, back to our HTTP request. I wrote

  { ok, {Status, Headers, Body }} = http:request(URL)

The left hand side matches a tuple. The first element of the tuple is the atom ok. The second element is another tuple. Thus the entire expression only succeeds if http:request returns a two element tuple with ok as the first element and a three element tuple as the second element. If the match succeeds, then the variables Status, Headers, and Body are set to the three elements of that second tuple.

(An aside for Ruby folk: you can do something similar using an obscure Ruby syntax. The statement

  rc, (status, headers, body) = [ 200, [ "status", "headers", "body" ] ]

will leave headers containing the string "headers".)

In our Erlang example, what happens if http:request returns an error? In this case, the first element of the returned tuple will be the atom error. This won't match the left hand side of our expression (because the atom error is not the same as the atom ok) and we'll get a badmatch error. We won't bother to handle this here—it's someone else's problem.

Let's compile our program.

  1> c(ranks).
  ./ranks.erl:13: Warning: variable 'Headers' is unused
  ./ranks.erl:13: Warning: variable 'Status' is unused
  {ok,ranks}

Erlang warns us that we have some unused variables. Do we care? To a point. I personally like my compilations to be clean, so let's fix this. If you prefix a variable's name with an underscore, it tells the Erlang compiler that you don't intend to use that variable. As a degenerate case, you can use just a lone underscore, which means “some anonymous variable.” So, we can fix our warnings by writing either

  { ok, {_Status, _Headers, Body }} = http:request(URL),

or

  { ok, {_, _, Body }} = http:request(URL),

I mildly prefer the first form, as it helps me remember what those elements in the tuple are when I reread the program later.

Now we can recompile and run our code:

  1> c(ranks).
  {ok,ranks}
  2> ranks:fetch_title_and_rank("0977616630").
  "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ItemLookupResponse
  xmlns=\"http://webservices.amazon.com/AWSECommerceService/2005-10-05\">
  <OperationRequest><HTTPHeaders><Header ...
     ... <SalesRank>1098</SalesRank>
     ... <Title>Agile Web Development with Rails (Pragmatic
          Programmers)</Title>...
     ...

Cool! We get a boatload of XML back. (If you're running this at home, you'll need to substitute your own AWS subscription ID into the request string to make this work.) Now we need to extract out the title and the sales rank. We could to this using string operations, but wouldn't it be more fun to use XPath? Another quick filesystem scan reveals the xmerl library, which both generates and parses XML. It also supports XPath queries. To use this, we first scan the Amazon response. This constructs an Erlang data structure representing the original XML. We then call the xmerl_xpath library to find the elements in this data structure matching our query.

  -include_lib("xmerl/include/xmerl.hrl").

  fetch_title_and_rank(ISBN) ->
    URL = amazon_url_for(ISBN),
    { ok, {_Status, _Headers, Body }} = http:request(URL),
    { Xml, _Rest } = xmerl_scan:string(Body),
    [ #xmlText{value=Rank} ]  = xmerl_xpath:string("//SalesRank/text()", Xml),
    [ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml),
    { Title, Rank }.

The -include line loads up the set of Erlang record definitions that xmerl uses to represent elements in the parsed XML. This isn't strictly necessary, but doing so makes the XPath stuff a little cleaner.

The line

    { Xml, _Rest } = xmerl_scan:string(Body),

parses the XML response, storing the internal Erlang representation in the variable Xml.

Then we have the line

    [ #xmlText{value=Rank} ]  = xmerl_xpath:string("//SalesRank/text()", Xml),

The notation #xmlText identifies an Erlang record. A record is basically a tuple where each element is given a name. In this case the tuple (defined in that .hrl file we included) is called xmlText. It represents a text node in the XML. The xmlText record has a field called value. We use pattern matching to assign whatever is in that field to the variable Rank. But there's one more twist. Did you notice that we wrote square brackets around the left hand side of the pattern match? Thats because XPath queries return an array of results. By writing the left hand side the way we did, we force Erlang to check that an array containing just one element was returned, and then to check that this element was a record of type xmlText. That's pretty cool stuff—pattern matching extends down deep into the bowels of Erlang.

The last line of the method simply returns a tuple containing the title and the sales rank.

We can call it from the command line:

  1> c(ranks).
  {ok,ranks}
  2> ranks:fetch_title_and_rank("0977616630").
  {"Agile Web Development with Rails (Pragmatic Programmers)","1098"}

That's it for now. Next we'll look at fetching the data for multiple books at once. In the meantime, here's the full listing of our ranks.erl file.

  -module(ranks).
  -export([fetch_title_and_rank/1]).
  -include_lib("xmerl/include/xmerl.hrl").

  -define(BASE_URL, 
        "http://webservices.amazon.com/onca/xml?" ++
        "Service=AWSECommerceService" ++
        "&SubscriptionId=<your id>" ++
        "&Operation=ItemLookup" ++
        "&ResponseGroup=SalesRank,Small" ++
        "&ItemId=").


  fetch_title_and_rank(ISBN) ->
    URL = amazon_url_for(ISBN),
    { ok, {_Status, _Headers, Body }} = http:request(URL),
    { Xml, _Rest } = xmerl_scan:string(Body),
    [ #xmlText{value=Rank} ]  = xmerl_xpath:string("//SalesRank/text()", Xml),
    [ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml),
    { Title, Rank }.

  amazon_url_for(ISBN) ->
    ?BASE_URL ++ ISBN.

Comments

[Ola Bini]

Nice writeup, Dave.

I've always been fond of Erlang, and it's great that you're publishing this book. Looking forward to reading it.

Just a nitpick; regard the http:request call, you say that = is a pattern matching operator, but isn't it more true to say it's doing unification in the Prolog manner, than pattern matching?

Cheers

[Dave Thomas]

Ola:

Yes, I believe the two are similar, although Prolog takes it a lot further in terms of backtracking and the like.

I use the phrase "pattern matching" because that's what Joe uses :)

[JEG2]

At one point the article refers to ranks.rb, but I believe you meant ranks.erl. You also miscapitalize XPath as XPAth in one instance.

Good example though. Thanks for sharing!

Dave says: both fixed--thanks.

[Giulio Piancastelli]

There still an instance of ranks.rb in the last line before the listing of the whole Erlang program.

(Dave says: thanks--fixed)

I found quite thrilling that some behavior of Erlang is similar to Prolog. Less things to grasp for folks coming from a logic programming background like myself.

[Shashy]

Great article. I was working with CouchDb (http://couchdb.com) and a Ruby client yesterday. I'll try CouchDb and an Erlang client today for a full Erlang experience.

[lninyo]

just got my beta pdf. This a great short tutorial on Erlang. Thanks for publishing the Erlang book!

[Ross]

Dave - the sample code for the amazon_url_for function has a real SubscriptionId rather than the your_id placeholder. Did you mean to do this?

Dave says: sigh....

[Tobbe]

Nice writeup!
Just a few comments:

1. You don't need the '++' in the BASE_ULR macro. The compiler will concatenate successive strings automatically.

2. Regarding Unification, Erlang reguire that the right-hand-side of the pattern-match operation is fully bound. With Unification you may have Unbound variables on both sides, resulting in interesting effects... :-)

Cheers, Tobbe

[g]

The photo is Felix Klein, yes? Ho ho.

[Tom]

#xmlText{value=Rank} looks strange to me. Why is the unbound variable on the right?

[Tobbe]

Perhaps a bit confusing, but the record syntax is just some sugar for Erlang tuples, so this:

[ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml),

actually mean something like this:

[{xmlText,_,Rank,_}] = xmerl_xpath:string("//SalesRank/text()", Xml),

I say something since I don't remember the correct number of elements in the xmlText tuple and which element that corresponds to the Rank value.
The record syntax helps you to focus on the important part though.

[acechase]

It's really great to hear a mainstream voice in the software industry touting the virtues of Erlang. I'm interested to see how this plays out as, just six months ago, I was discussing Erlang with one of the "visionaries of our field" and his response to the benefits of Erlang was that it is too foreign for the average developer to take hold of. I'm hoping that proves untrue, and I think a good introductory book is a good start towards that end.

One style note in regards to the example: Where you're matching only on the "ok" response from http:request(URL) and letting an "error" response generate a bad match, I think it is clearer to match on the error and then return it unmodified. This approach still fits with the fail early philosophy of Erlang, but it provides a more direct indication of what the error was, rather than the more opaque matching error.

[alex]

Ola, in typical use of terminology, unification is working in both directions, whereas pattern matching only tries to match lhs with rhs.

So unification would bind X to 1 and Y to 2 for {X, 1} = {2, Y}, whereas (pattern) matching would complain about an unbound Y on the rhs.

So that's why it is actually correct to say that Erlang's = is a pattern matching operator:

1> {X, 1} = {2, Y}.
** 1: variable 'Y' is unbound **

[Ivan]

As acechase wrote, this language looks very strange. For me as I'm an average developer has very ugly syntax and I think that mainstream stays untouched.

[john]

great article, it might have been better to use a free service as opposed to amazon so its easier for people to follow. I dont want to signup for an amazon key to test this code out. :(

[Rafael de F. Ferreira]

Excellent tutorial. I took the liberty of translating the code to Scala, I hope you don't mind. My translation is here: http://blogs.sun.com/rafaelferreira/entry/pragmatic_scala .

[George]

Erlang has beautiful syntax, to me at least.

[cbmeeks]

Very interesting read!

I am trying to learn Erlang at the moment and it's hard to find practical examples.

I hope to create an Amazon AWS library (S3, EC2, etc) in Erlang as an exercise in learning Erlang.

cbmeeks
http://codershangout.com

[wasiq]

thanks for a nice tutorial

i get the following error when try to execute

3> ranks:fetch_title_and_rank("0977616630").
** exception exit: {noproc,{gen_server,call,
[httpc_manager,
{request,{request,undefined,,0,http, {"webservices.amazon.com",80}, "/onca/xml", "?Service=AWSECommerceService&SubscriptionId=XXXXXXXXXXXXXXXXX&Operation=ItemLookup&ResponseGroup=SalesRank,Small&ItemId=0977616630",
get,
{http_request_h,undefined,"keep-alive",undefined,undefined,
undefined,undefined,undefined,undefined,undefined,...},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,[],undefined,false},
"http://webservices.amazon.com/onca/xml?Service=AWSECommerceService&SubscriptionId=1QKKA9Z4JR3MCE2DKZ02&Operation=ItemLookup&ResponseGroup=SalesRank,Small&ItemId=0977616630",
[],none,[]}},
infinity]}}
in function gen_server:call/3
in call from http:handle_request/6
in call from ranks:fetch_title_and_rank/1

[Tom]

You are missing an initial inets:start(). Just enter that once in your Erlang console before calling ranks:fetch....