There was a time when no one knew how to write JavaScript, but those times are long since gone. People know how to write JavaScript now (some), and that's good!!1ELEVEN! Unfortunately, there's still a large portion of people who don't know how to write applications¹.

As a result of this, you often end up with applications that do too much, or applications that do too little. But the worst problem of all is when you end up with applications that you can only use through some human interface of sorts, and can't easily manipulate the stuff you're interested in with different things. Mind you, programmatic extensions matter a lot!

Thus, in this blog post I'll try to provide a few hints on how to achieve small, composable and extensible applications. Stick with me!

Don't start with an Application, start with the API. This might seem weird, but it actually makes a lot of sense, because the API defines how other computer things can interact with the data your application manipulates. But hey, why should we leave that as “other computer things”? Let me rephrase:

The API defines how your application (and other computer things) can interact with the data you're interested in.

My usual design approach is to examine the inputs and outputs of the application, and then design an API that handles such data. Later I'll stack the user experience and the “glue” code on top of such API to form an actual application.

This fits nicely with the "One Module Does One Thing" approach because then the API defines which kind of data you're dealing with, and all the meaningful transformations you can apply to such data. Transformations in turn do one thing, and might be grouped logically. Modules are really just a logical group of computations, that it happens to be a file in many languages is just an accident.

For example, let's say we want to write an application to display Tweets to the user. First and foremost, we examine the inputs and outputs, and draft some types to encode the data in our application (I'm using a type notation inspired by Haskell, but this should be nonetheless pretty straightforward):

-- | A TwitterTweet is something Twitter gives us (the input)
-- (note that this is a stripped down version)
type TwitterTweet
  favorited     :: Boolean
  retweeted     :: Boolean
  retweet_count :: Number

  created_at :: String  -- ^ Serialised Date
  source     :: String
  user       :: User
  id_str     :: String

  in_reply_to_user_id_str   :: String
  in_reply_to_status_id_str :: String
  in_reply_to_screen_name   :: String

  entities   :: { "urls"          :: [Entity]
                , "hashtags"      :: [Entity]
                , "user_mentions" :: [Entity]
                }

  text :: String    

So, the data Twitter gives us is quite a mess, and it'd be really difficult to manipulate that kind of data in our application. We can do better, so let's define a better type to encode a Tweet:

type User
  name :: String -- ^ The users' screen name
  id   :: String -- ^ Twitter's user ID

type Entity
  url          :: String
  expandedUrl  :: String
  displayUrl   :: String
  indices      :: (Number, Number)


type Text
  plain    :: String
  entities :: { "urls"          :: [Entity]
              , "hashtags"      :: [Entity]
              , "user_mentions" :: [Entity]
              }

type Tweet
  id            :: String -- ^ Unique identifier for this tweet
  user          :: User   -- ^ The user that tweeted this
  inReplyTo     :: Tweet
  source        :: String -- ^ Which application was used to compose this
  date          :: Date   -- ^ When this tweet was crafted
  favourited    :: Boolean
  retweeted     :: Boolean
  retweetCount  :: Number
  text          :: Text

So, now User and Text are separate types, this is because they make sense outside of the context of a Tweet and we might want to manipulate them separately. There's no reason to provide a complected type to a function that only needs to know the name of a user, for example.

Once we're done with the types our application needs to manipulate, we can draft an API that provides the primitives to manipulate these types, given the operations we'll be applying to them and the output.

-- * Type conversions

-- | We need to convert from Twitter format to ours
normaliseTweet :: TwitterTweet -> Tweet

-- | Convert Twitter Date serialisation to actual DateTime
parseDate :: String -> Date

-- | Extract the User that composed the tweet
twittedBy :: TwitterText -> User

-- | Extract reply information
repliedToUser :: TwitterText -> User
repliedToTweet :: TwitterText -> Tweet

-- | Extract the Text
textFor :: TwitterText -> Text


-- * Display transformations
-- (These are application-specific because they only make sense in the
-- context of user-facing HTML pages)

-- | We want to display a Tweet as HTML
renderTweet :: Tweet -> HTML

-- | We want to display a Text as HTML
textToHTML :: Text -> HTML

-- | We want to know the relative time since the tweet
fromNow :: Date -> String

-- | We want to display a link to a User
linkToUser :: User -> HTML

-- | We also want to display a link to a Tweet
linkToTweet :: Tweet -> HTML

If there's one hint I can provide when doing the initial API design, it would be:

Extract the most basic, orthogonal concepts, and provide primitives to encode them.

You can always add combinators on top of those minimal and simple primitives to let them do more stuff. Working with reeeeally small set of primitives and a lot of combinators means you get to write simple code that actually scales! But then, picking the right primitives can be really hard at times, so you need to have a good deal of knowledge about the domain you're trying to encode in your API.

Compositionality is a big thing. Compositionality is what you want in a big application. Compositionality is what will save your bacon when you have to actually maintain all the shit you've written. This is one of the reasons we don't put them in the first API draft, we want to get the primitives right first, and make sure they don't overlap!

Back to our Twitter example, when you retrieve data from Twitter, you usually get a List of tweets. Notice that nothing in the previous API allows you to take a list of Tweets and spits back a list of HTMLs, but it can take a single tweet and spit back a single HTML. We also have baked right into the standard library a function that takes a List of things, a function that transforms a thing A into thing B, and returns a list of things B. Well, this is enough to derive our transformation for lists of Tweets:

// Renders a list of Tweets
// renderTweetList :: [Tweet] -> [HTML]
function renderTweetList(tweets) {
  return tweets.map(renderTweet)
}

// Or we can use a better version of Map (if you know functional
// programming) 
var map = curry(2, Function.call.bind([].map))
var renderTweetList = map(renderTweet)


// -- An aside: ------------------------------------------------------

// If you don't know what `curry` is, well... A minimal explanation
// would be that functions in JavaScript actually takes a List of
// arguments. You should think about:
function add(a, b) { return a + b }

// As being actually:
function add(arguments){ return arguments[0] + arguments[1] }

// And when you're calling it as: add(1, 2) you're actually saying
// add([1, 2]).

// Currying takes a different route. Functions takes only one
// argument:
function itself(a) { return a }

// If you need to create a function that takes more than one argument,
// you use closures:
function add(a){ return function to(b) { return a + b }}

// And when you're calling it as: add(1, 2) you're actually saying
// add(1)(2).

// You can see an implementation of currying here:
// https://github.com/killdream/athena/blob/master/src/higher-order.ls#L56-L81

But this doesn't display anything in the screen yet, mostly because that's not the job of renderTweetList — it already does everything it needs to do. A thing that displays tweets on the screen should be something that takes an HTML, a container and adds that HTML to the container:

// addTo :: HTML, HTML -> HTML
function addTo(container, html) {
  $(container).append(html)
  return container
}

Now we can derive a simple function that will take a list of HTML things, and add them to a container:

// addAllTo :: HTML, [HTML] -> HTML
function addAllTo(container, htmls) {
  htmls.map(function(html){ addTo(container, html) })
  return container
}

// Or, we can go use our Curry friend and make it better-er
var addTo = curry(2, addTo)
var addAllTo = curry(2, function(container, htmls) {
  htmls.map(addTo(container))
  return container
})

I can't stress this point enough! If you want people to actually use your API in a meaningful way, you must work with structured data. Please don't “but strings are easy!” me. Strings might be easy, but we don't want easy when designing an API, we want simple ². Simple stuff sometimes means you get to write more, but also means that you get something that's more meaningful overall, that's extensible and that composes well with other things without randomly breaking for no good reason. When you pass Strings around for other people to parse you lose all the guarantees that they'll agree with each other on the structure your API (and external APIs) expect.

In the case of our API example, it would mean passing around Tweet types, rather than the HTML representation of them!. All of the central points of the API should accept one of our types (Tweet, User, Text), not arbitrary HTML or plain text strings, because then everyone can encode that slightly different.

“So, what if I want to send it over to someone else over the wire? Wouldn't it be better if I just use the representation that the other side will use to display the thing?”

Well, think about the following scenario: You have your application sending tweets to a logger that will display them. You want to “Keep It Easy”, and so decides it's a good idea to just send the way you want tweets to be displayed on the other side, so people don't need to write anything besides console.log.

A few weeks later someone comes up to those guys and say, “Hey, we're going to log only stuff that got retweeted at least 50 times.” The other-side guys quickly hack together a regular expression that looks for /(\d+) retweet/ and call it a day.

Some days later you decide retweet is too long and it's taking valuable space on the screen of your application now that you're porting it over to mobile devices. Then you decide to shorten that to rt. Guess who just got all their system's screwed?

If you pass over structured data, then it's simple. They wouldn't even need to touch their main system if they didn't want to, just put on a proxy in front of the service with this code:

next(tweets.filter(function(tweet){ tweet.retweetCount > 50 }))

If you need to communicate data with other services, you should just encode a structured representation using the best serialisation format for the job. JSON everywhere won't cut it, as won't XML. JSON is a generic data serialisation format as plain text, and XML is a document serialisation format as plain text. They're cool if they fit your data, and you don't care about the additional bandwidth/encoding time. Otherwise there are other stuff like Protocol Buffers to take a look at.

Warning

Please please please please please please! For the love of God, don't use XML to encode general data. XML is a document serialisation format, it's something you use to serialise TREE STRUCTURES. Mind you, Lists are not the best case for XML, dictionaries aren't either. Use a general data serialisation format for everything that isn't a tree.

You've gone through great lengths to create a minimal and polished API, now it's finally time to use it by writing your Application on top of it. Why, you might ask? Well, because Applications are the human-facing interface to your data. Applications talk to humans, and only ever to humans, because they choose a format that is difficult or impossible to use to talk to other application. APIs on the other hand talk only to applications using structured data, which is not the best format to present to the user for most types of data.

Say we want an application that will get the timeline of a given user and display it on a webpage. This can be encoded in a simple way using our API:

var dataP = twitter.statuses('notSorella')

dataP.then(function(data) {
  var tweets = data.map(normaliseTweet)
  addAllTo(twitterContainer, renderTweetList(tweets))
}).done()

If we then are tasked with displaying the same set of Tweets on the command line, we can just use the primitives, which are not HTML-specific!

var dataP = twitter.statuses('notSorella')

function renderTweet(tweet) { 
  return '@' + tweet.user.name + ': ' + tweet.text.plain
}

dataP.then(function(data) {
  var tweets = data.map(normaliseTweet)
  tweets.map(compose(print, renderTweet))
}).done()

This is it, me dears. This is the key to write large applications, this is the key to write extensible applications, and the key to write easily maintainable applications: compositionality.

You start with an idea, extract the key components of that idea (primitives), provide combinators to compose ideas together, and only then provide additional transformations for the user-facing interface.