Since most of my work is in a overtly dynamic language like JavaScript (granted I use LiveScript now, but same semantics), parsing the source and trying to make sense of it isn't always easy. Say you have the following:

/// Given an ID, sanitises it, otherwise generates an unique ID.
var id = function() { var guid = 0

                      return function(name) {
                               return name?    sanitise(name)
                               :      /* _ */  (++guid).toString(16) }

                      // Sanitises an ID
                      function sanitise(name) {
                        return name.replace(/\s/, '-')
                                   .replace(/\W/, '')
                                   .toLowerCase() }}()

Even though this use of the "Module Pattern" is way too common, it's not obvious right away that the documentation refers to the inner function rather than the immediately invoked function expression that surrounds it. Or say you have something like this:

/// Takes the first `n` items of a list.
var take = curry(function(n, xs) {
                   return xs.slice(0, n) })

Which is also fairly common (not just the currying, but decorating a function in general), not just in JavaScript, but also in Python. This doesn't make the type of take immediately obvious, unless you happen to have figured out the type of curry already, or if your documentation tool evaluates a certain sub-set of JavaScript to figure out these things.

So, you have to annotate your code anyways with types, but there might be cases where your documentation tool will have some trouble figuring out the structure of your code anyways. Not to say you'd need extremely language-specific tools to extract this information. In the end I ditched this approach and decided for a generalisation of all languages I worked mostly with (which at the time were mainly JavaScript and Python), not just on the way entities are annotated, but also in the way structure itself is annotated.

The documentation format I came up with (and evolved informally through the course of this year) is heavily inspired by ReStructuredText and Markdown. The basic idea is that you have sections with varying depth levels (nested sections yay!), and each section can have a range of information belonging to it, including other sections. This way, entities are just special kinds of sections. The main difference from systems like Docco, etc. is that there are subtle annotations that make it possible to figure out the structure of the code and extract meta-data concerning a particular entity.

The whole parser fits in under 100-loc of LiveScript code. You can get a feeling of the documentation format by reading Kalamoi, Papyr°, Calliope or Boo source code. Boo is written in JavaScript, the others are all LiveScript. Adapting the parser to other languages can be done in three lines of code, but until I refactor stuff this is still a pain:

// in parser.ls
exports.luaSyntax = boo.derive(baseSyntax, { comment: '--' })

// in index.ls
var languageFor = function(extension){ ...
  case '.lua': return 'Lua'

var syntaxFor = function(language){ ...
  case: 'lua': return parser.luaSyntax

You probably guessed the parser relies on single-line comments, and yes. That's because the parser is line-based for simplicity. Basically, every line that starts with <comment-character> is treated as special meta-data for the parser, and the rest is treated as code. Lines that start with a repetition of the last <comment-character> are treated as sections, and the number of <comment-character> defines the level of that section (just as Markdown headings).

## Module foo
# Description of `foo`.

### -- Internal functions

#### Function id
# Description of `id`
id = (x) -> x

### -- Public functions

#### Function k
# Description of `k`
k = (x) -> (y) -> x

As you can probably guess, the special "kind" of section is defined by the first <word> after the comment character. A <word> here is more in the sense of Lisp, except the only thing that works as a word boundary is white-space. Sections where the kind is -- are special headers, that have no special entity meaning, but can be used to logically group things within a source code to make it easier to navigate and understand the structure.

There are only two other special constructs that can be used inside of a section to declare special meta-data about an entity. The <generic meta data>, in the form of :keys: optional value, and the <type signature>, in the Haskell-veins of :: type, although the format of the type signatures themselves is not specified — not a job for this particular language:

### Function id
# :internal, deprecated:
# The identity function.
#
# :: a -> a
id = (x) -> x

This all works well for well structured source code, and probably somewhat well for pseudo-literate code.

I said before that all the benefits of this approach derive from a standard and simple format to exchange meta-data about entities. And there isn't anything easier nowadays for this kind of structured data than JSON. It's available everywhere, easy to parse, easy to write, and it usually maps nicely to lots of language's core data structures (at least dynamic ones) since it uses Maps, Vectors, Strings and Numbers as types, rather than a document tree .

This format is devised from an overt generalisation of all entities, with basis on what users of the API might be looking for when they browse through API references. This includes ideas that are common in other documentation systems, and kind of obvious, like an entity's name, its type or a description about what it means:

type URI = String

type EntityKind = Data
                | Module
                | Function
                | Object
                | Type

type Author = { name    :: String
              ; email   :: String
              ; website :: String
              }

type Entity = { id         :: URI  -- An unique ID for the entity
              ; parent     :: URI -- The entity that contains this entity
              ; kind       :: EntityKind -- The type of the entity being represented
              ; name       :: String -- The name of this entity
              ; signatures :: [String] -- The types of the entity
              ; text       :: String -- A text describing this entity
              ; code       :: String -- The source code associated with this entity
              ; meta       :: { String -> String | Bool } -- Additional metadata
              ; language   :: String -- The language of the source code
              ; line       :: Number -- Where the source code starts
              ; end-line   :: Number -- Where the source code ends
              ; file       :: String -- The file where the entity appears
              ; copyright  :: String -- Short copyright notice (if any)
              ; repository :: String -- The canonical repository for the code
              ; authors    :: [Author] -- Who contributed to the code
              ; licence    :: String -- The terms in which the code is released
              }

type Entities = [Entity]

Note

Yes, I do realise it makes more sense to create a new data structure to store all information related to code, I'll likely do it when refactoring these tools. I'll probably make code a List as well, so it can handle multi-methods defined in different parts of the code and such.

This, however does not cover several kinds of entities, specially entities that have intrinsic relationships with others in a way that it changes their behaviour, like Classes and SuperClasses/Mixins/Traits/Delegative & Concatenative Prototypes, etc.. It could just well be encoded in the type annotation, but then you lose the ability of using these relationships to convey additional information for the user when visualising the documentation, and encoding it as an additional but non-specified meta-data means that, while other tools can easily make sense of the data, we don't have a common-ground for them to collaborate with every other tool in this environment, so that's a no-go.

As it current stands, these things would need to be devised on top of the current standard and be added for the specification, so we don't have the risk of conflicts between different notations, and tools that don't understand that "extension" can just ignore the additional meta-data.

Well, meta-data is good for applications to manipulate them, but not good for the end user, thus we are left with the problem of "how do we visualise this meta-data". Turns out, there might be many different cases where we want to do different visualisations. The goal of Papyr° is to cover one of them: finding some entity that fits the problem the user is trying to solve in a potentially large and complex API, and discovering how to use that entity. In the current form, Papyr° manages to achieve almost none of the goals outlined above, although it is a starting point.

First, helping the user find the right component to use for a certain problem involves first knowing how the user searches for information about that particular problem. In some cases it's easy to guess, for example, the user might be looking for a stable sort, and as such they'll type "stable sort" in the search box and expect to see classes, functions, procedures, methods, whatever that will do the thing they want. But perhaps the user isn't really looking for a stable sort, perhaps in their case some other sorting algorithm will do the job better or faster because they don't have the constraints they gave in the first place. In this case, the system has to guess the intentions of the user from the provided context, and suggest things that fit the context but provide different constraints, or different benefits.

This also kind of asks for a more expressive search. One where the user can define their constraints better. For example, if they know they're working with a list of Elements, they might be looking for a sorting function that is efficient for those, so they could provide a basic type, say: sort :: [Element] -> [Element], and the search engine would attempt to match sorting functions that provide an approximate type to that. Hoogle does this kind of approximate type searches. Other constraints might involve platforms, mutability, etc. Ranking the search results by favouring the ones that fit the constraints the user provided better is also an interesting addition.

Currently Papyr° does not provide automatic contextual hints for potentially useful components, but this is definitely a feature I want to work on, and that could perhaps be branched in other projects. So, for example, an analysis tool could be ran on the entities meta-data earlier on to determine some possible logical tags for each entity, which could be used by the visualisation tool for relating different entities and ranking them. Approximate type searches are something that are planned but not yet implemented, this is however something I need to give a little more thought since the type signatures are not defined, so it's not something as straight-forward as Hoogle's type search. The problem is also more with analysing the user input and determining the types rather than having different type annotations used by the entities, since you could probably determine that by some meta-data or language.

So, the second goal of Papyr° is helping the user to use a particular entity to solve their problem. This is partially done by automatically loading Markdown literate examples and displaying them alongside the source code and other details for an entity. The literate examples are, however, static. There could be some use in defining a format for interactive examples, or even examples that would be ran by a particular engine when building the documentation JSON, for the cases where fully interactive examples in the browsers aren't possible.

The only other visualisation format I plan on writing is Reedox, a minimal visualisation format for overly small APIs (one or two functions), where it doesn't make much sense to have the user load an entire new page to search for something where the whole API fits in two paragraphs in your project's README. And this is the case for quite a few packages on NPM, including many I plan on writing in order to refactor the projects outlined above.