The Twitter API journey: a glimpse back to the future of Twitter that never was, as laid out in 2009.

In 2009 I interviewed Alex Payne, then Twitter API lead. The interview was published, in French, on now defunct ReadWriteWeb France. It was never released in its original english version.

Following the publication of the new Twitter API roadmap, and apparently much rejoicing from developers, it’s a good time to revisit the future of Twitter that was laid out in the interview. It was a future with the API becoming a central utility of the modern web, and a core value of Twitter itself. But of course we all know that since then, Twitter alienated many of the developers using its API by closing or limiting many uses initially fully allowed.

I hope the optimism about open web APIs, this optimism that Alex Payne showed back in 2009, can be brought back to the web — many developers and startuppers are now wary of web APIs, having had their code or product killed instantly when an API owner changed its policy without warnings. New optimism toward open web APIs would probably needs stronger, more binding commitments from organizations opening their APIs. A roadmap is a good start.

Alex Payne, Twitter API lead: “We want Twitter to be the message bus of the web” [2009]

Alex Payne and his team are changing the way you experience the web. They are adding more realtimeness to it.

(and yes, I’m amongst the pioneers that proudly use realtimeness as a word. There is only 18200 result on google at the moment [2009], but expect that to increase. [nope.])

You don’t even have to go to twitter.com for that new realtimeness to reach you. You just have to use something that use the Twitter API. And today, that could be a plant — or a social platform about stocks.

The Twitter API team is one of the most effective at changing the web. Imagine: a little group of developers, invisible to everyone but other developers. On top of their invisible work, thousands can build new, realtime applications.

And thousands did it, are doing it and will continue to do it.

The Twitter API is the best case scenario in term of an Open Web API: useful, usable and actually used.

(In fact, I first got the idea to interview Alex because I had a client that needed help in defining its web API strategy. But that was too important to keep it for myself ;-)

So, let’s get to know Alex’s work a little more.

The message bus of the web

Julien: What is the most surprising use of your API, the use that stroke you as “I would have never thought of that!”

Alex: The most surprising to me has been the work that InSTEDD has done using Twitter as a cheap, pervasive communication mechanism for humanitarian work (http://instedd.org/twitterbots).

I expected client applications, visualizations, data mining, and so forth, but seeing Twitter and our API used for disaster relief was certainly a pleasant surprise.

Julien: It’s a very interesting use: Twitter as core infrastructure.

What I love here, it’s how it allows them to prototype the emergency communication process. Iterating is so rare when it comes to communication networks, it’s precious! (because we all love prototypes and iterations, needless to say:-)

But it’s a lot of responsibility for Twitter. And it seems people are asking more and more on that front (see the #nomaintenance meme that emerged just after the Iranian election — I suppose it won’t be the last time we see it).

How do you relate with that responsibility, being part of the team that built the infrastructure to support so many tools?

Alex: It’s a lot to contend with, but we’re up to the challenge. We know that everything from emergency communication to essential business transactions are happening on Twitter.

And we’re doing our best to provide a stable, high-performance platform for those messages. We’ve made a considerable investment in our infrastructure, from an operational perspective, a software development perspective, and a talent perspective.

Julien: How do you relate to this challenges as a person?

For example, how do you personnally feel API-wise about the #nomaintenance meme [after the iranian election] or other signs of twitter becoming perceived as a public service?

Alex: I think that’s what we want.

We want Twitter to be the message bus of the web, and I think our users and developers should demand the utmost of our service. We need that pressure to guide us.

Julien: How do you think one can plan for “surprising uses” of an API ?

Alex: Just provide as much data, as many methods, as much access as possible.

Let developers do everything a user can do, and great projects will emerge. Don’t treat your API as a second-class feature; always launch new features with accompanying API methods.

The mistake you shouldn’t do

Julien: What are the errors you won’t make again, but see others do again and again ! I bring this question from those older slides

Alex: Versioning, versioning, versioning. I will never again design a web API without versioning from Day 1!

In this day and age, I also think it’s crazy not to provide OAuth from the get-go if you’re just starting from scratch.

We didn’t have that option when we started providing our API, but I think some sort of token-based authentication is the way to go.

Julien: I can imagine no versioning leads to very uncomfortable situations for devs, because you change things and it breaks their apps.

How did it became more and more of a roadblock? Does or did it restrain you, the API team?

Alex: It has became a roadblock as Twitter applications have moved from the web, for which upgrades and quick changes are easy, to phones and even embedded hardware devices, which in some cases can only be updated by re-flashing. The lack of versioning has definitely restrained us from making more drastic changes.

Julien: But why was it not practicable to just add /1.1/ in the URL, fall back to a frozen version if not stated — and start versioning like that ?

Alex: We want client authors to be explicit in their code about which version they’re using.

About OAuth

Julien: As an user, I love OAuth. OAuth is a much better solution than give your login and password to a third party!

But the very basic auth process was part of the twitter API success, as it lowered drastically the implementation barrier to build authenticated app or data flows.

In retrospect, you were right to go dead easy first, and more secure second.

How do you find that balance?

Alex: I think Basic Auth is certainly easier for developers (and, in some cases, users) to get started with.

We’re going to try to maintain that low barrier to entry by providing a Basic Auth to OAuth proxy for local development use, and by providing a web-based way to explore Twitter API methods (a test console).

[Note: a good example of this kind of console is the SoundCloud test console]

The developper community

Julien: Was it easy at first to gather feedback of the API users, AKA as the devs ?

Alex: Yes, definitely. We had a much smaller developer community when I first started spending time on our API, but they were very vocal and active. The foundation of our API grew in response to their needs and preferences.

Julien: What was the more surprising feedback/demand on the API?

Alex: As we’ve grown, I’ve been surprised at how much our developer community has grown with us.

Two years ago, we never would have needed API methods that return huge denormalized lists of user IDs, for example.

Developers could get by with a little bit of pagination. It’s been surprising and gratifying to see that our developers are “scaling up” with us.

A tale of two APIs

Julien: The problem of merging two APIs is something you face with the Search API…

Planning for consistency, do you just iterate, or do you think there is a right way to integrate the Search API — and even converge with external-but-friend APIs like bit.ly?

Alex: I think that by providing consistent, well-documented APIs, we can satisfy everyone. The stronger our conventions, the easier it is to learn your way around our APIs.

That’s our goal, and we’re engaging our developer community whenever possible to be sure that we’re delivering what people want.

We’re pretty clear on what we need to do for the Search API. It’s a very powerful API, but thankfully it’s also pretty simple to use.

Integration, in this case, is mostly about ensuring that the data you get back is the same no matter which API you’re talking to.

In terms of integrating with external APIs, that’s mostly a matter of pointing people in the right direction. We generally do very little facilitation there; client authors find other APIs that work well with our APIs, and word gets around.

We love REST

Julien: Was there times when you felt the RESTful paradigm was a limitation (Over more complex/complete choices like SOAP?) ?

Alex: No, not really.

We’ve had the occasional request to provide SOAP or another more heavyweight transport, but most developers seem happy to be working with a RESTful API.

If anything, our API is only going to get more RESTful as it evolves. We’re not taking full advantage of everything the REST model has to offer, at present.

Julien: Critics frequently say that many web APIs are not true REST at all (similarly to the web app MVC debate/confusion).

Do you feel that the Twitter API is not really RESTful, but that your goal is to tend to a real RESTful implementation? Could you elaborate on that point?

Alex: I think some methods in the Twitter API are closer to the academic definition of rest (in which resources are exposed as plural nouns which are modified by HTTP method verbs)

But many of the more recent methods are not.

Our goal is to normalize our method names around RESTful conventions.