Skip to content


What’s wrong with the Twitter API

I want to make something clear from the start: I love Twitter, though sometimes I wonder if I’m suffering from Stockholm syndrome. I have devoted much of the last three years of my life to working with the Twitter API and continue to pursue building the world’s best Twitter client for professionals in the form of Siftee. Recently Twitter staff has been reaching out to developers with a renewed vigor in the hopes of recapturing some of the goodwill and enthusiasm that has been squandered in the past two years. I applaud them for that. The new developer discussion site and documentation portal are significant improvements. Jack Dorsey has reached out for feedback. I recently spent time on the phone with Jason Costa, Twitter’s developer relations manager, at his request.  I think these are all good signs for the ecosystem.

With that said, there is a lot of feedback to give. This post is a technical one focused on the API itself, not on Twitter’s relationship with developers (I’ll save that for another day). Although it’s technical I’ve tried really hard to make it readable to “normals” :)

Before I start I want to thank Ryan Sarver for lending me his ear in the past, and especially Taylor Singletary for providing lots of great support over the years to not just me but the entire developer community.

Without further ado, here’s what’s wrong with the Twitter API…

Users can’t fetch all their old content.

Twitter imposes a variety of artificial limitations on how far back you can access tweets. Users can only access the most recent 800 tweets in their home timeline, their most recent 800 mentions and the most recent 3200 tweets they’ve sent. Additionally a user may have favorited tweets that have become “too old” to be retrieved. What this effectively means is most long-term users of Twitter will never be able to access all of their old content.

Twitter imposed these constraints early on due to their limited infrastructure capacity and a need to focus on reliable accessibility for recent content. While I can understand an ongoing need to prevent third parties from crawling all of Twitter’s old content, I believe that individual users should be able to access all old content directly relating to their account, which means all their Mentions, Direct Messages, Favorites and sent Tweets, without restriction.

With Siftee we attempt to archive as much of this content as possible for our users so they can search over it and see old conversations.

You can’t fetch all the replies to a tweet.

Twitter’s Issue 142 is one of the oldest and most infamous shortcomings of the API: there is no way to retrieve all the replies to a particular tweet. Issue 142 is about to celebrate it’s third birthday and has the dubious honor of being assigned to a programmer who no longer works for Twitter and it has been given a status of “WontFix”. I have long felt this is the most obvious shortcoming of the entire API and addressing it has more potential than any other item in this post to fundamentally change the nature of the Twitter experience. Consider that Twitter allows you to see everyone who has retweeted a tweet – allowing you to see all the replies to a tweet would revitalize Twitter as a medium for conversation rather than just broadcasting. Unfortunately Twitter sees itself as an information delivery system rather than a social network so this is likely to continue to go unresolved.

You can’t see who favorited a tweet.

Similar to not being able to fetch all the replies to a particular tweet, you cannot fetch a list of users who have favorited a particular tweet. While Twitter’s User Streams API supports notifying users when their tweets are favorited in real-time, there are no methods for finding out who favorited your tweets (or anyone else’s) in the past. This seems like a missed opportunity for user discovery.

Native retweets don’t allow for comments.

When Twitter takes a phenomenon that is naturally emerging from the ecosystem and attempts to formalize it, a good rule of thumb would be that the result should keep all the existing functionality of the phenomenon and ideally make some or all of it better. When Twitter rolled out native retweeting they solved a number of “problems” that most users probably didn’t care about (making tweets look like they came from the original author; making sure the tweet text wasn’t tampered with) while eliminating a very significant element of the original phenomenon (adding your own commentary to someone else’s).

We now have a situation where most Twitter clients support both the original “RT” approach to retweeting as well as the native retweet functionality. This sort of potential UI confusion is cited as one of the main reasons why Twitter wants to stop third parties from developing new clients. Of course this completely fails to acknowledge the reality that if Twitter’s own solution to retweeting was actually better across the board than the original behavior it would have been near universally adopted. I don’t mean to be harsh but I feel like I’m in pretty good company when Twitter’s own co-founder and Executive Chairman Jack Dorsey said he doesn’t use the native retweet functionality because it doesn’t fit how he retweets, by which he means he likes to include his own comments. I’m pretty sure he’s not the only one.

Think of it this way: the current system only allows for implicit agreement. There is no way to natively retweet someone while stating “I totally disagree with this”. Imagine only being allowed to quote political candidates you agree with. This is similar to Facebook’s problematic decision to create a Like button but no corresponding Dislike. It artificially skews potential activity in the system. This doesn’t make sense for a social network, but as I noted earlier, Twitter doesn’t consider itself a social network.

At this point it might be very difficult for Twitter to improve the situation. One idea is that Twitter clients could implement a behavior where a user could follow up a retweet with a separate reply to the original tweet and have the two visually linked.

DMs can’t be marked as replies.

Direct messages do not have the in_reply_to property that tweets have – in other words there is no way to explicitly link one direct message as being a reply to another. This means there is no way to break up direct message conversations between two users except perhaps by how much time has passed between messages, which is a very unreliable way of breaking up conversations. All this requires as a fix is implementing the exact same functionality that already exists for tweets to mark one as in reply to another.

DMs aren’t threaded as conversations.

Developers access direct messages via two endpoints: direct_messages and direct_messages/sent. The first represents a user’s inbound direct messages and the second represents all their outbound direct messages. This is, unfortunately, the entirely wrong model for how private messaging should be represented. This model makes it extremely difficult to surface old conversations between the user and another specific person because it requires the developer to go back in time by loading all old direct messages the user has sent and received just to find the ones sent to or received from a specific account. This is why Twitter clients that show DMs as per-user conversations (including twitter.com) don’t let you go as far back in time as you would like to go. It simply becomes unwieldy and requires too many potentially extraneous calls to the API.

The right model would allow developers to fetch a list of accounts the user has had DM conversations with in reverse chronological order (the same way your phone shows you who you’ve been texting) and then fetch just the messages between the user and another specific account (again, the same way your phone does it).

Search results aren’t tweets.

Twitter search does something very weird. It doesn’t return tweets! It returns information that roughly resembles tweets but leaves out many of the standard fields and totally changes one very important one – the user id of the sender. This is noted in a warning on the Search API page, which notes that the issue (Issue 214) is being “tracked”. Unfortunately this issue has been “tracked” since 2008, when Twitter acquired Summize to form the core of its search capabilities. The last comment on Issue 214 sums things up pretty well: it’s “more or less obvious that they are not ever going to fix it”.

Twitter developers could pretty easily fix the issues of missing and incorrect fields in search results by taking all the tweet IDs returned in a search and looking up the original tweets, but unfortunately…

Tweets can’t be looked up in bulk.

Twitter provides some nifty bulk lookup tools such as users/lookup which lets a developer send up to 100 user IDs or screen names to Twitter and get back a full representation of that user (their real name, bio, friends and follower counts, etc). Unfortunately there is no equivalent for tweets. If you have a list of tweet IDs (unique numbers linked to each tweet) and you want to look up the actual corresponding tweets, you have to do it one at a time.

This wasn’t really such a big deal for a long time, as there was rarely a situation where you would need to look up lots of tweets using ids. Twitter makes lots of different services available for getting tweets in bulk such as fetching a user’s home timeline or their mentions. But what if a developer wanted to look up something that Twitter didn’t provide a service for, say for example the top most favorited tweets? At one point in time they might have turned to a third party such as Favstar. Unfortunately (I seem to be using that word a lot) Twitter made what I consider to be one of their greatest strategic errors by changing their Terms of Service to prevent third parties from making tweets available via their own APIs. Instead, “If you provide an API that returns Twitter data, you may only return IDs (including tweet IDs and user IDs)” (section 4.A. of the ToS). Twitter allows developers to get up to 200 tweets at a time for things like a user’s home timeline, counting it as a single request against your rate limit (the number of times you can request certain things from Twitter per hour). But getting a similar 200 tweets from a third party service requires getting 200 tweet IDs and then requesting each tweet individually from Twitter, using up over half the current rate limit of 350 requests per hour.

Talking about Twitter’s recent terms of service changes and their implications is fodder for a different post. The takeaway here is that Twitter should never have forced third parties to make only tweet IDs available through their own APIs if Twitter wasn’t ready to release a corresponding bulk tweet lookup service.

Lists don’t show @replies to people you follow but are not members of the list.

There are a number of characteristics of Twitter lists that can be very confusing for users. One thing many people don’t realize is you do not need to follow the accounts you put on a list. In fact, lists can be a great way to keep track of things people are saying without following them and cluttering up your home timeline. Another thing many people don’t realize about lists is that they will never show tweets that are replies to accounts that are not on the list. This, in my opinion, is not a good thing.

The reason this is not a good thing is because many people think of lists as a way to organize the people they are already following into more manageable groups. If I follow someone and I put them on a list, such as my “Twitter Developers” list, I would expect any tweets of theirs that I see in my home timeline to also appear on my Twitter Developers timeline. But that is not the case. For example, if I didn’t put myself on my Twitter Developers list I would not see any @replies to me from anyone on my Twitter Developers list when looking at that list, even though I would see them in my home timeline. This is confusing.

There are good reasons from an infrastructure perspective as to why Twitter may have had to build lists this way. It would be great if Twitter would enable an option to request list tweets as either filtered or unfiltered for @replies to accounts you follow whether or not they are on the list. This is not too likely to happen, but the missing @replies could be pretty easily restored on the client side by merging the appropriate tweets from a user’s home timeline into their applicable list timelines.

Lists should not be capped.

Twitter imposes two limitations on lists: a user cannot create more than 20 of them, and each of them cannot have more than 500 members. Neither of these restrictions make much sense and my guess is that they do more harm toward user adoption of lists than they do good in terms of preventing Twitter’s infrastructure from being overloaded (which is the only justification for these restrictions in the first place).

Capping lists at 500 members doesn’t make sense given that Twitter can clearly already handle generating timelines for accounts that are following tens of thousands of people. Most users will not build such large lists for personal organization anyway, but consider cases such as building a list of conference attendees (Twitter arguably had its first major spurt of adoption at the 2007 SXSW conference as a way for attendees to communicate) or maintaining a company directory (Twitter maintains a list of all its employees – a list which apparently has special privileges as it currently has more then 500 members).

As for limiting accounts to 20 lists, again this seems very arbitrary and unnecessary. Most users will never create more than 20 lists, but those that would want to would likely have good reason to. What if a university wanted to create a list of all the students in each of its courses? What if a large company wanted to create a list for each of its departments? What if the US government wanted to create a list of government officials on Twitter in each of the 50 states? As we continue to generate more and more information, curation becomes ever more necessary. Twitter should be trying to get ahead of the curve on this, especially with new services like Google+ entering the fray.

Errors aren’t consistent.

This is purely anecdotal but from my extensive experience in building Siftee I can say that the Twitter API throws a lot of random errors that frequently don’t accurately reflect whatever the problem may be. Sometimes Twitter is having a capacity issue but the API tells you you’re requesting some information that doesn’t exist (when it does). Sometimes it spits HTML at you rather than a properly formed error response. Experience indicates there are a wide variety of possible error messages one can receive from the Twitter API but unfortunately these messages aren’t documented anywhere. Yes, there is an error codes and responses documentation page but this seems to just scratch the surface of actual error messages you may receive.

The crossdomain.xml for non-search APIs is too restrictive.

I saved this one for last as this is a pet peeve that only impacts developers using Flash (and possibly Silverlight). Twitter hates Flash (along with everybody else these days). Why do I say that? Because Twitter has since day one made it impossible for Flash to directly access any Twitter services except the Search API. This is very simply due to the lack of a sufficiently open crossdomain.xml file on Twitter servers which Flash needs to satisfy security constraints. I’ve been bringing this issue up for three years here, here, here, here and here. Siftee is currently built in Flash using the Flex framework and every request it makes cannot go directly to Twitter but instead has to be routed through a PHP proxy due to this extremely stupid constraint that no other major web service imposes. Regardless of how you feel about Flash, it makes no sense for Twitter to alienate a whole industry of developers simply because it can’t get around to reviewing this issue.

Conclusion

This post is not meant to be exhaustive. There are lots of other things Twitter could be doing. In fact we’re doing lots of great stuff with Siftee that can’t currently be done with the Twitter API. My goal was to cover what I see as some of the long-standing issues that should have been addressed a long time ago. I don’t mean to suggest that all of these things are easy to fix. I don’t work for Twitter and I am no expert in building large-scale web services. However after several years and raising more than a billion dollars I would have expected many of these issues to be non-existent.

I’m sure I’ll be adding to this as I expand my own knowledge of the Twitter API. In the meantime I look forward to seeing what comes of Twitter’s renewed interest in gathering feedback from developers.

Posted in Uncategorized.

  • Anonymous

    Very thorough and well articulated, Orian.  Some are definitely more critical than others but it’s great to see a list like this in one place.  

  • http://tomakefast.com PJ Brunet

    Nice analysis.  

    You forgot “No basic authorization.” and “Randomly shutting off apps.” and “Making it too difficult to get whitelisted.” and “Designating official apps and telling developers to stop making Twitter clients.” and “Lame quotas that severely limit the capabilities of apps.”

    Bring on the decentralized P2P microblogging–Twitter is so 2009.  

    Also Matt says WordPress is about to announce some “social” features.  That could be interesting.  I’d much rather use an Automattic API.  

    • http://tomakefast.com PJ Brunet

      I almost forgot.  We don’t want forced Twitter URL shortening.  All I see now is t.co and that doesn’t tell me anything about what I’m clicking on.  I trust the people I’m following more than I trust Twitter the corporation.  If anything, we need to be protected from Twitter–they fired the guy who created Twitter on his ThinkPad–then went on to take all the credit and lie to everyone.

      • http://shkspr.mobi/ TerenceEden

        Regarding t.co – your client can show you the original URL – it doesn’t have to display the t.co.

        For example, I’ve coded my client to show
        <a href=”http://t.co/foo”>example.com/whatever</a>

    • http://www.orianmarx.com/ Orian Marx

      Believe me I’ve found plenty of this frustrating. But I wanted to separate out the more “political” issues from the technical ones. Hopefully I’ll write about some of these other topics soon.

  • http://epcostello.com/ e.p.c.

    Bravo. 

    The favorites thing is especially annoying since you can’t even tell if a tweet has been favorited by anyone (other than yourself).

    I’ve stopped developing twitter applications or integrations with twitter and moved on.

  • http://twitter.com/GrayDear Glenn Dearing

    This post very well articulates many of my annoyances with Twitter. I would add 1 more: Native RTs don’t show up in lists. If I put someone in my list, I’d like to see the tweets they choose to RT there as well.

    • http://www.orianmarx.com/ Orian Marx

      I would have included that but Twitter actually just fixed that issue. It is now up to clients to implement the proper code. See here: https://dev.twitter.com/docs/api/1/get/lists/statuses

  • http://wearenytech.com/64-mark-birch-investor-entrepreneur-trader Mark Birch

    This is very comprehensive and I too have found some of this behavior strange even from the user perspective.  Thanks for posting!

    • http://www.orianmarx.com/ Orian Marx

      Thanks for linking!

  • Anonymous

    “You can’t fetch all the replies to a tweet.”

    How does Tweetbot do it?

    • http://www.orianmarx.com/ Orian Marx

      There is one experimental service Twitter offers that they may be using, but Twitter has clearly stated it is not a reliable service for fetching all replies to a tweet and could go away at any point in time. There are other ways this can be done but it would require very inefficient use of the API.

  • http://twitter.com/toloughlin Tim O’Loughlin

    Excellent piece!…the fact that native retweets don’t allow for comments drives most people to other clients

  • http://www.daviddalka.com/ Dalka

    What are your thoughts and opinions on:
    - Fraudulent bots created by the API that created funding and stories of “user adoption” in the early days?
    - The social media “experts” these API’s created?

    I think to discuss the current state one needs to consider the flaws that existed on day 1 that should have never occurred???

  • http://shkspr.mobi/ TerenceEden

    From what I can see, the public issues list hasn’t been looked at for about a month by anyone from Twitter.

    So many bugs, and no one who seems to care.  Such a pity.

    • http://www.orianmarx.com/ Orian Marx

      Yeah that’s something I would talk about in a separate developer relations post.

  • Anonymous

    Fantastic post.  

  • http://twitter.com/fluxresearch Flux Research

    This is really awesome and I’m glad you made it understandable!

    Some of the points you mention have kept me off Twitter as a user though I’m gradually getting more involved now that I’m doing a project that benefits from my involvement.

    The thing about Twitter maintaining that they’re not a social network is a good example of how rigid forms of self-identity can get in the way of dealing with the actual users of one’s service.  I can understand that they need a clear self-identity but, given that social features are now a part of almost all services, it almost sounds like they have psychological blinders that stop them from observing actual user behavior.

    • http://www.orianmarx.com/ Orian Marx

      I agree, I’ve always been pretty stumped by this. However if you take a look at much of Twitter’s leadership you’ll see that historically they were never very active tweeters, at least not to the degree that constitutes active in a social vs broadcast sense. 

  • Anonymous

    Amazing article. I didn’t know the Twitter API was so buggy. If Twitter wants to stay above Google+ and other services, it must fix these glaring issues. I mean, seriously, I can’t even fetch all replies to a tweet?!

  • Pingback: 9/11 coffee notes | Echo of Scripting News

  • Anonymous

    I agree that the limited history is irritating.

  • http://www.dancingbison.com Vasudev Ram

    Hey, good post. I’ve not evaluated all your points but some at least definitely resonated with me, not as a developer (I don’t develop for Twitter as of now), but as a user. Not sure about the Flash issue though – don’t have enough knowledge on the topic, but it *seems* from your description that Flash may have specific requirements like that .xml file you mention, that others don’t. Imposing such requirements may be against open standards (which I understand Flash may not (at least fully) support), and also seems detrimental (for those reasons) to the benefits and success of the Web / Internet.

    My 2c …

  • http://andrewphelps.com Andrew Phelps

    I find the constant, seemingly random, and utterly erroneous 502′s infuriating.

  • http://miralize.com Sean O’Grady

    Al ot of these problem have actually been fixed with new new twitter. Looks like they listened!

  • Pingback: What Twitter Wants – Orian Marx