Skip to content

What Twitter Wants

I have had a love / hate relationship with Twitter for four years. As a technologist, it is impossible not to be enamored with the transformative effect Twitter has had not just within my industry but the world at large. As an entrepreneur and perhaps an idealist, it is impossible not to be embittered by the trajectory upon which Twitter has set itself as a company.

In a recent ominous blog post, Twitter hinted at a further shift toward becoming a media portal and away from being an open platform for communication. Twitter has already earned its place in the history of the Internet and I have been very glad to be involved with it as a user and a developer during its exciting stages of rapid growth and innovation. What saddens me is that as a company Twitter seems hell-bent on relegating itself to being a precursor for something else, something better, abandoning its radical and innovative roots for staid ideas of commercialization in order to emulate a decade old model that will make it just another media entity if not completely defunct.

When Steve Jobs died, much was said about the fact that as a visionary he changed not one, but five industries. Few other entrepreneurs can make such a claim. In a similar sense, Twitter revolutionized five different areas of the web: real-time, mobile, non-reciprocal social networking, short-form communication, and the use of APIs. In comparison I would say that Google and Facebook each revolutionized two. The former in search (PageRank) and advertising (AdWords). The latter in activity streams (the news feed) and content sharing (tagging people in photos and posts).  The use of a social network identity as an authentication mechanism for other services is another major innovation, but I’m really not sure who deserves the most credit for that, Facebook or Twitter.

I am not claiming that Twitter was the first to introduce any of these ideas nor that their implementations were the best. What they did do is solidify these ideas in the new technological order. Twitter demonstrated the feasibility and value of delivering streams of information to the masses in real-time. It unlocked the extraordinary information dissemination potential of social networks that did not require reciprocal connections among users. It established the role of the mobile device in producing content for the web and disseminating news. It validated the idea that artificial constraints on the length of content (where we thought the web clearly had an advantage over paper) could lead to an explosion in the creation of content (how many people have written a blog post vs. posted a tweet?). Finally, it established the concept of web services as robust open platforms from which thousands of programs and startups could bloom.

It is because of this that as a web developer and an entrepreneur I feel I owe such a debt to Twitter. It has opened my eyes to so many exciting possibilities. And that is why I am so sad to think that some time in the near future I might not be using it.

My story as a Twitter developer

In 2009, my then girlfriend Whitney Hess was well on her way to prominence as a new voice in the field of User Experience. She attributed much of her success to her prolific blogging and her tireless engagement with a rapidly growing collection of followers on Twitter. Yet as this follower-base grew she was increasingly frustrated with how cumbersome the experience of using Twitter was for a professional. I could see how powerful this new communications channel was and yet how far it had to go to reach its potential.

Information management has always fascinated me. My previous startup was focused on building an intelligent RSS reader that would reduce information overload by collating related posts across the blogosphere. I was excited to turn my attention to trying to solve all sorts of interesting problems with the creation and consumption of tweets. How do you make it easy to filter tweets by keywords or relevance? How do you figure out who to follow? How do you discover what your followers are interested in? Where they are? What they talk about? Who they know that you might want to know? How do you find that important thing that someone said weeks ago? Is it through searching just your own streams? Is it through tagging tweets in order to “bookmark” them? How do you see all your old conversations with someone else?

Working on these problems became my passion for the next two years, pouring hundreds of hours into my little Twitter client, Siftee. It meant I got to learn all the ins-and-outs of the Twitter API, including all its ugly parts (I’ve already written about what’s wrong with the Twitter API). It meant I had to design and build my own infrastructure and APIs to do the things it seemed Twitter was never going to do, like enable search in your sent tweets and DMs, sorting and filtering of followers, tagging tweets, seeing conversation histories, et cetera (I owe a debt to Stephen Michaels over at Fusebox for developing the backend components that made those features a reality).

I arrived at the SXSW 2011 technology conference excited to show off my progress on Siftee. The angst over the acquisition of Tweetie the previous year, just before the announcement of promoted tweets at Twitter’s Chirp developer conference, had long ago created a gloom over the developer community but I had pressed on in light of the fact that I saw no signs that Twitter was interested in addressing the needs of the professional market any time soon. A day or two later, however, the infamous Ryan Sarver memo would drop, informing developers that continuing to build 3rd party Twitter clients was not in their best interests. I tried to press on, but that’s when potential investors left and right started telling me “it’s over.” I probably would have stopped working on my project if it hadn’t been for Rachel Sklar, a media insider, TechStars mentor and Twitter power user who immediately signed on as an advisor after a demonstration I gave her last summer.

Although Siftee was getting traction and continues to get new signups every day, friends and investors continued to tell me that building on Twitter’s platform was crazy, and, well, I had rent to pay and mouths to feed (I have a cat). Siftee is now more of a side project for me, but I can’t quite let go of it because I know that most of the problems I was trying to solve have still not been addressed, and they likely never will be.

I continue to believe that they are worth solving.

What just happened

A few days ago, Twitter’s consumer product manager Michael Sippey (formerly the VP of product at Six Apart) published a post on Twitter’s developer blog entitled Delivering a consistent Twitter experience. It reiterated 3rd party developers’ subordinate status as first detailed in the Ryan Sarver memo and laid the groundwork for Twitter to further clarify its vision of a more tightly-controlled experience.

It has created a firestorm of concern and criticism. Mathew Ingram warned Twitter to remember what happened to MySpace and Digg (and AOL, and AIM, and…). He wonders if anyone would use a truly open alternative. Dave Winer reminded developers of the folly of building on a corporate API. Ben Popper and Tim Carmody lamented as Twitter follows Facebook down the walled garden path. Anil Dash (who spent years working with Mr. Sippey at Six Apart) thinks we’re all just a bunch of whiny bitches.

But let’s step back for a second…

Twitter’s purpose

Back in 2009 Whitney and I started playing a little game. We would post messages like “NE corner 14st 6th ave #godontgo”. The walk signals in New York City had recently started acting glitchy, showing with increasing frequency both the walk and don’t walk signs at the same time. We decided to track our encounters with these confused boxes in real-time using Twitter. We wondered if our friends would catch on and join in the fun (a few did). We thought one day we could present the accumulated data to the city as a crowdsourced effort to help solve a municipal problem. I’d love to show you one of those tweets now, but I can’t.

We knew about Twitter’s technical limitations at the time – search didn’t go back more than a few days and users weren’t allowed to retrieve more than their last 3,200 tweets. We weren’t that concerned, after all Twitter was still a fledgling company with limited resources and they issued constant reminders that all these tweets were safely locked away somewhere. What we never imagined was that three years, a billion dollars and hundreds of new engineers later things would be exactly the same.

Of course, there is a simple solution for retrieving all your old tweets. All you have to do is ask nicely – and be a senior journalist at a respected news organization like National Public Radio. Andy Carvin was able to retrieve his archive of (at the time) 95,000 tweets doing just that. Now, the purpose of this request was so that Mr. Carvin could study his massive amount of first hand reporting on the Arab Spring, an effort for which some said he should earn a Pulitzer, and I applaud his monumental efforts. As a side effect he got to revisit pleasant memories of his daughter in his very first tweet, something I imagine a lot of other people with thousands of old tweets would like to be able to do as well.

Being an acclaimed journalist wasn’t the only way to get special treatment with Twitter data. As an early developer on the platform, I have the privilege of “whitelisting”. My @orian account can make up to 10,000 requests for data from Twitter every hour, whereas yours most likely can only make 350. Additionally I have access to the “Site Streams” service which enables developers to track tweets and keywords across thousands of users in real-time. Whitelisting is no longer available, and Site Streams (which was intended to serve as somewhat of a replacement) remains in perpetual beta with signs that it too will likely be shut down. At the time, whitelisting was granted to developers on the premise that they would build interesting applications and expand the ecosystem. Nowadays Twitter would prefer that that be left to the pros – as in them and a handful of designated partners who will charge you enterprise level rates to retrieve precious Twitter data, even if it’s your own.

The beauty of Twitter was that it didn’t have a purpose. This of course is what led to the common characterization of Twitter as being a platform for telling people what you had for lunch. The reality is that this is what separated Twitter from Facebook. It didn’t provide a context that said “here, this is for sharing stuff between friends.” Want to see what someone is saying? Go ahead and follow them, they don’t need to follow you back. Want to provide customer support for your product? Have at it. Want to build an emergency broadcasting service for your town? By all means. Want to track your dieting progress? Good idea! Want to “hear” what time it is according to Big Ben? Um, sure, why not?

Providing a better user experience

Ever since the Ryan Sarver memo, the party line at Twitter has been that reducing the diversity in the ecosystem will result in a better user experience, one guided by Twitter’s expert hand. Yet, as Nick Bilton of the New York Times points out, Twitter’s sites and apps remain a cacophony of confusion more than a year later.

More importantly, nearly all of the useful additions to the original idea of Twitter as a one-to-many short message service originated outside of Twitter HQ. The @reply? Outside Twitter HQ. The hashtag? Outside Twitter HQ. The retweet? Outside Twitter HQ. Search? That came from Summize, which Twitter acquired. Analytics? Twitters current efforts are based off their acquisition of Backtype. Lists? Twibes and TweepML (now dead) did it first. In-line images? Brizzly (now dead) did it first. Pull-to-refresh? Loren Brichter, creator of Tweetie, invented that. Twitter acquired Loren’s company atebits and made its products the first official Twitter apps for the iPhone and iPad, and then set about patenting the pull-to-refresh feature (to Twitter’s credit, their approach to patents seems to be innovative although that’s up for debate). By the way, Loren quit working for Twitter back in November.

Luckily, those are all examples of ideas that have made it into the core of Twitter. What about everything else? What about Proxlet, a popular service that let people filter their timelines so they didn’t have to see Foursquare checkins or dopey hashtag memes if they didn’t want to? Twitter shut them down with no advance warning. It never came back. What about Favstar, a service that scours Twitter activity to find the best stuff that users are favoriting? Developer Tim Haines put his API plans for the service on hold for fear that Twitter would shut him down. And then there’s TweetDeck, which Twitter hurriedly acquired to keep from the hands of Bill Gross out of fear he might roll out an ad network faster (better?) than they would, or even possibly a competing social network. This was a move that pretty clearly did not benefit end users.

So what does Twitter believe to be the next big step in improving the user experience? Primarily it seems to be the, er, expansion of content delivered in expanded tweets. I’ve watched as developers in the Twitter forums have pleaded to be able to make their media content appear in Twitter’s side pane and expanded tweet views, only to receive the same response: “we’re not taking more partners at this time”. Twitter could have gone the way of supporting existing embedded media standards years ago by utilizing a service like Embedly, which now supports 218 different content providers at last count, but instead they are focused on getting developers to use their proprietary “Twitter cards” which, not surprisingly, requires a formal agreement with and approval from Twitter. [edit: Ryan Sarver wants to correct some inaccuracies here, noting that there are hundreds of partners for Twitter Cards and joining doesn't require a deal of any kind. However, here is the form for participating in Twitter Cards which explicitly states you need approval from Twitter, so I don't see how my original statement is inaccurate.]

That’s a far cry from annotations, an idea Twitter announced at the Chirp conference. With annotations, developers would be able to add any arbitrary metadata they wanted to tweets, so tweets could suddenly carry any kind of additional meaning their creators wanted to add to them. It was an idea so radical I could barely wrap my head around it. I remember meeting a bunch of Twitter engineers in the lobby at the conference. When I introduced myself, they said “oh, you’re that guy who sent the letter to Ryan.” Just before the conference I had sent Ryan Sarver my thoughts on how Twitter was mismanaging the developer community, which, to his credit, was at his request. Apparently the letter had circulated. They then asked me what I thought was more important, user streams or annotations (the two major engineering efforts announced at the conference). When I said “annotations”, the engineers on that team cheered. Twitter took to its blog to detail how this would be part of their strategy to create enduring value. Annotations never happened.

What Twitter wants

Twitter has figured out what it wants to be when it grows up. It wants to be The New Media. Twitter doesn’t want to come to you. It wants you to come to it. This, we’re told, will provide a better experience. And it will – for their advertisers and their investors.

In its youth Twitter might have thought it purpose was to empower us as creators. But it has grown up, and the conventional wisdom of grownups is that it’s far more profitable to think of everyone as consumers.

This is just the natural outcome for a social network once it leaves the hands of its early adopters and reaches scale. Twitter isn’t trying to create a Horrible Network for Kardashians. It’s just a reflection of all the normal people that showed up to use the service. At least, that’s what Anil Dash argues. But Anil is wrong. Twitter has the hand that guides the users, not the other way around. Twitter gets to decide whether the engineering priority is to support Justin Bieber having millions of followers (here’s @biz with the Bieber boxes) or whether all users should be able to access all their old tweets.

Anil takes issue with programmers like Dalton Caldwell (and presumably myself) who are lamenting what Twitter could have been, claiming we’re just a bunch of “hippie utopian technological triumphalists” who are upset because Twitter isn’t just for geeks anymore. Well, programmers will always be in the minority in established social networks. Concluding that we’re just resentful of this status is wrong. I certainly can’t speak for everyone, but I know I was trying to make things better for whoever wanted to use Siftee, even if they liked Kardashians.

Others are saying that Twitter is just taking a cue from Apple, which clearly stands out as an example of the benefits of top-down control. Color me skeptical. I certainly believe Twitter would be a better service and product today if they had continued to work with the developer community rather than against it. Some see Jack Dorsey as the heir apparent to Steve Jobs (though some do not). And, ya know, I understand where Twitter is coming from. What company ever made good money focusing on building great products that made their users more productive? I guess that’s why Apple became the worlds largest company by selling ads.

What Twitter should do

I hope that the contents of this blog post adds some useful context to the vibrant discussion that is emerging about what Twitter is, has been, and may become. If Twitter wants to remain a truly open network and bring 3rd-party developers back to the platform, here are three things it should do:

First, Twitter should issue a memo explicitly reversing course on the policy that 3rd parties should not develop Twitter clients and acknowledge that this policy was not the right approach to fostering future innovation on the Twitter platform. This memo should be accompanied by a strategy document detailing how Twitter will work with 3rd parties to develop guidelines for the rollout of new platform features and branding requirements while maintaining overall implementation flexibility.

Second, Twitter should announce a revised ad distribution model, and a timeline for implementation, which provides for free ad-supported access to tweet streams as well as paid access to tweet streams without ads. In addition, some portion of revenue generated from ads delivered through 3rd parties should revert to those 3rd parties. For further discussion on the benefits of this approach, see Nova Spivack’s excellent piece A Solution to the Twitter API Problem. By the way, the revenue share model is actually what Twitter CEO Dick Costolo said he was going to do two years ago “in a very transparent way.”

Third, Twitter should announce an effort similar to Google’s Data Liberation Front to provide users complete access to the content which they have created and continue to own under the current Terms of Service. Nova Spivack’s piece makes salient points regarding the law around content ownership and distribution, and I do hope that someone with expertise starts to take a closer look at Twitter’s artificial restrictions on retrieving your own content. At a minimum, Twitter should provide a way to request and receive a one-time data dump of your own Tweets and Direct Messages should you choose to take them elsewhere (and without qualification as to who you are or why you want them, so long as you are the author).

Note that these are not feature suggestions. There’s a reason for that. Everyone does not need the same features. That’s the beauty of an open ecosystem. At some point Twitter’s thinking on open ecosystems flipped, where the costs are now perceived to outweigh the benefits. My suggestions only reflect how to flip it back. The rest takes care of itself.


Through Twitter I’ve met hundreds of interesting people, spawning thousand of conversations. I’ve used it to grow my skills as a programmer, an entrepreneur and maybe even a writer. It has been fascinating to watch its growth and see its impact on the world.

Obviously I’m not happy about the direction Twitter is headed in, and I haven’t been for a long a time. Do I think any of my suggestions above will happen? No, I don’t. I think Twitter will continue to spread FUD until what’s left of the ecosystem remains wilting in the carefully arranged flower beds of its walled garden, foregoing the legacy of all the good ideas that got it to where it is today.

Mind you, I don’t think we’re losing these good ideas. The web doesn’t seem to let good ideas go. That’s why I continue to remain excited about how they might take hold elsewhere. For that, I’ll leave you with some final thoughts on Twitter from someone who articulated them way better than I could:

Twitter needs to decentralize or it will die. Maybe not tomorrow, maybe not even in a decade, but it was (and, I think, remains) my belief that all communications media will inevitably be decentralized, and that all businesses who build walled gardens will eventually see them torn down. [...]

The call for a decentralized Twitter speaks to deeper motives than profit: good engineering and social justice. Done right, a decentralized one-to-many communications mechanism could boast a resilience and efficiency that the current centralized Twitter does not. Decentralization isn’t just a better architecture, it’s an architecture that resists censorship and the corrupting influences of capital and marketing. At the very least, decentralization would make tweeting as fundamental and irrevocable a part of the Internet as email. [...]

So while I don’t expect Twitter to master its own destiny as far as the decentralization of the medium goes, I do support the idea, and I hope that Twitter as a business can coexist with the need for the world to have a free, open, reliable, and verifiable way for humans to instantly communicate in a one-to-many fashion.

- Alex Payne, original engineer of the Twitter API

Posted in Twitter.