Web Daily Find

Last Friday was a hot day in Sebastopol, California. Eran Hammer-Lahav rolled into town hours after finding out that there was a security hole in his pet project for the last few months, a new way to use Twitter to log in to third party sites using the OAuth protocol instead of user names and passwords. Working as the Open Web Evangelist at Yahoo, Hammer-Lahav was relieved to have been told about the hole so he could help fix it. When he arrived in Sebastopol at a small event of industry leaders called Social Web FOO Camp, he talked with friends and colleagues about it.

At some point in conversation Hammer-Lahav realized that the problem went far beyond the Twitter implementation. The OAuth protocol had an inherent vulnerability; big companies like Google, Netflix and Yahoo had implemented OAuth and scores of tiny startups had too.

OAuth has support, but it doesn't have a centralized authority ready to deal with problems like this. Over the next week a story unfolded as the community moved to deal with the security issue. It's a dramatic story. Fifty people from thirty companies mobilized to quickly and quietly respond. Big companies came to the aid of small ones. Twitter willingly took another major PR hit for the good of the open web community. Journalists circled around hints of a story. The decentralized community of open web and data portability advocates and engineers figured out on the fly how to protect users' control over their own accounts and company trust in the new protocol. This is the story of how they did it.

The Nature of The Problem

The problem was a vulnerability to something called a "Session Fixation Attack." The gist of it is this. Services supporting OAuth let their users pull data into other websites for reuse around the web. In order to do this securely, the 3rd party site has to ask the original site for permission. This might be a new little website asking permission to import your Gmail contacts or to post to Twitter through their site instead of Twitter.com. OAuth was born from the work that Flickr did to create a secure way that other applications could be granted permission to access your photos for printing, editing or posting elsewhere.

The problem arose if an attacker could convince you to complete their request for account permission with your login. At the end of the process they would have access to your account.

Hammer-Lahav explains how this works in detail and offers flow charts in his blog post explaining the technical nature of the problem. For another explanation of this kind of attack see Mitja Kolsek's paper titled Session Fixation Vulnerability in
Web-based Applications (PDF), which was published in 2002. In other words, this is not a new problem - it was just newly discovered to be an OAuth vulnerability.

How It All Went Down

Eran Hammer-Lahav was at FOOCamp when he realized this was a problem that extended far beyond Twitter's implementation. All 30 companies currently offering OAuth were vulnerable. MySpace, Yammer, PhotoBucket. Google, Netflix, Yahoo. Millions of peoples' accounts were at some risk.

If OAuth was software, a fix could be implemented and pushed out to everyone who was using it. But it's not, it's just a standard-based specification implemented out in the wild and no one party is in charge of it. Someone had to do something though, and they had to do it fast.

The first thing Hammer-Lahav decided to do was call up Alex Payne, API lead at Twitter. Though Twitter had done everything right, it was a particular Twitter implementation that revealed the whole problem and it had only been out for a few days. (We thought it a big enough deal that we wrote a whole post about that implementation.)

Twitter shut down the OAuth option for login within 30 seconds of his phone call, Hammer-Lahav says. They did it without explanation, because they were asked to keep quiet about the security problem for one week - in order for all the providers to get a chance to respond before the security problem went public and could be exploited.

Developers cried out that Twitter was shutting down technology essential to their business without warning - and not for the first time. Robin Wauters wrote a post on TechCrunch channeling developer anger over the shut-off. (Lest we imply too much criticism we'll note that we've written very similar stories ourselves.)

Twitter was widely criticized - and they kept their mouth shut, saying only that it was a temporary problem that would soon be resolved. "I can't stress enough how noble Twitter's behavior was yesterday," Hammer-Lahav told us. "Twitter bashing is a sport now and it's a sport that sells ads. Techcrunch wasn't aware of the security threat but it put Twitter in a position where if they were going to talk about it then they would put other companies at risk. We told Twitter that it was going to go public so do your own PR management and they did a good job. The emails sent by other providers to Twitter thanking them for taking that hit have been amazing."

After contacting Twitter, Hammer-Lahav started emailing all 30 companies listed as OAuth providers with Chris Messina's help. Half of them had representatives at FOOCamp, the event he was calling from. He explained the problem to them as he was able to reach people and asked them not to discuss it until next Thursday, one week later. He knew it would be a difficult secret to keep with so many parties involved, including the frustrated developers trying access all of those companies' OAuth APIs.

"At first it took me half an hour to explain the problem," he says. "By the next day I had the explanation down to 30 seconds." Within 12 hours the group discussing the problem knew there was no simple solution - it could require changes by OAuth providers and outside applications that consume OAuth permission in order for everything working again.

The group of OAuth providers formed an email list to discuss the problem and fifty people from 30 companies joined in. Deciding to focus on communicating with the initial service providers was a decision that had to be made. "You have to triage the parties involved," Hammer-Lahav says. Providers needed extra time to deal with the problem because they couldn't just plug the hole or pull the plug easily; FireEagle, for example, only has an OAuth API - there's no other way for the service to function.

OAuth is being advanced by a decentralized community of developers and other parties, but Eran Hammer-Lahav has been its most visible advocate. He's gained years of experience in the trenches fighting for a variety of open standards. He talked to every OAuth provider on the list and volunteered to act as the Community Threat Response Contact. Yahoo, his employer, told him to take as much time and do whatever he needed to deal with the problem. The company put Allen Tom in charge of Yahoo's response and donated Hammer-Lahav's paid time to the community effort. "If I was working for a different company this might not have been possible," he says. "Yahoo! had a whole team of people managing their own response to the situation."

All thirty companies sprung into action to neutralize the security risk and prepare their respective technical responses. Mashery co-founder Clay Loveless and team pushed back other work to pull all nighters and others pitched in as well. Everyone was an equal participant in working together, from single person startups to multibillion dollar companies. "Yahoo and Google put engineers on the line helping people with small startups to review solutions they were going to deploy," Hammer-Lahav says. "Usually the big guys figure it out amongst themselves and leave everyone else to their own devices. This felt like a real community. There was no liability because it was casual advice. Security people are expensive. Some startups don't even have in-house engineers, they are entirely outsourced."

One by one many of the providers shut down their APIs and one by one they implemented solutions.

By Wednesday, one day before the self-imposed period of silence was over, there had to be a lot of pressure built up behind the scenes. Alex Payne, the man in charge of the Twitter API and a guy who is much less grumpy than you'd probably be if you had his job, started getting visibly frustrated. "The view from under this bus is really something," he said on Twitter. "Nobody in the tech press has bothered to contact me for comment on the OAuth issue. Why bother with facts when speculation drives clicks?"

Just after noon on Wednesday, CNet's Carolyn McCarthy reported that Twitter and others had pulled OAuth support because of a security problem in the spec. "In the interest of online safety," she wrote, "CNET News has chosen not to make the details of the security hole public." McCarthy was at FOOCamp as well and may have heard about the security issue then, but decided to more or less respect the wishes of the developer community and hold off writing about the issue at all until just before the deadline lifted. If that was the case then she both won accolades from involved parties for her discretion and got a lot of pageviews for jumping deftly on the story after the threat had mostly passed but before others wrote about it.

Minutes after the real story was out, Twitter posted about it on the company blog. Then the official OAuth blog posted about it, linking to McCarthy's post and publicly thanking Twitter for taking all the heat for days. Chris Messina worked fast to update the site and co-ordinate the community response. Then API service provider Mashery, the company that powers OAuth APIs for Netflix and many other companies, posted about it on its blog, assuring customers that the problem was small and under control and thanking Twitter as well. Finally Dave Winer, a web forefather and hardcore Twitter critic, made a post on his blog urging people to lay off Twitter and appreciate the way they were communicating with people about a number of intersecting and difficult technical problems.

One day later, one week after the community responding to the OAuth threat called for a week of silence to come up with a solution - Twitter announced that its OAuth API was back.

That was yesterday and by today almost all of the 30 OAuth providers have OAuth back up and running. There are two different long-term solutions in the works that are being debated on the email list as we speak. Hammer-Lahav says he expects a revised draft of the spec will be ready next week.

And that's how a decentralized community solved a security threat in an open identity spec, quickly. One company (Twitter) took a risk at implementing a new technology advocated by an employee of another company (Yahoo's Hammer-Lahav), then an engineer at yet another company found the beginning of the security hole, then news of the whole problem was sent out to contacts on a Wiki, an email list was formed, companies donated their employees valuable time to aid in the effort, everyone more or less kept their mouths shut (including the unfairly criticized Twitter) and then everyone worked together to find a solution just in time. I think that's a pretty cool story.

Lessons for the Future

Hammer-Lahav took the lead in responding to this crisis and says he did it with the future of crisis response in open web communities in mind. Creating a template now for the future is only so possible, though. "In a year this same approach isn't going to work because too many businesses are going to depend on the providers," he says. "If we don't find a way to deal with this in the future then companies will remain very cautious about relying on multiple data sources." He says that people want to create a database listing all the parties involved in technologies like this, but prioritizing who gets talked to first will depend on the nature of the threat.

Finally, Hammer-Lahav says that more companies need to empower more employees to step up and take leadership in this kind of situation. The combination of technical, people and process skills is rare but those people need to be found. "It's not sufficient to have only Chris Messina and I as the two people who can do this," he told us. "We need other companies to step up and say there are people in their organization that can support the community. Yahoo said 'you're going to go do this for the community for as long as it takes,' Yahoo was paying me to manage the community threat in a way that was not purely in their self interest."

Can open communities advocating for an open web respond quickly and effectively to inevitable security issues? It sounds easier said than done, but for now we've got at least one very interesting story that says it is possible.