STE WILLIAMS

Big-name websites leaked people’s private session keys and personal information into strangers’ browsers, due to a Cloudflare bug uncovered by Google security researchers.

Cloudflare helps companies spread their websites and online services across the internet. Due to a programming blunder, for several months Cloudflare’s systems slipped random chunks of server memory into webpages, under certain circumstances. That means if you visited a website powered by Cloudflare, you may have ended up getting chunks of someone else’s web traffic hidden in your browser page.

For example, Cloudflare hosts Uber, OK Cupid, and Fitbit. It was discovered that visiting any site hosted by Cloudflare would sometimes cough up sensitive information from strangers’ Uber, OK Cupid, and Fitbit sessions. Think of it as sitting down at a restaurant, supposedly at a clean table, and in addition to being handed a menu, you’re also handed the contents of the previous diner’s wallet or purse.

This leak was triggered when webpages had a particular combination of unbalanced HTML tags, which confused Cloudflare’s proxy servers and caused them to spit out data belonging to other people – even if that data was protected by HTTPS.

Leaked … Some unlucky punter’s Fitbit session slips into another a random visitor’s web browser. Click to enlarge (Source: Google Project Zero)

Normally, this injected information would have gone unnoticed, hidden away in the webpage source, but the leak was noticed by security researchers – and the escaped data made its way into the Google cache and the hands of other bots trawling the web.

Timeline

The blunder was first spotted by Tavis Ormandy, the British bug hunter at Google’s Project Zero security team, when he was working on a side project last week. He found large chunks of data including session and API keys, cookies and passwords in cached pages crawled by the Google search engine.

“The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to clean up,” he said today in an advisory explaining the issue.

“I’ve informed Cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”

Ormandy said that the Google team worked quickly to clear any private information and that Cloudflare assembled a team to deal with it. He provisionally identified the source of the leaks as Cloudflare’s ScrapeShield application, which is designed to stop bots copying information from websites wholesale, but it turns out the problems ran deeper than that.

Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. https://t.co/wjwE4M3Pbk

— Tavis Ormandy (@taviso) February 23, 2017

On Thursday afternoon, Cloudflare published a highly detailed incident report into the issue: it happens that Cloudflare’s Email Obfuscation, Server-Side Excludes and Automatic HTTPS Rewrites functions were the culprits.

The problem occurred when the company decided to develop a new HTML parser for its edge servers. The parser was written using Ragel, and turned into machine-generated C code. This code suffered from a buffer overflow vulnerability triggered by unbalanced HTML tags on pages. This is the broken pointer-checking source that is supposed to stop the program from overwriting its memory:

/* generated code. p = pointer, pe = end of buffer */
if ( ++p == pe )
    goto _test_eof;

What happens is that elsewhere p becomes greater than pe, thus it avoids the length check, allowing the buffer to overflow with extra information. This eventually leads to the above web session leaks.

“The root cause of the bug was that reaching the end of a buffer was checked using the equality operator and a pointer was able to step past the end of the buffer,” said Cloudflare’s head of engineering John Graham-Cumming in the biz’s incident report.

“Had the check been done using = instead of == jumping over the buffer end would have been caught.”

According to Graham-Cumming, for data to leak, the final buffer had to finish with a malformed script or img tag, be less than 4KB in length (otherwise Nginx would crash), and be running the three functions.

The new cf-html parser was added to Cloudflare’s Automatic HTTP Rewrites function on September 22, 2016, to its Server-Side Excludes app on January 30 this year, and partially added to the biz’s Email Obfuscation feature on February 13. It was only in the Email Obfuscation case that significant memory leakage appears to have happened, which tipped off Ormandy.

Cloudflare does have a kill switch for the more recent of its functions and shut down Email Obfuscation within 47 minutes of hearing from Ormandy. It did the same for Automatic HTTPS Rewrites a little over three hours later. Server-Side Excludes couldn’t be killed, but the company says it developed a patch within three hours.

Cloudflare says SAFE_CHAR logging shows that the period of greatest leakage occurred between February 13 and 18, and even then that around 1 in every 3,300,000 HTTP requests through Cloudflare leaked data across 3,438 unique domains. It says that it held off disclosing the issue until it was sure that search engines had cleared their caches.

In posts on Hacker News, Ormandy pointed out that the 3,438 figure only covers direct queries, and that any information that passed through Cloudflare’s hands was vulnerable to leakage.

He has also noted that the top award for Cloudflare’s bug bounty program is a t-shirt. Maybe the web giant will reconsider that in the future. ®

Article source: http://go.theregister.com/feed/www.theregister.co.uk/2017/02/24/cloudbleed_buffer_overflow_bug_spaffs_personal_data/