CloudBleed: CloudFlare Leaks Sensitive Information

The internet infrastructure provider CloudFlare was recently discovered to be leaking server memory, potentially exposing sensitive information such as user passwords, authentication tokens, and other data. Cloudflare’s reverse proxies systems inserted random amount of server memory into webpages. As a result, visitors to these websites had unencrypted memory appended to the bottom of their browser page.

Google and other search engines responded by purging cached copies of web pages that were affected by this flaw. Google discovered (and purged) cached pages that contain private messages from well-known services, personally identifiable information (PII) from major sites that use CloudFlare, and even plaintext API requests from a popular password manager that were sent over https.

What do I do about it?

At this time it is not clear the extent user credentials and server certificates were compromised. Nevertheless, many security experts recommend changing all passwords and certificates as a precautionary measure. Researchers at Arbor Networks, in an alert, suggested that “For most of us, the only truly safe response to this large-scale information leak is to update our passwords for the Web sites and app-related services we use every day…Pretty much all of them.” [19

What is CloudFlare?

CloudFlare provides a content delivery network (CDN), Internet security services, and distributed domain name server services, sitting between the visitor and their customer’s hosting provider. Also CloudFlare is a leading provider of protection against distributed denial of service attacks (DDOS).

CloudFlare customers run the gamut of major enterprises and institutions with a major web presence and seeking resistance from high volume attacks. These include Uber, OK Cupid, Fitbit, 1Password, and 2 million others. You can check if a website uses CloudFlare by going to http://www.doesitusecloudflare.com/

What caused the problem?

Dubbed “CloudBleed” owing to its resemblance to the 2014 HeartBleed vulnerability in OpenSSL, these leaks were recently identified by Tavis Ormandy of Google Project Zero. Ormandy traced the memory over-read bug in a server-side HTML parser on CloudFlare’s systems. CloudFlare promptly responded by disabling and patching the vulnerable functionality.

A single character – ‘>’ rather than ‘=’ – in Cloudflare’s software source code caused the problem. CloudFlare used a open source library to parse HTML code. Because both the CloudFlare code and the library made equality checks instead of greater than check, the pointer stepped past the end of a buffer. This programming error is known as a buffer overrun. Had either check been done using >= instead of ==, the buffer overrun would have been caught.

What can be done to prevent this from happening again?

A number of software security practices should have identified this weakness. Manual code review might have missed it as the code spanned a library call. Status analysis tools perform data and control flow analysis so are better than most reviewers in parsing nested logic. Because the library CloudFlare used was open source, it is likely it could be readily included in a scan. Proprietary libraries typically present a greater challenge for code analysis. Lastly, a quality testing tool like Selenium might have identified the problem if tests had been written to detect the leaked data. At present it is not publicly known whether any or all of these methods were applied by CloudFlare’s developers.