ServerCentral works around-the-clock to help keep our clients secure. Here’s some insight into the time taken to perform a massive, network-wide scan and notify our affected clients:
ServerCentral’s response summary
You don’t spend 15 years in IT without one of “those” moments that stops you dead in your tracks. A few years ago, it was when a client crashed a live SAN because the storage vendor labeled it “DEV” instead of “PROD.” Last Tuesday, it was when a customer asked me to test his server for vulnerabilities and I was able to read somebody’s email (presumably one of his clients’) with the results of a paternity test.
Despite me doing it with permission to help diagnose a security vulnerability, there was still the residual sick feeling of actually being able to read someone’s private information so easily. Even though we patched his servers in twenty minutes, we knew there could be a thousand people reading personal correspondences like that one all over the world. More than anything, we felt powerless knowing we could have done nothing to correct this issue in advance.
My heart went out to him. His clients paid him to protect the confidential data they exchange with their customers, and the most trusted tool on the Internet for securing communications—SSL—had failed us all.
This customer thought his system patched against the Heartbleed vulnerability, tested it multiple times with the online tools available to him, and believed he was safe. When he discovered he wasn’t, he asked us to protect his systems. We immediately added filters to his firewalls to block the vulnerable responses, which helped as long as no one got past the firewall. Of course, he still needed to fix the underlying issues on his servers.
The last thing we wanted was to have another client learn that their server was open to the Internet, so we starting scanning.
On any given day, ServerCentral announces almost a million IP addresses to the Internet at large distributed between our POPs and data centers around the globe. Scanning that many IP addresses for a vulnerability like Heartbleed is no easy task, but our Managed Services and Network Engineering teams were more than up to it.
Our Response
At 3 PM Tuesday, our BGP sessions offered 986,113 IP addresses aggregated across our own IP space and that announced by our clients. Utilizing Nmap, an open source network scanning tool, we scanned every IP in our announcement lists for open port 443 web servers.
All better.
Using a high-end virtual server based in our own cloud environment, we completed the scan in approximately seven hours overnight Tuesday—just in time for our first shift to analyze the results.
While ACLs, firewalls, and intrusion detection systems could account for errors in accessing certain clients and IPs, we detected only 48,026 servers listening on port 443. That represented just shy of 5% of the total systems running an SSL web server. From there, we fed the discovered IP addresses into an exploit script that analyzed SSL handshake information for the Heartbleed vulnerability.
With the help of our Development team, we created a script that scanned up to a thousand hosts at a time for Heartbleed. Results came in slowly because of the nature of the scan, but by Wednesday evening, our team identified 7,553 vulnerable hosts in our network.
Our last step was to inform our customers, a task our Data Center Operations and Managed Services teams completed overnight Wednesday. We sent out a general announcement about Heartbleed with information on how to mitigate the threat to all of our customers. We then split up the list of vulnerable hosts and personally contacted each customer with specific hosts to patch.
Moving Forward
We faced one of the biggest threats to Internet security and made it through, but the fallout is just beginning.
The nature of Heartbleed still makes me uncomfortable to know that so much private, sensitive data could have been exposed to hackers or other nefarious parties. Still, I know that’s where ServerCentral makes the biggest difference.
Our people are here for you around the clock to stop every vulnerability dead in its tracks before you even detect an issue. We have the experience and technical skill to fix these problems and keep you and your clients safe.
