By now, you’ve probably heard about the bash vulnerability that’s kicking around on all major news sources. Even today, CVE-2014-6278 remains exceedingly relevant to many environments—and it’s only one of a handful of new weaknesses found beyond the original bug.
Quick Recap If You Haven’t Heard of This “Shellshock” Thing Before
This vulnerability was initially disclosed on September 24th. Patches were made immediately available by most OS vendors, but early on September 25th, it was disclosed that the patch was incomplete. Updated patches were available later that day. These are treated as two separate vulnerabilities and are referenced by:
If the last update you applied was on September 24th, you may still be vulnerable.
There many blog and news articles that already do a great job explaining the in-depth technical tails. I can recommend the following:
It’s absolutely critical that all externally-facing systems be updated ASAP. We also recommend doing a full review of internal infrastructure in some not-so-obvious places, like wireless APs and out-of-band management devices. Make sure to run the latest updates on shellshocker.net.
Testing for CVE-2014-6271 Vulnerability
A quick test to see if you are vulnerable to the original September 24th CVE-2014-6271 is to run the following via ssh/console:
env x='() { :;}; echo vulnerable' bash -c 'echo hello'
If you are NOT vulnerable, that will print:
bash: warning: x: ignoring function definition attempt bash:
error importing function definition for `x' hello
If you ARE vulnerable, that will print:
vulnerable hello
Testing for CVE-2014-7169 Vulnerability
Testing for CVE-2014-7169 is more involved.
The following is an example of an exploitable system vulnerable to CVE-2014-7169:
test@vulnerable:~$ mkdir empty test@vulnerable:~$ cd empty/ test@vulnerable:~/empty$ X='() { function a a>\' bash -c vulnerable bash: X: line 1: syntax error near unexpected token `a' bash: X: line 1: `' bash: error importing function definition for `X' test@vulnerable:~/empty$ ls vulnerable test@vulnerable:~/empty$
The following is an example of a patched system not vulnerable to CVE-2014-7169:
test@patched:~/empty$ X='() { function a a>\' bash -c vulnerable bash: vulnerable: command not found test@patched:~/empty$ ls test@patched:~/empty$
Note the file named “vulnerable” in the newly created, empty directory. When the same payload is ran on a patched system, this file is not created and the directory remains empty.
If you test vulnerable to either test, please update your bash packages ASAP. If you’re a ServerCentral customer, please contact us should you need assistance or advice.
How Something like This Gets Handled Behind the Scenes at An IaaS Provider
| 09/24 10:30 AM: | See a bash vulnerability announced, it might be big but very little information is immediately known. These notices come across dozens of times each day, but each much be evaluated to see if it applies to our environment, and how severe an impact if so. Bash is certainly ubiquitous but generally an attacker must have authenticated system access in order to exploit it. Sounds nasty, but since it most likely requires an authenticated user to actively exploit this reduces the attack surface considerably. Make note to watch it more closely, prepare for 11am meeting. |
| 09/24 12:30 PM: | Check on it at various security forums and industry news sites. Start to realize this is going to be big, and get creative in thinking what may be exploitable by this bug. |
| 09/24 1:00 PM: | Local public mirrors server has been force-updated to latest released patches on all available Operating Systems. |
| 09/24 1:20 PM: | Industry peers in technical roles at our customers, vendors, and competitors begin exchanging information and testing tools. New attack vectors are identified in real-time and the realization that this exploit is extremely serious begins to set in. |
| 09/24 1:30 PM: | All-hands-on deck escalation to update all externally facing systems – regardless of if we knew specific applications were vulnerable or not. Puppet utilized to automatically deploy updates on large server farms, and forced-updates applied on all individual application clusters. |
| 9/24 2:30 PM: | All servers updated, more in-depth internal auditing begins and systems updated as deemed appropriate. |
| 9/25 10:30 AM: | Read news that the initially deployed patch is incomplete, confirming rumors spreading overnight. While the initial patch fixed the most obviously exploitable portion of the code, there remain functions which are still vulnerable in some situations. Fix seems to be more complex than the first, and is some time out. |
| 9/25 10:00 PM: | Updates to most worldwide mirrors start hitting for CVE-2014-7169. |
| 9/25 10:30 PM: | Public mirrors server is synced with all major OS distributions. Internal departments notified that systems must be immediately updated. |
| 9/26 11:00 PM: | External-facing services updated, updates ongoing for internal equipment. Puppet updates already installed on large clusters. |
| 9/27 11:30 AM: | All departments confirm updates applied and testing clean. Ongoing internal assessment for embedded systems. |
Lessons Learned
- Treat public mirror repositories as front-line services. Current update methods are too slow to pick up changes (on both our mirror server and on many others worldwide) to mainline repositories. Fixes for this should be implemented both on the server and client side for robustness.
- Automation is good. Continue implementing in more areas.
- Formal inter-departmental notification processes can speed up response times.
While this issue is still ongoing, some general conclusions can start to be drawn at this time. The scope of long-tail software that may in some way use bash in an insecure manner has yet to be known, and we will likely see exploitation of this flaw for years into the future as new holes get exposed.
The most prudent thing to do on any machine that has bash installed is to ensure bash is patched to the most up to date version. While some combinations of application and OS are less vulnerable than others, it is too early to be sure of anything. The safest course of action is to immediately patch all vulnerable bash installations.
Conclusion
Each major exploit is a major opportunity for evaluating your organizational and technical capacity to address major issues in a safe and timely manner. This vulnerability is no different, and we expect to continue learning moving forward.
