This week VMware was in the negative limelight because of hacker Hardcore Charlie (clearly a hacker who watched to much Snoopy in his live). He claims to have stolen VMware ESX source code, not directly from VMware (that would have been really bad) but from the China Electronics Import & Export Corporation (CEIEC).
So far he only released publicly some 300MB of the source code, but he is saying he will release all on may 5th.
If source code of ESX and vSphere is out in the open, this of course can become a serious problem as it would make it much easier for hackers to figure out where the weak spots are. (the image supposedly is proof from mr Hardcore charlie that he has the code).
I wrote a while back an article about how many people have their ESX and vCenter boxes directly attached to the internet… you might want to reconsider implementing a firewall in between
Read more about the ‘hack’ in information week.
or visit my article about unsafe placement of ESX servers
A while back I wrote an utility to scan for VMware servers using the VMware API. I was curious how many people would have their ESX(i) servers directly attached to the internet. Shockingly I found many I have received quite a few emails in the last few weeks if I wanted to share the program, so IT Admin can check their network for ‘rogue’ vmware servers in their environments. Well here it is I did some bug fixes and the program now seems to run fine. How does it work? It first checks if a server responds to port 443 (ssl), if so, it send a VMware API call to the server to query the product name, version and build number (this does NOT require any credentials), if a server reponds back this is shown and written to a file. To speed up the scanning process, I made the software highly multi threaded. It can spin up up to 750 parallel threads for scanning. I would only recommend these high thread counts on windows server operating systems, found that a desktop windows OS becomes unstable above 500 threads.
So here a great challenge for you!
1. Download the VMware scanner
2. sit down behind any normal desktop in your corporate environment
3. scan your entire network
find any vmware servers? Then start worrying!!! why would you connect your management interface on your easily accessibly network???? Do not come complaining to me that your servers got hacked
Download the VMware scanner on the main download page
I really hope 2011 will be the year the security industry will design their solutions specific for the virtualization platform. And we are good on track. HP released their tipping point solution for VMware, Trend Micro has their antivirus solution for vShield End point called Deep Security 7.5 and now (today) kaspersky announced that they will also support vShield End Point, meaning that they will also release this year an Anti-Malware solution designed to run via the virtualization layer, so that you do not have to install anti virtus software inside your VMs any more.
“Kaspersky Lab is truly excited to support VMware vShield for our mutual customers. Our support of VMware vShield enables our clients to securely maintain state-of-the-art, multi-layered anti-malware strategies across virtual environments and heterogeneous IT infrastructures, with a level of performance and Total Cost of Protection that truly meets their business needs,” commented Nikolay Grebennikov, Chief Technology Officer at Kaspersky Lab.
No information yet on pricing or an actual shipping date. The kaspersky folks are saying it is hard to get commitment from the russian programmers but the company does see it as one of their top priorities.
Read more about kaspersky’s announcement here.
Trend Micro, Kaspersky…. where is the rest? I know sophos has done technical previews of their solution as well, so hopefully they will soon come with a product to market. Here at the infosecurity show in the UK, McAfee is showing off a solution with Citrix’s virtualization platform, Symantec has been quite, but have also claimed they are working on a solution. With a bit of luck, by the end of 2011 all major anti-virus / malware companies should have a product designed for virtualization.. Or would that be a too big request for Santa
After the release 2 weeks ago of the vSphere hardening guide for 4.1, you might feel overwhelmed with the information in the hardening guide. To help you understand how ‘bad’ your systems are configured compared to the hardening guide, VMware released today a free Compliance Checker for vSphere
So of course I had to run the easy-to-install (just requires java) tool against my vSphere environment and had to find out I am a bad admin, well at least from a security perspective. The compliance checker does not only check the config of the ESX boxes but also the .vmx parameters of the individual VMs. Unfortunately the tool is not smart enough to show which VMs are lacking in security, it just shows that 50% or 3 out of 6 VMs are not ‘compliant’. I guess you have to figure out yourself which of the 3 VMs are correctly configures and which are not
you can download the compliance checker here: http://www.vmware.com/products/datacenter-virtualization/vsphere-compliance-checker/overview.html
Not everyone seems to be aware, or take security seriously enough , but VMware does release hardening guides for vSphere. The vSphere 4.1 guide was a while in the making and getting feedback from the community, but yesterday VMware released the official version.
The guide covers topics from VMX parameters (special VM configuration settings), ESX host settings, vCenter setup and Virtual Networking guidelines.
One of the topics covered in the guide is ESXi logging (like I discussed about in my post from yesterday):
Another point is that, by default, the logs on ESXi are stored only in the in-memory file system. They are lost upon reboot, and only one day’s worth of logs are stored. Persistent logging to a datastore can be configured. It is recommended that this be done so that a dedicated record of server activity is available for that host.
Reading thru the Hardening Guide does give me the feeling it requires a lot of manual tuning As I only have 2 hosts in my environment, so not a big deal. But I do feel sorry for some of you with 100+ hosts. Maybe someone should write a cool powershell script for this??? (any volunteers?). Update: just got feedback that a perl script does excist and can help you apply the hardening guide. I have not tested it myself, but you can find it here: http://communities.vmware.com/docs/DOC-11901
Eric sloof just posted an video about VMinformer, which is an automated tool to apply the security hardening guidelines. It is a normal windows application and looking at the video seems to work easy. VMinformer is not a free tool, when I run their TCO calculator it claims the licensing cost for 100 hosts is $27600 USD. It does save a huge amount of manual labor, but I do find the cost quite high, especially as this license cost is per annum.
More info about VMinformer: http://www.vminformer.com/products/vmi-professional/
Doing it your-self with help, using powershell
Well my question about anyone making a powershell script for this, did give me some replies I understand a new book is about to be released called: VMware VSphere PowerCLI Reference: Automating VSphere Administration. Chapter 12 in this book dedicated to Hardening the vSphere Environment. As the book is not published yet, I am not sure how much details if will cover, but might be worth a read
More about the book: http://www.powerclibook.com/
2 Weeks ago I attended the Blackhat security event in Barcelona and got me thinking about security around the VMware platform. At the show there was even a session about vulnerabilities in virtualized environments. Happy to hear there where no major leaks in the ESX layer, it still got me thinking. What is someone right now is trying to hack into my ESX hosts or my vCenter environment, would I know??
During blackhat I also saw a tool demonstrated that I never had heard of. It is called OSSEC and is made by Trend, but is free to use by anyone. OSSEC is designed to collect centrally all the logs from servers, including ESX, ESXi and Windows (vCenter) servers and analyze the entries to raise alerts when it detects something wrong. For instance if someone was not successful in logging in. It can even increase the alert when it notices if multiple failed login attempts happen in a short time span; this could indicate that a bruteforce attack is taking place.
I found that OSSEC worked quite simple in defining rules and was even more surprised that it came with default build-in VMware rules. unfortunately designed I think for 4.0 as I had to make some small changes for my 4.1 environment to accomodate VMware’s updated log syntax.
OSSEC’s success depends on the quality of the rule set. So I am really hoping more people with real world VMware environments have a look at this and share their experiences against what would make good rules. To make this a bit easier, I have written a simple ‘manual’ on how to install ossec and configure ESXi host and vCenter servers to remote log against the OSSEC server. You can find the instructions here: http://www.run-virtual.com/?page_id=690
You might wonder (like I did), who cares? Should the VMware management environment not be on its own isolated network? Well I found out that in the real world this is not the case. Last week I did a workshop around security with 15 customers and I think all had an environment that any desktop in their company could access their vCenter and/or ESX hosts. So if any of these clients would get infected by some kid of VMware Virus, it could attack!
After hearing that so many people had their VMware environment ‘so’ exposed, I wondered. Are there any ESX / vCenter servers directly attached to the internet? So the hacker in me decided to spend an evening building a ‘vmware scanner’. I made it fully multi threading (about 800 threads per program) and spinned up 5 virtual machines running this scanner. This allowed me to scan roughly 4000 IP addresses every 30 seconds (max winsocket timeout)… The scanner was build using the VMware APIs, so it could not only detect if an IP was a VMware server, but also what version and build number. Yes, requesting this information can be done ‘anonymously’ with no special permissions.
… 24 hours after scanning and suffering from having bad internet at home because all the scanner bots where eating up my outgoing bandwidth I was completely shocked. Each of my scanner bot had long lists of ESX, ESXi and even vCenter server IP addresses, including some very old versions (which I wonder if they have them fully patched).
Well I do hope that none of your VMware boxes are directly internet attached, but the real world has shown that most people do have their VMware servers attached to their entire internal network. I understand the practical reason behind this, but you should then also make sure you have the right measurements in place to detect any attacks.
I would suggest, tryout OSSEC, let me know what you think and let’s share some good rule sets, so we all can detect the instant something ‘strange’ is going on, from security attacks to hardware failures.
The OSSEC/VMware install manual: http://www.run-virtual.com/?page_id=690
Since the release of vSphere 4.1 a new API for security is available called EPSEC (End Point Security). Together with the already existing VMSafe API security vendors can make security products designed for virtualized worlds.
You might ask, why do security products need a special virtualization approach? Well of course you do not have to, but certain things will not perform so well. Especially if you talk about desktop virtualization. Imagine running 50+ desktop VMs on a single server and all the virus scanners inside those operating systems decide to do their daily scan at the same time. So Trend and Symantec already have ‘random’ start features in their products to prevent this kind of behavior. But can it be done even more efficient?
Yes! by using this new EPSec API. It allows security vendors to build anti virus/malware solutions in a single virtual appliance that can protect all VMs running on a single server, with NO AGENTS installed inside the VMs it self. Trend Micro is the first to have this to market with their currently available DeepSecurity 7.5 product. Symantec and McAfee will hopefully follow soon as well.
To proof that this new way of implementing AV protection is working efficiently, trend asked the Tolly group to research/benchmark this, and what a results did they found! The ‘old school’ scenario consumed 1.7 to 8.5 times more resources. This resulted in 29% to 275% improved workload density!
Especially if you are currently running a VDI environment or planning soon to implement one, I would highly recommend you read the tolly report and plan your Anti Virus/Malware strategy accordingly.
Since the introduction of VMware vSphere 4.0 a new API became available to the security world know as VMsafe. This new API can provide security vendors with an interface inside the virtualization stack where they can build there security solutions like firewalls, virus scanning, intruder detection systems, etc. Unlike the physical world, if would allow security products to be 100% isolated from the environment they are trying to protect while having a complete understanding of that same environment.
So what is happening with all the VMsafe stuff? I personally was hoping on many releases during VMworld in San Francisco, but this did not happen Just to give you some background on the VMsafe program, the security vendors have different ways to build security solutions on top of VMware. The new thing about VMsafe is that they can build a module that sits inside the VMkernel and inspect virtual machines, this is called fast-path. Besides fast-path, there is also a slow-path option, which is more a virtual machine approach, where the actual security product sits inside a Virtual Machine and can be information/copied about all the actions the other Virtual Machines take. Separately of the fast or slow patch concept, there is the VMware VDDK (also being seen as part of the VMsafe initiative, but has been available for longer). The VDDK is an disk API, that allows other programs to access a virtual machine’s harddisk like the VMware Consolidated Backup solution does. It does not matter is the VM is powered on of off, but a disk can just be ‘extra’ mounted to an other virtual machine that for instance runs a virus scanner. The clear downside of VDDK is that nothing is realtime.
So with that in mind, here a status update of what I know to date. (please feel free to correct me and add information).
Altor Networks, almost a neighbor from VMware in California, was Founded in 2007 by security and networking experts from Check Point Software, Cisco and Oracle, released their Altor VF 3.0 product at VMworld. Well almost, the product is finished and has been submitted to VMware for certification (a requirement that is part of the VMsafe program to ensure stability of the ESX server). Altor is using the VMsafe Fast-Path API to build their firewall, claiming that they can handle much more throughput because of this. Besides being a firewall it will also be able to segment Virtual Machines without having to change your networking layout and do intruder detection. The product will work on a distributed switch and with the Cisco Nexus 1000v virtual switch.
Reflex Systems, is like Altor Networks a company focused on security for the virtualized datacenter. They have a product called VMC (Virtualization Management Center) that offers security for VMware, Citrix and Microsoft. A new component in this security product is vTrust, which is their component that is based on VMsafe. vTrust provides Dynamic Policy Enforcement and Management, Virtual Segmentation, Virtual Quarantine and Virtual Networking Policies. It is unclear if this is done with fast or slow path, but their website states it is done at the VMkernel level, so I would assume they are using fast-path. I am not clear if this vTrust component is already released, the software manual on the site does not talk about this component yet, so I guess it still needs to be released (or their manual needs to be updated on their site).
Trend Micro, a more broader security company is developing 2 solutions based on the VMsafe program. The upcoming product (within 2 months or so) called Deep security will use VMsafe slow path (based in a virtual appliance) to do network security like firewall and intruder detection. The product that they have on the market today is called ‘Core Protection for virtual Machines‘ which uses the VDDK API to do offline virus scanning of powered on and off virtual machines.
More security companies to follow in next article…