Tools and Utilities
Virtualization Tools and Utilities
As I received many positive reactions on my Analog Alarm Box to monitor your VMware vSphere environment, I decided for the first “finalized” product to not include the split-flip display, as I still need to work out a lot of things to be able to “mass” produce it. The slim box will have 4 analog meters and you will be able to decide yourself what each meter displays. Like 4x the CPU usage of 4 different clusters or the CPU and Memory Usage of 2 clusters, storage consumption, or what ever.
I am adding a 2 line 16 chars LCD display in the box, that can show the names of the meters that are being displayed and can also display alarm notifications. The box will get a buzzer, for when an alarm condition is met. You will be able to configure and setup the box using the build-in webserver.
I will make easy kits that people will be able to assemble that will contain everything but the Raspberry PI board, including a breakout board will all the electronics that can just be put on the header pins or the Raspberry PI, 4x the analog meters, the LCD display, buttons and the wooden box itself.
I know this is supposed to be a weekend project, but I had some spare time on my Thursday evening So here an update on my vSphere Alarm Box.
I received my analog panel meters this week. Instead of them displaying 0 to 5v I created some new face plates for them using my automatic paper cutter so they can display CPU and Memory consumption. These meters are very easy to control by using a PWM signal, but the challenge is that the Raspberry PI only has one PWM port. So I decided I want to use an arduino for this. Making the Raspberry PI talk to the arduino is fairly straight foward. I am using I2C, but as the arduino uses 5v for its logic circuits and the Raspberry 3.3v I had to build a small level shifter to allow both both safely to communicate over I2C.
Everything now connected hardware wise, now I just had to make the meters work, displaying the right information from VC. I am writing my code in PHP and already had last weekend written the PHP code to use the QueryPerf API call to retrieve the last 5 minute average for CPU and memory of my cluster. But PHP does not allow me to send I2C calls, so I had to write an extension module for PHP to enable me to do this. As I never had written an PHP module, thanks to google, this did not take too long
So there it is.. IT IS WORKING Every minute Raspberry PI uses QueryPerf to retrieve information from VC. Using my PHP I2C extension it sends an I2C call, via the level shifter to my arduino. The arduino uses the value received via I2C to set the PWM for the specific pins and tha dah… working
As I am happy with the progress my little project is taking, I thought I would share with you all what I am working on. I like to make physical things and one of the things I have been developing is and old-school split-flap display (You know from those boards in old train stations). Some day I like to make 140 units, so I can have a live twitter board in my living room, but for now I have settled with one
I needed a purpose for my display… So I thought I would make a vSphere / Datacenter Alarm Box, based on some good old-school components. At first I wanted to use my good trusted Arduino, but unfortunately it does not support HTTPS/SSL (not powerful cpu to deal with the encryption). And if I want to make an alarm box for the vSphere environment, it need to at least be able to get all kinds of info from your virtual center server. So I ordered an Raspberry PI, a new open hardware platform like the Arduino, but based on a much more powerful processor and running linux (and all this for $30!).
After some tinkering around this weekend, I was able to get my Raspberry PI to talk to Virtual Center using the official APIs So I can now get alarms, check for other things like retrieve the CPU and Memory Usage of the entire cluster (last 5 minute average). So besides using only my split-flap display, I decided to also build 2 analog panel meters in my box, as these measure 0 to 5volt, I can easily control them just using PWM, so they can show (in percentage) at any time the CPU and Memory consumption of your Cluster.
I am not 100% finished yet, but all the components are starting to fall in place So I thought I would share with you and of course if you have any great ideas about what this Alarm box should monitor/detect, please let me know. I am also putting in a 7-segment display, to show the amount of running Virtual Machines.
Here a video of the split-flap display part I have build for testing.
Let me know if you like this project and if you maybe someday want an Alarm box like this, as I can consider making a few.
I plan on making the face plate from aluminum or really nice oiled wood. Their will be a serine on top of the box, that can go off on certain events you selected, with just the light and/or sound going off. So they box will be completely stand-alone (NO PC required) and will be about $75 in parts. Just plug in power and ethernet, it will run a webserver that you can connect to to configure it.
to be continued…
I thought I would share with you a simple way to copy a VM from your ESXi servers to your desktop. In my case, my local NAS (qnap) device had a diskfailure, so it is rebuilding its disks for the next 2 days, which slows down IO for my VMs (mainly for my work VM), so I temporary wanted to copy the VM to my local PC and run it in workstation.
Using the build-in download option in the vsphere client sucks. It was super slow. I tried using sftp, again slow (bit faster then vsphere client). My first tought was to google “enable ftp server on esxi”, which I did, and did find ways to get an ftp server installed on my ESXi boxes, but while I opened the ESXi firewall, it still had problems opening a data connection. So after 15 minutes fussing with that I quit that road.
In the forums about FTP on ESXi I came across posts to use Veeam FastSCP, so let’s give that a try. Turns out that the software does not exists anymore and is integrated in a 500mb install of Veeam Backup. I have fast internet, so sure I can download then, but then during the install.. “do you want to install local sql server or use existing??” Hello, I am just trying to copy a single VM, I do not want to screw up my local machine with all that stuff, so cancelled that as well.
Mmm.. what else to do… went googling again and then stumbled on a Freeware bit of software called FreeNFS (http://freenfs.sourceforge.net/) It seems very recent as the note from the developer is from August 14th 2012. It is the most simplest plain NFS server software for windows. its a single executable, click on it, and it runs. done
So now it was easy. As my local machine suddenly was an NFS server, I add the NFS datastore to my servers and just used a clone VM operation, to thin disk, to my local datastore
Thank you Lawrence for making this great Free NFS software! Works great with ESXi (version 5).
FreeNFS – http://freenfs.sourceforge.net/
Well I started work in my new version of vAudit, making more functionality and most important support for View 5. In case you do not know what vAudit is, it is mainly a tool to understand who and when is using your View environment. This can help you see how well the adoption of your VDI systems is going, when not to plan maintenance, etc.
In the old version of vAudit I used WMI to query the event logs of the brokers to see who was logging in and out. Since View 4 the products comes with the option to store all events in an events database (Microsoft SQL or Oracle). So for this new version this is what I want to use. Unfortunately VMware’s View engineers are not easily storing the event data (and this is an understatement!). So it took me a while to even get some basic information out of the system. As my vAudit is not ready for release yet, I thought I would at least share some SQL statements with you, in case you want to start cracking
Checking daily max concurrent users.
Well this is the easiest (and the only easy thing) to do. As the broker service daily writes an event 5 minutes to midnight in the events database on that days max concurrent user. You can easily get this using a query like this: (MS SQL)
select Count, Time from(select top 30 dbo.view_event_data_historical.IntValue as 'Count', dbo.view_event_historical.Time as 'Time' from dbo.view_event_historical,dbo.view_event_data_historical where dbo.view_event_historical.EventID = dbo.view_event_data_historical.EventID and dbo.view_event_data_historical.Name = 'UserCount' and dbo.view_event_historical.EventType='BROKER_DAILY_MAX_USERS' order by dbo.view_event_historical.Time DESC) A Order by Time
This might seem like an extensive way of doing this, but it will allow you to control the last so many days to query. Just change the ‘top 30′ to any amount of days.
With this data you can chart a nice daily max concurrent user overview. See my screenshot of the new vAudit release. (I did alter the data in the database to make it look like I have a lot of sessions).
NOTE: also on the name of the tables. When you install view, it asks for a table prefix. In my case I supplied “view” as prefix. So make sure to check your prefix and modify the table names based on that.
If you want to make more sense of the sessions used by your users, it becomes more of a challenge. Each time a users logins to the broker an events is written. The same for when they session is broken/loggedout. In the event table these events are not easily connected so you would have to query for each the event_data table as well to get the broker session ID and match them all up. The super irritating thing is that the time stamp is NOT in the event_data table but just in the events table. This would else have made it into a very easy query. So after a lot of googling, cursing and wishing I wan an SQL expert, here is what I came up with.
drop table #sessions drop table #logoffs drop table #users create table #sessions (SessionID varChar(32), StartSession datetime, EventID int) insert into #sessions (SessionID, StartSession, EventID) select view_event_data_historical.StrValue, view_event_historical.Time, view_event_data_historical.EventID from view_event_historical, view_event_data_historical where view_event_historical.EventID = view_event_data_historical.EventID and view_event_historical.EventType = 'BROKER_USERLOGGEDIN' and view_event_data_historical.Name = 'BrokerSessionId' create table #users (EventID int, username VarChar(512)) insert into #users (EventID, username) select view_event_data_historical.EventID, view_event_data_historical.StrValue from view_event_data_historical, view_event_historical where view_event_historical.EventID = view_event_data_historical.EventID and view_event_historical.EventType = 'BROKER_USERLOGGEDIN' and view_event_data_historical.Name = 'UserDisplayName' create table #logoffs (SessionID varChar(32), EventID int, EndSession datetime) insert into #logoffs (SessionID, EventID, EndSession)select view_event_data_historical.StrValue, view_event_historical.EventID, view_event_historical.Time from view_event_historical, view_event_data_historical where view_event_historical.EventID = view_event_data_historical.EventID and view_event_historical.EventType = 'BROKER_USERLOGGEDOUT' and view_event_data_historical.Name = 'BrokerSessionId' select #sessions.SessionID, #sessions.StartSession, #logoffs.EndSession, #users.username from #logoffs, #users, #sessions where #logoffs.SessionID = #sessions.SessionID and #users.EventID = #sessions.EventID
The query uses some temporary tables to help match the session ID’s, get the Start time from one event, the end time of an other event and get the displayname of the user. See the picture of my result.
Next step would be to use this data to make daily charts on when most uses are logged in and out, users stats like average session time, etc. So more to come
Just to extra clarify what this query does, it check sessions to the Broker! not to the actual VMs. You can do this as well, but even more complicated (as those do not have unique IDs like BrokerSessionID). The the list shows people logging into the broker, but you will NOT know if they started one, two or more virtual machine sessions.
If you have any SQL work for View you want to share, please do! If you can optimize my lame-ass sql queries, please do
I will hopefully soon be posting a beta of the next vAudit that does all this and translates it into nice charts.
2 Weeks ago I attended the Blackhat security event in Barcelona and got me thinking about security around the VMware platform. At the show there was even a session about vulnerabilities in virtualized environments. Happy to hear there where no major leaks in the ESX layer, it still got me thinking. What is someone right now is trying to hack into my ESX hosts or my vCenter environment, would I know??
During blackhat I also saw a tool demonstrated that I never had heard of. It is called OSSEC and is made by Trend, but is free to use by anyone. OSSEC is designed to collect centrally all the logs from servers, including ESX, ESXi and Windows (vCenter) servers and analyze the entries to raise alerts when it detects something wrong. For instance if someone was not successful in logging in. It can even increase the alert when it notices if multiple failed login attempts happen in a short time span; this could indicate that a bruteforce attack is taking place.
I found that OSSEC worked quite simple in defining rules and was even more surprised that it came with default build-in VMware rules. unfortunately designed I think for 4.0 as I had to make some small changes for my 4.1 environment to accomodate VMware’s updated log syntax.
OSSEC’s success depends on the quality of the rule set. So I am really hoping more people with real world VMware environments have a look at this and share their experiences against what would make good rules. To make this a bit easier, I have written a simple ‘manual’ on how to install ossec and configure ESXi host and vCenter servers to remote log against the OSSEC server. You can find the instructions here: http://www.run-virtual.com/?page_id=690
You might wonder (like I did), who cares? Should the VMware management environment not be on its own isolated network? Well I found out that in the real world this is not the case. Last week I did a workshop around security with 15 customers and I think all had an environment that any desktop in their company could access their vCenter and/or ESX hosts. So if any of these clients would get infected by some kid of VMware Virus, it could attack!
After hearing that so many people had their VMware environment ‘so’ exposed, I wondered. Are there any ESX / vCenter servers directly attached to the internet? So the hacker in me decided to spend an evening building a ‘vmware scanner’. I made it fully multi threading (about 800 threads per program) and spinned up 5 virtual machines running this scanner. This allowed me to scan roughly 4000 IP addresses every 30 seconds (max winsocket timeout)… The scanner was build using the VMware APIs, so it could not only detect if an IP was a VMware server, but also what version and build number. Yes, requesting this information can be done ‘anonymously’ with no special permissions.
… 24 hours after scanning and suffering from having bad internet at home because all the scanner bots where eating up my outgoing bandwidth I was completely shocked. Each of my scanner bot had long lists of ESX, ESXi and even vCenter server IP addresses, including some very old versions (which I wonder if they have them fully patched).
Well I do hope that none of your VMware boxes are directly internet attached, but the real world has shown that most people do have their VMware servers attached to their entire internal network. I understand the practical reason behind this, but you should then also make sure you have the right measurements in place to detect any attacks.
I would suggest, tryout OSSEC, let me know what you think and let’s share some good rule sets, so we all can detect the instant something ‘strange’ is going on, from security attacks to hardware failures.
The OSSEC/VMware install manual: http://www.run-virtual.com/?page_id=690
Ever need to change the size of your Virtual Machines? Well, changing the size is fairly easy, but what about the data and partitions inside your Virtual Harddisks? Paragon has made their “partion manager 10 for Virtual Machines” available for free! (well at least the first 5000 copies). Besides changing partitions, it can also do other handy tricks like backups and merging partitions. The product supports the ‘new’ GPT partitions that operating systems like Windows 7 and OSX uses.
Get here your free copy
Partition Manager 10 for Virtual Machines product page
Happy New Year to you all! CES just started and IOMEGA, now like VMware an EMC company just released a new product called v.Clone. It’s simple, take any IOMAGE external harddisk and you can clone your PC and later on keep it insync (2way), take the external disk anywhere else and with the embedded vmware technology (a custom version of VMware Player) you can run your PC on anyone’s system! Wow nice.
There are some catches, the V.clone software will only work with IOMEGA harddrives and you will need local admin rights on the computer where you want to run the virtual machine. Win7, Vista and XP are supported.
According to IOMEGA, v.Clone is the first product designed for individuals to move seamlessly back and forth between a physical environment (home or office PC) and a virtual environment (Iomega Hard Drive connected to any PC). This allows the user to have a complete, usable virtual copy of their physical PC with them at all times. v.Clone keeps your data in sync between your primary and secondary computer so your virtual environment is ready to go at a moment’s notice.
No pricing information yet, but IOMEGA claims it will be coming in January.
I just finished an updated version of vAudit. In case you do not know what vAudit is, it is an utility that can connect to your VMware Connection Broker and report about user activity. This new version includes 2 major updates; 1. the program can be resized 2. the information can be exported to a comma seperated file.
You can find vAudit 1.5 on the download page
Thanks to Jeff and Sudharsan for your feedback! I just made a new version available of vAudit with 3 new improvements:
- vAudit now also checks for login failures, so you can detect if people are trying to hack into your system
- You can now resize the username column, so you can actually read the username if you have long domain names
- When you MouseOver a session, it will display the machine name and time information
In the next release, I will try to add information about your top users, make an export function and at some point will give your a graph with daily concurrent usage.
You can download vAudit 1.00 here