I've recently released a tool called vm-truck-loader to automate virtual machine creation with VMware vCenter. Using a simple CSV file, and leveraging the API exposed by vCenter, this Java based command line tool enables you to design a fully automated deployment process, and can be a key to implement a proper Standard Operating Environment around vCenter. In this regard, I thought I'd do a quick entry on this blog, to briefly describe what the tool can do.
Instrumenting virtual machine creation
What is SOE ?
Generally, to achieve that, people use some template base system. For most of our customers, this is a build or base image of RHEL. Some go even further, having the creation of this base system automated. Anyhow, this base system is just a blank page, a ready to run server lacking:If you wish to build a proper standard operating environment - and who would not ;) - you need to be able to spawn a new system very easily. In an ideal world, creating a machine from start should be so easy, that it should be easier than fixing a problem on a running system. (of course, the issue may lie in the build of the machine, but that's not the point here !).
- recent software update and security fixes
- any kind of application specific software
It's more or less a Linux system with 'vi' and a 'sshd' server running on it. But, on first boot, it runs kickstart, which takes care of fixing the first issue - i.e. updating the system, and installing all the software you want to have on any of your servers. For the ones who have been one step further down this road, the kickstart main purpose is to install the Puppet client agent (or the Chef agent, CFengine client, etc).
Once the client is installed, Puppet (or equivalent) will take care of setting up the host properly and, more importantly, automatically, consistently, and reliablely. Then you can start deploying applications or services, such as databases or web servers, on the running system.
This is just to give a quick overview of what I mean here by standard operating environment (or SOE). As one can imagine, the focus here is on having a fully automated chain of software process. The automation is required to spawn a new, ready to be used, system, reliably.
All that I have described up until now is not at all tied or specific to virtualizing solutions (and even works for bare metal). So what does all of this have to do with this newly released project and virtual machines from VMware vCenter?
Pushing the envelope
If you are running on top of virtualized systems, it's most likely because you want to be flexible, and easily reallocate and add resources. And if you have implemented a strong SOE, as described above, this will pay off, in sorts, that people will ask you to set up more and more virtual machines because the setup is flexible and resilient. And here comes the point of VM Truckloader.
As I use 'mutt' to read my email, and I practically never use my mouse for anything, you can imagine how I react to such a situation. But more importantly, each 'click' increases the chance of human error. And if something goes wrong, and you have to destroy and recreate all the VMs, you'll again have to click your way out of it...If you are using vCenter, you have a UI to create and manage your virtual machine(s), but when it comes to, let's say, order the creation of 10 or 20 virtual machines, each with a different name, along with a proper FQDN and other specific settings, you have a lot of clicks to go through.
Of course, vCenter offers a rather feature complete remote API to instrument everything, but it's SOAP based (if I'm not mistaken) and therefore not that easy to instrument using 'curl' or other classic admin scripting tools. There is an Open Source Java API for this, called vijava and used by VM Truckloader, but Java is rarely a language easily used by admin people, and the API itself clearly requires a good knowledge of both the Managed Object Reference and Java to be used.
This is why I recently built - for a customer kind enough to agree to release it as an Open Source tool, a small Java command line tool, that allows one to easily instrument the creation of a large set of virtual machines, each with some specific and distinct data and parameter (name, number of CPUs, and so on). The information is simply contained in a CSV file, one line per machine, and VM truckloader will take care of creating each virtual machine in vCenter - based on a template that provides the basic VM.
At the end of the process, the tool will update the CSV file to add some generated information (such as the new host MAC address), so that you can keep track of those. Also, it can be configured to fire up an external script, executed after each VM creation, and pass all the information available in the updated CSV line to the script!
This "post install script" is crucial, as it can finish the preparation required before firing up the new system. For instance:
- associating the MAC address with the appropriate FQDN in a DNS server
- registering the host in the monitoring system - whether this is Nagios, Icinga or Shinken (or RHQ)
- and so on...
The tool can even be used to batch some operation on all the virtual machines specified in the CSV file - such as START, STOP or RESTART. Once the external script has finished its job, you can just call VM Truckloader again to fire up all your newly, freshly created systems. From this point, Kickstart and then Puppet will take care of the rest.
So I hope that by now you have understood why I developed and released this tool, and why I find it crucial in the setup of a proper standard operating environment with VMware vCenter. As it only requires a simple CSV file - which can be easily generated from a database or simple spreadsheet, I dare to think it's generic enough to be useful in many places. And, because it's a command line tool, it can be easily integrated in to a script, and run behind a simple web form to order new systems (for instance) or embedded in a more complex Java application to instrument your virtual data center.
What about Red Hat CloudForms?
It's important to note that if you are using CloudForms, you can instrument all the virtual machines from it directly, and more likely make this tool pretty useless to you. :) However, if your manager has not yet agreed to using CloudForms, you can certainly have a taste of what CloudForms could bring you, in terms of automation, using VM Truckloader ;) !
Using VM Truckloader with Satelitte (RHNS)
If you are running Satellite (RHNS) to provision your vCenter system, you'll probably want to have VM Truckloader interact with it. For instance, when you create a VM, you will want to register this newly created system into the Cobbler instance used as CMDB by RHNS.
Doing this is fairly easy, thanks to the capacity of Truckloader to execute a script after executing any command. As the script will be invoked with all the information on the machine passed as an argument, it will be simple to build the appropriate command:
#!/bin/bash
#
if [ -z "${TEMPLATE}" ]; then
echo "Template name has not been defined."
exit 1
fi
readonly HOSTNAME=${2}
readonly MAC=${4}
if [ -z "${HOSTNAME} ]; then
echo "No hostname provided"
exit 2
fi
if [ -z "${MAC}" ]; then
echo "No MAC address provided."
exit 3
fi
cobbler system add --name="${hostname}" --profile="${TEMPLATE}" --mac="${MAC}"
Using VM Truckloader with OpenShift
If you are using OpenShift on top of vCenter, VM Truckloader may also come in quite handy to automate node creation and interact with essential parts of the OpenShift infrastructure such as the DNS server. Indeed, to scale up the system, an automated process (maybe running on the broker?) could trigger VM Truckloader to create new virtual machines. Those new instances could, on first boot, register with the broker.
Go for it !
All of that to say - please go and download the first release, Blue Thunder, of VM Truckloader (releases), and let me know what you think of it!
Links
Truck images are taken from the Open Clipart project.
Last updated: February 26, 2024