By Doug Baer

Maybe the better question is, “Got Documentation?”  

These days, when I talk to people in IT, I hear a lot of requests for “automation.”  This buzzword isn’t nearly as sexy as “virtualization,” but it comes up in the same conversations fairly regularly, especially when the discussion inevitably turns towards cloud computing.  

automation. (2010). In Merriam-Webster Online Dictionary.
“the technique of making an apparatus, a process, or a system operate automatically.”  

Retrieved August 9, 2010, from http://www.merriam-webster.com/dictionary/automation  

Who wouldn’t want some of that?  Work happening automatically? That’s got to be all kinds of good. Normally, the conversation goes something like this, “We’ve got X people managing Y servers and workstations across Z sites.  All of this overhead and running around is killing us!  If there was just some way we could apply automation, we could probably save time and be able to do our jobs better.”  That sounds fairly reasonable to me.  Unfortunately, there seems to be a prevalent misconception that automation is as simple as buying some software or creating a script and running it periodically.   

Write it down!

Automation of certain tasks may require nothing more than a simple script, but I posit that anything more than one or two lines of code requires a plan.  I have written my fair share of both simple and complex scripts, and my experience leads me to believe that automation does not come in a box.  Furthermore, a documented process for performing the task must exist in order for the automation effort to have a chance at success.  That last bit is the gap many organizations struggle with since tribal knowledge still reigns supreme.  Do not despair, for all is not lost!  If someone is currently doing this work, these processes must exist in someone’s head.  They can be written down to create initial documentation as the first step of the automation.  

The “D” Word!

I know, I know, I used the “D” word that is anathema to IT staff everywhere: documentation.  Before you dismiss me as impractical and close your browser, give me a chance.  Perhaps you have heard this before and still don’t believe it.  I was there once.  I have scoffed, fought, tried, and learned.  In the end, I have come out better, wiser and happier.  You probably realize that once you take the time to document something in sufficient detail, you own it.  Talk about a scary proposition: you become the go-to guy (or girl) for that process and you may become the recipient of calls at all hours from your coworkers.  Consider how that can be a good thing, too – not the phone calls, the other part.  Questions from your coworkers occur because they recognize that you have taken the time to learn the process and are willing to share.  Questions often mean that they are using your documentation and can be considered for inclusion in your work as clarification or enhancement.  On the other hand, receiving questions that could be answered by reading your documentation can be frustrating.  At that point, it is time to introduce them to the documentation repository – teach a man to fish and all.  It isn’t your job to read the documentation to them.  

Something that took me awhile to realize is that inquiries about a process that you documented means that you are not being required to perform that work every time it is needed.  As time goes on, others will be able to take increasing responsibility in performing the process you designed and/or documented.  Trust me; you don’t want to be the only one who understands the process.  You may think that makes you irreplaceable, but it also creates a dependency.  Take it from someone who has fielded support calls while on the road during my vacation because I was that guy.  If you want that responsibility, it means that you can never be guaranteed a quiet evening, weekend, or vacation.  Not my idea of a good time.  

Assuming that you are interested or are otherwise still around to see where this is going, consider that documentation of a process is pretty much a functional specification for the automation components (scripts, compiled code, parameters, etc.) that you want to create or configure. Once the documentation is done, as Yoda would say, “one step closer to automation we are!”  

What does documentation have to do with automation?

Nonexistent, incomplete or inaccurate documentation makes automation difficult or impossible and normally results in a big, kludgy mess to troubleshoot and maintain.  That is, if you were fortunate enough to actually hack something functional together in the first place.  Don’t just start writing code – that comes later.  As Miracle Max says in The Princess Bride, “You rush a miracle man, you get rotten miracles.”  The same is true for automation.  Take your time and get everything together before you start.  Have a plan.  

I have a background in software development, but would not insult software development professionals by calling myself a developer.  As long as I can remember, I have had more of a focus on system administration than development, so scripting is the form of automation that I leverage most frequently.  When designing a new script, I utilize the existing documentation to walk the process I want to automate.  During this exercise, I log all steps and pay special attention to any requirements for authentication, authorization, and tools. Next is high-level pseudo-code and identification of command line tools or APIs that will be used.  

Scripting is automation

Scripting can be considered a form of automation because its purpose is to reduce the amount of manual work necessary.  A script gets its name from the performing arts discipline: a written version of a play or other dramatic composition; used in preparing for a performance.  Effectively, a set of instructions used to perform a specific task.  Does this sound like documentation to anyone else?  This also looks very similar to the definition of process: a particular course of action intended to achieve a result.  A script is very simply a processes documented in a language that may be executed to achieve the desired result.  That is not to say that scripts are self-documenting, merely that they are a structured, executable way of encapsulating the steps needed to perform a process.   

As a general rule, I will try to create a script if there is a task that I need to do more than once, frequently, or if there is a large possibility for errors.  If nothing else, it helps me gain a better understanding of the process.  Based on that experience, let’s consider some use cases for scripting:  

• I need to do something many times: adding 2000 users to 25 new security groups or mail distribution lists or updating the mailing address for 1100 users in Active Directory.
• I need to do something frequently (or for compliance purposes): ensure that all of my VMs have Cut/Copy/Paste disabled via the remote console.
• I have to do something that involves a tedious GUI and/or many manual steps: updating the values of many custom fields in a Microsoft Word document and refreshing the fields within the document to integrate the changes.

Taking the next step: Orchestration

The goal of automation is to reduce the number of manual steps required to perform a process.  In my view, automation is something that generally deals with a single system or process.  Once there is a need to coordinate activities between multiple systems or processes, we have entered the realm of orchestration.  Traditional definitions of orchestration lean towards the arrangement or harmonious organization of a musical composition.  There are obvious parallels here, and a more generic definition could take the following form: an arrangement of events to achieve a desired effect.   

The example I like to use here is that of a universal remote control.  Some of these remote controls allow recording of a series of steps so that a single button press can turn on my television, change the channel to my favorite one, and set the volume to my preferred level.  While definitely nifty, this is pretty much a simple script attached to a button.  That is a useful step in the right direction, but it could be so much more.  The better universal remote controls allow access to several devices simultaneously.  Extending the scripting concept to multiple devices makes this so much more interesting.  Now, I can press a button that turns on the TV, powers up the DVD player, switches the TV’s input to the DVD player, turns on the receiver and speaker system, adjusts its input to the TV/DVD player, adjusts the appropriate volume controls for my setup, and begins playing the disc; all of that with the press of a single button.  Now, it wouldn’t be catastrophic if some of those actions were performed the wrong order, but attempting to play the disc before the player was powered on would not achieve the desired result.  Sufficiently complicated orchestration involves both arrangement and synchronization.  Sharing of status information between components, perhaps via the orchestrator, may be required in order to ensure commands are not sent until the targeted device or system is ready to process them.  This just got a whole lot more complicated.

I recently attended a “cloud” event and staffed our booth there.  I am not a big fan of booth duty, but it provided me the opportunity to talk with several people regarding their interpretations of the great and powerful Cloud. The event was somewhat sparsely attended and a majority of the discussion occurred between various exhibitors, but there was quite a bit of interest and among those who did attend.

In the end, the mystery and reverence surrounding the Cloud reminded me of those little three-eyed green aliens in the Toy Story movies:

Buzz: Who’s in charge here?

Aliens: [pointing up] The Claw!

Alien #1: The Claw is our master.

Ailen #2: The Claw chooses who will go and who will stay.

…

Aliens: Do not fight the Claw…

The Cloud is coming and we need to trust the Cloud: it will solve all of our problems, controls our destiny and we need it to survive.  At least there is no shortage of hype here. 

If the corridor of vendors at the event was any indication of the industry as a whole, there is no shortage of vendors selling Cloud.  Many of these companies seem to be simply repackaging old solutions in shiny new marketing slicks containing this latest technology buzzword.  In these cases, customers get the same thing they would have gotten two years ago, but with a new name. I guess that’s one take on green IT: recycled technology solutions!  An example here is some hosting companies retooling their offerings to use virtual machines rather than physical rack units as billing increments.  This is a good place to start, but is hardly the place they want to be.

On the other hand, some entirely new concepts are surfacing.  In this arena, I noticed companies that outsource instant messaging, online meetings, and even help desk solutions.  The business case here is pretty much the same as the one for non-cloud-based outsourcing: moving some IT support to a third party lets businesses reduce IT spend and internal time spent maintaining servers and applications.  I agree with the concept, but believe the pitfalls of that outsourcing model still apply.  From the most basic perspective, Who has access to my data? Where is it? Who do I call for support? Are they qualified?

There are still others who want you to send them all you’ve got and let their team of ‘experts’ handle everything.  I don’t know of anyone yet who is willing to bet their entire business on a startup cloud provider, but some smaller shops are probably considering it to save costs.  Datacenter facilities, infrastructure equipment, and IT ‘experts’ are not cheap.  Add to that the cost and potential complexity of maintaining reliable backups for compliance and DR purposes and pitching everything into the cloud sounds pretty slick.

Would you like fries with that?

A cloud solution is a combination of technologies and processes rather than an object that can be packaged up and handed over.  I can’t give someone a cloud like I can give them a hamburger, and they surely can’t buy one off the shelf.  There is quite a bit of unknown territory between current state and cloud state.  I recommend that anyone planning to run business applications in a public cloud should take a look at an internal cloud first.  Elasticity and utility pricing are two main principles of ‘the cloud,’ but these require certain levels of standardization, automation, and documentation to be truly effective.  Are you there yet?

Specific benefits and drawbacks depend on which cloud type is chosen – these types are frequently described with the acronym ‘XaaS’ where the ‘X’ is ‘A’ for Application, ‘P’ for Platform, ‘I’ for Infrastructure, and so on for whatever is being provided ‘as a service’ from the cloud.  Regardless of the type, consider your environment’s current state, operational practices, maturity and direction to determine whether a compelling need exists.  To continue the analogy, perhaps a burrito or chicken sandwich is more appropriate than a hamburger right now. 

You need a plan to ensure success.  I encourage a cautious approach that takes into account your environment as well as the state of the technology being considered.  To that end, I submit the following along with my apologies to David Letterman:

Top 10 Reasons You are NOT Ready for the Cloud

10. You do not have a reliable, redundant network (or Internet) connection.

If you use Google Apps or Salesforce.com, two popular Software as a Service (SaaS) offerings, and your office loses its connection to the Internet, how much work can your staff do?  What would happen if your email server, file server or databases were in the cloud as well?  100% uptime of the cloud-based service does not mean anything if your users can’t access those systems. 

Ensure that you are not architecting less reliability into your solution than you have today by adding a dependency on a component with less resiliency than the current implementation.

Different implementations of cloud-based services have different requirements, but all of them require connectivity between the service hosting the resources and the resource consumer.  Dial-up networking is probably not going to cut it, and some of those wireless AirCards aren’t much better.

9. You have specific compliance requirements and don’t want to go to jail.

Who is responsible for the security of your data when it lives in the cloud?  Do you, your auditors/regulators and the provider all see this the same way?  When backup tapes fall off the back of a truck, someone takes the heat, but getting data off of tapes requires some effort.  Data hosted by virtual machines in a cloud environment is much more easily accessible (I’m thinking “silver platter”).  That, and it could potentially be shipped across or even out of the country depending on your provider’s infrastructure.  Is that acceptable for the types of data you are storing or manipulating in the cloud? Are you sure?

8. Your current workloads aren’t smart enough to take full advantage of cloud infrastructure.

As the infrastructure becomes more advanced and flexible, applications designed to run on physical hardware are falling behind: they’re just not built to take advantage of the new levels of flexibility.  Can your Exchange server request the provisioning of an additional Client Access server when it realizes there is a bottleneck?  Can it even realize that there is such a bottleneck?

Existing workloads may be moved into a cloud infrastructure, but all of the nifty features may not be available unless the applications have been updated to support them.  It may make more sense to wait until your applications have been upgraded, or maybe now is the time to look at alternatives that have already been developed to run in a cloud.  Some of the SaaS solutions are maturing nicely – check out Google Apps Premier or Zimbra for email and collaboration.

Not to spoil the surprise, but I conducted a few simple performance tests and performance was comparable in all three environments.  There are many people in the community who are much more qualified than I am to run detailed performance evaluations and I will defer to them in that regard.  I am not looking to stress any particular environment or unnecessarily showcase features of one that the others don’t have.  I am looking for usability and general performance of my Windows-based workloads. 

Regardless of the platform, a strong argument can be made that creation of a single virtual machine should be a straightforward process.  If it is not, you should seriouslycarefully consider the maturity and enterprise readiness of the platform.  Fortunately, each of these platforms supported a wizard-like method of creating a new virtual machine, attaching an ISO image, and installing the operating system inside the VM.  This process was similar on each platform, but vSphere and XenServer provided additional guidance by first asking which OS would be installed into the VM.  This helped the environment recommend sane and supported defaults for the virtual machine hardware, including number of virtual CPUs, amount of RAM, and SCSI/IDE configurations.  Once the OS is installed, the specialized drivers need to be loaded to provide the best user experience and performance: VMware Tools, XenServer Tools or Microsoft Hyper-V Integration Services.

In most production environments, there will be a need to create several VMs, and doing that by hand each time is so prehistoric.  With each platform, I created 10 VMs using the best method I could discern – if there are better methods, I’d love to hear ideas, but I .  Using XenCenter, I was able to select a powered-off VM, right-click it, and select Copy.  If I had enough foresight to sysprep the Windows image ahead of time, I could get a unique VM without too much additional work.  In the case of XenServer, there are additional deployment options available using the for-pay StorageLink integration and certain storage arrays (HP P4000, EMC Celerra, NetApp, etc).  Those are interesting, but not within the scope of my testing.

For vSphere and Hyper-V, I am able to clone VMs or define templates using the for-pay management products vCenter and SCVMM, respectively.  Using the free ESXi and Hyper-V hypervisors, however, I was left to more creative solutions.  In both cases, I used the management GUI to create 9 new virtual machines, then resorted to the command line to copy my base machine’s VMDK or VHD file into each new VM’s home directory.  Once I powered the new machines up and sysprep ran inside each one, I was good to go.  This, however, required a bit more intimate knowledge of the underlying OSes and filesystems than most people care to learn. 

Of note during this process, I learned that Hyper-V appears to thin provision its VHD files by default, vSphere asks each time, and XenServer fully allocates.  Also, using Hyper-V, VM creation must be performed via Failover Cluster Manager in order for the VM to be highly available, but disk attachment had to be done via Hyper-V Manager if the VHD was not attached at the time of VM creation.  I ended up creating 9 new VMs with empty disks, then replacing that VHD file with a copy of my base VHD.

It seems common practice to highlight the (currently unique) memory over-commit capability of vSphere, which makes it possible to pack more VMs per host.  So, a common question is, “How many VMs with 2 GB RAM can I create and run on a host containing 24 GB RAM?”  VMware vSphere allows the creation of a seemingly unlimited number of VMs and will swap to disk as needed. Hey, I asked how many would run, not how many would perform well!. With the Citrix and Microsoft offerings the answer is 11. 

XenCenter will prevent you from starting a VM on a host with insufficient memory and will automatically move the VM to another host within the Resource Pool to start it up.  If a Resource Pool is not setup, the following error shows up in the Event Log, as would be expected.

In Hyper-V, attempting to power up VM#12 from Failover Cluster Manager will transparently fail and move the VM to another host in the cluster with capacity to bring it online.  Simple, elegant, and unobtrusive – I like that.  However, attempting to start VM#12 from the Hyper-V Manager console, will fail with an error that insufficient system resources exist. 

Good information to know.  As a side note, I had forgotten how much RAM was installed in my host machines and had to figure out how to determine that information from a Windows Server Core install without simply creating VMs until I ran out of RAM and received the error above.  I apologize, but if you have seen the Disney movie Up, you’ll understand that this is my equivalent of, “SQUIRREL!”

With that in mind, I armed myself with 6 new HP BL460c G6 blade servers in an HP c7000 chassis, a few hundred gigabytes of LeftHand iSCSI storage, and the installation media for the hypervisors and their management tools. These ramblings should hardly be considered scientific, but I found inspiration and guidance in the excellent work produced by Chris Wolf of the Burton Group. Anyone serious about comparing virtualization platforms should check it out.

I am most familiar with the VMware vSphere product, and my installations of ESXi proceeded as usual and I added them to a vCenter server using the 60-day evaluation licenses. I have not had experience with XenServer since the Citrix acquisition, but did not run into anything unexpected with that installation either: pop in the install media, boot the host, answer a few questions, done. Within a workday, I was able to install and configure two each of my ESXi and XenServer hosts and present them with shared storage from the LeftHand.

With XenServer, I spent a little time getting the StorageLink and Workload Balancing (features up and running. Both of these were licensed via a trial version of Citrix Essentials for XenServer, which was fairly simple to obtain from the Citrix website. I was able to create a Windows 2003 VM on each platform and create several clones on shared storage that, with a little planning, came up nicely sysprepped and ready to go. I was even able to get vMotion and XenMotion working and had VMs moving back and forth between hosts of the same type. Keeping in mind that the VMware VMotion capability will go away when my trial runs out, Citrix gets points for having that available for use in the free version. More on that later.

The next day, I spent installing my first (and second) Hyper-V Server 2008 R2 software and learning about my options for management – see, since I chose to perform the install using the Hyper-V Server 2008 R2 media rather than a licensed version of Windows Server 2008, I received a Server Core install on the hardware and didn’t have a full Windows GUI to work with. I am comfortable at the command line, so I didn’t see this as a big deal. With R2, Microsoft provided some configuration assistance in the form of the sconfig.cmd script/utility that automatically launches on the Hyper-V server console when the host boots. This is nice and I give them points for keeping things simple. I would appreciate it if they added another option –one that would launch the iscsicpl.exe GUI so I can configure the iSCSI initiator without resorting to Google to figure out the name of that executable. I guess some people already know that stuff, prefer to use pure command line tools like iscsicli.exe to perform the configuration, don’t use shared storage, or simply use FC. I’m learning, so I expected stuff like this.

No big deal. I was able to enable the Failover Clustering feature using option 11 from sconfig and likewise used option 4 to open the firewall ports and enable remote MMC management of the Hyper-V host. Once that part was done, I was able to hit the host remotely with Disk Manager on my Windows desktop machine and configure the shared (iSCSI) storage. Awesome. Does anyone else find it strange that I can use RDP to get a GUI connection to a box that basically runs a command line? It took a few steps that I think are a little convoluted, but I had two machines ready to go with Hyper-V.

Then I hit a speed bump. In past lives, I have probably used cluster.exe more than most people, but I had not done so since Windows 2003 and I know a few changes have been made (no need for a cluster service account, spiffy new cluster shared volumes (CSV), etc.). Consequently, I was looking for a little GUI action to help me out. Can I get some cluadmin.exe? Not so much. Bummer.

With either VMware ESX or XenServer, I fire up my trusty web browser, point it at a host, and I can download the vSphere Client or XenCenter installer and can use those tools to manage the hosts. I tried that with my Hyper-V installations and got nothing. Hey, who wants a web server running on their hypervisor machine anyway? That’s just a security exploit waiting to happen.

So, Google me this… I discovered that there is a remote admin pack for Windows 7 that I could install, or I could manage my Hyper-V boxes by loading the Failover Clustering and Hyper-V Management roles on another Windows Server 2008 R2 machine. I have a 32-bit Windows XP desktop machine – I’m well aware that I need to upgrade and get out of 2001. The problem here is that I don’t have any Windows 7 and I also didn’t have an R2 install of Windows Server 2008 sitting around yet. For the record, the 2008 (non-R2) version of Failover Clustering refuses to configure Hyper-V R2 machines into a cluster (unrecognized version). Fine, I’ll cluster them later. Without spare hardware to use for a Windows 2008 R2 machine, and wanting to kick the tires on this new platform, I decided to create a Hyper-V VM, install Windows Server 2008 R2 into the VM, and use that to create my cluster.

Once I had my new VM up and running (using the non-R2 version of Hyper-V Manager to build it), I loaded the Failover Cluster Manager and Hyper-V Tools Features into it. I was able to point at my two Hyper-V hosts, create a new MSCS cluster, enable CSV, and create a highly-available VM on the shared storage. It is interesting to note that a VM that is configured to be “live migratable” must be created via the Failover Clustering Manager and is essentially a cluster resource that swings between cluster nodes. It was a little tricky to get my management VM protected since I had originally created it using the Hyper-V Manager, but the process made sense and I did it without a trip to Google. Like the other two environments, I moved the VM between nodes and didn’t have a problem. One of my coworkers started building a new server to host the Microsoft System Center management tools like Virtual Machine Manager (SCVMM), which I understand makes a this whole process a lot more friendly.

So, while the Hyper-V environment took a little more work out of the box and from scratch (no existing Windows 2008 R2 machines), I was able to get all three environments up and hosting a VM without too much work. Sure, I am using the 60-day trial versions of vCenter and XenServer Essentials, but I am now able to look at each platform and compare features across the board.

Whew! Next up is the comparison.

Doug Baer is a Principal Consultant and leader of the Virtualization Practice at IT Partners in Phoenix, AZ. He holds a BS in Computer Science from the University of Arizona, is a regular contributor on the VMTN forums in between customer engagements and is VCDX #19.