TestingBot has moved to its own cloud!

private cloud

Ever since we started TestingBot (almost 2 years ago!) we’ve been running TestingBot on Amazon AWS (EC2 + S3 + other services).
These last few months however, we’ve been moving everything from Amazon to our own private cloud.

Originally Amazon AWS seemed like a good fit for us: easy to setup, manage and maintain.
We’d scale up and down, depending on the number of tests our customers were running.
As it turned out, AWS has its disadvantages:

  • Noisy neighbors: sometimes instances would behave much slower than usual, because other people on the same hypervisor were using all the hypervisor’s resources.
  • Expensive: AWS is expensive, as soon as we start an instance, we’re billed for the entire hour, even if we only need to run a 2 minute test on it.

In July we started looking into running our own private cloud on a bunch of dedicated servers. Originally we planned to use VMWare’s vSphere and vCenter, but after testing VMWare for 2 weeks we concluded it would not satisfy our needs:

  • Expensive: complicated/expensive licensing + expensive support
  • Black box: whenever something went wrong it was hard to troubleshoot since we can’t look at the code. VMWare does have good documentation though.
  • Complicated API: we needed an API that would help us automate launching/destroying VMs. VMWare’s APIs are complicated to test and use.

After we ditched VMWare, we decided to look into an open-source solution: KVM + Qemu.
This turned out to be an instant success: easy to install, setup and use.
Together with libvirt we quickly had a proof of concept system where we could easily launch and destroy virtual machines.

Everything was looking good, we just had one more wish: eliminate booting the VMs.
Since booting the VMs takes time and resources from the CPU, eliminating it would mean faster VM turnaround and less IO on our VM host servers.

We eventually stumbled upon GridCentric. They’re working on VMS, which eliminates boot storms (for example: a big spike in VM boots early in the morning when people start their work-day). After a proof of concept we quickly had a system where we could launch a VM with RAM already loaded into the VM, ready to immediately run the test.

Now we’re running our own cloud; as soon as a customer wants to run a test we spin up a VM in less than 10 seconds, run the test and destroy the VM after the test has finished. This way we guarantee pristine VMs, fast tests and a secure environment.

Together with these changes, we’ve changed some more things on TestingBot:

  • Updated our OSX VMs to OSX Mavericks
  • Added IE11 to our grid
  • Created a “prerun” capability to download and run any program you specify before running your test so you can customize the VM to your liking.


  • Max December 6, 2013 Reply

    Out of curiosity, I didn’t think you were allowed to virtualize OSX unless you’re running it on Apple hardware? Is this info wrong?

  • Brandon December 6, 2013 Reply

    Do you run your OSX VMs on top of Apple hardware?

  • Thomas Orozco December 6, 2013 Reply

    Did you guys evaluate any of the competing private cloud orchestration platforms, such as CloudStack, OpenStack, or Eucalyptus?

    If you did, what did you think?

  • nelson December 6, 2013 Reply

    Great share. I have a question:

    Could something like Docker be part of your stack in the future, would a Container based approach a fit for you guys? I am asking because you are mentioning VM startup time, which would be a perfect use-case for Docker.

    • jochen December 9, 2013 Reply Author

      I don’t think it’d be possible to run IE/Microsoft tests with Docker?

  • nelson December 6, 2013 Reply

    Another question:

    How much are you saving in terms of cost compared to AWS?

  • Greg December 6, 2013 Reply

    This is a great write up, thanks for sharing. How did building out and maintaining your own internal cloud affect your internal IT budget? Do you have a projection of where the cost of the machines will be offset by not shelling out cash for AWS? Finally, do you have other locations to support failover of your data center? This last bit has been cost prohibitive for us.

    Thanks again.

  • Gaurav Patel December 6, 2013 Reply

    Have you thought about Docker? It would save you time on bootup and let you spin up/spin down many containers on a single VM.

    • jochen December 9, 2013 Reply Author

      I don’t think it’d be possible to run IE/Microsoft tests with Docker?

  • rafirafi December 6, 2013 Reply

    I’m interested to know how do you do to run OSX in kvm without breaking the Apple eula, some time ago I tried to see if it was possible and I concluded that it’s only possible to run one vm on a apple tied hardware.

    And from ready the kvm mailing list it looks like they think the same, so I’m really curious and interested to know which are the requirement.

  • Tom December 6, 2013 Reply

    Can you explain more about how you’re running your OS X VMs?

  • jochen December 9, 2013 Reply Author

    Our OSX VMs currently run on Apple Mac Mini running Ubuntu 12.04 with VirtualBox. We tried to get it working with KVM but failed, though this does look interesting: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/

Leave a Reply

Your email address will not be published. Required fields are marked *