About TestingBot


TestingBot offers easy online Selenium testing.
Use our Selenium grid to run tests across multiple browsers (over 58 combinations!).

We can run your Selenium tests on a daily basis and alert you when a test fails.

Interested? Take a look at our features.


TestingBot has moved to its own cloud!

private cloud

Ever since we started TestingBot (almost 2 years ago!) we’ve been running TestingBot on Amazon AWS (EC2 + S3 + other services).
These last few months however, we’ve been moving everything from Amazon to our own private cloud.

Originally Amazon AWS seemed like a good fit for us: easy to setup, manage and maintain.
We’d scale up and down, depending on the number of tests our customers were running.
As it turned out, AWS has its disadvantages:

  • Noisy neighbors: sometimes instances would behave much slower than usual, because other people on the same hypervisor were using all the hypervisor’s resources.
  • Expensive: AWS is expensive, as soon as we start an instance, we’re billed for the entire hour, even if we only need to run a 2 minute test on it.

In July we started looking into running our own private cloud on a bunch of dedicated servers. Originally we planned to use VMWare’s vSphere and vCenter, but after testing VMWare for 2 weeks we concluded it would not satisfy our needs:

  • Expensive: complicated/expensive licensing + expensive support
  • Black box: whenever something went wrong it was hard to troubleshoot since we can’t look at the code. VMWare does have good documentation though.
  • Complicated API: we needed an API that would help us automate launching/destroying VMs. VMWare’s APIs are complicated to test and use.

After we ditched VMWare, we decided to look into an open-source solution: KVM + Qemu.
This turned out to be an instant success: easy to install, setup and use.
Together with libvirt we quickly had a proof of concept system where we could easily launch and destroy virtual machines.

Everything was looking good, we just had one more wish: eliminate booting the VMs.
Since booting the VMs takes time and resources from the CPU, eliminating it would mean faster VM turnaround and less IO on our VM host servers.

We eventually stumbled upon GridCentric. They’re working on VMS, which eliminates boot storms (for example: a big spike in VM boots early in the morning when people start their work-day). After a proof of concept we quickly had a system where we could launch a VM with RAM already loaded into the VM, ready to immediately run the test.

Now we’re running our own cloud; as soon as a customer wants to run a test we spin up a VM in less than 10 seconds, run the test and destroy the VM after the test has finished. This way we guarantee pristine VMs, fast tests and a secure environment.

Together with these changes, we’ve changed some more things on TestingBot:

  • Updated our OSX VMs to OSX Mavericks
  • Added IE11 to our grid
  • Created a “prerun” capability to download and run any program you specify before running your test so you can customize the VM to your liking.
VN:F [1.9.22_1171]
Rating: 7.8/10 (33 votes cast)
VN:F [1.9.22_1171]
Rating: +10 (from 18 votes)
TestingBot has moved to its own cloud!, 7.8 out of 10 based on 33 ratings

About the Author

Selenium fanatic at TestingBot

Comments (12)

Trackback URL | Comments RSS Feed

  1. Max says:

    Out of curiosity, I didn’t think you were allowed to virtualize OSX unless you’re running it on Apple hardware? Is this info wrong?

    VA:F [1.9.22_1171]
    Rating: 5.0/5 (1 vote cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  2. Brandon says:

    Do you run your OSX VMs on top of Apple hardware?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  3. Did you guys evaluate any of the competing private cloud orchestration platforms, such as CloudStack, OpenStack, or Eucalyptus?

    If you did, what did you think?

    VA:F [1.9.22_1171]
    Rating: 5.0/5 (1 vote cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  4. nelson says:

    Great share. I have a question:

    Could something like Docker be part of your stack in the future, would a Container based approach a fit for you guys? I am asking because you are mentioning VM startup time, which would be a perfect use-case for Docker.

    VA:F [1.9.22_1171]
    Rating: 5.0/5 (1 vote cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • jochen says:

      I don’t think it’d be possible to run IE/Microsoft tests with Docker?

      VN:F [1.9.22_1171]
      Rating: 5.0/5 (1 vote cast)
      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  5. nelson says:

    Another question:

    How much are you saving in terms of cost compared to AWS?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  6. Greg says:

    This is a great write up, thanks for sharing. How did building out and maintaining your own internal cloud affect your internal IT budget? Do you have a projection of where the cost of the machines will be offset by not shelling out cash for AWS? Finally, do you have other locations to support failover of your data center? This last bit has been cost prohibitive for us.

    Thanks again.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  7. Gaurav Patel says:

    Have you thought about Docker? It would save you time on bootup and let you spin up/spin down many containers on a single VM.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • jochen says:

      I don’t think it’d be possible to run IE/Microsoft tests with Docker?

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  8. rafirafi says:

    I’m interested to know how do you do to run OSX in kvm without breaking the Apple eula, some time ago I tried to see if it was possible and I concluded that it’s only possible to run one vm on a apple tied hardware.

    And from ready the kvm mailing list it looks like they think the same, so I’m really curious and interested to know which are the requirement.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  9. Tom says:

    Can you explain more about how you’re running your OS X VMs?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  10. jochen says:

    Our OSX VMs currently run on Apple Mac Mini running Ubuntu 12.04 with VirtualBox. We tried to get it working with KVM but failed, though this does look interesting: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VN:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

Leave a Reply




If you want a picture to show with your comment, go get a Gravatar.