About TestingBot


TestingBot offers easy online Selenium testing.
Use our Selenium grid to run tests across multiple browsers (over 58 combinations!).

We can run your Selenium tests on a daily basis and alert you when a test fails.

Interested? Take a look at our features.

Run tests against any Selenium version

Our Selenium grid now supports running your Selenium tests with an older version of Selenium.
By default, we always run your tests on the most recent version of Selenium, to ensure maximum performance and stability.

Some customers however like to run their tests on an older version of Selenium, this is now possible by specifying the desired Selenium version in the desired capabilities option.

For example this Python snippet which will run the test on Selenium 2.31.0

import unittest
from selenium import webdriver
from testingbot import driver

class Selenium2TestingBot(unittest.TestCase):

    def setUp(self):
        desired_capabilities = webdriver.DesiredCapabilities.FIREFOX
        desired_capabilities['version'] = '12'
        desired_capabilities['platform'] = 'WINDOWS'
        desired_capabilities['name'] = 'Testing Selenium 2 in Python'
        desired_capabilities['selenium-version'] = '2.31.0'

        self.driver = webdriver.Remote(
            desired_capabilities=desired_capabilities,
            command_executor="http://key:secret@hub.testingbot.com/wd/hub"
        )
        self.driver.implicitly_wait(30)

    def test_google(self):
        self.driver.get('http://www.google.com')
        assert "Google" in self.driver.title

    def tearDown(self):
        self.driver.quit()
        unittest.TestCase.tearDown(self)

if __name__ == '__main__':
    unittest.main()
VN:F [1.9.22_1171]
Rating: 5.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

TestingBot has moved to its own cloud!

private cloud

Ever since we started TestingBot (almost 2 years ago!) we’ve been running TestingBot on Amazon AWS (EC2 + S3 + other services).
These last few months however, we’ve been moving everything from Amazon to our own private cloud.

Originally Amazon AWS seemed like a good fit for us: easy to setup, manage and maintain.
We’d scale up and down, depending on the number of tests our customers were running.
As it turned out, AWS has its disadvantages:

  • Noisy neighbors: sometimes instances would behave much slower than usual, because other people on the same hypervisor were using all the hypervisor’s resources.
  • Expensive: AWS is expensive, as soon as we start an instance, we’re billed for the entire hour, even if we only need to run a 2 minute test on it.

In July we started looking into running our own private cloud on a bunch of dedicated servers. Originally we planned to use VMWare’s vSphere and vCenter, but after testing VMWare for 2 weeks we concluded it would not satisfy our needs:

  • Expensive: complicated/expensive licensing + expensive support
  • Black box: whenever something went wrong it was hard to troubleshoot since we can’t look at the code. VMWare does have good documentation though.
  • Complicated API: we needed an API that would help us automate launching/destroying VMs. VMWare’s APIs are complicated to test and use.

After we ditched VMWare, we decided to look into an open-source solution: KVM + Qemu.
This turned out to be an instant success: easy to install, setup and use.
Together with libvirt we quickly had a proof of concept system where we could easily launch and destroy virtual machines.

Everything was looking good, we just had one more wish: eliminate booting the VMs.
Since booting the VMs takes time and resources from the CPU, eliminating it would mean faster VM turnaround and less IO on our VM host servers.

We eventually stumbled upon GridCentric. They’re working on VMS, which eliminates boot storms (for example: a big spike in VM boots early in the morning when people start their work-day). After a proof of concept we quickly had a system where we could launch a VM with RAM already loaded into the VM, ready to immediately run the test.

Now we’re running our own cloud; as soon as a customer wants to run a test we spin up a VM in less than 10 seconds, run the test and destroy the VM after the test has finished. This way we guarantee pristine VMs, fast tests and a secure environment.

Together with these changes, we’ve changed some more things on TestingBot:

  • Updated our OSX VMs to OSX Mavericks
  • Added IE11 to our grid
  • Created a “prerun” capability to download and run any program you specify before running your test so you can customize the VM to your liking.
VN:F [1.9.22_1171]
Rating: 7.8/10 (33 votes cast)
VN:F [1.9.22_1171]
Rating: +10 (from 18 votes)

TestingBot open-sources NodeJS based Selenium grid

node-js-S

Today we’ve released our NodeJS code which runs our Selenium hub.
This code has been running in production for over 11 months now on a single CPU server for all our customers.

Using the code is really simple, you can either download it straight from GitHub or use npm to install the package.

sudo npm -g install node-seleniumgrid

Once the hub is running on your own computer, you can connect Selenium nodes to it and run tests against the grid.

The original Selenium project already contains a Selenium grid (written in Java), but we wanted to build our own in NodeJS.
The code is simple to read and works fully asynchronous. Mocha tests are included which makes sure the hub behaves like it should.

We’ve also added a feature where the hub will forwards tests to the TestingBot.com grid when it can not find a Selenium node on your local hub.
For example, you could run this grid on your local network and attach Windows/Linux nodes residing in your network to the grid.
All tests running on Windows/Linux will run on your local Selenium nodes, once you run a test which needs a Mac node, the grid will automatically forward the request to the TestingBot grid.

The source code comes with a Apache License, Version 2.0 license.
We welcome any patches/pull-requests and are happy to help out with problems (please use the issues section on Github)

node-seleniumgrid on GitHub

VN:F [1.9.22_1171]
Rating: 8.3/10 (3 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Manual Cross Browser Testing

uii

Today we’re happy to announce the launch of a new feature: live manual testing.

With this new feature you can interact with any browser in our cloud, straight from inside your own browser.
Once you click a browser and operating system in our new Manual Testing Member Area, we’ll spin up a new Virtual Machine and launch the browser you picked in less than 30 seconds.

You’ll be able to control the Virtual Machine from inside your browser, take screenshots, change the screen resolution and collaborate live with other co-workers.
Since TestingBot runs on two datacenters: one in the US and one in Europe, we provide the fastest experience with a minimum of latency.

When you start a tunnel on your computer, you’ll be able to manually test your internal websites from inside our cloud.

Ready to get started? Head on over to our feature page or immediately start testing!

VN:F [1.9.22_1171]
Rating: 9.0/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)

TestingBot extends its cloud to Europe

europe

We’re excited to announce that starting from today, our customers can choose to either run their tests on our existing US cloud or on our new European based cloud!

With this new datacenter in Europe (Ireland), we’ll provide faster test times for our customers residing in Europe.
A quick comparison on the difference in latency when residing in Europe:

Location Ping Result
EU 36.454 ms
US 96.838 ms

For testers residing in Europe, using our European cloud will mean a decrease in latency of over 50%!

The service and features we provide are identical across both clouds. When running a test in Europe, your request will only go through European servers to guarantee maximum speed.
Our Tunnel has been updated to provide a “region” option, which means you can choose to run your local tests through a European or US based pristine tunnel.

To get started, all you need to do is use our European Selenium Grid, which runs on: europe.testingbot.com on ports 4444 and 80.
More information and examples are available on our Europe page.

VN:F [1.9.22_1171]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Screen resolution option now available for all Selenium tests

resolution

You can now specify a custom screen resolution when running any Selenium test on our grid.
TestingBot can now modify the screen-resolution on Windows, Linux and Mac!

These are the available screen resolutions:

  • 800×600
  • 1024×768
  • 1280×960
  • 1280×1024
  • 1600×1200
  • 1920×1200
  • 2560×1440

You can find info and more test options on our support options page.
The same screen resolution settings are available in our TestLab as well.

Enjoy!

VN:F [1.9.22_1171]
Rating: 7.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Appium testing on TestingBot: automated hybrid/native iOS testing.

appium test

With Appium on TestingBot you can now run WebDriver tests against hybrid and native iOS apps.
There’s no need to do anything special to your iOS app, just supply an URL to the zipped version of the .app file in the desired capabilities and Appium will extract your app and run tests against it in an iOS Simulator.

Since Appium uses the WebDriver protocol, you are free to choose any programming language you want to run tests against it: PHP, Ruby, Java, .NET, NodeJS, Python, …
Because Appium uses Apple’s UIAutomation, there’s no need to recompile or modify your app to run tests against it.

Starting from today, all our customers can run Appium WebDriver tests against iPhone/iPad simulators in our cloud. Run iOS tests in parallel without having to worry about setup or maintenance.

You can find more info and an example video of an Appium test on our Appium page.

VN:F [1.9.22_1171]
Rating: 4.0/10 (3 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Travis CI Selenium testing with TestingBot

Travis CI is a continuous integration service, you can use this free service to run (Selenium) tests from your GitHub repository.
It is very easy to use the TestingBot Selenium grid together with Travis CI.

Below is a guide to run your Selenium tests with Travis CI.
We offer TestingBot status badges to indicate your Selenium tests status on your GitHub pages.

Set up Travis CI

Sign up at Travis CI and connect your GitHub project with Travis CI.
You can find information on how to do this at the Travis CI help section.

Sample Selenium Test

Add a simple Selenium test to your GitHub project, modify the .travis.yml file in your repository to indicate you want to run a Selenium test.

language: node_js
node_js:
  - 0.8
env:
- [
    {secure: "akuE0dld1Ke9mahjUUrQhUZRYWasdfewfwefewfZlbvOx\nqaPybirPGsDmImvcktaAkjLxpePd0V1+ak+4dws7dTrFfEsdvsdsdvsdvds2\nud1q5oGOzEqfiRGxY/fJHLWlaQ609Bsdfdsfds2VeY1Z/V7N9iQ="},
    {secure: "JGfkAfr/SOlzV+NpgNi3fxP4F2usdfdsveGAppugHj1IxhoyjY\nOp07x4p1hdIfWVF03RqUrPNXkl72+yh53pv2fUzsdfsd3434GRjGy6J6\notuA/N+xs0+TP2ENlCmDauwO32Okfojvj7CgvsdfdsfRyaFzIGWPdw="}
  ]
script:
- "node tests/examples/*.js"

The two secure lines in the above example are your TestingBot key and secret, which are used to run a test on our TestingBot Selenium Grid.
To generate these 2 lines, you need to install the Travis CI gem and run these commands with your own key and secret:

travis encrypt username/projectname TESTINGBOT_KEY=key
travis encrypt username/projectname TESTINGBOT_SECRET=secret

Now you can use the encrypted TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables in your tests.
Remember to indicate your test’s privacy setting as “public” (privacy = true) if you want to use the test status badges we provide.

Here’s an example in NodeJS:

var webdriverjs = require('tbwdjs');
var client = webdriverjs.remote({
    host: 'hub.testingbot.com',
    desiredCapabilities: {
        browserName: 'internet explorer',
        version: 9,
        platform: 'WINDOWS',
        api_key: process.env.TESTINGBOT_KEY,
        api_secret: process.env.TESTINGBOT_SECRET,
        name: (process.env.TRAVIS_JOB_ID ? ("Travis Build " + process.env.TRAVIS_JOB_ID) : "Simple Test"),
        privacy: true
    }
});

client
    .testMode()
    .init()
    .url('http://google.com/')
    .titleEquals('Google')
    .end();

Get your TestingBot badge

Once your first Selenium test has ran via Travis, you can start displaying its status with our Status Badges:

<a href="http://testingbot.com/u/key">
  <img src="http://testingbot.com/buildstatus/key" alt="Selenium Test Status">
</a>

More information

Our help page regarding Travis CI integration has more info: Travis CI + TestingBot

VN:F [1.9.22_1171]
Rating: 5.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

PhantomJS headless testing with Selenium WebDriver and TestingBot

Today we have added PhantomJS to our Linux VMs on our Selenium Grid.
You can now run your WebDriver tests on PhantomJS + Ubuntu.

PhantomJS is a headless WebKit solution which is very fast, it does not suffer from slow browser startup times or other issues you might encounter with browsers.
Together with Ivan De Marino’s GhostDriver, we can now run Selenium WebDriver tests with PhantomJS.

An example of how to run a simple test with Ruby and PhantomJS on our grid:

caps = {
  :browserName => "phantomjs",
  :platform => "LINUX"
}

urlhub = "http://key:secret@hub.testingbot.com:4444/wd/hub"

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 120

@webdriver = Selenium::WebDriver.for :remote, :url => urlhub, :desired_capabilities => caps, :http_client => client
@webdriver.navigate.to "http://www.google.com/"
puts @webdriver.title
@webdriver.save_screenshot("./screenshot.png")
@webdriver.quit

The example above will open the homepage of Google, print the title of the homepage and save a screenshot of the page on your harddisk.

As PhantomJS and GhostDriver will be improved and updated in the future, we will keep on updating these on our Linux VMs.

Together with Ubuntu, PhantomJS is very fast:

PhantomJS + GhostDriver Ubuntu on our grid:
real 0m11.395s
user 0m2.588s
sys 0m0.225s

Google Chrome Ubuntu on our grid:
real 0m24.249s
user 0m2.476s
sys 0m0.128s

If you have any questions or suggestions for new features, please let us know.

VN:F [1.9.22_1171]
Rating: 8.5/10 (6 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Windows 8/Windows 2012 Selenium Testing

Today we have added Windows 2012 (= Windows 8) VMs to our grid.
You can now run Firefox/Chrome and IE10 tests on Windows 2012 VMs.

To run a test on a Windows 2012 VM, you need to use WIN8 as the platform desired capability, see the following example:

caps = {
  :browserName => "internet explorer",
  :version => "10",
  :screenshot => true,
  :platform => "WIN8"
}

urlhub = "http://key:secret@hub.testingbot.com:4444/wd/hub"
client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 120
@webdriver = Selenium::WebDriver.for :remote, :url => urlhub, :desired_capabilities => caps, :http_client => client
@webdriver.navigate.to "http://testingbot.com/"
puts @webdriver.title
@webdriver.quit

The full list of all browser combinations we support: 91 browser combinations

We’ve also updated the Selenium version on all VMs has been updated from 2.25.0 to 2.26.0 (most recent version), together with the most recent IEDriver and ChromeDriver versions.

Happy testing!

VN:F [1.9.22_1171]
Rating: 9.8/10 (4 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)