About TestingBot


TestingBot offers easy online Selenium testing.
Use our Selenium grid to run tests across multiple browsers (over 58 combinations!).

We can run your Selenium tests on a daily basis and alert you when a test fails.

Interested? Take a look at our features.

How we make our interactive testing super fast

At TestingBot, we like our users to profit from fast access to our VMs.

In our manual testing area, we provide a way to pick an Operating System, Browser and browser version.
As soon as you picked a combination you’d like to use, we spin up a Virtual Machine, which is a virtual computer in our cloud system. This computer is fresh, not used by anyone before you and will not be used by anyone after you. As soon as you close the manual test session, the computer and all its data is destroyed.

To keep this entire process very fast, we do various things:

  • We start the virtual machine with its RAM memory already loaded, meaning we can skip the entire boot process. Booting up a computer takes a couple of minutes, by eliminating this, we can provide you with a fresh computer on the fly.
  • For Windows testing, we use the very fast RDP protocol to provide remote desktop. RDP is created by Microsoft and works very fast with Windows.
  • Contrary to our competitors, we stream these RDP packets immediately to your browser, by using FreeRDP and Guacamole. This means less lag during your test session, fast and responsive mouse to control our computers in your browser.
  • We utilize websockets to transfer data between our cloud and your browser, which means realtime data transfer instead of polling for data, which is much slower.
  • Our MAC/Linux/Android machines are equipped with an optimized VNC server which will first compress data and keep the transferred data to a minimum. This keeps your manual test session responsive.
  • The machines are equipped with a lot of RAM and a fast CPU, which means the manual test will not be slow due to slow computer speed, browser speed or any other bottleneck.

We’re constantly working on improving our system to keep everything very fast.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

2014 for TestingBot

As we’re approaching the end of 2014, we like to list a few goals TestingBot achieved this year:

  • The number of paying customers increased by 43% in 2014.
  • We improved our manual testing software: less latencies, higher responsiveness and better quality when controlling one of our browsers with your mouse.
  • Big speedup in running Android tests, together with a bunch of new Android devices. Testing on these devices will be a lot faster now
  • Added Yosemite and iOS 8 to the mix
  • … lots of minor tweaks, improvements and fixes…

We’re already working on new features and enhancements which will appear in the first few months of 2015.

Happy holidays!

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Run tests against any Selenium version

Our Selenium grid now supports running your Selenium tests with an older version of Selenium.
By default, we always run your tests on the most recent version of Selenium, to ensure maximum performance and stability.

Some customers however like to run their tests on an older version of Selenium, this is now possible by specifying the desired Selenium version in the desired capabilities option.

For example this Python snippet which will run the test on Selenium 2.31.0

import unittest
from selenium import webdriver
from testingbot import driver

class Selenium2TestingBot(unittest.TestCase):

    def setUp(self):
        desired_capabilities = webdriver.DesiredCapabilities.FIREFOX
        desired_capabilities['version'] = '12'
        desired_capabilities['platform'] = 'WINDOWS'
        desired_capabilities['name'] = 'Testing Selenium 2 in Python'
        desired_capabilities['selenium-version'] = '2.31.0'

        self.driver = webdriver.Remote(
            desired_capabilities=desired_capabilities,
            command_executor="http://key:secret@hub.testingbot.com/wd/hub"
        )
        self.driver.implicitly_wait(30)

    def test_google(self):
        self.driver.get('http://www.google.com')
        assert "Google" in self.driver.title

    def tearDown(self):
        self.driver.quit()
        unittest.TestCase.tearDown(self)

if __name__ == '__main__':
    unittest.main()
VN:F [1.9.22_1171]
Rating: 5.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

TestingBot has moved to its own cloud!

private cloud

Ever since we started TestingBot (almost 2 years ago!) we’ve been running TestingBot on Amazon AWS (EC2 + S3 + other services).
These last few months however, we’ve been moving everything from Amazon to our own private cloud.

Originally Amazon AWS seemed like a good fit for us: easy to setup, manage and maintain.
We’d scale up and down, depending on the number of tests our customers were running.
As it turned out, AWS has its disadvantages:

  • Noisy neighbors: sometimes instances would behave much slower than usual, because other people on the same hypervisor were using all the hypervisor’s resources.
  • Expensive: AWS is expensive, as soon as we start an instance, we’re billed for the entire hour, even if we only need to run a 2 minute test on it.

In July we started looking into running our own private cloud on a bunch of dedicated servers. Originally we planned to use VMWare’s vSphere and vCenter, but after testing VMWare for 2 weeks we concluded it would not satisfy our needs:

  • Expensive: complicated/expensive licensing + expensive support
  • Black box: whenever something went wrong it was hard to troubleshoot since we can’t look at the code. VMWare does have good documentation though.
  • Complicated API: we needed an API that would help us automate launching/destroying VMs. VMWare’s APIs are complicated to test and use.

After we ditched VMWare, we decided to look into an open-source solution: KVM + Qemu.
This turned out to be an instant success: easy to install, setup and use.
Together with libvirt we quickly had a proof of concept system where we could easily launch and destroy virtual machines.

Everything was looking good, we just had one more wish: eliminate booting the VMs.
Since booting the VMs takes time and resources from the CPU, eliminating it would mean faster VM turnaround and less IO on our VM host servers.

We eventually stumbled upon GridCentric. They’re working on VMS, which eliminates boot storms (for example: a big spike in VM boots early in the morning when people start their work-day). After a proof of concept we quickly had a system where we could launch a VM with RAM already loaded into the VM, ready to immediately run the test.

Now we’re running our own cloud; as soon as a customer wants to run a test we spin up a VM in less than 10 seconds, run the test and destroy the VM after the test has finished. This way we guarantee pristine VMs, fast tests and a secure environment.

Together with these changes, we’ve changed some more things on TestingBot:

  • Updated our OSX VMs to OSX Mavericks
  • Added IE11 to our grid
  • Created a “prerun” capability to download and run any program you specify before running your test so you can customize the VM to your liking.
VN:F [1.9.22_1171]
Rating: 7.8/10 (33 votes cast)
VN:F [1.9.22_1171]
Rating: +10 (from 18 votes)

TestingBot open-sources NodeJS based Selenium grid

node-js-S

Today we’ve released our NodeJS code which runs our Selenium hub.
This code has been running in production for over 11 months now on a single CPU server for all our customers.

Using the code is really simple, you can either download it straight from GitHub or use npm to install the package.

sudo npm -g install node-seleniumgrid

Once the hub is running on your own computer, you can connect Selenium nodes to it and run tests against the grid.

The original Selenium project already contains a Selenium grid (written in Java), but we wanted to build our own in NodeJS.
The code is simple to read and works fully asynchronous. Mocha tests are included which makes sure the hub behaves like it should.

We’ve also added a feature where the hub will forwards tests to the TestingBot.com grid when it can not find a Selenium node on your local hub.
For example, you could run this grid on your local network and attach Windows/Linux nodes residing in your network to the grid.
All tests running on Windows/Linux will run on your local Selenium nodes, once you run a test which needs a Mac node, the grid will automatically forward the request to the TestingBot grid.

The source code comes with a Apache License, Version 2.0 license.
We welcome any patches/pull-requests and are happy to help out with problems (please use the issues section on Github)

node-seleniumgrid on GitHub

VN:F [1.9.22_1171]
Rating: 8.8/10 (4 votes cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Manual Cross Browser Testing

uii

Today we’re happy to announce the launch of a new feature: live manual testing.

With this new feature you can interact with any browser in our cloud, straight from inside your own browser.
Once you click a browser and operating system in our new Manual Testing Member Area, we’ll spin up a new Virtual Machine and launch the browser you picked in less than 30 seconds.

You’ll be able to control the Virtual Machine from inside your browser, take screenshots, change the screen resolution and collaborate live with other co-workers.
Since TestingBot runs on two datacenters: one in the US and one in Europe, we provide the fastest experience with a minimum of latency.

When you start a tunnel on your computer, you’ll be able to manually test your internal websites from inside our cloud.

Ready to get started? Head on over to our feature page or immediately start testing!

VN:F [1.9.22_1171]
Rating: 9.0/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)

TestingBot extends its cloud to Europe

europe

We’re excited to announce that starting from today, our customers can choose to either run their tests on our existing US cloud or on our new European based cloud!

With this new datacenter in Europe (Ireland), we’ll provide faster test times for our customers residing in Europe.
A quick comparison on the difference in latency when residing in Europe:

Location Ping Result
EU 36.454 ms
US 96.838 ms

For testers residing in Europe, using our European cloud will mean a decrease in latency of over 50%!

The service and features we provide are identical across both clouds. When running a test in Europe, your request will only go through European servers to guarantee maximum speed.
Our Tunnel has been updated to provide a “region” option, which means you can choose to run your local tests through a European or US based pristine tunnel.

To get started, all you need to do is use our European Selenium Grid, which runs on: europe.testingbot.com on ports 4444 and 80.
More information and examples are available on our Europe page.

VN:F [1.9.22_1171]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Screen resolution option now available for all Selenium tests

resolution

You can now specify a custom screen resolution when running any Selenium test on our grid.
TestingBot can now modify the screen-resolution on Windows, Linux and Mac!

These are the available screen resolutions:

  • 800×600
  • 1024×768
  • 1280×960
  • 1280×1024
  • 1600×1200
  • 1920×1200
  • 2560×1440

You can find info and more test options on our support options page.
The same screen resolution settings are available in our TestLab as well.

Enjoy!

VN:F [1.9.22_1171]
Rating: 7.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Appium testing on TestingBot: automated hybrid/native iOS testing.

appium test

With Appium on TestingBot you can now run WebDriver tests against hybrid and native iOS apps.
There’s no need to do anything special to your iOS app, just supply an URL to the zipped version of the .app file in the desired capabilities and Appium will extract your app and run tests against it in an iOS Simulator.

Since Appium uses the WebDriver protocol, you are free to choose any programming language you want to run tests against it: PHP, Ruby, Java, .NET, NodeJS, Python, …
Because Appium uses Apple’s UIAutomation, there’s no need to recompile or modify your app to run tests against it.

Starting from today, all our customers can run Appium WebDriver tests against iPhone/iPad simulators in our cloud. Run iOS tests in parallel without having to worry about setup or maintenance.

You can find more info and an example video of an Appium test on our Appium page.

VN:F [1.9.22_1171]
Rating: 5.5/10 (4 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Travis CI Selenium testing with TestingBot

Travis CI is a continuous integration service, you can use this free service to run (Selenium) tests from your GitHub repository.
It is very easy to use the TestingBot Selenium grid together with Travis CI.

Below is a guide to run your Selenium tests with Travis CI.
We offer TestingBot status badges to indicate your Selenium tests status on your GitHub pages.

Set up Travis CI

Sign up at Travis CI and connect your GitHub project with Travis CI.
You can find information on how to do this at the Travis CI help section.

Sample Selenium Test

Add a simple Selenium test to your GitHub project, modify the .travis.yml file in your repository to indicate you want to run a Selenium test.

language: node_js
node_js:
  - 0.8
env:
- [
    {secure: "akuE0dld1Ke9mahjUUrQhUZRYWasdfewfwefewfZlbvOx\nqaPybirPGsDmImvcktaAkjLxpePd0V1+ak+4dws7dTrFfEsdvsdsdvsdvds2\nud1q5oGOzEqfiRGxY/fJHLWlaQ609Bsdfdsfds2VeY1Z/V7N9iQ="},
    {secure: "JGfkAfr/SOlzV+NpgNi3fxP4F2usdfdsveGAppugHj1IxhoyjY\nOp07x4p1hdIfWVF03RqUrPNXkl72+yh53pv2fUzsdfsd3434GRjGy6J6\notuA/N+xs0+TP2ENlCmDauwO32Okfojvj7CgvsdfdsfRyaFzIGWPdw="}
  ]
script:
- "node tests/examples/*.js"

The two secure lines in the above example are your TestingBot key and secret, which are used to run a test on our TestingBot Selenium Grid.
To generate these 2 lines, you need to install the Travis CI gem and run these commands with your own key and secret:

travis encrypt username/projectname TESTINGBOT_KEY=key
travis encrypt username/projectname TESTINGBOT_SECRET=secret

Now you can use the encrypted TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables in your tests.
Remember to indicate your test’s privacy setting as “public” (privacy = true) if you want to use the test status badges we provide.

Here’s an example in NodeJS:

var webdriverjs = require('tbwdjs');
var client = webdriverjs.remote({
    host: 'hub.testingbot.com',
    desiredCapabilities: {
        browserName: 'internet explorer',
        version: 9,
        platform: 'WINDOWS',
        api_key: process.env.TESTINGBOT_KEY,
        api_secret: process.env.TESTINGBOT_SECRET,
        name: (process.env.TRAVIS_JOB_ID ? ("Travis Build " + process.env.TRAVIS_JOB_ID) : "Simple Test"),
        privacy: true
    }
});

client
    .testMode()
    .init()
    .url('http://google.com/')
    .titleEquals('Google')
    .end();

Get your TestingBot badge

Once your first Selenium test has ran via Travis, you can start displaying its status with our Status Badges:

<a href="http://testingbot.com/u/key">
  <img src="http://testingbot.com/buildstatus/key" alt="Selenium Test Status">
</a>

More information

Our help page regarding Travis CI integration has more info: Travis CI + TestingBot

VN:F [1.9.22_1171]
Rating: 5.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)