Today we’ve added a form on the homepage for visitors to try out our service.
This simple demo will open the website you’ve entered in Internet Explorer 8 and perform a basic test.
It should give you a sense of how easy it is to use our service and test your websites.
Once you entered a website address, you will see this page. It will take up to one minute for the test to finish.
As soon as the test is finished, you will see the results together with screenshots and a video.
In this blogpost we’ll talk about creating your first Selenium test, running the test on your own computer and then later run it on several browsers at once.
It shouldn’t be too hard to follow along, but if you have any questions or remarks, please leave a comment.
Prerequisites
Before we can create a test, we first need a website and an idea of what to test on the website.
Let’s say we have an e-commerce website selling products.
An important part of your e-commerce website is the shopping cart, this virtual cart keeps track of the products the visitor indicated he/she wants to purchase.
If this shopping cart functionality should fail, for whatever reason (bug, code changes, human error, …) the owner of the website will immediately start losing money and customers.
In an effort to prevent this, we decide to create a functional test with Selenium to verify if the shopping cart is still functioning like it should.
The first thing we need is to download the Selenium IDE addon for Firefox. It will help us in creating our first Selenium test.
The IDE has a function to record your browsing activity. All you need to do is launch the addon, click the record button and start performing actions.
Our first test
In our case, we would click the record button, go to our e-commerce website and click “Add to cart” on a specific product.
We’ll then go to the shopping cart page and verify if our product is in the cart. Right click the title of the product in the cart and select “verifyTextPresent”.
This will verify if the product is in fact in our cart.
Stop the recording of the test. Your first Selenium test is ready!
You can now choose to run this test in your browser. It will mimic the actions you just performed and verify that the product is in fact in your shopping cart.
We could stop here and run this test manually every time we want to test our shopping cart. But what about other browsers, maybe the cart is broken in Internet Explorer, a browser used by a lot of people.
Run your test on multiple browsers
That’s where TestingBot.com shines. You simply add your test to our website, choose which browsers you want to run it on and how many times per day/week/month.
We run the tests for you on the browsers you want. If a test should fail, we’ll alert you so you know that something is wrong with your shopping cart.
You can then fix the problem without losing visitors and revenue.
Screencast
We’ve made a screencast of the above example so you can follow along easily:
As mentioned before on this blog, we use a homemade utility to take screenshots of browser windows.
Our tool, built in C#, will take a screenshot of any browser window on Windows.
The way it works is: we changed some Selenium code to capture the PID (Process ID) of the browser that is being used by using Java’s JNA library.
We pass this PID to our tool, which uses the PID to correctly identify the unique browser window. We then get its Hwnd and use PrintWindow to capture the contents of the window.
By releasing this code as open source, we thought it would possibly help others. Let us know what you think!
It’s been a busy week on TestingBot, here’s what we did this week:
Updated our grid from Selenium 2.15 to 2.16, the changelog reads:
Issue 2985 fixed, Calling quit() on the IE Driver in a shutdown hook will kill the JVM.
Issue 2958 fixed, Selenium RC fails to close IE HTA after tests (mshta.exe doesn’t get killed)
We added raw log support, so you can now download the raw log files from the Selenium grid.
For the time being we’re redirecting all safari calls to safariproxy because safariproxy is much more stable.
We optimized captureEntirePageScreenshot and captureEntirePageScreenshotToString to support all browsers.
The standard Selenium configuration only supports Internet Explorer and Firefox.
We introduced a sub-account feature, you can now add members from your team.
It is now possible to share test results with the public. Test results now come with an option to make results public or private. Default is private.
Today we’re examining the captureNetworkTraffic selenium command, a little known gem in Selenium’s command list.
Selenium offers a feature, captureNetworkTraffic, which allows you to intercept the network traffic as seen by the browser running your test.
The response includes headers, status codes, timings and ajax requests. It’s especially useful if you’re debugging requests in your browser.
We have just finished optimizing the video recording in our Selenium grid.
Implementing this was harder than we thought, so here are some tips on how we implemented this.
To create a video, we used AForge’s AviWriter, which makes creating avi videos very easy in C#.
We first started off with using C# and Windows’ PrintWindow function. This worked in some cases but would sometimes show a dark square in the content area.
PrintWindow was also very slow when appending it to a video.
Next, we tried to use Bitblt and GDI with C#, together with AviWriter. This was still too slow for us as it caused tests to run twice as long. We want your tests to run as fast as possible and did not want to hog the CPU because of video recording.
After searching on Google, we found several costly screen capture SDKs which promised fast video recording by using special drivers. This turned out to be working in some cases, but was still too slow.
Finally, we investigated Microsoft Expression Encoder which offers a screen-capture SDK. This turned out to be a winner since it recorded video without hogging the CPU.
The only pitfall is that the video encoding used by this SDK is a proprietary encoding by Windows. To convert the video to FLV, we needed to encode it after recording, something that requires quite a lot of CPU.
We choose to do the video recording on a different high CPU server, so that your tests will run as fast as possible without a screen recording slowing down the CPU.
The program we’re using to take screenshots is MiniCap, which is a lightweight command line program to easily take screenshots. Highly recommended.
It was quite a journey, but we think we now have a good system to record tests without slowing down your tests.
After some more optimizations in our code and infrastructure we’ve now reached the point where the average simple test takes 9 seconds.
Compared with competitors, we’re 2 to 3 times faster in running your test.
This means that you can run more tests in less time at testingbot.com
Here are some benchmarks with a simple test that starts an Internet Explorer browser (7) and verifies the title of google.
competitors:
0.0496 tests/s, 0.0496 assertions/s.
real 0m22.299s
user 0m2.064s
sys 0m0.112s
testingbot.com:
0.12 tests/s, 0.12 assertions/s
real 0m8.833s
user 0m0.212s
sys 0m0.028s
We’ll be experimenting with a quick-launch feature soon where we’ll speed up the startup of a browser (the most time-consuming event during testing).
We’ve just finished upgrading our Selenium grid (the hub and all its RC nodes) to Selenium v2.15.0
This upgrade should provide improved stability.
According to the Selenium changelog these are the changes in the v.2.15 version:
WebDriver:
Now supports up to and including Firefox 11.
Changed order of FirefoxProfile initialization attempts on a
remote host: 1) try to load from capabilities, 2) try to load
locally if webdriver.firefox.profile is specified, 3) create an
empty profile.
Added the beginning of infrastructure for gathering logs from
webdriver tests.
Added an UnreachableBrowserException.
Additional work on the WebDriver JS console.
Implicit waits now change how long we wait for alerts. This
functionality will change in 2.16
FIXED: 2700: The infamous scrolling issue.
FIXED: 2816: ClassCastException when calling switchTo().alert().
Grid:
Grid now distinguishes between CLIENT_GONE and
FORWARDING_TO_NODE_FAILED.
FIXED: issue where older RCs could not connect to a newer hub.
RC:
Selenium-backed WebDriver now uses atoms for typing.
Automation Atoms:
Significantly reworked input device abstractions.
Known Issues:
HTA mode for Selenium RC (*iexplore) leaves a browser window open.
Selenium is the industry standard for automatically testing websites in browsers.
A selenium test is a set of instructions that will be sent to a browser of your choice.
Most major programming language offer support for Selenium. Either by plugins or by default.
These plugins will translate language specific syntax to Selenium commands which can be understood by a Selenium Grid.
For example, let’s say we want a test where we will test Google on Firefox. We want to check if the title of the homepage on Google is “Google”.
We start of by specifying which browser we want to use. In Selenese (the syntax used by Selenium), we would use the command getNewBrowserSession together with the options: Firefox and browser version.
This command is sent to the Selenium grid, which is managed by a hub.
The hub will receive the command, check if its grid supports the browser and assign the requester (client) a session.
The client now needs to use this session to further instruct the grid. An example of a session-id would be sofpa92f3fj023jfodsf9.
When we’ve reached this point, the hub has instructed one of its node on the grid (which supports the browser we requested) to start the browser.
Since we want to test Google, we need to instruct the browser on the node to go to google.com.
We do this by sending the “open” command to the grid, together with the URL and our sessionid.
The hub will receive the request, check the sessionid and knows that we previously requested a Firefox browser on a specific node.
Our hub will forward the request to the node, which in its turn will instruct Firefox to open the page and go to the requested URL.
To end our test, we need to verify if the title of the page which is now opened on the node does in fact equal “Google”.
We send a request to the grid, asking our hub to contact the correct node and ask for the title of the current page.
We receive the title back from the grid and verify if the title does in fact equal the title we expect.
