Thursday, December 9, 2010

Web UI driver comparison

I have had the (dis?)pleasure of working with 3 of the major Web testing drivers out there on my day job. I want to do a Selenium RC vs. Webdriver vs. Sahi post. A lot of people at work ask our team this question as we have used all 3. After doing a search, I realized that there are a lot of biased posts out there, so I have tried to be as objective as I can.

Selenium RC (pre 2.0)

Good
  • Supports pretty much every major browser that exists out there
  • Supports writing code in Java, Ruby, Python, PHP and a few other languages
  • Has been around for quiet sometime and the community is pretty active. If you get stuck with something, the chances are, others have already faced the same and have talked about it on some Selenium forum
  • Provides different kinds of locators to identify elements on the browser - Name, ID, XPath and Dom.
  • Provides the ability to either inject JS or write extensions through user-extensions in order to enhance Selenium[1]
  • Has a recorder but only for Firefox
Bad
  • Has a notion of modes: Vanilla, Chrome, IEHTA, PI etc. Its very confusing to figure out what you want to use when you start off
  • Has a need for explicit waits for page loads, elements to appear etc.[2]
  • The architecture uses a proxy server that injects Selenium JS to each page in a different frame or into the page itself depending on the mode. On IE, one needs to setup the proxy server settings. This is very painful if you have a build farm with 20 IE machines. Also, one needs to manage the life cycle of the server which is again, well, work. 
  • Uses JS event emulation in order to do user actions[3]. This can cause 2 issues:
    • Selenium tries and sends all events that make sense for a given action. In order to click, for example, it would have to do a "mouse in", "mouse down", "mouse up", "click" etc. But, a user may have any sort of event listener like "blur" which is not possible to simulate with Selenium.
    • I have faced some issues with testing frameworks like DOJO using Selenium (IDE & RC)
  •  Frames and Windows are not easily testable.
  • You need to accept a Selenium certificate if you want to test HTTPS because of the proxy server.
  • Has a big flat interface with no notion of Browser elements.
My personal take on Selenium RC pre 2.0 is, it was a good contender, but there are better options now. I would not want to use this version anymore.


Webdriver (Pre Selenium 2.0)

Good
  • Takes a different approach to implementation. Tries to be as close to native as possible with every browser i.e. IE has a plugin, Firefox has an addon etc. This gets rid of the need for a proxy server.
  • Tests are faster compared to Selenium RC
  • Creating a new webdriver instance is as simple as saying "new FirefoxDriver();", for example. Does not have the notion of modes
  • Though not very useful in an AJAX heavy application, supports HTTPUnit which means you can run headless tests.
  • Has a nice abstraction for UI elements like Button, Checkbox etc. Nicer API which allows for nicer OO code.
  • Dealing with HTTPS is straightforward as you just need to accept the real certificate from the application under test
  • The driver development is very active. The last time I checked Simon Stewart was working full time on this in Google.
Bad
  • Support on IE has a very major roadblock which has not been fixed in the last 5 months. This was the major reason why we had to ditch Webdriver. Basically the test hangs and we do not know what the problem is.
  • StaleElementException: When you do a getElement operation, Webdriver returns a list of elements. However, if due to some JS activity, the element gets replaced, you get an exception. This can be very tricky to deal with.
  • Though one does not need to wait for page loads, one still needs to explicitly wait for elements to appear. For example, if you do an action which results in Ajax fetching a link and then you want to click on the link, you have to wait for the link to appear. There is no implicit waiting. Apparently, this is in the backlog, but it was not there 3 months ago when I last worked on webdriver.
  • API unimplemented on IE! Though, the latest version has some of this fixed, there are some API which throw an exception on IE.
  • We realized that "class name" locators do not work on IE. So, we had to basically resort to XPaths for all locators. This can be tricky if you are not familiar with XPaths. Also, the locators become very verbose.
  • The move to Selenium 2.0 was not very well documented about 4 months back. However, I think it might have changed now.
  • No recorder yet
Overall, I like Webdriver for its simplicity in startup and usage. If only the IE issues were ironed out and the locators were made nicer, it would have been a keeper for me.


Sahi

Good
  • Provides all the features that Selenium does
  • Has a recorder that works on IE, FF and Safari
  • Implicitly tries out different locator strategies. It tries id, name, text and class name - in that order for a given locator. It also supports regular expression syntax for locators. This is immensely powerful.
  • Follows a concept of Element Stubs in its Java driver. This is very powerful. What it does is, when you say "driver.div('foo')", it returns an ElementStub. Whatever operation you perform on it, Sahi sends it over to the browser as a command and evaluates it there with implicit waits. This gets rid of
    • Need for explicit waiting because of JS or Ajax
    • The Stale Element issue faced in Webdriver.
  • Gets rid of all explicit waits. Sahi takes care of blocking when a page reloads, if there are ajax requests in progress and when you are trying to act on an element which is not present yet. This makes the tests a lot terser and stabler.
  • Has different locator mechanism - In, Under and Near. Also supports the traditional XPaths & Dom locators.
Bad
  • Uses the same architecture of Selenium RC for its Java driver because of which one needs to deal with the proxy issues. Especially painful for IE.
  • Under and Near locators are devious. Though they give a good mileage to start with, if your page has a lot of repeated entries (like a list of sorts), you may be hitting the wrong element. Since Sahi returns the first element that matches by default, your tests may become flaky because a wrong element got matched for your locator. Only 'In' is deterministic, while 'Under' and 'Near' are not. My advice is do not use them if you can avoid them.
  • HTTPS is again painful because one has to accept the Sahi certificates. Though this should be one time thing, somehow, I always have to do this on every Sahi upgrade. Doing this on a build farm is very painful.
  • There are some issues which are not solved yet because of which builds hang. The ElementStubs do not have a time out and they can potentially get stuck for ever. We have had builds which have stuck overnight and had to be manually killed.
  • The community is not big and its just a few people, albeit full time, working on the tool. Turn around time can be a little big.
Just the implicit waits and the locator friendliness (the strategy that goes through name etc) is so powerful that I am sticking with Sahi for now.

May be if I get time, I can post the code from our git history to show how webdriver and Sahi code looks like for the same operation. Sahi code is a lot smaller.

If any of you have evaluated Wati[rjn], Krypton or any other drivers, please do leave a comment.

[1] - Hakan and I actually wrote an extension that waits for any open Ajax calls implicitly so that the users do not have to explicitly wait.The user-extension concept is pretty useful.

[2] - Explicit waits in tests are a bad idea. I did a quick Google search and did not find many entries. A new one coming up.

[3] - I will talk about the implication of this in a different post.

7 comments:

Pankaj Nakhat said...

Good post. I have used all three in the past and currently settled with Webdriver.

To me it offers clean API, much better then Sahi API.

Offers single interface for all browser type.

Doesn't use hacky JS to do stuff which old selenium and now sahi does at times.

It can test NON JS mode.

Supports page object out of the box.

But on contrast I like StaleElementException, as it tells you since you last read the DOM, the element has changed. Now to handle this you can easily catch this exception and re read the DOM.


Also, IE driver in 2.0.a7 is much better then it used to be.

Pavan said...

Hey Pankaj,

Thanks.

Like I said, I liked Webdriver a lot. In fact, a friend of mine who also happens to be a friend of Simon introduced us and we were in touch about the issues we faced. Given a chance, I want to get back to Webdriver and check it out again.

However, I still think the Sahi's ElementStub approach is more elegant compared to catching StaleElementException - which of course is a trivial thing to do.

I will try and checkout 2.0 sometime soon.

V. Narayan Raman said...

Hi Pavan,
Nice post, and thanks for your comments.

I think you should also mention these points:
1) All these tools were used in combination with Twist.
2) The Java drivers were used.

A few comments on Sahi
1) HTTPS certificates are automatically handled in the Sahi Pro version. (Still undergoing some ironing)
2) Proxy switching in IE has been made invisible to users (will be available in the next OS release.)
3) Sahi kills a browser if it has not been active for more than 2 (and half) minutes. The timeout logic needs to be added to Twist too.
4) About the support: We at Tyto are adding more support personnel, so your turn around time should become smaller. Sahi is the only OS web testing tool (of these 3, at least) completely backed by a commercial organization, and we are growing.
5) I would not be so harsh on "near" and "in" APIs.
6) You can run Sahi tests on HTMLUnit too. But I doubt your app would work on that combo.

On Webdriver:
I spoke to Simon during GTAC and he proposed our using Webdriver underneath Sahi. We tried it on our system and found a couple of issues.
1) IE did not complete requests when there were many images.
2) Focusing issues prevent our multithreaded playback.

Given that we have commercial users, we could not move to Webdriver because we would not have control over bugfixes, and we will not be able to meet our SLAs. Some day when Webdriver matures, on all our supported browsers and platforms, and we are guaranteed support at the pace we need, we may move to it too.

@Pankaj Nakhat
Are you mentioning Sahi's Java APIs?

Pankaj Nakhat said...

@Narayan

I am specifically talking about Java driver. Also, i think the place where webdriver has an edge with using Native OS events o driver the browser, which is close to what a user would do.

And last but not the least, don't like to specify browser paths manually in the configuration, though its very trivial.


Again I liked Sahi, and currently doing a POC of Sahi with Jbehave integrating with Hudson.

Pavan said...

[Narayan] I have used Selenium and Webdriver without Twist. But Sahi, yes, with Twist only. And yes, Java driver was used.

[Pankaj] I agree with you. I am fan of native events myself, as long as they do not impose the restriction of needing focus.

Crazy said...

Hi Pankaj, I m QTP guy and now I learned selenium webdriver and Sahi. Would you please let me know which is the good tool that I can use in my next project.

I found this post is very useful and appreciate your efforts.

Thanks & Regards,
NV

renderinglife said...

Hello Guys,

First Thanks to Pankaj for this wonderful post.

Second Thanks to Narayan for Sahi.

I preferred Sahi over other tools as it is very friendly for testers like me and others in my team who get frightened when it comes to coding , java, etc.

I have automated around 40% of my web application with Sahi which was unmanageable with Selenium and Web Driver (Other teams in my company used Selenium and Web Driver investing a lot of time in initial setup and still they do not having a good recorder).

The best part of Sahi is the controller which helps recording click on span on the fly.

While other teams are still struggling I am happily recording my stuff.

Just one request to Narayan, please sir release a new version of Sahi OS and also please support C# coz I like C# a lot.
regards,
rahoolm