Web UI driver comparison

I have had the (dis?)pleasure of working with 3 of the major Web testing drivers out there on my day job. I want to do a Selenium RC vs. Webdriver vs. Sahi post. A lot of people at work ask our team this question as we have used all 3. After doing a search, I realized that there are a lot of biased posts out there, so I have tried to be as objective as I can.

Selenium RC (pre 2.0)

Good

• Supports pretty much every major browser that exists out there
• Supports writing code in Java, Ruby, Python, PHP and a few other languages
• Has been around for quiet sometime and the community is pretty active. If you get stuck with something, the chances are, others have already faced the same and have talked about it on some Selenium forum
• Provides different kinds of locators to identify elements on the browser - Name, ID, XPath and Dom.
• Provides the ability to either inject JS or write extensions through user-extensions in order to enhance Selenium^[1]
• Has a recorder but only for Firefox

Bad

• Has a notion of modes: Vanilla, Chrome, IEHTA, PI etc. Its very confusing to figure out what you want to use when you start off
• Has a need for explicit waits for page loads, elements to appear etc.^[2]
• The architecture uses a proxy server that injects Selenium JS to each page in a different frame or into the page itself depending on the mode. On IE, one needs to setup the proxy server settings. This is very painful if you have a build farm with 20 IE machines. Also, one needs to manage the life cycle of the server which is again, well, work. 
• Uses JS event emulation in order to do user actions^[3]. This can cause 2 issues:
• Selenium tries and sends all events that make sense for a given action. In order to click, for example, it would have to do a "mouse in", "mouse down", "mouse up", "click" etc. But, a user may have any sort of event listener like "blur" which is not possible to simulate with Selenium.
• I have faced some issues with testing frameworks like DOJO using Selenium (IDE & RC)
•  Frames and Windows are not easily testable.
• You need to accept a Selenium certificate if you want to test HTTPS because of the proxy server.
• Has a big flat interface with no notion of Browser elements.

My personal take on Selenium RC pre 2.0 is, it was a good contender, but there are better options now. I would not want to use this version anymore.


Webdriver (Pre Selenium 2.0)

Good

• Takes a different approach to implementation. Tries to be as close to native as possible with every browser i.e. IE has a plugin, Firefox has an addon etc. This gets rid of the need for a proxy server.
• Tests are faster compared to Selenium RC
• Creating a new webdriver instance is as simple as saying "new FirefoxDriver();", for example. Does not have the notion of modes
• Though not very useful in an AJAX heavy application, supports HTTPUnit which means you can run headless tests.
• Has a nice abstraction for UI elements like Button, Checkbox etc. Nicer API which allows for nicer OO code.
• Dealing with HTTPS is straightforward as you just need to accept the real certificate from the application under test
• The driver development is very active. The last time I checked Simon Stewart was working full time on this in Google.

Bad

• Support on IE has a very major roadblock which has not been fixed in the last 5 months. This was the major reason why we had to ditch Webdriver. Basically the test hangs and we do not know what the problem is.
• StaleElementException: When you do a getElement operation, Webdriver returns a list of elements. However, if due to some JS activity, the element gets replaced, you get an exception. This can be very tricky to deal with.
• Though one does not need to wait for page loads, one still needs to explicitly wait for elements to appear. For example, if you do an action which results in Ajax fetching a link and then you want to click on the link, you have to wait for the link to appear. There is no implicit waiting. Apparently, this is in the backlog, but it was not there 3 months ago when I last worked on webdriver.
• API unimplemented on IE! Though, the latest version has some of this fixed, there are some API which throw an exception on IE.
• We realized that "class name" locators do not work on IE. So, we had to basically resort to XPaths for all locators. This can be tricky if you are not familiar with XPaths. Also, the locators become very verbose.
• The move to Selenium 2.0 was not very well documented about 4 months back. However, I think it might have changed now.
• No recorder yet

Overall, I like Webdriver for its simplicity in startup and usage. If only the IE issues were ironed out and the locators were made nicer, it would have been a keeper for me.


Sahi

Good

• Provides all the features that Selenium does
• Has a recorder that works on IE, FF and Safari
• Implicitly tries out different locator strategies. It tries id, name, text and class name - in that order for a given locator. It also supports regular expression syntax for locators. This is immensely powerful.
• Follows a concept of Element Stubs in its Java driver. This is very powerful. What it does is, when you say "driver.div('foo')", it returns an ElementStub. Whatever operation you perform on it, Sahi sends it over to the browser as a command and evaluates it there with implicit waits. This gets rid of
• Need for explicit waiting because of JS or Ajax
• The Stale Element issue faced in Webdriver.
• Gets rid of all explicit waits. Sahi takes care of blocking when a page reloads, if there are ajax requests in progress and when you are trying to act on an element which is not present yet. This makes the tests a lot terser and stabler.
• Has different locator mechanism - In, Under and Near. Also supports the traditional XPaths & Dom locators.

Bad

• Uses the same architecture of Selenium RC for its Java driver because of which one needs to deal with the proxy issues. Especially painful for IE.
• Under and Near locators are devious. Though they give a good mileage to start with, if your page has a lot of repeated entries (like a list of sorts), you may be hitting the wrong element. Since Sahi returns the first element that matches by default, your tests may become flaky because a wrong element got matched for your locator. Only 'In' is deterministic, while 'Under' and 'Near' are not. My advice is do not use them if you can avoid them.
• HTTPS is again painful because one has to accept the Sahi certificates. Though this should be one time thing, somehow, I always have to do this on every Sahi upgrade. Doing this on a build farm is very painful.
• There are some issues which are not solved yet because of which builds hang. The ElementStubs do not have a time out and they can potentially get stuck for ever. We have had builds which have stuck overnight and had to be manually killed.
• The community is not big and its just a few people, albeit full time, working on the tool. Turn around time can be a little big.

Just the implicit waits and the locator friendliness (the strategy that goes through name etc) is so powerful that I am sticking with Sahi for now.

May be if I get time, I can post the code from our git history to show how webdriver and Sahi code looks like for the same operation. Sahi code is a lot smaller.

If any of you have evaluated Wati[rjn], Krypton or any other drivers, please do leave a comment.

[1] - Hakan and I actually wrote an extension that waits for any open Ajax calls implicitly so that the users do not have to explicitly wait.The user-extension concept is pretty useful.

[2] - Explicit waits in tests are a bad idea. I did a quick Google search and did not find many entries. A new one coming up.

[3] - I will talk about the implication of this in a different post.