Replace QTP with Selenium + AutoIt + Applescript - Part I


I hate QTP. Why? Because it sucks. Of course, QTP fanboys will immediately jump up & down stating that I don’t know how to use QTP. On the contrary, I know exactly what I am talking about. As of this writing, QTP still does not support Windows 7, heck it still doesn’t support Firefox 3.6! Are you kidding me? Well, apart from HP’s snail pace development process, I have other problems with the tool itself. Like its really retarded scripting engine (which uses vbscript), which does not provide you any real mechanism to maintain frameworks.

Another example - CreateObject(“WScript.Shell”) - what do you think will happen if you used that in QTP? Any programmer who knows vbscript, will say that it creates a wscript object but she’d be so wrong. It rather creates a native windows shell automation object. WScript CreateObject() is simply not supported in QTP as QTP scripting engine overrides WScript.

Let us get one thing straight - Automation is programming, now let that sink in for a second…again - automation is programming. If your automation tool does not provide a real good programming interface, it is not fit for automation. Obviously in my books, QTP falls way short of that goal. One of the statements I consistenly hear is - “oh we don’t have programmers in our automation team”. If you cannot see the fallacy in that statement, no one can help your team - not even QTP. And of course, support from HP is bad too. Case in point - recently our team encountered a bug in QTP 10 where it had memory allocation issues & the workaround offered to us - “restart QTP after every 4 test case runs”. I am not joking.

QTP does few things really good vis-a-vis record & playback (and they make it real simple for non technical users). And that also includes support for various enterprise applications both web based & win32. That means, they have to cover a lot of territory before they can release something and that explains why Windows 7 support is still lacking. But in your case, do you need Sharepoint support on Windows 7? If all you’re testing is your own web app, why do you have to wait for HP to finish support for say Oracle enterprise apps? At this juncture, the only reason your team is still sticking to QTP is either because you have no real developers in your QA team and/or you have a lot of test cases automated in QTP. The later is a pain initially to convert to something else, but if you plan it out correctly you will save tons of headache in future.

I could go on & on about all that is wrong with QTP, but this article is not about that. This article is about getting rid of QTP & using alternatives in place of it to achieve a truly cross platform solution. After joining my current company, one of my first goal was to do exactly that. And this article describes what we did & how we did it.

What are we automating?

I work for the SSL-VPN group at Juniper. Not many people (outside of my group) know all the capabilities of this box even if they use our box. The reason is simple - you don’t need to use all the features and you can’t fit them all in a simple data sheet or a marketing slide. But its my group’s responsibility to make sure everything is tested and for that matter automated. Right from the start our group was a Mercury shop & that means QTP infestation was really thick. Well you can’t really blame anyone as our product, like many others in the market, is incremental in nature and the tools that I am championing right now were not even created. Back in early 2000’s, this was the first SSL VPN product, which was designed primarily to provide remote access to web-enabled enterprise application. But now, it provides remote access solutions from L3-L7 with support for a really dizzying array of backend auth servers, authentication mechanisms & enterprise applications like Sharepoint & even VMware solutions. This box allows you to access these applications either strictly through web (if it is web enabled) or through traditional vpn solutions a la vpn client that is supported on Windows, Mac & Linux. Admin configuration is strictly through web, which can get really hairy depending on how complex your network is, what kind of (or combinations of) auth servers you use, what kind of policies you want to enforce etc. Users can access the box (or in essence the network) via web or through the client directly.

We not only have to test the admin & user accessible web pages of our box, we even have to test enterprise applications that are rewritten through our box. For e.g. when you access OWA through our box using the rewriter, it rewrites links, flash etc so that the app is forced to pass through the box. And that means, not only are we testing our web application, we also have to test 3rd party web application that is accesses through the box. So yes, at the end of the day we end up writing test cases to test OWA application itself (through our box of course). And just to emphasize, if we get enough complaints that a site like Youtube is not working properly through our box, we will create test cases to test Youtube through our box. And we have to test all these on all versions of IE & latest versions of Firefox on Windows XP, Vista & 7; Safari & Firefox on Mac and Firefox on Linux (Ubuntu). You can easily how complex our testing infrastructure & framework becomes & so far we have just tested one part of our product - web. We also have 3 different windows clients, 2 Mac clients, one linux client & a java client. And recently, we announced mobile clients for all the major smartphone platforms (you heard Steve Jobs mention it - didn’t you? :). You can launch them through the web portal or through the system itself. In the former case, it automatically gets pushed to the system (not the mobile) & installed if it is not already & supports two delivery mechanisms - activex & java.

So how do we approach this automation?

Essentially, the whole automation can be broken down in to two segments - web & system. And we needed tools that had good APIs and were as cross-platform as possible, more importantly for web. We simply don’t have the time or the man power to maintain separate testsuites for each platform. That means, a lot of abstraction has to take place to facilitate such environment where we write one test & can run on all platforms unmodified (or with very minimal set of if-then-else statements). Juniper uses Perl for all of its testing & since our group ended up at Juniper through acquisition, we had no choice but to move to Perl. But I don’t hate Perl, in fact it used to be my goto language during my Sysadmin days. And the community support for Perl is second to none. So we (or at lease I) didn’t really have any qualms to move to it from python. So now that we decided on Perl, it was time to go look for tools.

In the past couple of years, if you search for web testing tools you simply cannot miss Selenium. If your team does web testing exclusively, this tool should address almost all of your needs but there are caveats. And the best part, it’s open source and that means you can modify it for your team’s unique needs. It was definitely invaluable in our case. The two most important thing that made Selenium a slam dunk was its support for all the mainstream browsers on all the 3 platforms & a server-client model with support for multitudes of languages - specifically Perl. The server-client model is very useful, since generally, your driver & the DUT are not the same machine - and in our case it certainly wasn’t.

Now that Selenium was going to address the web part (which was actually rather easy decision), we needed to look at tools for automating system applications - in our case, vpn clients & to some extent 3rd party applications like Outlook, Word etc. Most of our test cases are for Windows platform, followed by Mac. Linux definitely needs some love, but with minuscule % of the customers using it, we couldn’t justify spending time on it. So we needed some tool that would allow us unified API (preferably with a server-client model) & work on Windows (all flavors & both 32-bit & 64-bit) and Mac OS X. We couldn’t find any single tool that met all those criteria. So we decided to end up using native OS capabilities to address our goal. To tie them all together, we created a RESTish automation server that uses native OS tools to provide automation for native apps. That is where AutoIt steps in. AutoIt, is more or less a convenient user friendly (well actually developer friendly) wrapper around Win32 API & its API works across all flavors of Windows. And the best part is that their API is also available through a COM dll, which makes it very easy for us to consume the API through Perl. But there are still some things that you can’t do through AutoIt, so you have to use Win32 API directly through Perl. In case of OS X, obviously Applescript is practically a no-brainer.

In Part-II of this article, I will discuss how our implementation of REST server provides automation for native system events & apps & goes hand-in-hand with Selenium to provide a very feature rich, faster & more importantly cross-platform alternative to QTP.