Just came across this article on LoadRunner against Visual Studio Team System, by Scott Moore.
NOTE: the article in question is not totally impartial (who is???), as Scott Moore is a consultant for a company partnering with HP/Mercury. The same goes for this blog, in a direction tending towards Microsoft.
There are some valid points made re. the versatility of LoadRunner - Microsoft is not going to get their hands on the innards of SAP and Oracle in the near(?) future, so it's unlikely that they will be able to compete on that front. That being the case, Scott Moore is (IMO) spot on with his call that VSTS is not going to be a LoadRunner killer. ERP solutions will always need load testing and LoadRunner will probably handle that admirably... but having tried both VSTS and LoadRunner, the ease with which a test can be recorded and parameterized without delving into script is pretty impressive. However, the lack of support for Opera/Firefox browsers on VSTS is a fairly large shortcoming, as-is the current shortcomings in data generation and binding.
Having read the article, I think I'll stand by my previous conclusion of horses for courses: ASP.NET or another web app, which can be tested off pure HTTP - use VSTS. Heavy duty requirements? Go with LoadRunner. Either option has a price tag beyond my personal reach, anyway...!
Thursday, August 16, 2007
Unit Testing in Visual Studio 2008 ('Orcas') - Professional Edition
The good word is that unit testing will be available in the professional editon of Visual Studio. This is a near mandatory feature in an IDE today, which most Java IDEs provide out-of-the-box; TDD in .NET would have been further along the road today if VS2005 Pro had included the unit testing feature.
Let's just hope that they don't provide unit testing while leaving out the code coverage feature! :P
Let's just hope that they don't provide unit testing while leaving out the code coverage feature! :P
Load testing with VSTS and LoadRunner
I've not been able to post for a while - been knee deep in work and life, while standing on my head...
Anyway, I've been working with the QA team at my workplace on load testing some of our applications headed for Production over the last few days, getting a taste of both LoadRunner and VSTS.
LoadRunner is *the* heavy duty tool supporting loads of protocols, including SAP and Siebel (amongst others), in addition to the basic requirement of HTTP. We were testing an ASP.NET application which came a cropper when ViewState values would not get replayed, but until then recording a test, replaying it and configuring a load was fairly simple. Debugging is not A-grade AFAIK - but breakpoints and analysis of variables is provided. We tested this off a 10 day trial license, which restricts load to 10 virtual users; the real thing is *said* to be expensive (claims unverified).
On to Visual Studio Team System (VSTS) - Microsoft have done a good job making setting up a test easy, but the bias towards testing Microsoft stuff is definitely visible (e.g. in browser support). Setting up a test was easy, as was parameterizing it. LoadRunner has lots of features to generate data, but VSTS 2005 required some effort to bind to a data source. Thankfully, VSTS 2008 ('Orcas') will have data binding to CSV/XML files in-built; VSTS does provide for extensibility via Request/Test Plugins, which can be used to provide data. I did hit an issue where if an asp:Buttonhas its UseSubmitBehavior switched off, the script records the Cancel button's existence on the page as well, leading to the subsequent HTTP request not being handled correctly by the server. Tests were conducted on a 180-day trial license, without any feature restrictions.
The comparison plays out like this - for basic web apps (definitely ASP.NET apps), VSTS should suffice. While not being cheap, VSTS does not cost as much as LoadRunner. However, LoadRunner is a far more mature product and versatile with it - having support for ERPs and protocols like CITRIX and RDP. The VSTS feature of rigging Unit Tests and Web Tests into Load Tests was pretty impressive and the .NET language + debugging features of the VS IDE outstripped LoadRunner by some distance.
Final word: select what you need for your specific requirement; but for the majority of small-medium business applications, VSTS should meet your need.
Anyway, I've been working with the QA team at my workplace on load testing some of our applications headed for Production over the last few days, getting a taste of both LoadRunner and VSTS.
LoadRunner is *the* heavy duty tool supporting loads of protocols, including SAP and Siebel (amongst others), in addition to the basic requirement of HTTP. We were testing an ASP.NET application which came a cropper when ViewState values would not get replayed, but until then recording a test, replaying it and configuring a load was fairly simple. Debugging is not A-grade AFAIK - but breakpoints and analysis of variables is provided. We tested this off a 10 day trial license, which restricts load to 10 virtual users; the real thing is *said* to be expensive (claims unverified).
On to Visual Studio Team System (VSTS) - Microsoft have done a good job making setting up a test easy, but the bias towards testing Microsoft stuff is definitely visible (e.g. in browser support). Setting up a test was easy, as was parameterizing it. LoadRunner has lots of features to generate data, but VSTS 2005 required some effort to bind to a data source. Thankfully, VSTS 2008 ('Orcas') will have data binding to CSV/XML files in-built; VSTS does provide for extensibility via Request/Test Plugins, which can be used to provide data. I did hit an issue where if an asp:Button
The comparison plays out like this - for basic web apps (definitely ASP.NET apps), VSTS should suffice. While not being cheap, VSTS does not cost as much as LoadRunner. However, LoadRunner is a far more mature product and versatile with it - having support for ERPs and protocols like CITRIX and RDP. The VSTS feature of rigging Unit Tests and Web Tests into Load Tests was pretty impressive and the .NET language + debugging features of the VS IDE outstripped LoadRunner by some distance.
Final word: select what you need for your specific requirement; but for the majority of small-medium business applications, VSTS should meet your need.
Subscribe to:
Posts (Atom)
