Saturday, October 1, 2011

Alleluia !! a Load Test tool for Silverlight 4 that gives results within 5 min ! {download, install and running the tests included}

After many searches, having read many people complaining about Load testing a Silverlight applications, watching nice demo of "Hello World" cases, installed commercial tools {but was stuck at some configuration points. See the notes at the end where I listed some of my tests}, and I finally …. gave up tonight !!



But thanks to a final search on Fiddler2 fields {because it gave me the most promising results}, I came across stresstimulus tool. So sorry for those of you who already know this free Fiddler add-on, and also may be its limitations {in which case, please tell us your view points}.

For the purists, I don't want to test Silverlight UI as such {i.e. for this type of test, I am not interested in Visual Studio 2010 Coded UI}, since it runs Client-side. Rather, I wanted to validate that, once deployed into production, this RIA will not put our IIS servers down. Hence, I want to simulate the load on Server-side, by generating lots of various calls to our web services easily.

For my first trial of this tools, I chose a complex Line Of Business application
{ Disclaimer for those who recognizes this application /!\ I just did it gently so that I do not break it}.

image


Within the next 5 min I got the detailed results !! I could have nearly cried so much it was easy and … cheap !! {I am moving to the Pro version since I checked that it works fine on our real application, even with STS WIF and WCF binary activated !! which seams to be so difficult to handle for other tools}.



image


Here are some results
Performance Graphs: Number of simultaneous users, request per seconds, Average response time, Bytes received per seconds, Errors per seconds.image


A summary page
Test Summary
Test Case Summary
File Name: Thavonekham-Tests.ssconfig
Primary Pages: 2
URLs : 39

Test Parameters
Think time between base pages:
As recorded

Iteration timing: Think time between iterations - 2s
Load pattern: Constant load
Users: 5
Browser Type: IE8
Network Type: Dial-Up 56k
Complete test after: 5 iterations
Warm-up time: 0 seconds
Test Run Information
Date: Oct-1 2011
Start Time: 21:46:34.19
End Time: 21:48:00.11
Warm-up (s): 0
Aggregate Session time (s): 851.53
Test run duration (s): 85.92

Overall Result
Status: Completed
Max User Load: 5
Total bytes sent: 138,960
Total bytes received: 41,191,492
Bytes sent/sec: 1,617.292
Bytes received/sec: 479,409.064
Errors
Http Errors: 0
Total Timeouts: 0
Page Timeouts: 0
Missed Goals: 0
Test Iterations
Iterations Started: 5
Iterations Completed: 5
Avg. iteration time (s): 78.76
Requests (HTTP transactions)
Total requests: 195
Requests/sec: 2.27
Avg. response time (s): 4.37


Page detailed {in terms of *.aspx for instance}: 
image


Request details {including resources such as xaml, png, xap, svc, …}
image

And finally
Iteration Details
image


And it does really stress my Windows 8 computer ! {I only ran the test below for 30 seconds}
image_thumb1

At least one limitation: The generated reports do not track down to a specific WebMethod…. except on the example above. Indeed, the calls to WebMethods were passed within the Query String.


Monitoring the load on the Database:
As a complementary tool, use Microsoft SQL Server Profiler {in profiling mode, with a filter set to the name of your database to filter out unwanted SQL queries}. When you play back your tests with StresStimulus, it does call my Web services WCF bin and generated large amount of hits on the database. Now, just spot your top slower queries, and here you go ! Optionally, you could save your trace and run SQL Tuning Advisor.


HOWEVER it is NOT a win yet, I have to try on our real life project. Indeed, I love Silverlight it is just a great RIA, but I also I apply even more the proverb “A bird in the hand is worth two in the bush”.
Finally, it seams that with Silverlight we are left behind within the .Net world {eg. Unit Testing, CodeCoverage, proper ALM, …}


If you find better tools, please let me know.




TIPS / IDEAS / Alternative solutions for testing SL apps|

NB : If you want to know what other tools I tried "{ Deviland start to feel the pain I went through}  see below:

Re: Load Testing vis Silverlight Client?
1 hour, 16 minutes ago | LINK
Hi,
I also struggled finding a proper load testing tool for Silverlight 4 {I know plenty of smart people would claim that there is no need to test RIA, but I want to test the WCF that my SL is using} :
Plenty of "Hello World" nice and sunny video, ... but the reality is so different.
1> Despite the fact I was using VS 2010 ultimate, it seams not to work "out of the box" for SL ! {despite its price}
2> I successfully used Fiddler2 {free of charge} to record aspx calls, and most importantly the binary WCF calls { svc}. Fiddler2 offers the surprising ability to "SAVE AS VISUAL STUDIO WEB TEST"
[Visual Studio WebTest http://www.fiddler2.com/fiddler/help/WebTest.asp].
The only problem I had is that VS 2010 did not understand the binary messages that was recorded ! Damned !! I guess I unfortunately have to add an plain text endpoint. That was my most promising experience in Load testing with SL.
3> Otherwise you can find at least 2 CodePlex projects {http://wcfloadtest.codeplex.com/ and http://teamtestplugins.codeplex.com/}.
that seams to be OK, but they are not really maintained. My first go with those tools did not succeed all the way through. only partially.
4> It seams HP Loadrunner 11 supports Silverlight 3 and 4 with WCF bin mode {but not advanced bindings, which is fine with me}.
I will try that in the following months {just the time for me to sort out the license and first install}.
5> Selenium Silverlight http://code.google.com/p/silverlight-selenium/updates/list and www.neotys.com
If you are more lucky with EASY tools {i.e. I know my VS 2010 would work with plenty of cheats and "CodePlexes" }, please HEEELLLLPPPPP !!
UPDATE > alleluia I just tried a Free {but with limited Virutal Users} add on to the great Fiddler2 beta.
It is stresstimulus > Follow the link here http://memoprojects.blogspot.com/2011/10/alleluia-load-test-tool-for-silverlight.html.
Vincent THAVONEKHAM.
SL4LoadTest@thavo.com
www.thavo.com

9 comments:

  1. "For the purists, I don't want to test Silverlight UI as such {i.e. for this type of test, I am not interested in Visual Studio 2010 Coded UI}, since it runs Client-side. Rather, I wanted to validate that, once deployed into production, this RIA will not put our IIS servers down. Hence, I want to simulate the load on Server-side, by generating lots of various calls to our web services easily."

    Sorry I still don't get it. What was not available but now achievable with your new tool?

    ReplyDelete
    Replies
    1. Thanks for your feedbacks,

      On large Silverlight project, "What was not available" ? I cannot achieve 2 things RAPIDLY :

      1) Silverlight UI tests that works first time easily and (without adding Dlls and hack in our code, finding objects in DOM, do it again once a UI element has changed or a x-y coordinate has moved, or a Browser zoom is not set to 100%, or pleinty of other small things that added all together just becomes a pain...).

      2) Testing RAPIDLY my Silverlight WCF web services mainly because they are in Binary mode activated (some tool would allow that, only after clicking many times on options & dropdown list for EVERY single Web Methods...)

      Well, this is applicable on at least the 4 major tools I came across, that are VS2010 SP1+FP2 with Coded UI, Telerik Studio (a very good product), HP QTP, HP Service Tests. I insist on RAPIDLY, as those tools work to a certain extend (although some performs very poorly).

      Vincent.

      Delete
  2. Hi Vincent,

    I am also trying to use Load Runner 11 (along with the multitude of patches on it) for performance testing a Silverlight application. Have to admit, HP didn't do a very good job of it. LR keeps hanging and crashing while working on a SL script, it's not at all stable. Running a single SL script in Vugen consumes more than 50% of cpu on a 4 cpu box and >400MB of memory. Also it's not possible to point your scripts to a different environment by importing another WSDL file. This is a serious drawback for the protocol, as also the fact that even minor changes to the script (cut paste some steps across actions) are also not supported.

    Wanted to know if you're also facing similar issues and has any workaround been found?
    Also, since this is a RIA app, the response times captured by LR are not really indicative of what the end user is experiencing due to a very heavy UI. Any means to incorporate client side performance testing to get a realistic picture of performance?

    Cheers,
    Manisha

    ReplyDelete
    Replies
    1. Hi, thanks for your comment.

      To be honest, since all those research of THE perfect tool for Silverlight takes for ever, I just gave up.

      A possible workaround:
      SQL Management tools (profiler, optimizer) + Stresstimulus + Create a volume database + tools to limit bandwidth ...

      Delete
  3. It is good to find the quality article related to software testing.

    ReplyDelete
  4. Actually VS2010 Ultimate would work well, if the Web Test Plugin were maintained and updated to the current state of the products. But it appears the developers of that product have pretty much abandoned listening to the user base...

    Thanks!
    ken

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. Hi Vincent,

    I have to load test silverlight 5 application. Need to perform both load and UI testing.

    Below are the app details-
    Silverght version 5, base 64 encoding with XAML serialization.
    Silverlight process is 64 bit.
    LoadRunner 11.5 doesnot support silverlight 5 app on 64 bit server.
    i am not able to decode the response.
    So, i have tried Fiddler2 with StressStimulus addin but not getting the required results. Please let me know if anything specific has to be done.

    Thanks,
    Swati

    ReplyDelete
  7. Hi Vincent,
    Thanks for the post! As a matter of update, StresStimulus has a new website www.stresstimulus.com
    New version 4.0 extended support for load testing of Silverlight 5 applications.
    Cheers,
    Tony

    ReplyDelete

Note: Only a member of this blog may post a comment.