The fundamental problems and impracticality of using MSTest

posted @ Monday, December 25, 2006 12:00 AM

 

I can't help but chime in about MSTest, in repsonse to the latest post by Ayende regarding the comments on one of Scott Bellware's posts.  This is just another area that Microsoft just doesn't seem to "get".  If you really understand the importance of the TDD rhythm, you know that speed is key.  That rapid feedback cycle is where, the high productivity paired with writing high quality code, happens.  If the time it takes you to write a test, make it fail and write the code to make it pass is more than a minute or two, you're not being as productive as you can be. 

TDD != Unit Test Generation

I've seen first hand (and am still seeing) how slow this process can be using MSTest and the "Test View" in Visual Studio 2005.  Sure it's got pretty colors but the important thing to remember is that Red-Green-Refactor is about a MINDSET, not COLORS!  Just because you've got reds and greens showing up in some window does NOT mean you're doing TDD.  Especially when Microsoft has made it so easy to generate unit tests for you.  As most of you know, this 'caused quite a backlash from those of us that are TDD practitioners in the real world.  Thank goodness Jeffrey Palermo set 'em straight.  :)

Miserably Slow Testing

But back to MSTest.  One of the primary reasons MSTest is so slow is that for every single test run, it creates copies of ALL of the assemblies into a unique testing folder under the "TestResults" folder.  It may not sound like a big deal, but I've seen this cause tests to run at least 3-4 times longer than with NUnit or TestDriven.NET (with MSTest).  But more on TestDriven.NET in a sec.  I understand that by creating copies of the assemblies for each test run, it better facilitates publishing those results into the Team Foundation Server data warehouse.  But at the very least, this should be optional.  It's highly impractical that you'll be publishing test results to the data warehouse while implementing a feature in a TDD session.  It would just break the whole TDD rhythm. 

Is There Hope For MSTest?

If you absolutely have to use MSTest, the tool that makes it bearable is TestDriven.NET.  I personally think that the TestDriven.NET plug-in has single handedly saved MSTest from imploding.  Just bind it to Ctrl-T (thanks to a tip from my co-worker, Paul) and you have a frictionless TDD experience.  It's hands down the fastest way to run tests (NUnit or MSTest) in Visual Studio 2005. 

Getting Into The Rhythm

What makes TestDriven.NET so great, is you can just simply write your test, Ctrl-T to see if fail very quickly, implement the code, Ctrl-T again and see it pass.  Usually all within a matter of seconds because it only runs the one test where your cursor is located.  One of the newest features (either it's new, or I've just recently discovered it) is that you don't even have to navigate back to your test method to re-run your last test.  Jamie's added an option for re-running your last test method no matter where your cursor is located.  This speeds things up even more because you can actually just stay in the code your implementing and repeatedly run the same test over and over.  This is very handy during the refactoring phase of TDD. 

Where MSTest (without TestDriven.NET) Misses The Mark

These simple features are what MSTest is missing.  You have to open up separate windows (i.e. Test View, Test Results, etc.) and highlight the test you want to run, etc.  It's just not the best way to do things.  As Scott noted, I have to wonder why Microsoft seemed to have developed this stuff in a vacuum, without consulting the folks who actually practice TDD.

So the bottom line is that if you want to be as productive as possible in your TDD adventures, use TestDriven.NET, preferrably with NUnit.  My TDD approach is kind of a hybrid between running tests using TestDriven.NET in Visual Studio 2005 and with my automated build script using nunit-console.exe. 

Perhaps I'll post a followup, if anyone's interested, detailing how I do TDD along with the tips and tools that I use to help me be productive as possible.

Comments
joeyDotNet - 6/2/2007 10:04 AM
# re: The fundamental problems and impracticality of using MSTest
Naysawn,
Judging from your article on the improvements, it sounds very promising. You've given me (yet another) reason to actually play with Orcas Beta 2. Just haven't yet found the time.

I'm going to try and give these features a test run (no pun int.) very soon and I'll post my thoughts about it here.

Freebee - 3/27/2008 1:06 PM
# re: The fundamental problems and impracticality of using MSTest
Thanks for great post
Post Comment






Please add 1 and 7 and type the answer here: