Continuous testing does not require static analysis
Ben Rady is the author of Infinitest, a continuous testing tool. This is an Eclipse or IntelliJ plug-in for Java that runs unit tests as you edit code. The key differentiator of Infinitest over other continuous testing plug-ins is that it runs only the unit tests that need to be run.
Ben describes how he accomplishes this feat. He uses static analysis of both the test and the code under test to determine whether tests need to run. Unfortunately, he's doing more work than he needs to. Continuous testing does not require static analysis.
Update Controls
I moderate an open source project called Update Controls for .NET. This project does data binding for WPF, Winforms, and Silverlight. It has absolutely nothing to do with a continuous testing plug-in for Java.
Or does it?
Update Controls automatically discovers the data that your properties depend upon. When that data changes, those properties are updated. You might think that it's doing some sort of static analysis to discover dependencies, but it's much simpler than that.
Say you have a dependent property "A". As the property getter for "A" is executing, Update Controls is watching. When another property called "B" is referenced, Update Controls says "Aha! A depends upon B." It's that simple.
Why does this work?
Most code is deterministic. If you call it providing the same inputs, it will produce the same outputs. Furthermore, it will do so in exactly the same way. So execute some code and it see what set of properties it reads. Execute it again, and it will still read the same set of properties.
Now, change a property not in the set that was read. Does the behavior of the code change?
No! The code only observed one set of properties. It doesn't care what happens to any other properties. It will still read that same set, and it will still produce the same results.
OK, this time change a property that is in the set. Does the behavior of the code change?
You bet it does. In fact, it may change drastically. It may even read an entirely different set of properties next time. So all bets are off. Once you change a property upon which it depends, you have to dump the entire set and run the code again. Then you can gather up a brand new set.
Back to Infinitest
So, Ben, my advice to you. Drop the static analysis. Instead pick up a code coverage tool. Look at all of the lines of code covered by each test. If any of those lines changes, you have to rerun the test. If not, you don't. Simple as that.
September 17th, 2009 at 9:51 pm
Mike,
So there's two problems we've found with using code coverage data for test selection in a continuous test runner. I covered some of this in the talk I gave at Agile2008 on continuous testing (http://submissions.agile2008.org/node/377).
One, if you're using code coverage to determine which tests to run, you need to run all the tests to get it (which kind of defeats the purpose of test selection). Static analysis allows you to pick which tests to run before actually running them. Sure, you can track the state of the line level dependencies and try to maintain it as classes change, but....
the other problem with this approach is the huge volume of data you generate. Because it's so simple, Infinitest is able to keep it's dependency graph in memory. That means it runs quickly and easily scales to workspaces with tens of thousands of classes. If we kept line level dependency information for each test, we'd have to store it on disk. That might be OK for a build tool or something like that, but for a CT runner, that kind of file IO would kill the responsiveness of the tool.
Thanks for the feedback.
Ben