Tests are a blessing and a curse. They are the developer’s friend – I can barely write code nowadays without a test suite. How else will I know when I’ve broken something? I’m very lazy – I can’t be bothered to manually run through every feature, or even to start up the web server (or whatever) in most cases. Edit, save, run tests, edit again, is my preferred cycle, and for this, the tests must be as comprehensive as possible – not only testing the features, but testing the things that shouldn’t work, and testing every nasty, hard-to-track-down bug you’ve ever found – there’s nothing worse than spending hours tracking down the same bug that you fixed last month.
But for deployment, tests are just annoying. Take CPAN modules, for instance. Most of the time, yay, tests pass, install the module. But as with all things, all the interesting things happen when things break, and the tests don’t pass. In my experience, if there are any failures at all, either all the tests fail, because there’s a dependancy that failed to build and you can’t even ‘use’ the module, or one or two out of 45,000 tests failed, because there’s a tiiiny little broken case on whatever bizarre architecture I’m using this week, and I’m just going to force install it anyway. This would seem to be served better by a much simpler ‘does the thing superficially work?’ test suite used for deployment, separate from the development test suite.