Waterfall is the best process: The analysis reveals all facets of the requirements, the design translates the requirements into technical blueprints, and later these are matched by the implementation. Testing is actually optional, given that everything is thought through upfront.

Waterfall is the best process, only not for humans. We just cannot think everything through. Even when we think we did take everything into account, we learn that we missed at least some details. The main reason is that our capacity is limited and thus we can only process a reduced amount of data. It is only logical that we get to miss details. Another reason is that we all perceive reality in different ways, and as soon as more people need to work together, the issue of communicating adds to the problem.

As a consequence, a more humane process is one that embraces our limitations. Agile processes do that. Instead of striving to conceive the perfect solution upfront, they promote iterations, feedback and the ability to react and correct the current situation.

However, to be able to react, you need to know what to react to. To take the right decision we need to be able to assess the situation accurately at all times, and given the fast development pace, we need to do it in a timely fashion, too. Effective assessment requires the relevant information to be available as fast as possible.

The only tiny problem is that software systems are large and complex and they handle large amounts of data. Thus, they present many details that make difficult the identification of the relevant information.

Approaching the problem in an ad-hoc manner does not benefit us. For example, when it comes to assessing software systems we mostly rely on code reading. Granted, modern IDEs do provide some help, but this solution does not scale when we want to reason about a system as a whole. To put it in perspective, a person that reads one line in two seconds would require approximately one month of work to read a quarter of a million lines of code. And this is just for reading the code.

Various studies report assessment to account for as much as 50% of the total development effort. Assessment is an important activity and it should be addressed explicitly in the development process. Especially in an agile development process.

On the one hand, we should have tools that do the grunt work, and that allow us to focus on the relevant information (yes, Moose is a great such a tool :)). On the other hand, we need better techniques and practices. Tools are great, but they are just tools. We should not forget that assessment is a human activity and the goal is taking better and timely decisions.