We need a radical change in the way we approach software assessment both in practice and in research.
Assessment is what we need to do before taking a decision. Assessment is a critical software engineering activity, often accounting to as much as 50% of the overall development effort. However, in practice this activity is regarded as secondary and it is dealt with in an ad-hoc way. This does not service. We should recognize it explicitly and approach it holistically as a discipline.
Why as a holistic discipline?
Because software evolution is a multidimensional phenomenon that exhibits itself in multiple forms and at multiple levels of abstraction. For example, software evolution spans over multiple topics such as modeling, data mining, visualization, human-computer interaction, or even language design. There exists an extensive body of research in each of these areas, but these approaches are mostly disparate and thus have little overall impact. We need a new kind of research effort that deals with their integration.
Ultimately, assessment is a human activity that concerns taking decisions in specific situations. Thus, to be effective, assessment must go beyond general technicalities and deal with those specific situations. For example, instead of having a predefined generic tool, we should be able to craft one that deals with the constraints of the system under study.
To accommodate the scale of the problem, the research methods should be adapted to the task as well. First, it is critical to integrate tool building into the research process, because without scalable tools we cannot handle large. Second, we have to work collaboratively both to share the practical costs and to integrate our approaches.
Holistic software assessment might sound too ambitious. But, it is simply too expensive to do it otherwise.
Comments
Hi Doru!
I agree with you. I have the feeling that having such holistic assessment is not easy to realize, however. For example, we have produced Profiling blueprints (@tools europe 2010). This involved several dimensions: code instrumentation, code analyze, visualization, benchmarking. We had to balance each of these dimensions to obtain something useful and practical. Did we had an holistic approach? Frankly speaking I have no idea. However, we made some practical contributions: Mondrian is 43% faster, and we have shown there are other way to visualize program execution. Our work, test blueprint, may not have changed the face of the World, but it’s application, which is a side effect of our work, has an impact on the moose community. Going from an idea to a general benefit is maybe the key to holistically assess. Closing the loop is always good.
Alexandre
By holistic I mean that the problem of assessment in general needs a broader perspective that should integrate multiple domains (like you mentioned) and more importantly include context as an explicit and central variable.
When you say that you improved Mondrian, dealing with the specifics of Mondrian is where the value actually is. The general technique must be applied in a context to be effective, and for this to happen we need those techniques to be parameterize-able.