The Security Testing Improvements Profile (STIP)
Fraunhofer has developed an assessment scheme called Security Testing Improvements Profile (STIP), that is dedicated to assess security testing processes. It can be used stand alone or in addition to established test process assessment approaches. STIP is based on the general ideas of TMMi and TPI. Thus, we have defined a set of key areas that we considered relevant for security testing. The key areas describe major aspects or activities in a security testing process.
The key areas are grouped in four main groups and defined to be self-contained and distinct so that each of the areas represents a relevant aspect of a security testing process. For each of the key areas we have defined a performance scale with up to four levels that are hierarchically organized and build on each other. The levels can be used to evaluate concrete security testing processes with respect to their performance in the belonging key area. Each level with a higher number represents an improvement for the underlying security testing process. Each higher level is better than its prior level in terms of time (faster), money (cheaper) and/or quality (better).
Application to the DIAMONDS Case Studies
Figure 2 shows the results of the case study on testing the application software for bank note sorting machines. The score before the project started is denoted in red and after the project in blue. We can observe that the case study advanced in nearly every aspect of security testing. The case study gained from nearly all the relevant innovations of the DIAMONDS project with the exception of monitoring. The biggest gains were made in the areas Fuzzing and Test generation where the case study was used as a driver for the research project.