In order to capture and minimize the risks, security-critical ICT systems should be subjected to careful risk management in accordance with the established industry standard ISO 31000. However, for complex systems, risk management can be very complex and difficult. While the subjective assessment of experienced experts can be an acceptable method for risk analysis in small scale, other approaches need to be chosen as size and complexity increase. One possibility for a more objective analysis is the use of security testing according to ISO 29119. However, the testing itself can also be complicated and expensive, in particular if unintended, unknown behavior is to be analyzed. Even highly unsafe systems provide many harmless test results, as long as the "wrong" test cases are created and executed.
RACOMAT – Risk Assessment COMbined with Automated Testing
The Challenge: Risks of complex ICT systems
Networked information and communication systems (ICT) conquer our daily lives and change the industry. Critical infrastructures, such as power grids or the banking system, are nowadays already heavily dependent on them. The importance of ICT systems will continue to rise with autonomous means of transport (e.g. autonomous cars and trains, drones) in the near future. The direct, existential well-being of persons is thereby increasingly entrusted to information and communication systems. The requirements for their security and reliability are correspondingly high.
Combine risk assessment and security testing
One way to deal with these difficulties is to combine the different approaches and try to use the strengths. In a complex system, it is first necessary to carry out a high-level assessment of the risks based on experience and literature. In order to make this initial risk assessment more precise, it is possible to use security testing exactly where the first high-level risk image shows the greatest uncertainties. The objective test results can then be used to extend, refine or correct the previous risk image. Economically applicable to large complex ICT systems, however, this method is only provided with adequate tool support.
The RACOMAT Tool
The RACOMAT tool supports a component-based, compliant risk assessment. The tool uses intuitively understandable risk graphs to model and visualize a risk image. For risk analysis, known methods such as Fault Tree Analysis (FTA), event tree analysis (ETA) and the CORAS method can be used in combination in order to benefit from the different strengths of the individual processes.
The RACOMAT tool calculates how much effort is required for security testing in order to improve the quality of the risk image by reducing uncertainties, starting from a total budget for the risk assessment. The tool provides recommendations on how to use these resources. RACOMAT identifies and prioritizes relevant tests.
Automatic Security Testing
In order to be able to automate risk-based testing, a low-level risk assessment is required. For this, RACOMAT allows to model relations between artefacts from the risk analysis and elements of the investigated system. Specifically, the tool uses Threat Interfaces. These can represent components with their input and output interfaces as well as the associated potential weaknesses and hazards.
RACOMAT supports the allocation of potential vulnerabilities and threats with existing expertise from existing libraries such as MITER CAPEC and CWE or BSI IT Grundschutz. Experience values for important safety features - such as probabilities and consequences - are already included in this. Depending on the interface and technology, various potential vulnerabilities and threats are more relevant. The RACOMAT Tool automatically suggests the most relevant vulnerabilities and threat scenarios for identified Threat Interfaces. Thus, the analyst has only to go through a manageable checklist to ensure that nothing is overlooked.
The RACOMAT Tool performs automatic or at least semi-automatic testing using the Security Test Pattern. For typical attack scenarios, RACOMAT provides a catalog of predefined test patterns that can be instantiated without manual effort to test selected components. If there are no matching test patterns yet, new test patterns can be created in the RACOMAT Tool.
The observed test results are used in particular to calculate probabilities that attackers can trigger certain unintended events.
Dependencies and simulations of events
The RACOMAT Tool uses Monte Carlo simulations to calculate probabilities of occurrence for dependent events. Randomly distributed input values are used in a large number of simple event simulations in order to be able to determine approximate entry probability values even in the case of complex dynamic interrelations, even if the exact calculation of the solution would be too difficult.
The RACOMAT Process
The RACOMAT Tool suggests an iterative approach to risk assessment. An initial, coarse risk image is gradually improved in several rounds. The concepts of risk-based security testing (RBST) and test-based risk assessment (TBRA), i.e. the improvement of risk assessment using safety tests, are combined:
1. Develop the initial risk model based on literature, empirical values and expert assessment
2. Use event simulations to calculate the consequences and overall risks of threats
3. Select threat scenarios with the greatest uncertainties that are to be analyzed more closely using security tests
4. Generate required test cases
5. Perform tests and use the results to improve the risk model
6. Once again, use an improved risk model to carry out event simulations in order to determine overall risks more precisely
Continue with step 3 until the budget for the risk analysis is exhausted
7. Final risk evaluation and measures to reduce unacceptably high risks
Support for higher management
The use of domains-specific information provides further opportunities for support for large organizations. A plug-in has already been developed for the finance and banking sector, which actively supports the modeling of the business scenarios in BPMN and the association with the domains-specific technical infrastructure.
In future, wizards for other domains should also be developed and deployed as plug-ins for the RACOMAT tool.