CCE Theses and Dissertations

Date of Award

2014

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Computer Science (CISD)

Department

Graduate School of Computer and Information Sciences

Advisor

Gregory E. Simcoe

Committee Member

Francisco J. Mitropoulos

Committee Member

Sumitra Mukherjee

Keywords

Information science, benchmarking cost, changeload, cost reduction, resilience, self-adaptive, testing

Abstract

Ensuring the resilience of self-adaptive systems used in critical infrastructure systems is a concern as their failure has severe societal and financial consequences. The current trends in the growth of the scale and complexity of society's workload demands and the systems built to cope with these demands increases the anxiety surrounding service disruptions. Self-adaptive mechanisms instill dynamic behavior to systems in an effort to improve their resilience to runtime changes that would otherwise result in service disruption or failure, such as faults, errors, and attacks. Thus, the evaluation of a self-adaptive system's resilience is critical to ensure expected operational qualities and elicit trust in their services. However, resilience benchmarking is often overlooked or avoided due to the high cost associated with evaluating the runtime behavior of large and complex self-adaptive systems against an almost infinite number of possible runtime changes.

Researchers have focused on techniques to reduce the overall costs of benchmarking while ensuring the comprehensiveness of the evaluation as testing costs have been found to account for 50 to 80% of total system costs. These test suite minimization techniques include the removal of irrelevant, redundant, and repetitive test cases to ensure that only relevant tests that adequately elicit the expected system responses are enumerated. However, these approaches require an exhaustive test suite be defined first and then the irrelevant tests are filtered out, potentially negating any cost savings.

This dissertation provides a new approach of defining a resilience changeload for self-adaptive systems by incorporating goal-oriented requirements engineering techniques to extract system information and guide the identification of relevant runtime changes. The approach constructs a goal refinement graph consisting of the system's refined goals, runtime actions, self-adaptive agents, and underlying runtime assumptions that is used to identify obstructing conditions to runtime goal attainment. Graph theory is then used to gauge the impact of obstacles on runtime goal attainment and those that exceed the relevance requirement are included in the resilience changeload for enumeration. The use of system knowledge to guide the changeload definition process increased the relevance of the resilience changeload while minimizing the test suite, resulting in a reduction of overall benchmarking costs. Analysis of case study results confirmed that the new approach was more cost effective on the same subject system over previous work. The new approach was shown to reduce the overall costs by 79.65%, increase the relevance of the defined test suite, reduce the amount of wasted effort, and provide a greater return on investment over previous work by a factor of two.

Download
Share Feedback

Share

COinS