Abstracts

Test-suite Augmentation for Evolving Software

R. Santelices, P.K. Chittimalli, T. Apiwattanapong, A. Orso, and M.J. Harrold

One activity performed by developers during regression testing is test-suite augmentation, which consists of assessing the adequacy of a test suite after a program is modified and identifying new or modified behaviors that are not adequately exercised by the existing test suite and, thus, require additional test cases. In previous work, we proposed MaTRIX, a technique for test-suite augmentation based on dependence analysis and partial symbolic execution. In this paper, we present the next step of our work, where we (1) improve the effectiveness of our technique by identifying all relevant change-propagation paths, (2) extend the technique to handle multiple and more complex changes, (3) introduce the first tool that fully implements the technique, (4) present an empirical evaluation performed on real software. Our results show that our technique is practical and more effective than existing test-suite augmentation approaches in identifying test cases with high fault-detection capabilities.

Identifying Testing Requirements for Modified Software

T. Apiwattanapong

Throughout its lifetime, software must be changed for many reasons, such as bug fixing, performance tuning, and code restructuring. Testing modified software is the main activity performed to gain confidence that changes behave as they are intended and do not have adverse effects on the rest of the software. A fundamental problem of testing evolving software is determining whether test suites adequately exercise changes and, if not, providing suitable guidance for generating new test inputs that target the modified behavior. Existing techniques evaluate the adequacy of test suites based only on control- and data-flow testing criteria. They do not consider the effects of changes on program states and, thus, are not sufficiently strict to guarantee that the modified behavior is exercised. Also, because of the lack of this guarantee, these techniques can provide only limited guidance for generating new test inputs.

This research has developed techniques that will assist testers in testing evolving software and providing confidence in the quality of modified versions. In particular, this research has developed a technique to identify testing requirements that ensure that the test cases satisfying them will result in different program states at preselected parts of the software. This research has also developed supporting techniques for identifying testing requirements. Such techniques include (1) a differencing technique, which computes differences and correspondences between two software versions and (2) two dynamic-impact-analysis techniques, which identify parts of software that are likely affected by changes with respect to a set of executions.

JDiff: A Differencing Technique and Tool for Object-Oriented Programs

T. Apiwattanapong, A. Orso, and M.J. Harrold

During software evolution, information about changes between different versions of a program is useful for a number of software engineering tasks. For example, configuration-management systems can use change information to assess possible conflicts among updates from different users. For another example, in regression testing, knowledge about which parts of a program are unchanged can help in identifying test cases that need not be rerun. For many of these tasks, a purely syntactic differencing may not provide enough information for the task to be performed effectively. This problem is especially relevant in the case of object-oriented software, for which a syntactic change can have subtle and unforeseen effects. In this paper, we present a technique for comparing object-oriented programs that identifies both differences and correspondences between two versions of a program. The technique is based on a representation that handles object-oriented features and thus, can capture the behavior of object-oriented programs. We also present JDIFF, a tool that implements the technique for Java programs. Finally, we present the results of four empirical studies, performed on many versions of two medium-sized subjects, that show the efficiency and effectiveness of the technique when used on real programs.

MaTRIX: Maintenance-oriented Testing Requirement Identifier and Examiner

T. Apiwattanapong, R. Santelices, P.K. Chittimalli, A. Orso, and M.J. Harrold

This paper presents a new test-suite augmentation technique for use in regression testing of software. Our technique combines dependence analysis and symbolic evaluation, and uses change information between old and new versions of a program to (1) identify parts of the program affected by the changes, (2) compute the conditions under which the effects of the changes are propagated to such parts, and (3) create a set of testing requirements based on the computed information. Testers can use these requirements to assess the effectiveness of the regression testing performed so far and to guide the selection of new test cases. The paper also presents MaTRIX, a tool that implements our technique, and its integration into a regression-testing environment. Finally, the paper presents two empirical studies that demonstrate both the effectiveness of our technique and the shortcomings of previous techniques in assessing the adequacy of a test suite in exercising program changes.

Efficient and Precise Dynamic Impact Analysis Using Execute-After Sequences

T. Apiwattanapong, A. Orso, and M.J. Harrold

As software evolves, impact analysis estimates the potential effects of changes, before or after they are made, by identifying which parts of the software may be affected by such changes. Traditional impact-analysis techniques are based on static analysis and tend to identify most of the software as affected by the changes, due to their conservative assumptions. More recently, researchers have begun to investigate dynamic impact-analysis techniques, which rely on dynamic, rather than static, information about software behavior. Existing dynamic impact-analysis techniques are either very expensive-in terms of execution overhead or amount of dynamic information collected-or imprecise. In this paper, we present a new technique for dynamic impact analysis that is almost as efficient as the most efficient existing technique and is as precise as the most precise existing technique. The technique is based on a novel algorithm that collects (and analyzes) only the essential dynamic information required for the analysis. We discuss our technique, prove its correctness, and present a set of empirical studies in which we compare our new technique with two existing techniques, in terms of performance and precision.

A Differencing Algorithm for Object-oriented Programs

T. Apiwattanapong, A. Orso, and M.J. Harrold

During software evolution, information about changes between different versions of a program is useful for a number of software engineering tasks. For example, in regression testing, knowing which parts of a program are unchanged can help identifying test cases that need not be rerun. For many of these tasks, a purely syntactic differencing may not provide enough information for the task to be performed effectively. This problem is especially relevant in the case of object-oriented software, for which a syntactic change can have subtle and unforeseen effects. In this paper, we present a technique for comparing object-oriented programs that identifies both differences and correspondences between two versions of a program. The technique is based on a representation that handles object-oriented features and, thus, can capture the behavior of object-oriented programs. We also present JDiff, a tool that implements the technique for Java programs, and empirical results that show the efficiency and effectiveness of the technique on a real program.

An Empirical Comparison of Dynamic Impact Analysis Algorithms

A. Orso, T. Apiwattanapong, J. Law, G. Rothermel, and M.J. Harrold

Impact analysis - determining the potential effects of changes on a software system - plays an important role in software engineering tasks such as maintenance, regression testing, and debugging. In previous work, two new dynamic impact analysis techniques, CoverageImpact and PathImpact, were presented. These techniques perform impact analysis based on data gathered about program behavior relative to specific inputs, such as inputs gathered from field data, operational profile data, or test-suite executions. Due to various characteristics of the algorithms they employ, CoverageImpact and PathImpact are expected to differ in terms of cost and precision; however, there have been no studies to date examining the extent to which such differences may emerge in practice. Since cost-precision tradeoffs may play an important role in technique selection and further research, we wished to examine these tradeoffs. We therefore designed and performed an empirical study, comparing the execution and space costs of the techniques, as well as the precisions of the impact analysis results that they report. This paper presents the results of this study.

Leveraging Field Data for Impact Analysis and Regression Testing

A. Orso, T. Apiwattanapong, and M.J. Harrold

Software products are often released with missing functionality, errors, or incompatibilities that may result in failures, inferior performances, or, more generally, user dissatisfaction. In previous work, we presented the Gamma approach, which facilitates remote analysis and measurement of deployed software and allows for gathering program-execution data from the field. In this paper, we investigate the use of the Gamma approach to support and improve two fundamental tasks performed by software engineers during maintenance: impact analysis and regression testing. We present a new approach that leverages field data to perform these two tasks. We also present a set of empirical studies that we performed to assess the usefulness of the approach. The studies were performed on a real subject and on a real user population. The results of the studies show that the use of field data is effective and, for the cases considered, can considerably affect the results of dynamic analyses. Moreover, the empirical studies show that the approach is also efficient: the kind of field data that we consider requires very limited space and little instrumentation to be collected.

Improving Impact Analysis and Regression Testing Using FieldData

A. Orso, T. Apiwattanapong, and M.J. Harrold

Software products are often released with missing functionality, errors, or incompatibilities that may result in failures, inferior performances, or, more generally, user dissatisfaction. In previous work, we presented the Gamma approach, which enables analyses that (1) rely on actual eld data instead of synthetic in-house data and (2) leverage the vast and heterogeneous resources of an entire user community. In this paper, we investigate the use of the Gamma approach to support and improve two fundamental tasks performed by software engineers during maintenance: impact analysis and regression testing. We propose a new approach that leverages eld data to perform these two tasks. We also discuss ongoing empirical studies, performed on a real subject and on a real user population, to assess the feasibility of the approach.

Selective Path Profiling

T. Apiwattanapong and M.J. Harrold

Recording dynamic information for only a subset of program entities can reduce monitoring overhead and can facilitate efficient monitoring of deployed software. Program entities, such as statements, can be monitored using probes that track the execution of those entities. Monitoring more complicated entities, such as paths or definition-use associations, requires more sophisticated techniques that track not only the execution of the desired entities but also the execution of other entities with which they interact. This paper presents an approach for monitoring subsets of one such program entity-acyclic paths in procedures. Our selective path profiling algorithm computes values for probes that guarantee that the sum of the assigned value along each acyclic path (path sum) in the subset is unique; acyclic paths not in the subset may or may not have unique path sums. The paper also presents the results of studies that compare the number of probes required for subsets of various sizes with the number of probes required for profiling all paths, computed using Ball and Larus' path profiling algorithm. Our results indicate that the algorithm performs well on many procedures by requiring only a small percentage of probes for monitoring the subset.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License