GrammaTech Bug-Injector Research Receives IEEE SCAM 2019 Distinguished Paper
During the International Working Conference on Source Code Analysis & Manipulation (SCAM), a GrammaTech research publication was awarded the Institute of Electrical and Electronics Engineers (IEEE) Computer Society TCSE (Technical Council on Software Engineering) Distinguished Paper Award.
The paper, “Automated Customized Bug-Benchmark Generation,” describes Bug-Injector, a system that automatically creates benchmarks for customized evaluation of static analysis tools. This work was motivated by a desire to improve performance measurement of static analysis tools. Usually, tools are evaluated against established benchmarks. However, these have proven incomplete for evaluating modern, sophisticated tools.
Bug-Injector operates in a pipeline with three stages: instrument, execute, and inject & validate, as shown below.
Bug-Injector takes three inputs: (1) a host program in source format, (2) a set of tests for this program, and (3) a set of bug templates. It attempts to inject bugs from the set of bug templates into the host program, and returns multiple different buggy versions of the host program. Each returned buggy program variant contains at least one known bug (the one that was injected), and is associated with a witness—a test input which is known to exercise the injected bug.
Injected bugs are used as test cases to build a static analysis tool evaluation benchmark. In the benchmark, Bug-Injector pairs every injected bug with the program input that exercises that bug. Our team identified a broad range of requirements for bug benchmarks; with Bug-Injector, we generated on-demand test benchmarks to meet these requirements. Bug-Injector also allowed us to create customized benchmarks suitable for evaluating tools for a specific use case (e.g., a given codebase and class of bugs). Our experimental evaluation demonstrates the suitability of our generated benchmarks for evaluating static bug-detection tools, and for comparing the performance of different tools.
This article was originally published in https://blogs.grammatech.com/grammatech-wins-ieee-scam-2019-distinguished-paper-award-for-bug-injector-research