Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks? (Video, ESEC/FSE 2020)

Опубликовано: 19 Август 2026
на канале: ACM SIGSOFT

869

"Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks? (Video, ESEC/FSE 2020)
Fabrice Harel-Canada, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim
(University of California at Los Angeles, USA; University of California at Los Angeles, USA; Virgina Tech, USA; University of California at Los Angeles, USA; University of California at Los Angeles, USA)

Abstract: Recent effort to test deep learning systems has produced an intuitive and compelling test criterion called neuron coverage (NC), which resembles the notion of traditional code coverage. NC measures the proportion of neurons activated in a neural network and it is implicitly assumed that increasing NC improves the quality of a test suite. In an attempt to automatically generate a test suite that increases NC, we design a novel diversity promoting regularizer that can be plugged into existing adversarial attack algorithms. We then assess whether such attempts to increase NC could generate a test suite that (1) detects adversarial attacks successfully, (2) produces natural inputs, and (3) is unbiased to particular class predictions. Contrary to expectation, our extensive evaluation finds that increasing NC actually makes it harder to generate an effective test suite: higher neuron coverage leads to fewer defects detected, less natural inputs, and more biased prediction preferences. Our results invoke skepticism that increasing neuron coverage may not be a meaningful objective for generating tests for deep neural networks and call for a new test generation technique that considers defect detection, naturalness, and output impartiality in tandem.

Article: https://doi.org/10.1145/3368089.3409754

Supplementary archive: https://doi.org/10.5281/zenodo.4021473 (Badges: Artifacts Available, Artifacts Evaluated — Reusable, Artifacts Evaluated — Functional)

Submitted to the conference by on 1970-01-01

Video Tags: Testing, Software Engineering, Machine Learning, Neuron Coverage, Adversarial Attack, fse20main-p661-p, DOI: 10.1145/3368089.3409754, DOI: 10.5281/zenodo.4021473, Artifacts Available, Artifacts Evaluated — Reusable, Artifacts Evaluated — Functional

Presentation at the ESEC/FSE 2020 conference, November 8–13, 2020, https://2020.esec-fse.org/
Sponsored by ACM SIGSOFT, https://www.sigsoft.org/

Twitter: / fseconf
Reddit: / esecfse "