I am trying to understand the linked paper at ICSE 2008 (Namin et al. 2008).

To provide an overview: A mutant is a variant program generated from a software program that has two distinct results -- killed or alive. The mutation score or adequacy ratio ($AM$) is the ratio between the number of mutants killed and the total number of mutants. That is, $AM = \frac{|killed(M)|}{|M|}$ where $M$ is the set of all generated mutants, and $killed(M)$ is the subset that were killed. The mutants may be divided into non-overlapping sets based on how they were created -- the mutation operators in the paper.

That is, given an operator $i$ which produces $m_i$ mutants, and $n$ operators in total, $\sum_{i=1}^{n} |m_i| = |M|$, and $\sum_{i=1}^{n} |killed(m_i)| = |killed(M)|$.

Any given program will generate a very large number of mutants. The key question is how best to estimate the mutation score of a mutant population generated from a given population by using only a small subset.

The paper uses the following approach: It splits the mutants generated by the mutation operator used. Then the authors try to fit a linear model for the complete mutation score:

$AM = k + c_1 Am_1 + c_2 Am_2 ... c_n Am_n$

where $Am_i$ represents the mutation score calculated from mutants from a particular operator. That is, for the operator $1$ which produces $m_1$ mutants,

$Am_1 = \frac{|killed(m_1)|}{|m_1|}$

The paper uses forward selection of models to find the best fit model that uses a smaller subset of mutation operators (and hence a smaller subset of mutants) to predict the full score.

From my understanding, one can predict the full mutation score with statistical confidence if one considers it as a strata sampling problem, with different mutation operators as different strata, and ensure the right ratio of mutants in the sample. My question is, how does the authors' approach compare to the strata sampling approach? Which one can be expected to be more accurate? I have not seen the authors' approach before, and lack a good understanding of its why it may be better than strata sampling, and I would be happy to see either an analysis of why one or the other approach is better, or reading materials for further understanding.

  • $\begingroup$ I am not sure what the research question is, but forward selection is a wrong answer. See @frankharrell's book on the topic. You could use lasso or ridge regression or whatever, but frankly this sounds like research design flaw -- "We have not collected enough data to estimate all the parameters, so we have to use a model selection procedure". In this case, what do you think you should stratify on? For variance estimation purposes, stratification is largely similar to linear regression on strata indicators, but then in practice, you stratify to oversample the strata with the highest variances. $\endgroup$ – StasK Oct 26 '17 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.