Fine-tuning of methods before main analysis: p-value hacking? I am reviewing a paper for a journal. The authors propose a new classification scheme. They then try to assess the classification accuracy using some novel predictor data. Standard classification techniques (SVM, KNN, etc) are used. I came to know of a detail of what they did, via the response document that they gave to my initial set of comments. Hence this message.
Let us say there are two ways to process the predictor data, and derive the prediction metrics (independent variables) associated with the classification. Let us call them M1 and M2 (method1 and method2). The authors first tried both the methods using the entire sample data set, and they concluded that M1 was better. This was done because M1 was giving better results (accuracy stats) with the data. Then, they did their "main analysis" with M1. And in the final paper, they just talk about M1, and they give a logical-sounding reason for selecting M1. That is, they do not mention that they had tested M1 and M2. They derived accuracy stats using cross validation (train/test ratio 80:20). I think that these accuracy stats (eg, the overall accuracy, kappa values) associated with sample/M1 will now not represent that of the population. Am I right? Or is it just the confidence intervals associated with those stats?
I also think it is a version of "p-value hacking / data dregring". But it is more like "methods dredging", where they first use the data to select a "good/better" method. And then in the main paper, they do not mention this step. They (rather) give a logical reason for selecting method M1, and they go on to state the results of using this method.
 A: This is a textbook case of HARKing: Hypothesizing After Results are Known (Kerr, 1998). Not disclosing this step is indeed more than borderline unethical. The authors should prominently note that they did this preprocessing step, because it makes all their results less reliable.
Essentially, all statistics (i.e., parameters estimated from data, like the accuracy of a classifier or many other parameters) are random variables. Evaluating statistics across multiple methods, models or samples and then cherry-picking the largest or most significant ones means that we tend to pick the ones that just randomly in this particular dataset turned out to be large. It may well be that in a similar new dataset, this statistic is lower and another one higher, simply due to sampling variability. Any estimate driven by this kind of cherry-picking will be biased, i.e., it will systematically over-estimate the true parameter value. And confidence intervals that are calculated without taking this additional "snooping" step into account will be too low, i.e., they will have lower than their nominal coverage. Correcting for this "path through the garden of forking paths" (take a look at Gelman's paper) is highly nontrivial.
(Also, accuracy is a seriously flawed evaluation measure.)
