Changing variable values and examine the outcome difference between the altered and original data

Question

I recently read an approach which is used to find the effect of changing an independent variable.

They are doing a classification problem, so each data row (or record) is associated with an outcome of YES or NO.

They take one data row (i.e., test row), and then build a model using the other n-1 rows (more specifically, they use k nearest neighbour to build the model). They use the model to compute the probability of the outcome of that one particular test row. They get a value of, for example, 51% chance that the outcome for this data row is YES.

They then modify the value of one of the independent variable in that test row by X, and the use the model to examine the outcome again. They get a value of 55%, for example.

After the steps, they say that changing the independent variable by X can increase the probability of this test row being YES by 4% (55-51).

Since I am new to the field of statistical analysis and data mining...I am not sure whether this analysis approach is sound. I tried to google for other references for this approach, but I could't find any...

Could someone please help me explain whether this technique is valid or point me to some references? Thank you very much in advance!

Answer 1

I would judge this method as a sound one if we know their aim of doing such manipulation to the variable. However, I can share two cases in which such a procedure is sound.

Case 1: Improving Prediction

In some problems, the practitioner might face a prediction problem in which the target label distribution is not well-balanced between the classes of interest (e.g. the positive to negative class ratio is 8:2). One approach to deal with this problem is over-sampling. In over-sampling, new samples are added to the minority class through manipulating the values of the variables in the original samples. SMOTE is a known technique for applying this kind of over-sampling.

Case2: Causation and Prediction

In other contexts, manipulating the values of a variable and monitoring the effect of such manipulation on predicting the target variable can reveal a cause-effect relationship. A good example stated in the Causality Challenge "For instance, both smoking and coughing may be predictive of lung cancer (the target) in the absence of external intervention; however, prohibiting smoking (a possible cause) may prevent lung cancer, but administering a cough medicine to stop coughing (a possible consequence) would not." Thus, manipulating the values of the stop coughing variable would not change the predictivity.

Going back to the approach stated in the question, their method could be sound in terms of revealing causality of the variables. For improving predictivity, it would be sound to apply different manipulations on the training data rather than the testing one as they did.

In summary, we need their exact conclusion about the result or finding stated in "they say that changing the independent variable by X can increase the probability of this test row being YES by 4% (55-51)." to judge accurately their work.