0
$\begingroup$

Few weeks ago I conducted a study where I measured the effectiveness of different interaction techniques for a class project. I measured users performance time and accuracy. In the report I wrote following description as a method I used for outlier detection.

"To account for data quality from online data collection, outlier handling was performed to account for trials where participants were likely to have disruptions or mistakes that were greater than would be expected with a usual attempt. For instance, trials having very long completion times were excluded because users likely did not spend the entire duration performing the single task in such cases. We excluded 268 (9%) of the collected responses as outliers based on interquartile range (IQR), where an outcome was considered an outlier if it was more than 1.5 times the size of the IQR away from either the lower or upper quartiles."

My instructor then gave me following comment:

Instructor Response: "please explain on which set(s) of observations the outlier removal procedure was applied. In case the procedure was applied on sets that contain observations from different techniques, this could highly bias the results. For example if a particular technique is much worse than others, then most of the removed outliers could come from that technique, and the procedure would make that technique appear much better than it is. Please double check and in case such an imbalance is found, consider re-analyzing your data. "

Any idea what should I answer? Do I need to run extra analysis?

$\endgroup$

1 Answer 1

1
$\begingroup$

Your instructor asked details of how outliers were defined. Was it done per group (per technique) or on whole dataset? In case of full dataset some data from techniques that may have tendency to show better/worser results could've been defined as outliers, when in reality it's their effect. So to quote your instructor: this could highly bias the results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.