Few weeks ago I conducted a study where I measured the effectiveness of different interaction techniques for a class project. I measured users performance time and accuracy. In the report I wrote following description as a method I used for outlier detection.
"To account for data quality from online data collection, outlier handling was performed to account for trials where participants were likely to have disruptions or mistakes that were greater than would be expected with a usual attempt. For instance, trials having very long completion times were excluded because users likely did not spend the entire duration performing the single task in such cases. We excluded 268 (9%) of the collected responses as outliers based on interquartile range (IQR), where an outcome was considered an outlier if it was more than 1.5 times the size of the IQR away from either the lower or upper quartiles."
My instructor then gave me following comment:
Instructor Response: "please explain on which set(s) of observations the outlier removal procedure was applied. In case the procedure was applied on sets that contain observations from different techniques, this could highly bias the results. For example if a particular technique is much worse than others, then most of the removed outliers could come from that technique, and the procedure would make that technique appear much better than it is. Please double check and in case such an imbalance is found, consider re-analyzing your data. "
Any idea what should I answer? Do I need to run extra analysis?
