I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case.
I have a dataset that I'm analyzing where in order to achieve "good" results, the data must be restricted in several ways. When I began restricting, there were over 140,000 entries, and after cutting out all the "bad" entries, there's about 2,000 entries. The next step in my analysis is to cut out the outliers in the data. This is done based on the percentile of a previous draft of the dataset that has about 4,000 entries. A colleague of mine did a similar analysis and cut out any outliers that were below the 1st percentile and above the 99th percentile. It's worth noting that their dataset was much smaller than mine, so they couldn't restrict much more beyond that point without getting into issues of sample size.
My conundrum: The more outliers I cut out, the better my results get. I get truly terrible results when I only cut out outliers below the 1st and above the 99th percentile. I've gone all the way up to cutting out below the 10th and above the 90th percentile and my results are much more realistic. My question is how far can I go with this? What basis do I have for deciding where the cut-off is for restricting the data? My gut says that data between the 10th-90th percentile is too restricted already, but it's tempting to report it because the results are truly so much better.