Let me clear up some misconceptions first. The estimate of your population variance is not high because your sample is small. In fact, just the opposite is often the case, the variance tends to be low because small samples over represent the peak of the distribution. The variances of larger samples are more representative and more accurate. And, as a corollary, small samples are less accurate and have a higher sampling error, usually measured as standard error.
Data are generally not discarded merely because they have something that might be judged as high variance. The variance is considered a property of the data that you need to discover and you attempt to get enough data to get a reasonable determination of that. Looking at your data these don't seem like very high variances at all. You can actually get very useful information out of data like that but you will need more of it.
If you throw away these data and just get another small sample and it has lower variance that doesn't tell you that the underlying distribution has lower variance, it's just sampling variability. Therefore, don't do that. Just keep collecting more data and noting properties of it like time of day. If it's relatively consistently noisy over a period of time you might just be able to average it all together and get great distributions of the two different kinds of signal.
Clearly there is overlap in your distributions and you're going to have to take some time to get this working correctly. You'll need to collect lots of samples of each of your different signals to see if your manipulations in the algorithm have effects. There's enough noise that it would be easy to fool yourself into thinking you had succeeded in solving the problem if you just throw away samples you don't like. There's also enough noise that you might not have much of a problem and continue to think you're failing if you throw away samples you don't like.
In short, keep all of the data and get more of it. Work out the distributions. Make adjustments to your algorithm. Collect more data. Repeat until you've solved the problem.
When you do get more data and you've tried a couple of algorithms come back and ask for help on exactly how to model your data so that you can make decisions about that algorithms to keep and what to reject. At that time you might post more summary type statistics with your question like means, variances, and perhaps a histogram.
You might want to also ask for help from a cognitive psychologist who specializes in applied issues. There will be some tradeoff between mean QoS and the variance that you may be better off minimizing the variance even if it drops the mean. But I'm betting that should have been done by people other than you.