Just a foreword, I'm not a mathematician or otherwise statistically skilled. I know my way around calculating standard deviations, but it's all self taught. I'm a programmer with limited stats knowledge.
Now, Assume that you run a "Widget Rental" website, and on your application you allow owners of "Widgets" to rent them out. You let the widget owners create seasons, (such as May - June, and August - September ), and then pick a price that it would cost to rent out the Widget during that time period.
With the prices that are entered, I want to be able to :
- Sort all widgets by "lowest price" available, so low-to-high in a result list of all widgets.
- Provide a "From price" for a widget. That is to say, "You can rent this widget for $XXX per day".
Each widget can have a different price based on the time of year. Some widgets will have dozens of different prices depending on the season as you get "high" seasons and "low" seasons.
However, the sellers of the "Widgets" are especially mischievous, and have realised that if they set their widget to be really expensive for one day of the year, and also really cheap one day of the year, then they can easily appear at the low and high sort ranges. And they are also using this to manipulate their "from Price".
Currently, I took a very naive solution in order to calculate the "lowest price" for a Widget, which is to just take the
lowest( N ) value from a dataset.
What I would like to is to get a "lowest from price" for a widget, which accurately portrays the price which it could be rented from.. and remove the lower/higher-band outliers.
Take a look at this chart... with values...
X Axis - Time (each significant interval is a day)
Y Axis - Price
The X axis is time, and the Y axis is the price. Now, this contains a normal distribution, and there aren't any real statistical outliers in that dataset. It's common to see the price between the lowest value and the upper value to fluctuate as much as 200% or a few standard deviations.
However, take a look at this second chart... It contains a single day tariff, which is only 20 ēuros...
I've played around with using Grubbs test and it seems to work quite well if you have a small dataset and don't take into account the duration of a date range. However, once you start trying to throw in a 3 days for a widget which is $1, it doesn't work well for removing the 3 clear outliers...
The important thing is that I want to get a "from price". That is to say, I want to be able to say, "You can rent this widget from XXXX". So it should be reflect the overall pricing taken as a whole and ignore clear outliers.
I'm going to implement this in PHP, so I don't have any amazing statistical tools at my disposal other than the core math libraries.