2
$\begingroup$

Just a foreword, I'm not a mathematician or otherwise statistically skilled. I know my way around calculating standard deviations, but it's all self taught. I'm a programmer with limited stats knowledge.

Now, Assume that you run a "Widget Rental" website, and on your application you allow owners of "Widgets" to rent them out. You let the widget owners create seasons, (such as May - June, and August - September ), and then pick a price that it would cost to rent out the Widget during that time period.

With the prices that are entered, I want to be able to :

  • Sort all widgets by "lowest price" available, so low-to-high in a result list of all widgets.
  • Provide a "From price" for a widget. That is to say, "You can rent this widget for $XXX per day".

Each widget can have a different price based on the time of year. Some widgets will have dozens of different prices depending on the season as you get "high" seasons and "low" seasons.

However, the sellers of the "Widgets" are especially mischievous, and have realised that if they set their widget to be really expensive for one day of the year, and also really cheap one day of the year, then they can easily appear at the low and high sort ranges. And they are also using this to manipulate their "from Price".

Currently, I took a very naive solution in order to calculate the "lowest price" for a Widget, which is to just take the lowest( N ) value from a dataset.

What I would like to is to get a "lowest from price" for a widget, which accurately portrays the price which it could be rented from.. and remove the lower/higher-band outliers.

Take a look at this chart... with values...

Chart

X Axis - Time (each significant interval is a day)

Y Axis - Price

The X axis is time, and the Y axis is the price. Now, this contains a normal distribution, and there aren't any real statistical outliers in that dataset. It's common to see the price between the lowest value and the upper value to fluctuate as much as 200% or a few standard deviations.

However, take a look at this second chart... It contains a single day tariff, which is only 20 ēuros...

enter image description here

I've played around with using Grubbs test and it seems to work quite well if you have a small dataset and don't take into account the duration of a date range. However, once you start trying to throw in a 3 days for a widget which is $1, it doesn't work well for removing the 3 clear outliers...

The important thing is that I want to get a "from price". That is to say, I want to be able to say, "You can rent this widget from XXXX". So it should be reflect the overall pricing taken as a whole and ignore clear outliers.

I'm going to implement this in PHP, so I don't have any amazing statistical tools at my disposal other than the core math libraries.

Improve this question
$\endgroup$
2
$\begingroup$

Since this value will not be used for statistical testing or any further analysis, I am not sure that going through all the trouble of thinking about what is or is not an outlier is necessary. It is a much more complex question than you might suppose and, in fact, in many cases not a recommended approach. Note that any number you might come up with is not as easily interpretable as the minimum price and therefore not more legitimate than any other as the “right” summary, because it necessarily results from some assumptions about what is “normal” variation.

As a low tech alternative, why not take the median and be done with it? That way, you get a rough idea of the price that cannot be arbitrarily manipulated by changing the price on a single day. Some more exotic things like the “Winsorized mean” might also be relevant.

One scenario in which the median is less useful is a comparison between (A) a widget you can rent for 10\$ a day for 182 days of the year and 100\$/day for another 182 days and 50\$/day for the last remaining day and (B) a widget available for 50\$/day all year round. The median price per day is the same but the actual price will differ depending on the period you want to rent the widget. Of course, you could create similar scenarios for the mean or the Winsorized mean. Fundamentally a single number cannot always capture the whole price distribution or correctly reflect each potential renter's situation.

Yet another solution would be to order widgets using the highest end of their price range (i.e. starting with the widget with the lowest highest daily price – hope that makes sense) or display the whole range (“Rent it between x and y\$/day”) or a little graph so that “cheaters” can be spotted immediately by the users. Basically, deal with this as a design problem rather than a purely statistical one.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your input. I like the winsorized mean idea. This would definitely work better than a median value. I have thought about using rent between $X - $Y.. will give more thought. $\endgroup$ – Layke Jul 5 '13 at 11:33
0
$\begingroup$

To add to Gael's response, you could calculate the standard deviation for each widget price and then say that the price is usually between one standard deviation below and above the mean, which would be 67% of the cases. Obviously, you would use the actual values rather than the term standard deviation. Alternately, you could use a range of two SDs above and below if preferred, or some other range such as 80% of the time.

Improve this answer
$\endgroup$
1
  • $\begingroup$ 67% here is presumably an echo of the 68.3% or so of cases being within 1 SD of the mean for a normal distribution; but that's not a universal result. The key point is that means and SDs are themselves typically inflated by outliers, so do not provide resistant criteria, which are needed here. $\endgroup$ – Nick Cox Jul 5 '13 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.