1
$\begingroup$

I'm creating a program where my goal is to analyse lists of ticket prices and locate tickets that are priced abnormally low in relation to the spread of prices.

My preliminary model does the following to the price list in this order..(examples are at the end of my writeup - they include the modifications made after running through the model)

  1. Removes any tickets that are priced over 999.00 . Sometimes tickets will be priced intentionally at 9999.99 to ensure they do not sell - so i don't want to use them in my calculations.
  2. Removes any tickets that are priced 1.5 standard deviations above the new mean. I found that unrealistically high priced tickets were unnecessarily influencing the mean.
  3. Recalculates the mean and SD and pulls tickets that are 2.9 standard deviations BELOW the mean (I'm calling this the "Price Floor".

This results in some success, however it also produces Price Floors that are negative in some cases.

I would like to improve my model to be able to take any price list and develop "Price Floors" that are more realistic.

My questions are:

  1. Am I going down the right path with my current model?
  2. Would you change any aspects of my current model?
  3. Are there other methods available that may be more effective than standard deviation?

Below are two samples of the data I'm working with using the model I described above. In Set 1, the Price Floor is 36.48, thus producing one result of 36.1.

In Set 2, the Price Floor is -3.45. While I would not expect a result from Set 2, I feel that my model is inadequate due to the unrealistic Price Floor.

Thank you so much for any help you can provide.

Set 1 (Price Floor: 36.48)

[0] => 36.1
[1] => 41.6
[2] => 43.8
[3] => 44.9
[4] => 45
[5] => 46
[6] => 46
[7] => 46
[8] => 46
[9] => 46
[10] => 48.2
[11] => 49.03
[12] => 49.3
[13] => 49.3
[14] => 49.3
[15] => 50.4
[16] => 51.5
[17] => 51.5
[18] => 51.5
[19] => 51.5
[20] => 51.5
[21] => 52.05
[22] => 52.47
[23] => 52.6
[24] => 52.6
[25] => 52.6
[26] => 52.6
[27] => 52.6
[28] => 52.6
[29] => 53.7
[30] => 53.7
[31] => 53.7
[32] => 53.7
[33] => 53.7
[34] => 53.7
[35] => 54.8
[36] => 54.8
[37] => 54.8
[38] => 55.9
[39] => 55.9
[40] => 55.9
[41] => 55.9
[42] => 55.9
[43] => 55.9
[44] => 55.9
[45] => 55.9
[46] => 55.9
[47] => 55.9
[48] => 55.9
[49] => 57
[50] => 57
[51] => 57
[52] => 57
[53] => 57
[54] => 57
[55] => 59.2
[56] => 59.2
[57] => 59.2
[58] => 59.2
[59] => 59.2
[60] => 60.3
[61] => 61.4
[62] => 62.5
[63] => 64
[64] => 64.15
[65] => 64.15
[66] => 64.15
[67] => 65
[68] => 65
[69] => 66
[70] => 67.45
[71] => 67.45

Set 2 (Price Floor -3.45)

[0] => 46
[1] => 58.1
[2] => 58.38
[3] => 59.2
[4] => 61.4
[5] => 62.5
[6] => 64.7
[7] => 65.25
[8] => 65.25
[9] => 65.25
[10] => 65.25
[11] => 66.35
[12] => 66.9
[13] => 67.99
[14] => 67.99
[15] => 67.99
[16] => 67.99
[17] => 67.99
[18] => 68.55
[19] => 70.2
[20] => 71.85
[21] => 71.85
[22] => 72.4
[23] => 72.4
[24] => 73.5
[25] => 73.5
[26] => 73.5
[27] => 74.83
[28] => 75.7
[29] => 79
[30] => 79
[31] => 79.55
[32] => 80.1
[33] => 84.5
[34] => 84.5
[35] => 84.5
[36] => 85.05
[37] => 87.8
[38] => 89.45
[39] => 93.3
[40] => 95.5
[41] => 97.15
[42] => 98.76
[43] => 98.8
[44] => 99.66
[45] => 106.5
[46] => 108.7
[47] => 112
[48] => 112
[49] => 119.15
[50] => 119.15
[51] => 122.45
[52] => 123
[53] => 124.65
[54] => 125.75
[55] => 125.75
[56] => 128.5
[57] => 128.5
[58] => 133.45
[59] => 134
[60] => 135.65
[61] => 136.2
[62] => 139.5
[63] => 139.5
[64] => 139.5
[65] => 140.6
[66] => 149.95
[67] => 150.5
[68] => 154.9
[69] => 160.4
[70] => 161.5
[71] => 164.25
[72] => 164.8
[73] => 165.9
[74] => 167
[75] => 167
[76] => 167
[77] => 167.55
[78] => 169.75
[79] => 171.4
[80] => 172.5
$\endgroup$
  • 1
    $\begingroup$ can you edit you question to only leave the numbers (no [#], no arrows). $\endgroup$ – user603 May 21 '13 at 7:56
  • 1
    $\begingroup$ @user603 See my answer below for cut-and-paste code fragment to read in his data. $\endgroup$ – Bill May 21 '13 at 16:25
  • $\begingroup$ Sorry for the formatting issues.. Thank you so much for your responses.. I'm far from fluent with stats and this is a huge help. I'm on the road right now and will take a closer look later today. $\endgroup$ – rizzle May 21 '13 at 19:27
3
$\begingroup$

You are worried about outliers (values that are "too high" or "too low") affecting the mean and standard deviation of ticket prices. This is a very reasonable worry, and there are statistical tools to deal with it. The tools go by the name of "robust statistics."

In your case, I would make three suggestions. First, instead of using the mean, use the median. It can't be moved around (much) by crazy high and crazy low values. Second, instead of using standard deviations to define your price floors, use percentiles. Third, don't manually throw away data: let the robust methods ignore the crazy outliers for you.

Maybe put your price floor at the 5th percentile. In set 1, that would make the price floor around 45. In set 2, that would make the price floor around 61.

R code, including code to read in inconveniently formatted data:

> # R script out of order.  Code to read in data is below
> library(stringr)
> summary(dataset1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  36.10   51.50   54.80   54.52   57.00   67.45 
> quantile(dataset1,c(0.05,0.50,0.95))
    5%    50%    95% 
44.955 54.800 65.000 
> summary(dataset2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  46.00   71.85   95.50  104.50  135.60  172.50 
> quantile(dataset2,c(0.05,0.50,0.95))
   5%   50%   95% 
 61.4  95.5 167.0 

# Code to read in inconveniently formatted data
set1 <- "[0] => 36.1
[1] => 41.6
[2] => 43.8
[3] => 44.9
[4] => 45
[5] => 46
[6] => 46
[7] => 46
[8] => 46
[9] => 46
[10] => 48.2
[11] => 49.03
[12] => 49.3
[13] => 49.3
[14] => 49.3
[15] => 50.4
[16] => 51.5
[17] => 51.5
[18] => 51.5
[19] => 51.5
[20] => 51.5
[21] => 52.05
[22] => 52.47
[23] => 52.6
[24] => 52.6
[25] => 52.6
[26] => 52.6
[27] => 52.6
[28] => 52.6
[29] => 53.7
[30] => 53.7
[31] => 53.7
[32] => 53.7
[33] => 53.7
[34] => 53.7
[35] => 54.8
[36] => 54.8
[37] => 54.8
[38] => 55.9
[39] => 55.9
[40] => 55.9
[41] => 55.9
[42] => 55.9
[43] => 55.9
[44] => 55.9
[45] => 55.9
[46] => 55.9
[47] => 55.9
[48] => 55.9
[49] => 57
[50] => 57
[51] => 57
[52] => 57
[53] => 57
[54] => 57
[55] => 59.2
[56] => 59.2
[57] => 59.2
[58] => 59.2
[59] => 59.2
[60] => 60.3
[61] => 61.4
[62] => 62.5
[63] => 64
[64] => 64.15
[65] => 64.15
[66] => 64.15
[67] => 65
[68] => 65
[69] => 66
[70] => 67.45
[71] => 67.45"

set2 <- "[0] => 46
[1] => 58.1
[2] => 58.38
[3] => 59.2
[4] => 61.4
[5] => 62.5
[6] => 64.7
[7] => 65.25
[8] => 65.25
[9] => 65.25
[10] => 65.25
[11] => 66.35
[12] => 66.9
[13] => 67.99
[14] => 67.99
[15] => 67.99
[16] => 67.99
[17] => 67.99
[18] => 68.55
[19] => 70.2
[20] => 71.85
[21] => 71.85
[22] => 72.4
[23] => 72.4
[24] => 73.5
[25] => 73.5
[26] => 73.5
[27] => 74.83
[28] => 75.7
[29] => 79
[30] => 79
[31] => 79.55
[32] => 80.1
[33] => 84.5
[34] => 84.5
[35] => 84.5
[36] => 85.05
[37] => 87.8
[38] => 89.45
[39] => 93.3
[40] => 95.5
[41] => 97.15
[42] => 98.76
[43] => 98.8
[44] => 99.66
[45] => 106.5
[46] => 108.7
[47] => 112
[48] => 112
[49] => 119.15
[50] => 119.15
[51] => 122.45
[52] => 123
[53] => 124.65
[54] => 125.75
[55] => 125.75
[56] => 128.5
[57] => 128.5
[58] => 133.45
[59] => 134
[60] => 135.65
[61] => 136.2
[62] => 139.5
[63] => 139.5
[64] => 139.5
[65] => 140.6
[66] => 149.95
[67] => 150.5
[68] => 154.9
[69] => 160.4
[70] => 161.5
[71] => 164.25
[72] => 164.8
[73] => 165.9
[74] => 167
[75] => 167
[76] => 167
[77] => 167.55
[78] => 169.75
[79] => 171.4
[80] => 172.5"

parse.set1 <- str_match(readLines(tc<-textConnection(set1)),"[[:digit:]\\.]+$")
    dataset1 <- as.numeric(parse.set1[,1])
    parse.set2 <- str_match(readLines(tc<-textConnection(set2)),"[[:digit:]\\.]+$")
dataset2 <- as.numeric(parse.set2[,1])
$\endgroup$
  • $\begingroup$ (+1). Also, the boxplot indicates that prices below 44 (first dataset) and 46 (second dataset) are suspiciously far from the main body of the data. Same for prices above 65 (dataset1) and 172 (dataset2). $\endgroup$ – user603 May 21 '13 at 16:52
  • 1
    $\begingroup$ Using the median and something based on percentiles sound like good ideas but setting the threshold directly at some low percentile assumes that a fixed proportions of prices is abnormal in each sample. Why would it be the case? The boxplot approach (x times the interquartile range) would avoid this problem. $\endgroup$ – Gala May 21 '13 at 16:59
  • $\begingroup$ @GaëlLaurans I agree with you and user603. The boxplot is a better approach than just using the 5th percentile. $\endgroup$ – Bill May 21 '13 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.