3
$\begingroup$

Box plot of the variable fare I have attached the boxplot of a variable called Fare(of a journey). This is a continuous variable which has outliers. According to some articles on outliers, I learned that any data point that is above/below the whiskers is an outlier. I also learned that the whisker distance is calculated by 75th percentile + 1.5*(Inter-Quartile Range).

In the case that I have attached, you can see there are too many outliers(200/891 observations). If I replace all these points with missing values(can be imputed later), won't it produce bias? Few articles asked to consider 3*IQR instead of 1.5*IQR. Should I do that way? How to deal when you have too many outliers?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ I would be very careful with the direction of your thinking. While some of your observations may fit some algebraic definition of an outlier, it seems to me that those "outliers" are in reality part of your data. Recall that the algebraic definitions are sensitive to the approximate symmetry or skew of the observations. I would first ask why you think you should remove these values or consider them outliers $\endgroup$ – Gabriel J. Odom Dec 8 '17 at 19:12
  • 1
    $\begingroup$ Agreed, these are not what would typically be considered outliers - it's just that your data are not well-described by a normal distribution. How to deal with them depends on the analysis you are trying to do. If you describe that, we may be able to advise you further. $\endgroup$ – mkt - Reinstate Monica Dec 8 '17 at 19:29
  • $\begingroup$ I would strongly recommend against replacing "outliers" with missing values. If anything it is worth considering a variance-stabilising transformation first. That said, please describe what you want to do with this data. $\endgroup$ – usεr11852 Dec 8 '17 at 22:40
  • $\begingroup$ As I am a beginner, I am working on the Titanic data set provided in kaggle. I learned from an online course that for any data science problem we need to tackle the outlier and then tackle the missing values. In the training dataset, there are 2 numeric variables. One is Age with many missing values and the other is Fare variable(box plot is attached) with 2 missing values. This is my motivation to detect or remove outliers as I don't want any bias while substituting the mean for the missing values. Is my approach right? Is it right to take the above values as it is? $\endgroup$ – Abiram Dec 9 '17 at 8:53
  • $\begingroup$ The objective is to predict the missing values in Age variable and also the overall target variable(whether a passenger has survived or not). To predict them I think the variable Fare is crucial. Pls advise me on this. $\endgroup$ – Abiram Dec 9 '17 at 8:55
4
$\begingroup$

These are not outliers. I am an economist and this is the way the data should look, based on your comments. It is a poor dataset to start a beginner on.

What you are looking at is called "price discrimination." In particular, it is third degree price discrimination. Another real world example, although it is an example of first degree price discrimination, is with the Apple i-phone. When it first came out they restricted production. As a consequence, the supply curve and the demand curve did not meet. Only those who valued it the most tried to buy it and they were willing to pay the most. Then they produced more, but still not enough for the supply curve and the demand curve to meet. People stood in line and those willing to pay the most got a phone. They continued this process until the price fell to the equilibrium price.

In doing this, they extracted as much revenue as possible from each person. There is a hidden structure in this data that you need to extract. It probably had to do with square footage, amenities and location. You do need to go and ask a new question as this won't get you where you are looking to go. The data has no outliers in it.

Without really looking at it closely, it is probably a Pareto distribution and not all Pareto distributions even have a mean, let along the nice properties you want a beginner to see.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.