You have two questions
What are outliers?
Which are outliers?
Should outliers be removed?
According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.
There can be different interpretations of 'outliers'
Outliers as extreme cases within a population: Outliers can be referred to the cases within a population in the outer regions. For instance the book 'Outliers' from Malcolm Gladwell speaks about outliers as extreme cases, cases that are relatively far from the mean.
Outliers as measurement errors: In statistics and experimental science/research, outliers can also refer to erroneous measurements. Data points that are not representing the population of interest because of some error in the sampling method or experimental execution.
The latter type of outliers, experimental errors, you might want to remove.
But the former type of outlier, extreme cases among the population of measurements, are not always the same as these latter types.
All measurements have some error* due to small variations in the population and experiments. But some measurements introduce error to a much larger extent and are a bad representation of the population that is being sampled or used for experimentation.
It is these types of measurement or sample errors that we may want to remove from the data because they cause bias and/or high variance in estimates of population parameters.
*It might be better to speak about 'disturbance' rather than 'error'. The variation of measurements is an error in a context of a physics or chemistry experiment, where some fixed value is being measured. But, in psychology, nutrition, health, econometrics, the variations are natural since there is no fixed value and a varied population is being investigated. We do not want to filter out all the errors if these errors are what is representing the population. Discarding data in preprocessing is about the wrong measurements.
For example, imagine some experiment that determines and compares the growth rate of two different strains of bacteria.
Some of the experiments might have a bias.
e.g. the temperature was not set correctly causing a consistent/systematic error.
Some of the experiments might have a large variance
e.g. some batches of bacteria were not kept under optimal conditions and have a relatively large variance in the measurements.
Outliers as a way to detect erroneous measurements
Detecting these cases of error in measurements and sampling is part of preprocessing the data and data cleansing.
One way to detect the erroneous measurements is by finding out the outliers in the first sense, finding extreme cases in the population of measurements.
- Outliers are extreme cases in a population.
- When outliers in a population of measurements can be considered as wrongful measurements/sampling, i.e. they are a part of the measurements/sampling that is not representing the population of interest, then they could be removed.
Should they be removed?
The choice to remove the outliers depends on whether they are likely wrong measurements/sampling and whether they are likely to influence the statistical analysis.
- Outliers in the population of measurements may not necessarily be equal to wrong measurements. If you are removing outliers too aggressively, then you might also remove data points that are not wrong measurements/sampling. The result is that the amount of data is decreasing. And also you can potentially introduce bias by having a selective/subjective sampling process that wrongly represents the population that is under investigation because there is too much censoring of extreme data points. Removing outliers in the measurements should be a balance between advantages and disadvantages.
- Besides removing outliers, you can also perform an analysis that is more robust towards outliers. For instance, you can consider the wrong measurements as 'correct' measurements (since all measurements have some 'error') but just with a different type of error. One alternative way to deal with it is, for instance, using a different cost function in regression (e.g. use absolute deviations instead of squared deviations).
In your example, it is difficult to say because there is no information about the data (what does it represent?). But we see so many 'outliers' that we may wonder whether they are not just naturally part of the population. They are outliers in the first sense (extreme data points) but that does not necessarily make them wrong measurements.