I have a large set of location data from a social network and would like to conduct a mobility study with it. For each object, I have up to several thousand locations where this object posted from. I quickly summarized properties of objects, which are
- location count (simply the number of locations I have of this object)
- distance traveled (all the distances between successive locations summed up)
- radius of gyration (standard deviation from the mean location, which is also termed the center of gravity)
- average velocity (self explaining)
Now, in order to study mobility, I really depend of having only humanoid objects in the dataset. The problem is that, on the particular social network, there are also non-humanoid objects contained (e.g. a radio station broadcasting always from the same location or an earthquake monitoring service broadcasting from the whole world). Thus, in the end, I have several classes of objects in my dataset, e.g. humans, stationary bots, dynamic bots, etc.
My strong assumption is that these different classes show very different values in the feature space, i.e. an earthquake monitoring service should have high location count, high distance traveled, high radius of gyration, etc.
Based on this assumption I wanted to use a simple supervised classifier (such as Naive Bayes) to classify my whole dataset so that I can remove non-humanoid objects. The training set would encompass about 200 objects which are gained by looking at the respective social network profiles of the objects.
But as you can see in the above figure, the distribution of the above-mentioned variables are extremely long-tailed, i.e. there are some extreme outliers but they are sparse, and the distributions are certainly not normal, which would be required for normal NB.
When I log10 transform the variables, some seem to approch a normal distribution (namely location count and distance traveled), but radius of gyration and average velocity are still not gaussian:
Thus the apparent problem is that there hardly will be any outliers in a training set of just 10^2 objects (and acquiring a larger training set is not easily feasible). Thus the priori probability of an outlier class will be very small.
- would this even make sense with this dataset? i.e. is there a classifier that can deal with these kinds of data or are the proportions of classes to unequal?
- One practical requirement would be that the technique is already implemented in some software (preferably R) because the removal of these outliers is just a small step in the huge process of my work...