1
$\begingroup$

I have a large set of location data from a social network and would like to conduct a mobility study with it. For each object, I have up to several thousand locations where this object posted from. I quickly summarized properties of objects, which are

  • location count (simply the number of locations I have of this object)
  • distance traveled (all the distances between successive locations summed up)
  • radius of gyration (standard deviation from the mean location, which is also termed the center of gravity)
  • average velocity (self explaining)

Now, in order to study mobility, I really depend of having only humanoid objects in the dataset. The problem is that, on the particular social network, there are also non-humanoid objects contained (e.g. a radio station broadcasting always from the same location or an earthquake monitoring service broadcasting from the whole world). Thus, in the end, I have several classes of objects in my dataset, e.g. humans, stationary bots, dynamic bots, etc.

My strong assumption is that these different classes show very different values in the feature space, i.e. an earthquake monitoring service should have high location count, high distance traveled, high radius of gyration, etc.

Based on this assumption I wanted to use a simple supervised classifier (such as Naive Bayes) to classify my whole dataset so that I can remove non-humanoid objects. The training set would encompass about 200 objects which are gained by looking at the respective social network profiles of the objects.

enter image description here N~2000

But as you can see in the above figure, the distribution of the above-mentioned variables are extremely long-tailed, i.e. there are some extreme outliers but they are sparse, and the distributions are certainly not normal, which would be required for normal NB.

When I log10 transform the variables, some seem to approch a normal distribution (namely location count and distance traveled), but radius of gyration and average velocity are still not gaussian:

enter image description here N~2000

Thus the apparent problem is that there hardly will be any outliers in a training set of just 10^2 objects (and acquiring a larger training set is not easily feasible). Thus the priori probability of an outlier class will be very small.

  • would this even make sense with this dataset? i.e. is there a classifier that can deal with these kinds of data or are the proportions of classes to unequal?
  • One practical requirement would be that the technique is already implemented in some software (preferably R) because the removal of these outliers is just a small step in the huge process of my work...
$\endgroup$
  • $\begingroup$ this subject has been discussed quite thoroughly in the past few days. My own view is that you should decide what's physically possible and exclude any that aren't. For example, air crew could only do a limited number of flights within a given period. Or you might decide that including air crew would not be appropriate. $\endgroup$ – Robert Jones Apr 7 '13 at 21:20
  • $\begingroup$ And of course astronauts who have come back from the international space station could have travelled even further. $\endgroup$ – Robert Jones Apr 7 '13 at 21:26
  • $\begingroup$ The ISS is not part of the WGS84 coordinate system I fear.. ;-) No, jokes aside, where has the discussion being going on? Do you have any link? I would like to see what people have concluded. And anyway, let's say I WANT to use a classification to solve the problem, then my question is still relevant. $\endgroup$ – wnstnsmth Apr 8 '13 at 5:23
-1
$\begingroup$

Sorry about the delay. If you type "outlier" in the top right hand search box, it should come up with a list of previous questions on the subject. Also if you scroll down a bit from the top, there will hopefully also be a list of topics that were categorised as relevant to your subject matter. I am afraid I am not a statistician, so wouldn't be able to personally provide more useful help.

Sorry folks, this should have been a comment

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.