Would it be possible to build anomaly detection algorithm to be able to tell apart real user from crawlers/spiders with high confidence? I have access to user tracking data where information about client session, IP address and weather client can process javascript. Based on that data I can calculate how many pages client visited and what is the entropy of time between requests. I also know which sessions converted which would be my non anomalous data I can use to train.

To summarize I have 3 features I think I can use:

- number of requests
- *calculated entropy based on time spent on each page
- can run JS

Are there any other possible or better features that could be used? Is this problem solvable with machine learning?

* crawlers usually don't spend too much time on the page so time between requests would be very short with similar intervals. That would give small entropy value. Real visitors would produce more random pattern.


I am pretty confident that it is possible to solve this task with high precision. Why don't you just try it? I would just use a linear method (linear SVM or LDA) on that and maybe create a binned representation (as in e.g. you take 30 bins that representing the number of requests. say each bin corresponds to 1 request. You initialize the bins with 0. Then you fill the bins up to the 'number of requests' bin).

You might also think about combining features like

  • number of pages visited / time spent on page
  • number of pages visited / available pages

Of course there is more - you could also compute n-gram features of the user agent etc.

But I would first try with few features and if performance is not good enough gradually increase complexity.


I agree with Soeren that you probably have, within your set of possible features, the means to construct a model for identifying real users (or crawlers if that's your intent). I flagged this as a question for CrossValidated the sister site for statistics and machine learning because the community there can provide you with detailed answers to questions like these. Unfortunately that flag was not yet accepted and cross posting is discouraged so I'll try to answer here.

If you were interested to try LDA I would consider QDA instead. Quadratic Discriminant Analysis is LDA without the assumption of both classes having the same variance. You had stated that humans have more randomness and different variances would better fit that assumption.

However you are calling this anomaly detection which is a type of problem neither QDA or LDA might best suited for. Usually this term implies that you feel that the majority class is identifiable but the other class(s) are either rare, or overly heterogenous (often the same thing relative to a captured dataset). I don't know which you are thinking of as the anomaly, crawlers or humans, but the non-anomaly class presumably has more self-identifying data than the other in an anomaly detection model and is thus unbalanced. To quote the Wikipedia definition (emphasis mine).

Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal by looking for instances that seem to fit least to the remainder of the data set. Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier (the key difference to many other statistical classification problems is the inherent unbalanced nature of outlier detection).

I would amend that definition to include that anomaly is useful if one class is clearly defined and there are many (possible infinite) classes that aren't (excessive heterogeneity). This doesn't imply that there are more than 50% of the majority class in your data capture but it still falls under anomaly detection. With the variety of human behaviors it is possible that humans would be the anomaly under that definition.

If one of these sounds like your situation then traditional classification algorithms may end up disappointing you. Consider for instance a dataset with 95% of Class A and 5% of class B. For supervised learning algorithms your model will very often resort to always guessing class A and thus netting a 95% accurate model that tells you absolutely nothing you didn't already know. In literature this referred to as class imbalance.

On the other hand if you are simply using the term 'anomaly' loosely then you have the canonical variety of classification algorithms to sample. In either case, I think you should pursue this problem because it will probably be both edifying and serve as a useful tool for you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.