1
$\begingroup$

I want to classify data into two classes based on three parameters given. My data is web server log and I want to classify it into potential user class and non potential user. The parameters are the time spent by the user in the website, the number of pages visited, and the number of transaction per session. All the parameters are discretized into categorical value.

What is the best and easiest method to do that?

EDIT

Clarification from comment by OP: I know how to identify some of the data as potential user or non potential user. The problem is my data has about a million records. How many records that I need to identify manually in order to build classifier? Is it still reliable to do classification method in this case? Which method that I should take if I have some defined parameters to identify the class?

$\endgroup$
1
$\begingroup$

What you're looking for is a clustering algorithm (i.e., unsupervised classification). If you use R, you can load your data into a data frame and apply a variety of clustering algorithms to get various clusters. You can inspect the cluster members and decide which cluster represents users. Some of the clustering algorithms I have personally used in R include kmeans, hclust, agnes, fuzzycmeans, and a few more. Since your data consists of millions of records, loading it all into memory might be an issue. For this, you might choose CLARA (http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/CLARA) which basically samples your data and uses the PAM clustering algorithm on the samples.

If you truly want to deal with "big-data" clustering (billions of records or something similarly ambitious), then I would recommend watching this youtube video (https://www.youtube.com/watch?v=FjhRkfAuU7I&index=9&list=PL3zSmKZQRgYTUa9yO7CEUFbrrN_GIjZNJ). Basically, the presenters use Apache Spark to load 60 GB of twitter data into memory and automatically cluster twitter records into different languages. I have personally tried to replicate their demo by following the youtube video and was successful (the demo starts half-way into the video).

$\endgroup$
0
$\begingroup$

Since the data is unlabeled you cannot classify the data in any straightforward way. Is there anyway you can manually identify a few of the data points as potential user class and non potential users? If so, you can build a classifier this way with your now known labels.

Another possible solution is to do a cluster analysis of your data and see if you can get good separation with two clusters. Try to see if the two clusters give you any separation of users that can be classified into these two groups. For new users you can then assign them to the existing clusters. This is very crude way of doing classification with unlabeled data.

EDIT: Based on your comments I think you are looking for semi-supervised learning. Here is an excerpt from wikipedia (http://en.wikipedia.org/wiki/Semi-supervised_learning):

Semi-supervised learning is a class of supervised learning tasks and techniques that also make use of unlabeled data for training - typically a small amount of labeled data with a large amount of unlabeled data. Semi-supervised learning falls between unsupervised learning (without any labeled training data) and supervised learning (with completely labeled training data). Many machine-learning researchers have found that unlabeled data, when used in conjunction with a small amount of labeled data, can produce considerable improvement in learning accuracy.

So basically you will try to label some of your data and then treat it as a semi-supervised problem. I am not sure what language you are using but you can take a look at python's scikit-learn library which has some methods for semi-supervised learning: http://scikit-learn.org/stable/modules/label_propagation.html

$\endgroup$
  • 1
    $\begingroup$ I know how to identify some of the data as potential user user or non potential. The problem is my data has about a million records. How many records that I need to identify manually in order to build classifier? Is it still reliable to do classification method in this case? Which method that I should take if I have some defined parameters to identify the class? $\endgroup$ – wakawaka Jun 3 '14 at 2:20
  • $\begingroup$ Ok, I edited my first post to address your question. $\endgroup$ – mike1886 Jun 3 '14 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.