0
$\begingroup$

I have a dataset with different types of user events, and I'd like to set a metric of user activity. I don't have any labeled data of the actual perceived activity level of users. The data consists of around 10 different types of events (count of events normalized by usage time). For example:

+---------+--------+--------+--------+--------+--------+
| UserId  | Event1 | Event2 | Event3 | Event4 | Event5 |
+---------+--------+--------+--------+--------+--------+
|   10252 |  0.048 |  1.266 |  0.777 |  1.224 |  0.551 |
|  982850 |  0.000 |  0.000 |  1.085 |  1.356 |  0.526 |
| 1009937 |  0.000 |  0.000 |  0.245 |  0.049 |  0.025 |
| 1029718 |  0.000 |  0.000 |  0.440 |  0.313 |  0.652 |
+---------+--------+--------+--------+--------+--------+

Looking at the distribution of the events, they all have an exponential-like type of distribution. Scatterplot matrix

Additionally, there's correlation between most of the events. Strongest correlation is around .3

I would like to be able to tell on a scale from 0 to 1, how active a user is, regardless of his/her usage time.

If I were to have only one event, I would fit a distribution to the data and use the probability as the predictor. Because I have multiple events and there's correlation between them, I wonder what would be the best way to model this.

I tried PCA (even though the dada is not normally distributed) and didn't get very promising results (no elbow, 95% of variance is explained by 7 out of 10 PCs).

Any advise would be greatly appreciated!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

As I understand it you are facing a problem of ordering(and then ranking) an unsupervised set of users. These type of problems can be tackled using a clustering algorithm.
There are many such algorithms (KMeans is pretty common).
Where a general approach would be, cluster all users and rank each cluster based on its centroid.
Since all of your features are on the same scale and have similar distribution, using an euclidean distance in the clustering might work well.

If you need to get more intuition into the 10 variables effect on clustering, you can try using an hierarchical clustering.

This is a great example(python)

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks @DaFanat. I also thought of this approach, but the questions is how to decide on the rank of each centroid. Why would clustering help here? I tried hierarchical clustering but it didn't seem (at least visually) very promising. $\endgroup$
    – Omri374
    Dec 25, 2017 at 10:12
  • $\begingroup$ Assuming the more activity on all events = more activity over all, you can use the centeroid distance from the origin as a ranking method. If this is not good enough, then maybe using weighting per each event? Do you have any prior knowledge/intuition about the events measured? $\endgroup$
    – yoav_aaa
    Dec 25, 2017 at 12:44
  • $\begingroup$ Using the centroid distance from the origin would assume that the distribution is uniform. I'd like to try to maintain the original distribution as much as possible (i.e. the exponential nature of the data) $\endgroup$
    – Omri374
    Dec 25, 2017 at 13:21
  • $\begingroup$ Right, the exponential nature is something worth considering. How about transforming(log) the data pre-clustering? Looks like a sound pre-processing step. $\endgroup$
    – yoav_aaa
    Dec 25, 2017 at 13:53
  • 1
    $\begingroup$ This is actually a good discussion :). Some of the problem is how to combine multiple features(dimensions) into one (ranking). I think a clustering algorithm, which make "most" order in multi dimensional space is a fair first step. One you say k-means(or any other clustering algorithm) bring noise into the decision, it might be cleaning noise. $\endgroup$
    – yoav_aaa
    Dec 26, 2017 at 7:27
1
$\begingroup$

Your data is nonnegative, so maybe you could try nonnegative matrix factorization?

Also if you think that you can model your data with some kind of distribution coming from exponential family, you could try Generalized Low Rank Models (see Exponential family PCA). They also cover NMF. An example implementation comes from H2O and it is possible to use it from Python and R.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.