2
$\begingroup$

I am trying to fit a mixture model to a dataset that consists of counts (so every record is a count of something, like the number of attempts by an IP address to connect to a website). I know, a priori, that each point belongs to one of two groups (for example malicious IPs and legitimate IPs), but I don't know which one without manually inspecting the record and determining the grouping. These are the things that I know though:

  • As the count value increases, it is more and more likely that the data point belongs to group 2.
  • 80 percent of values are 1 and 85% are less than 2.
  • Most values for group 1 are 1, but there are values as high as 10,000 that are from group 1.

Ideally, I'd like to be able to draw a line somewhere and say if the count is higher than some $n$, then I am $p$% sure that this record is from group 2.

I think a mixture model is a good place to look at to model this data, but looking at the histogram and kernel density plot, I don't see any distinct clumps in data. Based on what I mentioned above, my guess is that a mixture of a log-logistic (for group 2) and pareto (for group 1) might be an ok approximation to the data. Is there an R package or python module that would estimate the parameters of such a mixture?

$\endgroup$

1 Answer 1

1
$\begingroup$

Ideally, I'd like to be able to draw a line somewhere and say if the count is higher than some n, then I am p% sure that this record is from group 2.

It sounds like you want a model that gives an estimate of

$P(Y=1|X=x)$

where $X$ is the variable representing the count, and $Y$ indicates membership of group 2 (Y=1 means "is a member").

When you say "A mixture of log-logistic and Pareto" it sounds like you're referring to $X$. But in the model, you condition on the value of $X$ so what its distribution is would be irrelevant (and strictly speaking, being a count, it can't be either log-logistic or Pareto since those are continuous) to that calculation.

There are a variety of tools for this job, but you might like to consider logistic regression as a first step. That will not require you to draw a line - it will give you an estimate of the probability at any $x$. If you want to pre-specify $p$, you can back out an estimate of the $x$ that corresponds to that $p$.

Is there an R package or python module that would estimate the parameters of such a mixture?

This is such a different question as to merit its own post. In the first question you're modelling $Y|X=x$. Now it seems you want to deal with $X|Y=y$ ($y$ takes the values 0 and 1). Have I understood? You want a model that conditions the other direction?

$\endgroup$
8
  • $\begingroup$ Thanks @Glen_b. My problem is that $Y$ is a latent variable. If I had the value of $Y$ for every record, then logistic regression would be a great way to start. But then if I had $Y$, I had no problem at all, since I knew exactly what are the malicious IP's. $\endgroup$
    – user765195
    May 27, 2014 at 0:44
  • $\begingroup$ Either you have some data with group membership on which to construct a classification rule or you don't, and your question clearly suggests that you do. If you have no data on group membership, how can the first question I quoted be answered at all? If you have some such data, what prevents applying logistic regression to that data? $\endgroup$
    – Glen_b
    May 27, 2014 at 0:53
  • $\begingroup$ I can inspect each record and dig into IP lists to see if it is malicious or not (if it is listed somewhere), but my data does not have value of the group for each record. The dataset has about 500K records. $\endgroup$
    – user765195
    May 27, 2014 at 0:57
  • $\begingroup$ I should also mention that I'm looking for $X|Y=y$ as a mixture. $\endgroup$
    – user765195
    May 27, 2014 at 0:58
  • 1
    $\begingroup$ Yes, if you bring a priori knowledge that clusters will be unimodal, and the situation is one where there can only be two clusters, and you observe a bimodal distribution, the problem is well-formed and has a solution. (But if that a priori knowledge comes from a sample from the population of interest, you should use that sample to do your inference, so you make efficient use of the actual information.) $\endgroup$
    – Glen_b
    May 27, 2014 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.