Mixture model fixed effects

One standard model used with panel data is fixed effects: $y_{it} = \mu_i + \theta_t + \epsilon_{it}$, where $i$ is the individual and $t$ is time subscripts. This can be estimated easily with OLS and dummy variables.

The model assumes that there is a single underlying time series, $\theta_t$ for $t = 1,2,...,T$. All individuals in the data are assumed to follow this time series plus some individual effect, which is constant relative to time.

Suppose, however, that there are two or more groups of individuals, that each group has its own time series, and that, before looking at the data, we don't know who is in which group. I would like to estimate the fixed effects model in this case and to figure out which individual is in which group / cluster.

The approach that I thought of is that the model should become: $y_{it} = \mu_i + \sum_j \pi_{ij} \theta_{jt} + \epsilon_{it}$. $j$ indicates the group $1,2,...,J$. $\pi_{ij}$ is the probability that individual $i$ is in group $j$. $\theta_{jt}$ is the time effect for group $j$ at time $t$.

In my particular application, $\{\pi_{ij}\}$ needs to be a set of additional parameters. However, I can see that in other applications, it could be modeled as a function of some covariates.

Is this model a good approach? Has anyone tried it before? How do I estimate this model? The model has a lot of parameters. I've tried basic optimization, and it has not worked. Ideally, I am looking for software to do the estimation, such as a package in R; or, a solid reference that I could use to program this.

If the model that I've proposed is not a good approach, how else could I solve this problem? One possibility that I could see is first figuring out which individual belongs to which cluster, and then estimating a regular fixed effects model on each cluster. The issue then is how to perform the data classification. Regardless of the approach, I am still looking either for software or a good reference.

• Jessica, of the tags you put on this, only "panel data" and "fixed effects" are relevant. "Estimation" is way too broad; "clustering" refers to uncovering unknown clusters in the data; and "time series" usually means one time series, while you are interested in many. I would suggest you retag this to show "mixture models" and "longitudinal data" (biostat synonym for economics panel data). I could do that, but since you put a bounty on this question, I don't want to screw it up for you :) – StasK Apr 30 '12 at 14:50
• "mixed model" is a biostat term for econometrics "random effect" model. Oddly, that's still relevant here :) – StasK Apr 30 '12 at 18:18
• Thank you everyone who has read my question and tried to answer. All of the answers gave me good ideas / starting points. Unfortunately, no answer gave a straightforward solution. I will therefore let StackExchange automatically assign the bounty based on the votes cast by the community. – Jessica May 8 '12 at 13:58

This type of model has been entertained by education researchers for years under the name of growth mixture model (do different students exhibit different rate of learning?), although they work with it as a random effects model. I don't think you'd be able to come up with a proper fixed effect estimation for this model, as it may lack the sufficient statistic that you could condition on. But a full blown ML model should not be extremely difficult to fit.

In Stata, you could try fmm with a full set of panel id dummies and/or interactions with time, although this is a very wasteful approach in terms of degrees of freedom.

You are trying to fit a "mixture model", though I would suggest not having a separate set of $\pi_{ij}$ parameters for every person, but rather model these probabilities as a function (i.e. logistic) of some covariates. You could even start with a constant $\pi_j$. The typical approach for fitting mixture models is the EM (expectation-maximization) algorithm, though for the simpler versions even direct maximization would work.

Perhaps searching on these terms will help you find a solution that already exists.

• @Aniko- Thank you! Any suggestion as to which package does this? (An R package, a MATLAB function, something else?) – Jessica Apr 27 '12 at 15:47
• Sorry, I don't know anything relevant, but that does not mean it does not exist. A quick Google search of "mixture model panel data" brought up some papers discussing this. – Aniko Apr 27 '12 at 21:03
• A question for Jessica: Do you have enough data so that all the πijs are estimable? – Michael R. Chernick May 4 '12 at 15:13
• @MichaelChernick, if you want to get a user's attention, consider using the '@' symbol immediately before their name so that they will be notified that they were mentioned (as I did with your name here). – Macro May 4 '12 at 16:17

I suspect you actually mean $y_{it}=μ_i+∑_j I_{ij}θ_{jt}+ϵ_{it}$ where $I_{ij}$ is an indicator for individual $i$ belonging to group $j$ (with probability $\pi_{ij}$. I also wonder if you really want to allow for different individuals to have different group assignment probabilities? I suspect $\pi_j$ will suffice?

I guess it could be seen as a random/mixed effect (not a mixture) with a multinomial effect. This might be an improper term since "random effects" is a typically reseved for Gaussian random effects.

Also note that you are not dealing with a Gaussian mixture, because you allow an individuals shift of the Gaussian ($\mu_i$).

Anyhow, I can think of two approaches:

1. Write the likelihood and maximize it using en EM algorithm which you can write yourself.
2. Compute $\mu_i$ by averaging within individuals (assuming you have enough observations). The residuals will be distributed like a Gaussian location mixture at each $t$. You can use a standard EM implementation such as mixtools R package to estimate $\{\theta_{jt}\}$ and $\{\pi_j\}$.