# How to eliminate the effect of a categorical variable in sparse data

I have data from many individuals. For each individual I have a predictor variable $D$ and a response variable $C$ which are continuous. Below I have plotted simulated data for three individuals. Every individual has a different sampling of $D$. I want to model the underlying relationship between $D$ and $C$ while removing the effects of individual (categorical). Matlab is preferred, when possible.

Thank you.

EDIT: Below is a more realistic simulated dataset of 40 individuals. In these data, each individual has between 1 and 5 values of $C$, where $C = 3*exp(-.01*D) + c$ where $c$ is a randomized constant. I think what I want to do is regenerate the exponential function (and its coefficients) from the data and then estimate $c$ for each individual.

[F,G] = fit(allD,allC,'exp1');%matlab


Yields the exponential function but I am not sure how to recover the $c$. It is not the residuals.

• What do you think you should do? Apr 13, 2017 at 2:18
• My naive solution would be to pool all the measurements together and fit a single curve to that. But I am almost certain that there are better solutions. Apr 13, 2017 at 16:51
• Why does the title say sparse? What is sparse? Apr 13, 2017 at 18:04
• For a particular value of D, I am very unlikely to have samples from more than one individual. For some individuals I will have insufficient samples to accurately represent a curve. But across all individuals, I will have a comprehensive sampling. Apr 13, 2017 at 22:48
• Is your response non-negative? And is the expected scatter for a category independent of $D$? (e.g. in many cases, the "one category model" might be fit to $\log{C}$, to account for non-negative response and constant %-error) Apr 13, 2017 at 23:12

Perhaps you want to run a panel regression like:

$$y_{it} = b_0 + b_1 x_{it} + b_2 x_{it}^2 + u_i + \epsilon_{it}$$

Where $u_i$ are either category specific fixed effects or, if you assume that they're mean zero and uncorrelated with $x$, random effects.

You would estimate a curve with different vertical offsets for each category. Notation note: $i$ indexes the category, $t$ indexes an observation within a category.

Given the plot and your question you are trying to unify all the predictor variables into one predictor and all the response variables into one response

This is possible. Mathematically speaking you want a curve which fits the green,blue and red one.

You can either do this by using the least squares method wherein other curves would serves as residuals.

I believe I have found the proper technique used for this kind of analysis: nonlinear mixed-effects estimation.

% matlab
mdl = @(PHI,D) PHI(1)*exp(PHI(2)*D)+PHI(3);
phi0 = [3 .01 0];
[beta,PSI,stats,B] = nlmefitsa(allD,allC,categorical(allS),[],mdl,phi0,...
'OptimFun','fminunc','Vectorization','SingleGroup','LogLikMethod','none');
plot(B(3,:) ,c,'.')
plot([-.5 .5],[-.5 .5],'k')


The fixed effects (effects of $D$) is found in beta and the random effects (effects of individual) are found in B. This method seems to fail when the number of individuals and/or measurements per individual are low. One reason I am starting with simulated data is to estimate how much data I will need to collect to isolate the individual effects from the population effects for a given sparsity of measurements. Below I have plotted the estimated vs true effect of subject for a simulated data set consisting of 200 individuals, each with between 20-30 measurements.

I hope this is useful to others, and that there are not too many errors of vocabulary.