# Predicting proportions from time with a discontinuity

Imagine

• having a dependent variable $Y$ that is a proportion (i.e., the proportion of observations made at the given time point that satisfy a condition, where each time point involves 50 to 250 observations)
• $Y$ is measured at a series of time time points $X$, where $X = 1, 2, 3, ...$, typically to around 400.
• At initial time points, $Y$ typically equals zero or close to zero
• After an extended period of time $Y$ typically equals one or close to one
• At some point in between a transition occurs where values of $Y$ increase
• Throughout there is considerable time point to time point variability and given that Y is a proportion, the distribution of errors is not normal. Note also that the values of zero and one are common.

Properties of the data vary across studies, such as:

• the initial value of $Y$
• the time point when the value of $Y$ starts to increase
• the duration of transition from values mainly around 0 to mainly around 1

Questions

• What would be a good modelling approach to such data?
• How could the onset of the transition from values close to zero to values close to one be detected, especially given the non-normal errors?
-
 Does $Y$ increases monotoneously ? (it's not clear from your description whether the first derivative of $Y$ is always $\geq0$) – user603 Sep 14 '10 at 14:31 @kwak The data itself varies from time point to time point. The line of best fit is generally either stable or monotonically increasing. – Jeromy Anglim Sep 15 '10 at 6:52 Jeromy:> then, i don't think a Sigmoid function is what you want (because of symmetry assumption built unto it). – user603 Sep 15 '10 at 15:02

Sounds to me that Y(X) is a sigmiodal process. Thus logistic regression should be suitable for this data. If you model this in R with:

glm(Y~X,family=binomial)


you will find that the "sharpness" of the transition is determined by the magnitude of the X coefficient, and the point of transition (technically the mid-point) is at the ratio of the intercept coefficient to the X coefficient times -1. I made an image to illustrate this but cannot seem to upload it for some reason.

-
Thanks for the suggestion. Perhaps, my description makes it sound like a sigmoidal process, but the data is typically not as smooth as a sigmoidal process might suggest. The theory and the data suggest that some form of relatively abrupt change occurs, which leads to the commencement of the rise from values around 0 to values around 1. Thus, it's something like a segmented nonlinear regression with unknown change points. – Jeromy Anglim Sep 14 '10 at 13:26
The sigmoidal functions become step functions with appropriate prarameters, but are you meaning you are wanting something like a level shift in time series analysis? – James Sep 14 '10 at 13:47
My assumption is that a latent change occurs that triggers the initial rising process. I'm also interested in identifying the exact time point when this occurs. Thus, I see this as a true step function, whereas I would have thought a sigmoidal function could be parameterised to approximate this, but actually, it would still be continuous in the sense that it has a second derivative. – Jeromy Anglim Sep 15 '10 at 14:10
@Jeromy I'm still not sure why a sigmoidal function isn't appropriate without seeing the shape of the data, but you might find this paper useful: unc.edu/~jbhill/tsay.pdf – James Sep 16 '10 at 9:23
Thanks the paper looks good – Jeromy Anglim Sep 17 '10 at 2:13

Ignoring the "change point" your description suggests to me a (non-linear) mixed effects model of the following form:

g(E(Yi)) = Xi*beta + Zi*U


Where The betas are fixed effects, the U's are random effects, g(E(yi)) is the (logit) link to a binomial mean.

This will deal with logitudinal correlation of data and the non-Guassian distribution issues.

This must be coupled with some form of change point model, probably a Hidden Markov Model (HMM).

http://en.wikipedia.org/wiki/Hidden_markov_model

It may be necessary to set-up the model as a Directed Acyclic Graph (DAG) in MCMC format, or even specify it fully in a Bayesian framework using software such as WinBUGS.

See:

http://en.wikipedia.org/wiki/Directed_Acyclic_Graph

http://en.wikipedia.org/wiki/MCMC

http://en.wikipedia.org/wiki/WinBUGS

-
 Thanks. I'll explore some of these models. – Jeromy Anglim Sep 15 '10 at 14:17

James' approach looks good: each observation, according to your description, might have a Binomial(n[i], p[i]) distribution where n[i] is known and--to be fully general--p[i] is a completely unknown function that rises from near 0 to near 1 as i increases. A logistic regression (GLM with binomial response and logistic link) against X[i]==i alone as the explanatory variable might even work. If it's a poor fit, you can introduce additional terms, such as higher powers or (better yet, given the nonparametric spirit) splines. This readily allows for incorporating any covariates into the model, too.

In effect, what appears to be an abrupt change in the response might really just be a natural linear (or nearly linear) progression of logit(p) on which is superimposed Binomial variability. It is this possibility that leads me slightly away from the direction indicated by Thylacoleo, whose approach clearly is valid and likely to be effective. I'm just suspecting (hoping?) that your situation might be amenable to this somewhat simpler analysis.

A complicating possibility concerns the possible autocorrelation of the responses, but that would need to be investigated only if the logistic residuals look strongly over- or under-dispersed.

As a matter of EDA, you could smooth successive observations in a natural way and plot their logits against i. For instance, to smooth observations y[i] and y[i+1] you would construct (n[i]*y[i] + n[i+1]*y[i+1])/(n[i] + n[i+1]), effectively pooling two successive batches; longer smoothing windows can be constructed the same way. (This would automatically cancel out any negative short-term temporal correlation, too.) The fit to the smooth wouldn't be quite right--it would be less steep than appropriate--but it would suggest choices for the general form of the covariates (i.e., functions of i) to use in the regression.

This is, of course, only one of many possible models. For example, the observations at each time point might be a binomial mixture, which would allow both for overdispersion and another way of getting nonlinear fits on the logit scale.

-