# Predicting proportions from time with a discontinuity

Imagine

• having a dependent variable $Y$ that is a proportion (i.e., the proportion of observations made at the given time point that satisfy a condition, where each time point involves 50 to 250 observations)
• $Y$ is measured at a series of time time points $X$, where $X = 1, 2, 3, ...$, typically to around 400.
• At initial time points, $Y$ typically equals zero or close to zero
• After an extended period of time $Y$ typically equals one or close to one
• At some point in between a transition occurs where values of $Y$ increase
• Throughout there is considerable time point to time point variability and given that Y is a proportion, the distribution of errors is not normal. Note also that the values of zero and one are common.

Properties of the data vary across studies, such as:

• the initial value of $Y$
• the time point when the value of $Y$ starts to increase
• the duration of transition from values mainly around 0 to mainly around 1

Questions

• What would be a good modelling approach to such data?
• How could the onset of the transition from values close to zero to values close to one be detected, especially given the non-normal errors?
-
Does $Y$ increases monotoneously ? (it's not clear from your description whether the first derivative of $Y$ is always $\geq0$) –  user603 Sep 14 '10 at 14:31
@kwak The data itself varies from time point to time point. The line of best fit is generally either stable or monotonically increasing. –  Jeromy Anglim Sep 15 '10 at 6:52
Jeromy:> then, i don't think a Sigmoid function is what you want (because of symmetry assumption built unto it). –  user603 Sep 15 '10 at 15:02

Sounds to me that Y(X) is a sigmiodal process. Thus logistic regression should be suitable for this data. If you model this in R with:

glm(Y~X,family=binomial)


you will find that the "sharpness" of the transition is determined by the magnitude of the X coefficient, and the point of transition (technically the mid-point) is at the ratio of the intercept coefficient to the X coefficient times -1. I made an image to illustrate this but cannot seem to upload it for some reason.

-
Thanks for the suggestion. Perhaps, my description makes it sound like a sigmoidal process, but the data is typically not as smooth as a sigmoidal process might suggest. The theory and the data suggest that some form of relatively abrupt change occurs, which leads to the commencement of the rise from values around 0 to values around 1. Thus, it's something like a segmented nonlinear regression with unknown change points. –  Jeromy Anglim Sep 14 '10 at 13:26
The sigmoidal functions become step functions with appropriate prarameters, but are you meaning you are wanting something like a level shift in time series analysis? –  James Sep 14 '10 at 13:47
My assumption is that a latent change occurs that triggers the initial rising process. I'm also interested in identifying the exact time point when this occurs. Thus, I see this as a true step function, whereas I would have thought a sigmoidal function could be parameterised to approximate this, but actually, it would still be continuous in the sense that it has a second derivative. –  Jeromy Anglim Sep 15 '10 at 14:10
@Jeromy I'm still not sure why a sigmoidal function isn't appropriate without seeing the shape of the data, but you might find this paper useful: unc.edu/~jbhill/tsay.pdf –  James Sep 16 '10 at 9:23
Thanks the paper looks good –  Jeromy Anglim Sep 17 '10 at 2:13

Ignoring the "change point" your description suggests to me a (non-linear) mixed effects model of the following form:

g(E(Yi)) = Xi*beta + Zi*U


Where The betas are fixed effects, the U's are random effects, g(E(yi)) is the (logit) link to a binomial mean.

This will deal with logitudinal correlation of data and the non-Guassian distribution issues.

This must be coupled with some form of change point model, probably a Hidden Markov Model (HMM).

http://en.wikipedia.org/wiki/Hidden_markov_model

It may be necessary to set-up the model as a Directed Acyclic Graph (DAG) in MCMC format, or even specify it fully in a Bayesian framework using software such as WinBUGS.

See:

http://en.wikipedia.org/wiki/Directed_Acyclic_Graph

http://en.wikipedia.org/wiki/MCMC

http://en.wikipedia.org/wiki/WinBUGS

-
Thanks. I'll explore some of these models. –  Jeromy Anglim Sep 15 '10 at 14:17