I have a time series of observations from a longitudinal study of individual objects. These observations are seen as discrete sequences of features, one sequence per object. The sequences have different lengths.

At each discrete point of time an individual sequence may turn out to be defective. An error on an individual sequence does not mean that the sequence terminates. It might be that other, correct observations might follow.

I am interested in the probability that a sequence is correct up to a certain length. For that reason I've modeled each sequence as a series of binary events where an error is a positive event.

The probability $P(E, S, L, T) = p$ of an error $E$ in the sequence $S$ of length $L$ at time $1 \le T \le L$ is independent from

  • the point in time when the error occurs
  • the point in time when the sequence starts
  • other errors at the same individual sequence
  • the length of the individual sequence
  • other sequences

I want to estimate the probabilty that a sequence contains $n$ consecutive and correct observations starting from the first one.

Can this be modeled by the probability $P(E|T \gt n)$? How is this probability estimated? In a first approach using binomial distribution I would think that it is simply $(1-p)^n$. However, I'm wondering how to consider the different lengths of the sequences? Do I have to account for the length only if the probability of an error ist not independent of the point in time when the error occurs?

Note: In this special case it is possible to use survival analysis. However, I am interested in other ways to estimate the probability.

Edit The events of interest are errors in data acquisition and in the way how observations of the same objects are related to each other: The observations are made on a yearly base. The data are pseudonomized on the base of name, birthday etc. and a irreversible identifier is generated. After that all personal data are removed. If an error occurs then this has nothing to do with any retained property of the individual object. Therefore time is the only variable and as mentioned the error is independent of it.

Edit 2 As Gilbert note in his answer I am actually asking for the survival probability $S(n)$ while $P(E|T \gt n)$ is different. I reformulate my question as follows: "What is the logical formula for the probability of (a) a sequence of length greater than or equal to $n$, and (b1) no error occuring at all, or (b2) an error occuring after $n$?"

  • $\begingroup$ This looks typically as a survival analysis setting. So I just do not understand why you do not want to use survival analysis. Survival analysis comprises plenty of methods from non parametric estimation of survival curves to regression models in either continuous or discrete time. Please explain in what survival analysis does not fit your expectations. $\endgroup$ – Gilbert Jun 12 '16 at 12:02
  • $\begingroup$ @Gilbert There are some reasons: 1st I already know how to model the problem using survival analysis and for comparison I want to apply a different approach, 2nd see edit above: There is neither the need, nor the chance to detect dependencies between features of the object resp. sequence and the time of event. I personally feel that under these circumstances KM is the only useful survival analysis method. 3rd I am just curious how to model the logic. $\endgroup$ – Claude Jun 13 '16 at 7:03
  • $\begingroup$ OK, but you are interested in probability to survive without error up to time n. This is survival analysis. KM is not the only way to estimate survival curves. There is, for example, the Nelson-Aalen non parametric estimator, and you can estimate parametric survival curves using functions such as Weibul or Gompertz distributions. $\endgroup$ – Gilbert Jun 14 '16 at 9:04
  • $\begingroup$ @Gilbert thanks for the additional hints. I will read more about them. Do you have an idea how to solve the conditional logic problem? $\endgroup$ – Claude Jun 14 '16 at 10:22

Letting $T$ be the time to the failure (the occurrence of the first error), the survival probability $S(n)$ is the probability $p(T>n)$ that no error occurs up to position $n$, i.e. 'the probability that a sequence contains n consecutive and correct observations starting from the first one' that you seem to be interested in.

Now, $P(E | T>n)$ is something different. It is the probability to get an error after position $n$. You get this conditional probability by dividing the probability $P(E)$ to have an error in the sequence by the survival probability $S(n)$. With your assumptions, $P(E) = 1 - (1-p)^L$. Then $P(E | T>n)$ would be $\frac{1 - (1-p)^L}{S(n)}$.

| cite | improve this answer | |
  • $\begingroup$ You are right: I am interested in S(n). I was wondering if S(n) could be modeled by $P(E|T \gt n)$. In the meantime I modelled S(n) as follows $S(n) = P(T \gt n|L \ge n)*P(L \ge n) = P(T \gt n \land L \ge n)$? Not sure if this is right. In your formular $L$ is fixed. However, I need S(n), given a set of sequences of arbitrary length. $\endgroup$ – Claude Jun 14 '16 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.