Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have collected psychophysiological data measuring the subjects' (two groups) ability to perceive vibration. A vibrating probe moves against the skin at smaller and smaller displacements, and the subject indicates when they feel the vibration. Unfortunately, at high frequencies, the probe can only move a short distance, and sometimes the largest distance that the probe can move is still not large enough for the subjects to perceive. Thus, I have accurate threshold values for some subjects, but for some who never felt the vibration, I simply have a value that I know their threshold is greater than. Is there any way for me to still include this data? And what is the best way to analyze it?

share|improve this question
These are called censored observations. How to include it depends on the kind of statistical analysis you are conducting. – user10525 Jun 14 '12 at 21:12
I agree with Procrastinator except that I would use the term truncated. The approach to a simlar problem called right censoring occurs in survival analysis there you keep the truncated value but have an indicator variable to tell you whether the value is a complete value or a censored one. In survival analysis there is a simple way to deal with this but that is because you are estimating a survival curve. Here you may be wanting to calculate averages. If you ignore the trucation you underestimate the average. If you throw out the truncated points you underestimate the average. – Michael Chernick Jun 14 '12 at 21:37
To properly incorporate the truncated values you would need to have a probability model for the probe distance given that it is greater than the threshold. You could then take the mean of that distribution and compute a weighted average using the average for the values that were not trncated with the average for the truncated distribution where the weighting is according to the proportion of cases truncated. – Michael Chernick Jun 14 '12 at 21:37
Truncation is what would happen if you threw away the unquantified data. You don't want to do that! You are correct, Cale, that there is information in these censored values and in suspecting that there are some standard ways to analyze them (and pitfalls for the unwary). But to provide a good answer we would need to know what kind of analysis you seek. In particular, the treatment of these data is fundamentally different depending on whether they appear as dependent or independent variables in a regression. Perhaps you could elaborate on this? – whuber Jun 27 '12 at 3:11
Small detail unrelated to the statistical question at hand but it might be useful to know: Data of this kind are usually called “psychophysical” data, not “psychophysiological” (which include things like heart rate or skin conductance measures but not subjective judgments about sensations). This could also help you look for literature on how people usually treat this type of data. – Gala Nov 28 '12 at 8:56

I like to use heterogeneous Mixture Models to describe combined effects from fundamentally different sources.

You might look at something like a "Zero Inflated Poisson" model in the style of Diane Lambert. "Zero-Inflated Poisson Regression, With an Application to Defects in Manufacturing", Diane Lambert, Technometrics, Vol. 34, Iss. 1, 1992

I find this idea particularly delightful because it seems to contradict the notion that application of statistical design of experiments to medicine cannot fully cure disease. Behind the notion is the idea that the scientific method cannot complete its purpose in medicine comes from the idea that there is no disease data from a "perfectly" healthy individual and so that data cannot inform remedy of disease. Without measurement there is no room to improve.

Using something like a zero-inflated model allows one to extract useful information from data that is partially "error free". It is using insight into the process to take the information that could be thought of as "silent" and make it speak. To me this is the kind of thing you are trying to do.

Now I can't begin to assert which combinations of models to use. I suspect that you could use a zero-inflated Gaussian Mixture Model (GMM) for starters. The GMM is a bit of an empirical universal approximator for continuous PDFs - like the PDF cousin of the Fourier Series approximation, but with the support of the central limit theorem to improve the global applicability and allow typically many fewer components in order to make a "good" approximation.

Best of Luck.


More on zero-inflated models:

share|improve this answer

Clustering the results and defining a scale might be a solution.

Make a category variable like so (or differently):

  1. High sensitivity
  2. Normal sensitivity
  3. Low sensitivity
  4. Insensitive (the ones that are off the scale in your case)

You could use this variable to do the analysis, but whether the results are meaningful depends on how well you define the categories.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.