1
$\begingroup$

Here's a question that I haven't come across in any statistics classes or books. Imagine I have $n$ number of samples and I subject them to a series of different tests to characterise them. For fun's sake let's say it's beer. For each sample I do sensory analyses and get e.g. 5 observations pr samples. I measure pH in duplicate. I measure viscosity in triplicate. I measure another variable in quadruples and so on to obtain the dataframe below:

enter image description here

How to best combine these results in a dataframe (rows as observations, columns as variables) for further analysis (i.e. multivariate) and have the same number of observations pr variable? The variables with lower number of measurements will inevitably have missing values (NAs) but they aren't as such "missing", there just aren't that many observations.

I see these solutions:

  1. I average the measurements with lower number of replicates and paste it.
  2. In order to retain the distribution of data (and not effectively falsify data) for further analyses I randomly distribute (copy paste) the lower replicate measurements to obtain the higher number of observations (e.g. 12).
  3. (or 2.b) I replace all NAs by the mean.
  4. (or 3.c) I calculate the mean and sd of the lower replicate measurements and use rnorm (or similar) to create randomly normally distributed observations (i.e. 12 in this case). This is assuming normally distributed sample populations and gets tricky when dealing with e.g. duplicate measurements.
  5. I do all measurements 12 times.

Each solution has definite drawbacks and possible ethical complications. Loss of variance (information), something suspiciously close to falsifying data (although replacing NAs by the mean is common practice), time and money issues etc. In cases where there are e.g. 100s or more observations for one variable and 2 and 3 for other variables it becomes very tricky.

What do other researchers do? Is there some standardised way of doing this since I can't be the first researcher to deal with this issue.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

What you have is called repeated measurements, and there are indeed special models for repeated measurements data. Nowadays we most use mixed models, and such models do not have any requirement for equal number of measurements pr sample.

There are already many questions here, see this list

The question about how to organize the data frame: Use long format, that is, one observation pr row, and an additional Group variable identifying to which group that obs belongs.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Sorry, I think my use of the word "series" may have been misleading. These are all stand alone measurements, i.e. there is no time dimension. I could (and will) do each measurement at different time points but that isn't what I'm asking about here. Sorry again, I know the question is tricky, I've discussed it with colleagues that all deal with the issue in various ways as listed above. I will look into whether mixed models might be useful, thank you. $\endgroup$
    – d3F
    Commented Jul 27, 2020 at 19:15
  • $\begingroup$ There is no implication that time is a variable in the use of repeated measurements models (time might be a variable). For a better answer we need more context. One very specific case is a components of variance model, for instance stats.stackexchange.com/questions/361174/… or stats.stackexchange.com/questions/460247/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 27, 2020 at 19:21
  • $\begingroup$ Would the above (added) long format be suitable for mixed models? Either way, I'm really looking for an answer to how I might deal with the NAs for further analyses such as PCA or LDA. I'm looking for predictive models and clustering. Also, am I wrong in thinking that repeated measurements relate to repeating the same measurement using the same instrument/technique/analyses? $\endgroup$
    – d3F
    Commented Jul 28, 2020 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.