I have been having a difficult time finding out if my random forest classifier is actually properly designed. I have built a random forest model, however the observations I have used are not independent i.e., they come from a repeat measures study. For example, I have 20 observations but those 20 observations actually arise from only 3 animals: 1 animal had 10 tests, another had 7, and the third had 3. I have seen mention that correlation between variables isn't a huge concern especially if the variables are selected randomly at each node but I still can't find a solid answer as to if having dependence between some observations will be an issue. My rational has been that yes they are from the same animal, but they represent a different stage during the animals life. Thanks in advance!
-
$\begingroup$ i have a similar issue. I am using overalpping temporal aggregation on time series. my data points are correlated by nature as time series as well as another level of correlation is added by overlapping aggregation of data point. I know how to use mixed models in the regression field to deal with residual correlation.but here I want to apply a random forest and my goal is prediction(not inference. orimportance of variable). how does the correlation between my data points affect my model and how can I deal with it? thankyou $\endgroup$– MahnazCommented Jul 28 at 21:46
1 Answer
I think beyond the base assumptions of a random forest (or any other model method really) should be a logical and practical awareness for what using just three animals that have bee repeatedly sampled means for the utility of the model, even if the model built is with the utmost attention to detail using best practices.
The entire idea of predictive models such as this (as opposed to parametric models used for statistical inference for example) is to generalize something from your data to other instances of some system of individuals or events.
On a purely pragmatic level, how confident would you be in assuming whatever you are estimating with just three sample animals can be fully summarized by just three test subjects?
And how does using different times in their life (read that as samples under different environmental conditions) help or hurt your study.
While I understand and sympathize with having limited data (it is something we all struggle with at some point) ultimately it comes down to what is the consequence for the utility of the model down the road?
-
$\begingroup$ Thank you very much for the reply! Just to clarify, I do have more than 3 animals in my dataset (4,008 observations, 1,197 animals, 8 herds) but I get your point! I think with only 8 herds represented here there is limited external validity, but I suppose that would be a limitation of a lot of studies. $\endgroup$– JamieCommented Oct 26, 2022 at 13:03