I have several datasets of measured outcomes from different subjects, all measured in the same experimental setup. There are many possible explanatory features that may predict the outcome, but the exact nature of the relationship may differ from subject to subject. I can group my explanatory features into different subsets. I am interested in determining what subset of explanatory features does the best job of predicting the outcome on average, across all subjects.

For example: I am trying to determine what set of features does the best job of explaining how 100 subjects have each rated 50 movies. My objective is to answer the question: "In this population, do (a) the actors, or (b) the genres do a better job of predicting movie score?"

Different subjects have different actor preferences, so I cannot generate a single model predicting score from the actors.

Instead, I first attempt to model movie score as a function of the actors in the movie. I can generate 100 different models of $score = f(actors)$, each with different beta coefficients (reflecting different actor preferences) and various measures of model quality ($R^2$, likelihood, AIC, etc.) I then generate 100 new models, this time with $score = f(genres)$, and generate the same model quality measures. Is there a way I can aggregate or pairwise-compare these models across all subjects to reach an answer to my research question? The number of actors and genres are not the same, so comparing average R2 is probably a poor choice. Is it valid to take the average difference in AIC across all subjects and conclude that, on average, actors are better than genres at predicting scores because $AIC_{actors} < AIC_{genres}$?

In another attempt, I try to generate one single model for each subject with all actors and genres as predictors. I standardize the predictors. (As above, since subject preferences are different, I cannot generate a single model for the entire population). I then extract the beta coefficients for each predictor and compare them to try to determine whether beta coefficients for actors tend to be larger or smaller than beta coefficients for genres. The problem here is that actors and genres are not entirely independent (e.g. "Reese Witherspoon" and "Romantic Comedy" are highly collinear).

Generally speaking - I have a family of $N$ observation vectors $y$ (each with $D$ observations). There are multiple feature spaces $\{X_1, X_2, ...\}$ that are not orthogonal and have varying dimensionality. Each of the $N$ vectors $y$ points in a different direction within a given feature space $X_i$. Is there a way to express some overall measure of how effective a given feature space is in capturing the population variance in $y$ such that I can conclude which feature space is "optimal"?

I am not sure if multilevel models are applicable, because the distinction of "actors" vs "genres" seems to be more of a factor cluster, rather than a factor level. I do not believe the discussion based on this question is directly applicable, because I don't want to average models.

Thanks in advance for any input!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.