I’m interested in using machine learning techniques such as subset selection, lasso and ridge regression to predict which words an individual kid will get wrong. I have about 300 predictors and all my data is from within one child. I want to find a model that works well for that child (and perhaps other models with other children). But I’m worried that key assumptions of the model might be violated since all the data is from a single person. Basically, does it make sense (and not violate the assumptions of the models) to create separate models one for each student if the end goal is accurate prediction?
4
-
$\begingroup$ Hi- thanks for this response. I’m not trying to generalize across students. I just want to find an accurate model for one student. My goal is to use sublexical factors of words to predict whether or not a kid will read a word correctly or not. Yes- across time would be an issue but if I have training data and test data from within the same week or day that shouldn’t be an issue I would think. $\endgroup$– sahanmCommented Sep 1, 2018 at 1:39
-
1$\begingroup$ Ah, I see. Well then if it's phonetic accuracy, then a similarity measure among words should be straight forward based on phonological and orthographic properties. I would venture to guess that you could either cluster or do something like a PCA on the set of words you would use, and provided you had enough words to provide a reasonable sample for each phonetic cluster / principal component, you would be able to predict whether the same kid would properly pronounce previously unseen words. This would still be neglecting sequential characteristics, as some words are polysyllabic, etc. $\endgroup$– Don WalpolaCommented Sep 1, 2018 at 3:39
-
$\begingroup$ Ok thanks! Yeah I need to find out ways to deal with sequential effects but it sounds like I could use these models on data from within a single person without violating key assumptions? Cluster and pca are slightly different than best subset/lasso/regression - but I’m guessing they are all ok to use on a single individual? $\endgroup$– sahanmCommented Sep 1, 2018 at 12:46
-
$\begingroup$ I don't see why not, I mean that's kind of the idea behind standardized testing - they can't test you on everything, just what's intended to be a representative sample of material. $\endgroup$– Don WalpolaCommented Sep 2, 2018 at 2:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There are several issues I see with your approach. The first is determining what you are trying to predict. Do you have a similarity measure for words, with which you can generalize from the words used in your training data words unseen by the child it may encounter in your testing data? Any sort of similarity measure would need to be informed by what you mean by 'wrong' - is this a spelling test, a vocabulary definition test, or what?
And as far as generalizations to the performance of other children, I don't think that should be something you should expect here. Even if the child were to 'typical' for whatever set of children you hoped to generalize your model to, sample bias induced by the fact that you have only one sample point in your child population would almost certainly skew your results. And lastly, this is assuming static behavior over time - if the kid studies, it would be reasonable to assume that his performance would improve to a certain extent.
I would suggest you get a larger random sample that is adequately representative of the set of children you are trying to model if that is your goal, and work out a similarity measure for the words so you have a more concrete measure of performance if you're doing something like a spelling test.
