I have a dataset where a child read a passage out loud. Each row is one word (I’m order of the passage). I have a 1/0 for whether they read the word incorrectly or correctly. I’m trying to predict correct/ incorrect based on word features. But I’m not sure how to address the fact that words are repeated in the passage and sometimes the child read them correctly and other times they read them incorrectly. Does it not matter that some words are read more than others and will influence the model more? Is it more ‘naturalistic’ to leave it as is because connected text reading will always have repeated words? I’m wondering how it will influence the model.
It's probable to have conflicting labels in real world datasets. And, it's natural that you leave them since this is your data collection experiment and it reflects the uncertainty. CART uses majority voting to decide if the predicted class is
1. So, let's say, if you have a node that could isolate a word from others that is read
5 times correct and
2 times wrong by the child, the predicted class will be
1 (i.e. correct). What you can also do is obtaining a probability estimate to model the uncertainty for the word being read correctly, using the CART in regression mode. This will result in
5/7 for the above example. Then, you can apply thresholding if you want.