I am working on a classification problem where I need to classify objects based on a visual data. There are a couple hundred different classifications to be made and I have around a million plus observations to draw upon currently. The data has 49 features and 1 label. The features are all continuous.
In order to begin working on my model I decided to focus on the top four most popular classes. My training data was about 100k observations scattered over time fairly evenly. When I fit and test my model on this data, I get very good performance(99% accuracy).
I was highly skeptical of this performance so I decided to pull some newer observations of those same four classification. When I ran the model on these observations, my performance dropped to something around 60%.
What could I be doing wrong? I am new to Machine Learning and this data. What could be some troubleshooting techniques to solve this? I am using both R and Python/Sklearn.
Thanks in advance