I am trying to perform real-time decision making on data from a radar sensor and trying to detect occupancy. I generated data using the same sensor annotated it manually as vacant or occupied. I partitioned the data into a train and test set. For the training data, I segmented the data into windows of approximately 1 second worth of data each (as some of the features I'm using are temporal, such as range or variance over time), and shuffled these windows around to create randomness in the train set. I fit the data using the RandomForestClassifier. I ran the classifier on my test set and got 81% accuracy. The dataset is about 50-50 split between the two classes 0 (vacant) and occupied or movement (1)
I now start streaming data and computing the same features over a smaller window (about 100 samples). I want to make a prediction on windows of 100 samples, and see the decision vary from 0 (vacant) to 1 (occupied). Even though I got a accuracy of 80% on my test data, when I use the trained model on streamed data, I am only getting a prediction of 1 even when the room is vacant.
I also checked the prediction probabilities and the probabilities ranged from [0.27 0.63] when the room was vacant and [0.11 0.89] when there was movement or the room was occupied. Thus, it never predicts a zero even when the room is truly vacant.
What could be the reason for this?