Say I have a collection of videos, each that have an action that occurs at time $t_i$ for video $i$.

So I could have $\{v_1, v_2, ..., v_n \}$ with times $\{3, 7, ..., t_n\}$ in seconds, for example.

Is there any sort of method that can take these videos and train a model to predict what time $t_k$ is for a test video $v_k$?

I was thinking perhaps of labeling all frames in video $i$ before time $t_i$ and labeling them 0 and then those after as 1. Then, train a model to classify images into before and after and go from there.

This seems almost like a survival analysis problem. Any suggestions?


There are only two cases that come to mind where your idea might work: 1) All frames labeled 1 are semantically connected to the "action"; 2) You feed the whole video to the model.

If you are going to build a model that classifies single frames out of context, and frames after the action have nothing to do with the action itself, your labeling won't get you anywhere. If your action is fairly easy to classify from a single frame, you should probably just label as positives the frames where the action actually appears (but not frames after the action), train an image classifier, then at test time run it over video frames sequentially until your model finds a positive (the time stamp of this frame is your action time). If you need multiple sequential frames to identify the action you can make a multiple inputs model (possibly easier to do, but will have the limitation of a fixed number of input frames) or a recurrent model (possibly more powerful, but also harder to design and train).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.