2
$\begingroup$

I'm doing 2-class image classification (determining whether an object is present or absent in images) with CNNs. The dataset is a bunch of photos with continuous timestamps. And I observed that timestamps could be used for prediction as well. For example, in a series of photos, if one image is predicted to have the object I want to capture, then the adjacent ones have more chance to contain that object than other images.

What kinds of machine learning techniques could take timestamps into consideration? It will be great if these techniques could be combined with CNNs.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ What is preventing you from using timestamps as a feature? You have a CNN piece which deals with the image, and an auxiliary set of neurons that deal with the timestamp; ultimately the two networks merge at some point before the classification layer. $\endgroup$
    – Alex R.
    Commented Jul 13, 2017 at 16:48
  • $\begingroup$ @AlexR. One problem with that is that if the network still looks at a single image+timestamp at a time, then when you apply it to test data, it doesn't know about the adjacent ones having that label, and the timestamp feature is going to be larger than anything it was trained on and will perform unreliably. $\endgroup$
    – Danica
    Commented Jul 13, 2017 at 17:06
  • 1
    $\begingroup$ So in addition to a timestamp, you can have a "prior detection" input, which carry timestamp(s) and prediction(s) of the last few images that you processed. $\endgroup$
    – Alex R.
    Commented Jul 13, 2017 at 17:18

1 Answer 1

Reset to default
0
$\begingroup$

Probably the easiest thing to do would be a two-level classification system. At a high level:

  • Train a CNN normally, ignoring the timestamp, to estimate its confidence that the object is present.
  • Decide on the full set of labels at once, taking into account both the timestamp and the confidence score. For example, the scores sequence $\begin{bmatrix}.8 & .6 & .9 & .1 \end{bmatrix}$ might be labeled as $\begin{bmatrix}1 & 1 & 1 & 0\end{bmatrix}$, while $\begin{bmatrix}.3 & .6 & .2 & .1\end{bmatrix}$ might get all zeroes.

There are several options for this scoring option. You could use something like a conditional random field, or train a little classifier with features as the input score and the adjacent scores, or various other things along those lines.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for your suggestions. But I'm not quite sure that I got your ideas. Correct me if I misunderstand any of your points. First, I use CNN to get the confidence scores(probability for each image). Then I compute the time difference between adjacent images as adjacent scores . Finally, I make some strategies or apply some techniques to integrate the confidence scores and adjacent scores. If I train a classier based on these two scores, then I can not shuffle the images. Otherwise the adjacent scores are not correct after shuffling. Thank you. $\endgroup$
    – Jane
    Commented Jul 14, 2017 at 14:51
  • $\begingroup$ Yes, if you wanted to do a classifier based on the adjacent scores then you'd have to give it the scores that are actually adjacent, or at least close. :) $\endgroup$
    – Danica
    Commented Jul 14, 2017 at 17:11
  • $\begingroup$ I found it is not applicable to my current CNN output. Our most errors happen in the false negative category. When looking into the class probabilities for these false negative cases, I found they are very close to 0 (0.5 is the threshold), which means CNN confidently classify them as negative cases. In this case, integrating adjacent scores will amplify the adjacent scores significantly and lead to a unstable model. What do you think? Thank you. $\endgroup$
    – Jane
    Commented Aug 1, 2017 at 14:38
  • $\begingroup$ If your model is so confident about the false negatives, then it's probably not going to be possible to take this kind of approach to bump them upwards. $\endgroup$
    – Danica
    Commented Aug 1, 2017 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.