I have a subset of data from the MovieLens database and have verified its accuracy. Essentially, I have a list of <user_id, movie_id, user_score> pairs. I want to be able to predict the rating some user will give to some movie but WITHOUT storing any information about the user (so no user embeddings or user vector information). The model will read a list of movies and their scores that some user gave, then it will try to predict the rating the user will give to some other movie, based on the ratings it gave to those movies. The architecture is as follows:

  • A sequence of movie_ids and ratings a user gave
  • For each movie_id, get the associated embedding vector for it
  • Now we have a list of <movie_embedding_vector, user_score> and we will try to predict the user_watch_vector. The core of this is passing in the list of <movie_embedding_vector, user_score> into stacked Convolutional layers and getting a 30 dimensional vector from it
  • Use that vector and concatenate it with the unknown movie embedding vector and connect them to 4 stacked Dense layers to predict the rating. 512 -> 264 -> 128 -> 1.

I am using mean-squared error as a loss function. This is what the training loss and validation loss look like: enter image description here The validation loss always starts lower. Moreover, no matter what model architecture I play with (even if I do categorical_crossentropy as a loss function and predict classes of ratings), the validation loss is ALWAYS lower than the training loss at the beginning. Furthermore, the model always converges to a constant loss very quickly (~20 iterations) before it stops learning. Has anyone seen a particular example of data like this? Why is this and how can I get it to learn properly? Why is the validation loss always lower?

Training on about 30 million ratings where each user has given on average, 90 ratings.

  • $\begingroup$ Maybe you just missed a minus sign on the loss calculation? $\endgroup$ – badmax Jun 23 '17 at 22:15
  • $\begingroup$ I'm using Keras framework with their built in mse $\endgroup$ – q.Then Jun 23 '17 at 22:16

Keras calculates the training MSE of an epoch as the average MSE over all batches in that epoch, while the test error is calculated right at the end of the epoch. This biases the training MSE numbers to be larger and the test MSE numbers to be smaller.

See the Keras FAQ - Why is my training loss much higher than my testing loss?

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.