2
$\begingroup$

For a university project, we want to use reinforcement learning from raw camera input to teach a robot to hit a ball. It works when we detect the ball and feed that to the algorithm, so raw data is the next step.

As this data is too high dimensional, we want to apply dimensionality reduction. In a first evaluation, we used PCA and other non-neural methods to reduce from 28x32 to around 20 dimensions while still be able to learn from it with a bit loss in precision. Currently, we want to see whether autoencoders also work for that.

The code base of the team was matlab, therefore we used in the first part matlab autoencoders. We used a dataset of 100 pictures and reduced to 200 dimensions. Options are mostly default, from what I remember it where up to 200 episodes.

The result of encoding and decoding for Matlab can be seen in the following picture: enter image description here

Right now, the lab is transitioning to Python instead of Matlab. Therefore, we used Keras to build a custom autoencoder, closely following this blog entry. But when we run our autoencoder on the same dataset to same dimensions, the following happens:

enter image description here

The ball disappears. We have created a bigger dataset with up to 2500 of these pictures, but the ball always disappears:

enter image description here

Also, we used the sparse autoencoder and deep autoencoder, played with the activations and loss function, but always the same result. Has anybody tips on how to improve the learning so we keep the ball? Why does it work on the small dataset in Matlab, but not in Keras on the same?


Edit: I posted the code and the old and new data set on my github. I tried to subtract the mean, add Dropout, change Relu in hidden layers to sigmoids like Matlab, but I always get the removed ball.

$\endgroup$
3
$\begingroup$

I guess it's the problem with the data: the background is the same, and the ball is small.

As shown in the blog you referenced, one application of autoencoders is image denoising. enter image description here
When you use the denoising autoencoder you actually add noise to the input images on purpose, so from your results it seems that the autoencoder only learns the background and the ball is treated as noise.

Maybe we could try the followings to see if we can get better.
1) I don't know if you've subtracted the mean (background) from the image already, if not we can try that so that the data will be almost zero everywhere except the ball.

2) Instead of using fully-connected autoencoders we can try the convolutional autoencoders, which works better with image data and does not really rely on the denoising part. There's example code in that blog for this as well.

$\endgroup$
  • $\begingroup$ I also implemented the convolutional autoencoders, but that also resulted in poor results, the reconstruction was especially extremely blurry. Also, it takes extremely long time to train these. When I subtract the mean, then I get sometimes really weird results (negative loss), or no improvement at all. $\endgroup$ – reindeer Jun 22 '16 at 18:38
  • $\begingroup$ @reindeer that sounds weird, which loss did you use? is it the sum-of-squares? how come the sum-of-squares is negative? could you provide a link of the data if possible? $\endgroup$ – dontloo Jun 23 '16 at 1:34
  • $\begingroup$ @reindeer OK in that blog it uses the binary_crossentropy loss, that's why it can be negative, and it is designed for binary images (like the pictures of the digits). while matlab uses mse+regularizor+sparse regularizor by default. did you try the same loss as in matlab? $\endgroup$ – dontloo Jun 23 '16 at 2:43
2
+50
$\begingroup$

To my knowledge, I think the result with Keras is in fact correct. Since you train the autoencoder with images of the same background, and only the ball is moving, the autoencoder will try to "generalize" the balls and thus produce the "blurry trail" on the reconstructed image.

Therefore, maybe you should not focus on the reconstructed result, but check the nodes at "bottle neck layer" of the autoencoder.

$\endgroup$
  • $\begingroup$ I thought the same, but why then does it work in Matlab? $\endgroup$ – reindeer Jun 22 '16 at 14:51
  • $\begingroup$ Maybe you can provide the sample with balls for Matlab. The sample you provided above has no ball :P or Maybe try to use the big data set (2500) with Matlab $\endgroup$ – Jotarun Jun 23 '16 at 1:28
  • $\begingroup$ I added the link to the code and the old and new dataset, I currently have no access to a computer with the Matlab toolbox, but I am working on a sample of new data + Matlab. $\endgroup$ – reindeer Jun 26 '16 at 12:13
1
$\begingroup$

There are many factors that can affect the result, beyond the fact that the network is an autoencoder. If you're happy with the way the Matlab network performs, it should be possible to replicate in python by making sure all of these factors are the same. Here are some:

  • Network architecture (how many layers, units per layer, connectivity pattern)
  • Activation function for each layer
  • Loss function
  • Update rule and all associated hyperparameters (e.g. if using stochastic gradient descent, then learning rate should be the same, as well as any procedure for changing the learning rate over time; if using a batch optimization algorithm like conjugate gradient or BFGS, then parameters of the optimization solver should match).
  • Procedure for determining when to stop training (e.g. number of training epochs, early stopping or other convergence criteria).
  • Minibatch size
  • Procedure for presenting training examples (e.g. if any shuffling procedure is used, or selective repetition of particular examples)
  • Weight initialization procedure
  • Data preprocessing (e.g. centering/scaling/whitening method)
  • Regularization method and all associated hyperparameters (e.g. $l_1$ / $l_2$ penalties, fraction of dropped units if using dropout, noise method and noise parameters if using denoising autoencoders, etc.)
  • Procedure for selecting hyperparameters (e.g. random vs. grid search, scaling/bounds of grid search, size of training vs. validation set, number of cross validation folds)

Some other things to consider:

  • Matlab and/or Keras might make some of these choices by default or under the hood, so you'd have to check carefully to make sure they're the same.
  • If using stochastic regularization methods like dropout or denoising autoencoders, the noise/dropout should only occur during training, and be turned off when running the network
  • When checking that the Matlab/python networks are equivalent, use the same dataset
  • The loss function is generally not convex (and the training procedure may get trapped at saddle points). So, it's likely that the final network will differ across training runs (and might not converge to a good solution on some runs). So, identical results can't be expected. But, error on the validation set and qualitative aspects like presence/absence of the ball should be similar. It might be a good idea to compare a number of training runs in both Matlab and python to account for run-to-run variability.

If you've already trained the Matlab network and just need to use it, another possible approach would be to 'port' the trained network to keras/python. This would involve constructing the exact same network (including all equations and parameters, but the factors involving initialization/training would be taken out of the picture). If everything is duplicated, it should produce exactly the same results. It could also make sense to do this as a test, even if you want to perform further training with the python network. The reason is that it isolates 'runtime' factors from training factors. If the python network produces different results despite having duplicated the network architecture and parameters, then something on this level is still unaccounted for. If results are the same, then you can move on to duplicating and testing initialization/training factors.

Another option could be continue to run your network in Matlab but port the rest of the code to python. To call Matlab from python, use the Matlab engine API for python or python-matlab-bridge. To call python from Matlab, use Matlab's Python API.

Here's something else to consider, along different lines. If your camera is fixed and the ball is the only thing that moves, then there's no reason to try to represent the constant background image. You could take a static image of the background (possibly averaging several frames to reduce noise), then subtract it from every frame of the video. The only thing that should remain is the ball, plus a small amount of noise (which could be thresholded out). Use these preprocessed images as input to the autoencoder. A better method might be to use a weighted loss function for each frame. Each pixel would have a weight that decreases its contribution to the error if it's similar to the corresponding pixel in the background image. The hope with either of these methods is that they might focus the autoencoder's efforts on the ball and not the background. Of course, there might not even be a need for a neural network in this case. Given such preprocessed images, you could use a simple centroid finding algorithm to obtain the ball coordinates and use these as input to your reinforcement learning algorithm.

$\endgroup$
  • $\begingroup$ Thank you very much for the input. I already compared all the parameters I can control, but the results are still not even close. Also, Keras does not have the activation functions from what I saw, like 'satlin'. I just wonder what exactly could cause such a massive difference, as Matlab does it with 100 images and Keras fails with even more. I cannot subtract the background, as in later steps, we let the robots actually play the game, i.e. they move. $\endgroup$ – reindeer Jun 21 '16 at 10:11
  • $\begingroup$ Hmm, it must mean that something is still different. The 'porting' approach I described might be useful to determine whether issue lies with the network code itself or the training procedure. You might also consider using Lasagne instead of Keras, which would give you more control by giving easier access to the internals of the model. For example, there's no 'satlin' nonlinearity, but it would be trivial to add one. $\endgroup$ – user20160 Jun 21 '16 at 10:38
  • $\begingroup$ Matlab uses sigmoid by default, I overlooked that when I read the doc. $\endgroup$ – reindeer Jun 26 '16 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.