Best ANN Architecture for high-energy physics problem

Question

First off, a disclaimer: I'm not sure if this is the right Stack Exchange for this question, but I'm not aware of a machine learning specific SE.

I am doing research into characterising particle jets in high-energy physics. I am trying to use image recognition techniques, in particular convolutional neural networks to characterize jets into two classes.

These classes can be distinguished by the following features:

• Sudden 'jump' in the number of hits between layers of a detector
• Radius of concentration of hits
• Energy deposited in each layer

I am using 123x123x4 images. Each pixel in each channel represents a level of energy deposited in a layer of the detector. I am concerned that it may even be impossible to do this in a deep-learning approach, as there are typically only 150-300 pixels filled in each image.

I would like to use a ConvNet to classify the two different types of jet. However, I am not sure what architecture to use.

There are other variables that might be of importance in classification, and I would like to be able to include these also (probably in the dense layer immediately before the output).

I tried the following architecture, and trained with Ada, Adamax and Adadelta with no convergence:

              ___________      _________      _________      _________     ________    ______
              | Conv    |     | Max    |     | Conv    |     | Max    |    |       |   |     |
    Image --> | Layer 1 | --> | Pool 1 | --> | Layer 2 | --> | Pool 2 | -->|       |   |     |
              |_________|     |________|     |_________|     |________|    | Dense |   | Out |
                                                                           | Layer |-->|_____|
   Other      ------------------------------------------------------------>|       |
   Data                                                                    |       |
                                                                           |_______|

Are there any suggestions for architectures I should try?

Answer 1

Your architecture looks fine. I mean, it's straight out of MNIST lenet. It's a good solid network to start from. You can then evolve it over time, according to your loss curves, by adding capacity, ie layers, channels per layer, etc.

You could also consider adding dropout, for regularization.

As far as convergence... it's pretty much impossible not to converge, unless you are using too high a learning rate. So, divide your learning rate by 10, until it starts converging. You can just pick some tiny subset of eg 32 images, and just train on those images, using smaller and smaller learning rates, until the error on those 32 images drops to zero (which it should, because you'll overfit them, easily).

Then, once the loss on 32 images is dropping to zero, ie you've picked a small enough learning rate, fixed any bugs etc, then you can add more and more data, and then start increasing capacity of your network, ie adding layers etc. And you probalby want to add dropout, it's really good at encouraging generalization to test data.

Edit: oh, you're using Adadelta etc, which should handle learning rate for you. Well... I've mostly used SGD, and SGD is somewhat standard for deep nets (though gradually falling out of favor a bit recently). You might consider trying SGD, with a small learning rate, and seeing what happens.

Answer 2

Thomas Russell

First of all it is very interesting problem to solve.

I think you should not merge features extracted from CNN and other variables. You can try training CNN end-to-end for predicting class scores and train a Neural Network using other variables for predicting class scores and then some how merge both predictions(e.g. take mean) for final prediction. But the performance of this strategy also depend also on how you normalize inputs and design your networks?