2
$\begingroup$

I am relatively new to CNNs, and I'm working on a machine learning classification problem with chemical data. I'm looking for advice on 1) how to structure the input data, and 2) architecture for the neural network classifier. For reference, I am using PyTorch to build this.

For this task I have raman spectra to use as input. I've already removed the baseline and normalized the spectra to lie between 0 and 1. Next I want to input them into the model. I have been operating under the assumption that the data should be input into the model in the shape [batch, 1, length_of_spectra], that is if each spectrum is length 1000, and I have a batch size of 32, then the input data shape would be [32, 1, 1000]. Is this the correct shape? I'm a little confused about the difference between the number of (input) channels and the signal length. Should I consider each wavenumber to be a channel or is the entire spectrum a single channel? Effectively I want the neural net operators to operate on each spectrum independently of the other spectra in a particular batch. That is, if I do a 1-D convolution layer, I want the kernel to operate only on a single spectrum at a time (not convolve multiple spectra together).

I've so far tried two different structures, all relatively simple and taken from literature on using CNNs with raman spectroscopy data. Structure 1 (taken from https://pubs.rsc.org/en/content/articlepdf/2019/an/c8an02212g) (The way I'm describing the CNN structures below is meant to be simply descriptive, not computationally accurate.) :

layer1 = Dropout(p=0.5)(
   Maxpool1d(kernel_size=2, stride=2, padding=1)(
      leaky_relu(
         Conv1d(in_channels=1, out_channels=32, kernel_size=5, stride=2, padding=2)
      )
   )
)

layer2 = Dropout(p=0.5)(
   Maxpool1d(kernel_size=2, stride=2, padding=1)(
      leaky_relu(
         Conv1d(in_channels=32, out_channels=64, kernel_size=5, stride=2, padding=2)
      )
   )
)

layer3 = Linear(8128, 1024)(
   torch.flatten(start_dim=1)
)

layer4 = Linear(1024, 1)

When I'm training this, I do get nan loss after some time, and I'm not entirely sure why. If you have any suggestions on that, I'd very much appreciate it. Structure 2 (taken from https://www.sciencedirect.com/science/article/pii/S1386142521003085):

layer1 =  maxpool1d(kernel_size=2, stride=2)(
   leaky_relu(
      batchnorm1d(num_features=5)(
         conv1d(in_channels=1, out_channels=5, kernel_size=10, stride=2)
      )
   )
)

layer2 = dropout(p=0.5)(
   leaky_relu(
      batchnorm1d(num_features=5)(
         Linear(in_features=2125, out_features=5)(torch.flatten(start_dim=1))
      )
   )
)

layer3 = Linear(in_features=5, out_features=1)

This model also reached nan loss after some time, and I'm not sure why. Anyhow, this is what I've done so far, and I appreciate any advice on either of these models, but especially on the data input format/shape. Thanks!

$\endgroup$

1 Answer 1

3
$\begingroup$

Cool project!

Regarding inputs:

It depends on how you want to treat the signal. Putting wavenumbers as channels would result essentially in a fully connected Linear layer.

In a deep learning convolution, each filter will have different weights for each channel. So if you orient the spectrum so that wavenumbers lie along the channels axis, your convolution will just be computing a dense dot product.

If you're using convolutions, then you probably don't want that. (Otherwise you'd just explicitly model as a Linear layer).

But, regarding architecture and nans:

Are you sure you want convolutions for this problem? Is there a reason to think that there are local patterns that repeat at different wavenumbers? Patterns that repeat within a signal are where convolutions provide modeling utility. But if the goal is something like fingerprinting, I would think a Linear layer is closer to what you want. (Have not looked at the citations, though - maybe they provide good justification for the convolutions!) Ditto pooling - if you're not using convolutions, might not be very useful.

I'd also reconsider whether batch norm is the ideal normalization for this problem - I would think layer normalization would be sufficient and generally easier to work with, especially at smaller batch sizes like 32.

As far as nans, these in my experience usually result from optimization issues, not the model. If you're using Adam or something else complicated, I'd try switching to SGD with momentum and lowering the learning rate. I'd also check your data and confirm what the model is receiving during training is what you expect.

$\endgroup$
4
  • $\begingroup$ My thinking on the convolutional layers is that raman spectroscopy is based on vibrational responses of the components of the chemical. These responses are spatially dependent and so I think there would be patterns that repeat within the signal, certainly for a classification problem, where I'm looking to find a specific chemical signature within a given spectrum. But, it's possible I'm misunderstanding how convolutions work in this domain; I'll do some more investigating. As for nans, I was getting them some with Adam but a lot with LBFGS. $\endgroup$
    – CopyOfA
    Commented Aug 25, 2021 at 11:54
  • $\begingroup$ Hm, thinking out loud briefly. I vaguely remember working with IR like a decade ago, and Raman seems similar. If you have a single spectrum, then you don't have spatial information represented directly within that spectrum, in which case you wouldn't expect there to really be repeated patterns. However, IIRC in IR you could see shifted peaks in a spectrum due to interactions with nearby functional groups, etc. If Raman is similar, then I can imagine seeing these kinds of repeated patterns s.t. a convolution might do useful work. $\endgroup$
    – Gianni
    Commented Aug 26, 2021 at 13:04
  • $\begingroup$ OTOH, I imagine you wouldn't expect to see peaks shift all that far. Convolutions will apply the same weights at every wavenumber. You can think of convolutions as a kind of induced sparsity that lets you save a lot of weights relative to dense layers. But not clear how much they actually help here, if learned patterns wouldn't apply equally at every point in the input signals. $\endgroup$
    – Gianni
    Commented Aug 26, 2021 at 13:18
  • $\begingroup$ Re optimizers: I would definitely test SGD instead of Adam and LBFGS if you're seeing nans. (Hessian-ish methods in particular are relatively uncommon for deep learning due to stability and memory.) You could try gradient clipping or some other tricks, but often SGD is just as fine and less hassle, and saving memory could let you scale the batch size instead. $\endgroup$
    – Gianni
    Commented Aug 26, 2021 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.