3
$\begingroup$

A well known solution to make Neural Networks (NNs) work with varying lengths of input data is to use padding and then masking the padded inputs (padded with zero, a common approach in natural language processing). It's being said that this approach makes the NN skip the padded inputs. I want to know what's the mechanism behind such skipping. For me NNs are all about matrix multiplication. So here are my questions:

  1. Is masking done by forcing some weights to zero?
  2. How is it different than simply using the padded values that are zero without masking (zero times anything is zero again, hence I suppose the answer to my 1st question should be no, because there is no point in forcing the weights to zero when the input is zero itself)
  3. I assume masking didn’t suddenly emerge out of thin air, so I want to know who invented the method? Are there any academic papers talking about it?
$\endgroup$

1 Answer 1

1
$\begingroup$

Masking is not enforcing weights to be zero, masking enforces network inputs and intermediate hidden states to be zero. Setting the inputs to be zero at the begging is not enough; many places in a NN would make the states non-zero.

A typical feed-forward layer with an activation function $\sigma$ is $\sigma(Wx + b)$, and the output will be non-zero because of the bias $b$. When using batch normalization, an estimated mean is subtracted from the activation, which makes it non-zero. Layer normalization has a learned bias term, which also would make the hidden state non-zero.

Masking is, however, only relevant when there is a danger that the invalid mask-out hidden states would influence the network output. This happens, e.g., in Transformers in the self-attention layers: you only want the attention to consider valid positions. This is done by modifying the attention distribution (before normalization) by setting zeros to invalid positions, so it only considers the positions that were part of the input. Hidden states on the invalid position will get some values in the feed-forward layers and during layer normalization, but it does not matter as long as it does not influence the valid states.

Masking invalid input positions is considered a technical detail, and such details are often not discussed in papers. The first time I encountered masking was in 2014 when the original attention paper appeared on arXiv. They released an implementation on Github which did the masking in the attention mechanism, but the paper did not say a word about it.

$\endgroup$
3
  • $\begingroup$ Thanks for your explanation. I've seen masking only implemented for Transformers, LSTM and etc. But not for simple Dense layers. I want to make a simple MLP work with missing values in the predictors by simply masking those inputs. Do you know if that's possible? $\endgroup$
    – Amin Shn
    Feb 20, 2023 at 10:46
  • $\begingroup$ In theory, I would say, why not, but I never say anything like this. IMO it would make more sense to make a special value for missing values for categorical features or a flag feature saying that some input values are missing for numerical features. $\endgroup$
    – Jindřich
    Feb 20, 2023 at 13:00
  • $\begingroup$ Thank you, I made a quick implementation. Could you please check and see whether it is OK? This is the link: stats.stackexchange.com/questions/606509/… $\endgroup$
    – Amin Shn
    Feb 24, 2023 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.