5
$\begingroup$

I am new to Graph Neural Networks and have a basic question. I read a lot about the term mask or masking in the papers (for example this paper) and tutorials about GNNs. What does the term mask/masking refer to?

$\endgroup$
0

1 Answer 1

3
+50
$\begingroup$

You can think of masking as a form of dropout where the contribution (output) of a node is nullified (made zero). This is similar to stochastic depth for residuals in ResNets if you are to consider ResNets as just a special case of GNNs that have no directed cycles.

Just like dropout, each node has a 'survival rate' which determines the probability of being masked. When the node is masked it is effectively removed from the graph along with any nodes which have been completely orphaned by the node's removal (and recursively the process is applied to their children too).

However there is a special case with node masking where the output is not nullified, this is when the masked node is a root of its own aggregation tree; in this case only the child occurances of the masked node are nullified but not the root occurrence.

Figure 7 from this paper illustrates the process fairly well for a 2-aggregation tree of an example graph. figure 7

And, again, just like dropout, masking helps a GNN generalise better by effectively treating the GNN as an ensemble of smaller GNNs with certain nodes removed during training. This helps reduce overfitting by reducing the model's effective capacity at during training, forcing the non-masked subset of nodes to try and learn to the best of their ability without 'help' from the nodes which have been nullified.

$\endgroup$
5
  • $\begingroup$ Thank you very much—helpful explanation with an image and reference. $\endgroup$
    – Coder
    Jun 28, 2021 at 23:55
  • $\begingroup$ I have a question about the masking approach that is being used in nlp to make the neural network work with varying input lengths. Do you know how that works? Any papers or references would be appreciated. $\endgroup$
    – Amin Shn
    Feb 14, 2023 at 12:22
  • $\begingroup$ @AminShn are you talking about specifically a GNN for NLP, or just any sort of NN architecture for variable length sequences of text? Because Transformers that use relative attention such as ALiBi, Transformer-XL and XLNet are sequence length agnostic. If you want arbitrary masking schemes look into XLNet because it handles masking in a similar way to GNNs, but if you only care about causal masking look into ALiBi. $\endgroup$
    – Avelina
    Feb 15, 2023 at 13:20
  • $\begingroup$ Thanks. No I don't care about GNNs. If you check here: tensorflow.org/guide/keras/masking_and_padding they show how to use masking with LSTM as an example. I don't care about the model per se, because I am going to develop my own model, I just want to understand the operation behind masking. In the link I shared it doesn't explain what's the math behind it that make LSTM skip certain time steps. $\endgroup$
    – Amin Shn
    Feb 15, 2023 at 14:03
  • $\begingroup$ I've asked my question in more detail here: stats.stackexchange.com/questions/605391/…. I'd really appreciate it if you could help me with that. $\endgroup$
    – Amin Shn
    Feb 15, 2023 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.