New answers tagged

0 votes

Is it possible to deal with datasets of graphs with different number of nodes in graph nural networks?

Is it possible to do this with graph neural networks? Yes, this is possible using various GNNs architectures, and you usually do not need to set a maximum number or nodes. For example, Tox21 dataset ...
user avatar
  • 51.1k
1 vote
Accepted

References for cross-validation implementations in Pytorch

All the same considerations for cross validation apply for neural networks as for any other type of model. I.e. the usual scikit-learn (or other options for special situations like grouped+stratified ...
user avatar
  • 22.3k
1 vote

Concatenation or separate channels for a CNN

I'm answering this purely from a deep learning architecture persepective. There may well be domain-specific reasons why the authors have chosen the architecture described. This review paper: ...
user avatar
  • 454
0 votes

Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?

Usually, it does introduce more parameters. The original definition of attention (by Bahdanau et al.) defines attention energies as a single-layer NN computation: $$e_i = v^T \tanh \left( Wq + Uk_i + ...
user avatar
  • 2,401
2 votes

Do I need to normalize data before applying L1, L2 norm in ANN

Even if you don't use regularization, it is highly advised that you normalize your data before inputting to a neural network as it'll significantly affect the gradients. So, yes, you should normalize ...
user avatar
  • 51.1k
1 vote

What is the origin of the autoencoder neural networks?

Reviving this thread - In "Neurocomputing" by Robert Hecht-Nielsen @ 1990 there is reference to a 1986 paper by Cottrell/Munro/Zipser that outlines use of a neural network that has the ...
user avatar
0 votes

Can a deep neural network be used for mining bitcoins?

I couldn't believe that this is bad connection. I even think that AI has own place on any task human can provide as it can offer much more than just algorithmic solution. My thanks to Alex R. for good ...
user avatar
  • 101
3 votes

Finding the position of the global optimum with Pytorch

The description of the particular network is not specific enough to understand what the model is, or how it works. Additionally, the terminology seems confused because datasets don't have parameters, ...
user avatar
  • 78.2k
0 votes

Should I join train and validation sets for final NN model training? If yes, when to stop training the final model?

In general, the second option is better b/c it theoretically results in a model with lower variance than the one trained only on the train dataset, even if you recycle hyperparameters. And as long as ...
user avatar
0 votes

Why is data augmentation classified as a type of regularization?

The objective of regularization is to improve the generalization capability of the model (in other words its ability to perform well on unseen data). Regularization can be explicit or implicit. The ...
user avatar
  • 411
1 vote
Accepted

In a tranformer, the same word can have different attention weights in different sentences?

Yes, and it is not only the case for Transformer but for nearly any deep learning NLP model. Only when treating natural language data as bag-of-words, the sentence is considered as a sum of ...
user avatar
  • 113k
2 votes
Accepted

Restrict output range of a neuron based on output of other neurons

You can enforce the constraint with an additional transformation. Suppose we have $x_i = \sigma(AH + b)$ where $\sigma$ is the sigmoid activation and $A, b$ are the weights and biases. We can ...
user avatar
  • 78.2k
1 vote

Do value and key of additive attention need to have the same dimension?

In this implementation, yes, but the query and key variables correspond to a linear projection of the decoder and encoder states ...
user avatar
  • 2,401
3 votes

Automatic NN Architecture Configuration

The two most common methods are Grid Search and Random Search. For both of these you specify a set or a range of hyperparameter values and then iteratively try combinations of hyperparameters to find ...
user avatar
  • 454
1 vote

Why does Group Normalization work?

I don't think there's any formal reasoning as to why GN would outperform both LN and IN. So this is all based on my intuition. The difference in LN, GN, and IN is essentially the number of groups you ...
user avatar
  • 111
1 vote

AUROC too high in image classification

It may be that your minority class(es) have poor performance in the hard classification (obtained by choosing the class with the largest predicted probability), but that their predicted probabilities ...
user avatar
  • 2,846
0 votes

what is the memory usage of the VGG-net or any other neural networks?

For the detailed VGG memory footprint please take a look into: http://graphics.stanford.edu/courses/cs348v-18-winter/lectures/09_dnntrain.pdf, slide 22.
user avatar
2 votes
Accepted

Model performance when ground truth is not available

I am not sure I understand your first technique, because, AFAIK, the reconstruction error of autoencoders is what is used as the score for anomaly detection in the first place, so your first technique ...
user avatar
  • 2,948
3 votes

Why is cross entropy loss better than MSE for multi-class classification?

Crossentropy loss is equivalent to maximum likelihood estimation in a multinomial logistic regression. Consequently, we get all of the wonderful features of maximum likelihood estimation. This topic ...
user avatar
  • 31k
1 vote
Accepted

Why does contrastive loss distinguish positive from negative samples?

The contrastive loss has 2 components: The positives should be close together, so minimize $\| f(A) - f(B) \|$. The negative portion is less obvious, but the idea is that we want negatives to be ...
user avatar
  • 78.2k
2 votes

What are the advantages of using a Machine Learning (NN) method instead of regression model in survival analysis?

The neural net will pick up also interaction effects, unlike your Cox model with (one-dimensional) splines. However, you will need a very large sample size to beat a well built additive Cox model in ...
user avatar
  • 10.7k
2 votes

exp(log_softmax) vs softmax as neural network activation

Numerical stability and relisience to underflow is preserved by undertaking all intermediate computations in log-space to avoid the loss of small numbers due to underflow (i.e., rounding small numbers ...
user avatar
  • 94.4k
0 votes

What is the role of feed forward layer in Transformer Neural Network architecture?

Consider encoder part of transformer. If there is no feed-forward layer, self-attention is simply performing re-averaging of value vectors. In order to add more model function, i.e. element-wise non-...
user avatar
1 vote
Accepted

Baseline model for predicting the load forecast

I´m pursuing a PhD in this area so far I can tell that: Industry is mostly based on robust yet simple models: AWS offers a solution that as far as I can tell uses ARMA based models (AWS Forecast) ETAP ...
user avatar
0 votes

What is a “center loss”?

In short, it tries to increase the inter-class distance of the embeddings using the softmax function and decrease the intra-class distance for embeddings of each class using the center loss. To make ...
user avatar
  • 5,151
0 votes

Why have predictions for neural network regression wider error margin for edge values?

In general, for debugging, you could have a look at the data that is so badly predicted to get more insight. In this case, it looks like this has to do with the fact that angles are not really an ...
user avatar
  • 2,948
0 votes

How can calculate number of weights in LSTM

Total number of weights in LSTM N/W = 4 x inp_dim x (inp_dim + out_dim + 1) So, in your first model: For Stage-1(input --> h1): inp_dim = 39; out_dim = 1024 Therefore, weights of stage-1 = 4 x 39 x ...
user avatar
  • 1
3 votes
Accepted

Big NN vs Ensemble of Small NNs

I agree with your thoughts. However, small models tend to have high bias and large models have high variance (see S. Geman et al.: Neural Networks and the Bias/Variance Dilemma). So, if the model is ...
user avatar
  • 454
0 votes

How to calculate Precision and Recall for an image classification problem?

First, you should know the concepts of True Positive(TP), False Negative(FN), False Positive(FP) and True Negative(TN). These four items form the confusion matrix. You can define a Positive class as ...
user avatar
11 votes

How to make regression results to be integers?

What you’re doing is an ordinal regression task, which TensorFlow seems to support, and I recommend looking into this approach. At the same time, remember Box’s famous quote. All models are wrong, ...
user avatar
  • 31k
0 votes
Accepted

Neural Nework and mathematical programming

We can use the ReLU activation function and derive a linear equation from a neural network. Further information is presented in this paper (doi: 10.1007/s10601-018-9285-6)
user avatar
  • 129
0 votes

What does it mean when all gradients of a neural network are 0?

Gradients all equal to zero does not necessarily imply any problem with the network. Both minima and maxima occur where the gradient is zero. So it’s possible that your network has arrived at a local ...
user avatar
  • 78.2k
-2 votes

What does it mean when all gradients of a neural network are 0?

If you're using "tensorflow" and giving your model a batch make sure you put model(batch, training=True). If training isn't set to true then the gradient ...
user avatar
0 votes

Why is DP-SGD differentially private?

Here is a non-technical answer: In each lot (subset of data) the authors of that paper average of the clipped gradients (scaled to be length $\leq C$) from that lot. The clipping allows us to know ...
user avatar
0 votes

Understanding neural network outputs in terms of causal models?

Are you familiar with methods of interpreting neural networks which construct "relative variable importance" scores from neural network weights? The basic logic is that NN weights are ...
user avatar
  • 29
0 votes
Accepted

Can self-supervised pretraining work with only labeled data?

Self-supervised learning is effective because it allows researchers to use much more data than they would have time/money to label. If you can't expand your training dataset using self-supervised ...
user avatar

Top 50 recent answers are included