Questions tagged [neural-networks]
Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.
7,977
questions
1
vote
0
answers
22
views
Neural Network ReLU majority of weights small
When I view a histogram of my weights it is very much centred at 0, with the overwhelming majority being very small. I want to ensure I do not have a vanishing gradient problem.
I must preface this ...
- 123
0
votes
0
answers
18
views
Neural Network Linear Activation Functions [duplicate]
I understand the intuition that the sum of linear functions is again linear, and that is why a neural network with linear activation functions yields a linear model. But what I'm confused about is ...
0
votes
0
answers
8
views
VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling
I have implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE ...
- 1
0
votes
0
answers
22
views
How to add multiple embeddings (layers) to LSTM layer [closed]
The similar question was asked before here https://stackoverflow.com/questions/52627739/how-to-merge-numerical-and-embedding-sequential-models-to-treat-categories-in-rn/52629902#...
0
votes
2
answers
54
views
Mean squared error (MSE) vs Least squares error (LSE)
From my understanding the only difference between MSE and LSE is that with MSE you divide the sum of squared errors by the total number of values to get an average rather than just using the sum.
This ...
- 2,091
-1
votes
0
answers
19
views
Self supervised learning [closed]
I want resources and code for using self supervised learning for counting small images in a picture . i already have a code for it but it is not sufficient to count small images .
1
vote
0
answers
12
views
RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?
I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
2
votes
1
answer
182
views
Is batching needed for the test set?
I'm just starting to learn about CNN (convolutional neural networks). Does the test data also need to be divided into batches, similar to how it's done with the training data?
- 21
0
votes
0
answers
27
views
Training a neural network to predict a same output on the input which i train it on [closed]
Let say I train a neural network with this input vector 1.45 2.78, .. and there another input like this and I want to have vector 2.36, 78.2, -4.5, ...."Simply speaking whenever I gave a input ...
0
votes
0
answers
8
views
Can LayerNorm be used with feed forward neural networks? [closed]
FFN usually uses batch norm, but is there any reason why layernorm isn't used?
- 225
0
votes
0
answers
11
views
What does it mean when the Neural Network frequently has poor weight initialization?
I've been training a neural network to learn simple functions like addition. When I train this neural network from scratch, about half the time the neural network gets great accuracy, and the other ...
- 225
1
vote
0
answers
28
views
Neural network as alternative to conventional nonlinear regression?
Can I use the strength of neural networks for data analysis? Suppose I want to find a fit to a noisy sine curve $1 - V \sin(w \theta+ \phi)$ and I want to find an estimate for parameters V, $w$ and $\...
- 417
0
votes
0
answers
27
views
Unexpected distribution of scores after using class-weighted loss, when data is highly imbalanced (2%), low N and high p
I won't go into the way the data is built because I want to keep the discussion general. Relative to balancing, I couldn't find a lot of materials online about the results of cost-sensitive learning.
...
- 306
-1
votes
1
answer
32
views
Encoding heavily skewed feature for neural network [closed]
I am trying to use a neural network to predict horse races. One of the features I am using is historical starting price (i.e. what the market thinks the probability of that horse is.) It is a very ...
- 247
-1
votes
0
answers
25
views
How to deal with negative loss during deep learning training, when using negative log likelihood loss? [duplicate]
I have a question regarding how to deal with negative loss during the training of my deep learning model. I've observed many instances of negative loss generated by the data points as outlined below.
...
- 133
0
votes
0
answers
10
views
Training an Image Captioning Model with variable number of captions per image
I am following this guide for training an Image Captioning model
It uses a dataset which always has 5 captions per image. My dataset greatly varies how many captions I have per image from 1-42. This ...
- 133
0
votes
0
answers
21
views
What is the “web” drawing for a graph neural network?
It is common to draw a neural network as a "web" of neurons and connections, such as the "web" below of a multilayer perception that has input neurons in white, hidden neurons in ...
- 55.1k
0
votes
0
answers
8
views
Toy dataset: Radial VAE
I'm evaluating disentanglement in toy datasets seeing as we have such little understanding of the phenomena. I'm using various tools from differential geometry. Now I want to train a VAE on the ...
- 131
0
votes
0
answers
21
views
Comparison of Squared Dice Loss vs. Standard Dice Loss
I've been diving into segmentation tasks and came across two variations of the Dice Loss that I'm considering for my neural network: the standard Dice Loss and the Squared Dice Loss.
The Standard Dice ...
- 118
2
votes
1
answer
35
views
Are Bagged Ensembles of Neural Networks Actually Helpful?
I've been looking into ways to estimate uncertainty for regression tasks on neural networks. One of the obvious options is ensemble modeling. Consider an ensemble of neural networks that all have ...
- 88
0
votes
1
answer
20
views
Is it unwise to create a predictive model based off 20 independent variables when only 10 variables will be available for future observations?
I've created a predictive model which is based off a historical dataset and has 20 independent variables as the dataset set is comprised of completed projects, so have full information and dataset of ...
2
votes
1
answer
33
views
Why do SAC and TD3 use multiple critic networks as opposed to single network with multiple outputs?
Q-function approximators based on neural networks tend to overestimate the Q-function. Accordingly, reinforcement learning algorithms such as Soft Actor-Critic (SAC) and Twin Delayed DDPG (TD3) use ...
- 387
0
votes
0
answers
8
views
MC Dropout without weight decay -- how is the model precision calculated?
In the original [MC Dropout paper][1], the variance is calculated as the sum of two contributions, an sample variance (over the multiple forward passes), var$(y)$, plus a term quantifying the inverse ...
- 1
0
votes
0
answers
34
views
How this is possible? Test loss is under train
I got this graph for my loss.
As you can see the distance between the two graphs is so much!
Can we say it shows bias is large and it's underfitting?
Is this thing that I just said true or isn't true?...
- 1
0
votes
1
answer
16
views
Can I transform my output variable in an imbalanced dataset?
I have a dataset that has an output variable that is quite right-skewed and imbalanced. I want to use a neural network as a regressor to predict the output variable. Visually, it looks like there may ...
- 101
0
votes
0
answers
14
views
Process Modelling with LSTMs vs Probabilistic Programming
I am trying to model an aircraft’s turnaround process from the beginning (in-block) to the end (off-block). The goal is
to gain transparency about the progress of the process / subprocesses and
to ...
- 1
0
votes
1
answer
41
views
Why not use input padding in the first attention block in transformer decoder
I was studying the transformer decoder code below in Keras/Tensorflow. It was not clear how they made making decisions.
In the first attention block below (self.attention_1), why did they use ...
- 1
6
votes
6
answers
747
views
Confused about the notion of overfitting and noisy target function
So I am reading a textbook called "Learning from Data" by Abu Mostafa et al.
I am confused about the following concepts:
According to the authors, most real-world target functions $f$ are ...
- 1,373
0
votes
2
answers
45
views
Can not understand a column in a paper about CNNs
I am reading the SqueezeNet paper and I do not get the parameter depth here:
There isn't a description under the table, and the only extra mention of the parameter is that it means the number of ...
- 135
0
votes
0
answers
12
views
Find parameters in an algorithm with Machine Learning
I work on an equalization project. I've implemented the Least Mean Squares (LMS) algorithm with a set of data. To initialize coefficients h: h[k+1] = h[k] + mu e[k] x[k].
I need to find a compromize ...
0
votes
0
answers
12
views
Can MDP or reinforcement learning method be used on an And-Or graph?
I want to perform a pathfinding task in a graph using RL, I consider each node of the graph as a state, but there are and/or relations between these states.So I'm wondering if I can still use the ...
- 1
0
votes
0
answers
6
views
Question about Bounding Box Regression in R-CNN
I am curious why, in that equation of Bounding Box Regression, the Feature Vector φ5 obtained from Pool 5 among the trained CNN Layers is multiplied by w. Also, I'm curious about the reason for adding ...
1
vote
0
answers
21
views
Should networks get wider as they get deeper or narrower?
In a network with multiple hidden layers, should the layers start with fewer neurons and gain more neurons before the output layer? Should they start with more neurons in the first few layers and ...
- 11
1
vote
0
answers
24
views
Does skipgram model uses backpropagation?
I just started to get interested in natural language processing and I was trying to understand the skipgram model from word2vec. I was reading this interesting website. However, in the mentioned ...
- 111
10
votes
4
answers
1k
views
I've always learned that data standardization is not necessary for OLS regression, but then recommended for neural networks. Intuitively, why is that?
I know for LASSO and elastic net regression, standardization is important, because coefficient penalization in regularization will be biased if the ranges of data are different.
Meanwhile, OLS ...
- 101
0
votes
0
answers
4
views
NN prediction of fault duration time
I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges:
Model Performance: My neural network seems to be ...
- 1
0
votes
1
answer
51
views
Performing a classification if having categorial labels and a distance matrix
I encountered a multi-class classification problem and I wonder which model would work the best in my scenario. I have around 50,000 vectors (each of size 200) with corresponding categorical labels ...
2
votes
1
answer
34
views
How do additional loss terms impact the parameter count in a Deep Neural Network?
I've been working on training DNNs for various tasks and recently started incorporating additional loss terms into my models to improve convergence and performance.
I'm curious to know if I need to ...
- 53
0
votes
0
answers
16
views
CNN regression: output mimic the input test data
I am working on convolutional neural network regression problem using U-Net architecture for computing poisson equations. The input data is particle distribution (rho) and the output is electrical ...
0
votes
0
answers
15
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 657
0
votes
0
answers
8
views
How to adjust the scaling of the new data while use Incremental training of a neural network?
I am planning to use incremental training of my neural network model since I continually get new data and at present retrain the model after a period of time but the training window shifts forward. To ...
- 109
2
votes
0
answers
44
views
Is there anything like spaced repetition learning for machine learning models?
I would like to know if something like spaced repetition exists but for helping machine learning models learn.
Spaced repetition is a flashcard learning method for humans where the algorithm tries to ...
- 141
0
votes
0
answers
23
views
Which meta-learning algorithms are well-suited for "many-shot learning" scenarios, where the target training set is large?
Much of the meta-learning literature deals with the few-shot learning problem of using data from a diverse set of "source" tasks (the meta-dataset) in order to train a model that can quickly ...
- 101
0
votes
0
answers
11
views
How to best perform early stopping and operation point choice when doing cross validation
I currently have my dataset split in three sets, training, validation, and a hold-out test set. I am training neural networks for a binary classification problem using the following protocol:
Train (...
- 115
0
votes
1
answer
33
views
Are Neural Networks with a softmax output layer considered Probabilistic Classifiers?
For example, is ResNet for image classification a type of Probabilistic Classifier when we take into account the softmax at the end of the network as part of the network? My intuition points in that ...
2
votes
1
answer
163
views
Fitting a simple model first, then training a neural network on the error
Can someone tell me what the name is for the following process?
I have some data with inputs $x_i$ and outputs $y_i$, and I fit a simple model (e.g. linear regression) to them. Then, I compute the ...
- 21
1
vote
0
answers
25
views
Why I get spikes during training with vanilla gradient descent? [closed]
I developed my own NN toolbox, and it seems it works fine.
But I am not sure why I get these spikes in my loss during training:
I a training for a classification task of 2 inputs and 2 classes, ...
- 119
0
votes
0
answers
6
views
How to transform ratio input features in deep learning
I am a recommender system study which predict how likely a user browsing a product A will but another product B. One of the features is the price ratio of A and B, i.e., PriceB/PriceA. The assumption ...
- 2,065
0
votes
0
answers
10
views
Kernel + Mutliple SVM's + Platt Scaling = 1 layer neural network?
I have built my own Support Vector Machine by using quadratic programming and I'm using Kernel PCA with SVM. The output is tanh e.g Platt scaling.
When I combinde ...
- 349
0
votes
1
answer
44
views
Machine learning method for predicting years ahead
Let's say I have Antarctic temperature data for 2020-2023, and based on this data I would like to forecast the temperature for the following years.
Which machine learning method will allow me to ...
- 11