Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

Filter by
Sorted by
Tagged with
0
votes
0answers
34 views

Class imbalance: training set is balanced but test set is imbalanced, how to train?

I have a huge dataset, say around 100M data points, with a class imbalance of 1 positive for every 100 negatives. It is very difficult to train on the entire dataset, so I tend to undersample the ...
1
vote
0answers
11 views

Overfitting a neural network to a single batch as a sanity check - how small a loss value is small enough and long to run for?

I'm currently developing a neural network for a regression task. Following on from the advice given in places like here, here, and here I'm attempting to overfit my model to a single batch of 5 ...
0
votes
0answers
15 views

Effect of averaging gradients and shuffling (PPO)

In PPO (reinforcement learning algorithm) one often takes large batchsizes like 100000. To do that one averages the gradients 100 minibatches of batchsize 1000. As I understand it is recommended to ...
0
votes
1answer
32 views

Getting binary class from continuous values of neural network output [duplicate]

I have a custom neural network that I wrote from scratch and it does lot of mathematical computations and the output is a continuous value. I want to get the binary class output from these continuous ...
0
votes
0answers
23 views

Cross Entropy for sigmoid/tanh regression

My neural network has a tanh activation function for the output layer. It would be no problem to change this to sigmoid. The labels are values in the same range. By this I mean that the target value ...
1
vote
0answers
17 views

DL back-propagation: Does the loss necessarily need to be calculated from the direct output from a network?

In deep learning, we typically pass the output of a network we are training along with a training sample (batched or not) into a loss function. We then take the gradient of this loss function with ...
1
vote
0answers
21 views

Model fails to learn constant values

I want to train a model that takes image as input and predicts 8 float numbers. Here's the model architecture: ...
1
vote
0answers
24 views

Determine the optimal threshold for sigmoid layer in neural network for binary classification [duplicate]

I have a neural network that gives out a continuous value as output and I need to classify it as class 0 or 1. I am currently using a sigmoid on this continuous output value but after sigmoid the ...
5
votes
1answer
80 views

Proper loss function for regression with uniform target distribution

I'm doing some simulations and I would like to estimate a real number that is uniformly distributed between minValue and maxValue. For instance, between 20 and 30 (it's not an angle, so estimating its ...
1
vote
0answers
36 views

Is backpropagation a fancy way of saying "calculate gradient by taking partial derivative w.r.t. all x's"

I understanding how gradient is calculated in the usual context--it is just taking partial derivative w.r.t. each element of the X vector. Say a function $f$ has $n$ independent variables, denoted by $...
0
votes
0answers
28 views

How do nested neural networks function? [closed]

Problem: I'm trying to create a model where I input a set of (x,y,z) coordinates to get desired/ transformed set of (x,y,z) coordinates. My challenge: I need to do this for multiple objects that ...
0
votes
0answers
11 views

Data Preprocessing for Logistic Regression [closed]

What kind of data preprocessing techniques are best for logistic regression problems? I have datasets with about 11 features (more than half of which have discrete values).
0
votes
0answers
14 views

I need help understanding the meaning of the loss values of a WGAN with Gradient Penalty

I am currently working on training a Auxiliary Classifier Wasserstein GAN with Gradient Penalty. I based my implementation off of https://keras.io/examples/generative/wgan_gp/ (to which I added the ...
3
votes
0answers
28 views

Intuition behind Dropout in CNNs

I recently attended a lecture on CNNs and was given a brief overview on the topic of dropout. I understood the logic behind the regularization and silencing the firing of neurons to prevent ...
1
vote
0answers
26 views

Validation loss is significantly higher than train loss

I'm trying to train several models to solve an image regression problem. I'm using MSE as my loss function. Here's a couple of charts illustrating the training process: As you can see, on both charts ...
3
votes
1answer
27 views

Why in some neural network training cases the outputs are assumed to be a probability distribution?

This might be a stupid question but this question is bugging me for a long time. When I first started working with neural networks we usually created a neural network which output vector of number ...
0
votes
0answers
7 views

Categorical variable converted to embeddings using Embedding layer visual depiction

Say I have a dataset with 3 features 1) Date 2) dayOfWeek with values Sunday to Saturday 3) Number_of_customers If I use One-hot-encoding to convert "dayOfWeek" feature to numeric ...
3
votes
1answer
36 views

Extrapolation using machine learning models under specific assumptions

I have a problem that requires inherently extrapolation. I am aware that this a crucial matter with most (if not all) machine learning models. Yet, given the physical phenomenon underlying the ...
0
votes
0answers
53 views

Mean average precision varying based on classes

In my dataset, I have 10 classes, which have some peculiar characteristics (e.g., a single "object" can be separated into multiple disconnected parts). Now, the goal is to perform weakly ...
1
vote
1answer
37 views

Objective functions of the flow network based generative model by Yoshua Bengio?

I am reading the Yoshua Bengio et al, Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. The objective function, Equation (11) and (12) are set for a given trajectory ...
0
votes
0answers
7 views

Why do transformers use separate K and V networks

In transformers, two separate networks $\phi_k, \phi_q$ compute the keys and queries. The attention weight is then the dot product of the keys and queries. (i.e. $score = \phi_k(z_1) \cdot \phi_q(z_2)$...
-1
votes
0answers
23 views

Cubic equation gets high loss [duplicate]

I'm trying to learn some machine learning and after looking up some tutorials I managed to train a linear regression and second degree equation with acceptable precision. I then decided to step it up ...
0
votes
0answers
10 views

An efficient way to encode & embed tabular data of a video into a transformer?

So a little bit of a background: I have a folder which contains video files of lets say humans doing a certain action (i.e. walking) where each .2 seconds is documented in a ...
2
votes
1answer
127 views

How to train a neural network to minimize two loss functions?

For TF/Keras (or in general), what is the best way to define a multidimensional y target? Should this even be done? The problem: Any sample x tries to predict several "values of interest". ...
1
vote
1answer
31 views

Rationale for the objective function in a flow network based generative model by Yoshua Bengio?

I am reading the Yoshua Bengio et al, Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. It seems to me the objective of the paper is to generate the flow $F$ given ...
1
vote
0answers
13 views

Online vs Offline Triplet Selection in FaceNet

I have been reading FaceNet. In the Triplet Selection section, it is written Generate triplets offline every n steps, using the most recent network checkpoint and computing the argmin and argmax on a ...
0
votes
0answers
8 views

What is the advantage of a mixture of experts (MoE) architecture for DNN?

Theoretically, a DNN with enough parameters can fit any training data. Thus, what is the advantage of using a mixture of experts (MoE) architecture for DNN? Is there any relevant paper about this? p.s....
0
votes
0answers
9 views

Is there any formal mathematical analysis or statistical analysis for transformer or attention mechanism? [closed]

Attention is prevailing these years in many fields, but hardly can I find any formal reference to prove its high performance.
2
votes
0answers
11 views

Why neural network convergence with large eigenvalues of $\nabla^2 f$?

This question is based on this answer about SGD, which says that gradient descent tends to converge to the eigenvector of parameters $\theta$ associated with eigenvalues of largest magnitude. ...
0
votes
0answers
7 views

Result differences in Generative Adversarial Networks (GAN) across epochs

I'm using MNIST data. I'm confused on how to evaluate results across epochs. For instance, below are 100th and 500th epoch outputs. Should I be worried that what looks like 7 (row 1 column 2) at epoch ...
1
vote
0answers
22 views

Multi-class multi-label with partial mutual exclusivity

Given an input, I want to predict 0/1 for each of N output classes. The output can be 1 for multiple classes. So I'm training with individual binary cross-entropies for each of the output classes. ...
1
vote
0answers
13 views

How many epochs should I iterate for tuning an ANN?

I have been using Keras-Tuner for tuning my ANN before going into training. The tuner seems to be iterating forever even though I set a limit of 1000 epochs. After that, I have decided to terminate ...
2
votes
1answer
20 views

Is there such a thing as an order-$k$ RNN?

In HMMs it's common to include edges from previous layers of the model. Looking back at the previous $k$ layers creates an order-$k$ Markov model. Is this commonly done in RNNs? Have you ever seen ...
1
vote
0answers
19 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
5
votes
2answers
334 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
1
vote
0answers
23 views

exp(log_softmax) vs softmax as neural network activation

I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
0
votes
1answer
38 views

Component sizes in vanilla RNN

I would like to seek some clarifications on the dimensionalities of the components and weight parameters in a vanilla RNN model performing text classification for the next word. I will present my ...
0
votes
0answers
14 views

What is a good deep learning approach for highly imbalanced data?

I am having a dataset with highly imbalanced classes (some classes have 4k examples while others have only one example). What is the best approach to handle such problem? Traditional approaches are ...
1
vote
0answers
32 views

Feed-Forward ReLU Networks as a matrix multiplication

When reading papers, Feed-Forward NN are often formalized as follows: $$\Phi(x) := \sigma(W_L\cdots \sigma(W_2\cdot \sigma(W_1x))\cdots) $$ i.e., the ReLU activation function $\sigma$ applied to the ...
1
vote
0answers
16 views

Bootstrapping allows retraining across the different bootstrapped datasets? [closed]

I am training a model and I would use bootstrapping since my dataset is really really small. Hence, I bootstrap a dataset, I train on it and then I get some validation error and metrics on the unseen ...
1
vote
1answer
63 views

understanding normalisations in StyleGan2 and on general

I was looking through labml's implementation of StyleGan2, and came across two normalisations: in the DiscriminatorBlock: "Scaling factor $\frac{1}{\sqrt 2}$ ...
3
votes
1answer
76 views

Recurrent neural networks of unspecified size

Are there recurrent neural networks (RNNs) of variable size? From what I've seen, RNNs are usually built using several nodes (or layers), in a manner similar to unrolled hidden Markov models (HMMs); ...
2
votes
1answer
62 views

Machine Learning with Aggregated Frequency Data as Training

I am trying to build a Deep Learning model in which I have the following structure user feature binary_label 1 100 0 2 200 1 3 140 0 ... ... ... 6000000 188 1 But the problem is that when I try ...
0
votes
0answers
9 views

Mixup VS CutMix Data Augumentation Method

I am looking for arguments on which Data augmentation (Mixup VS CutMix) method would be preferable for Image data and Time-series classification data. As for as I know, both have the following ...
1
vote
1answer
16 views

Dropout value - increase, decrease, keep the same across layers

I'm confused about dropout values that people set. Sometimes it's the same value, say 0.4. Sometimes they increase them gradually from 0.2 to 0.5. For example after ...
0
votes
0answers
10 views

Study Resources for optimizing(increasing/decreasing) a certain feature using AI

I am looking for resources that talk about how we can optimize(increase/decrease) a certain independent variable using other independent variables and the dependent variable. For example, we have 3 ...
2
votes
0answers
22 views

Is the VC dimension of a MLP regressor a valid upper bound on how many points it can exactly fit?

I want to calculate an upper bound on how many training points an MLP regressor can fit with ~0 error. I don't care about the test error, I want to overfit as much as possible the (few) training ...
1
vote
1answer
27 views

Single-layer perceptron mathematical formulation

I'm trying to btter understand the formalism under the following compact formulation of a single-layer perceptron. If we consider $V=\mathbb{R}^d$, then $$\hat{f}(x_1, \dots, x_d) = \sum_{i=1}^Nc_i\...
0
votes
0answers
20 views

Rolling average plus exponential weighted average - Crazy?

Hi I am trying to predict covid data with neural network models for a Uni project. The covid death data is not reported in Scotland so a 7 day rolling average is definitely needed. No smoothing: 7 ...
1
vote
1answer
20 views

Soft Target in Knowledge Distillation

I am currently reading the paper Distilling the Knowledge in a Neural Network and in the introduction I came across the following sentence - When the soft targets have high entropy, they provide much ...

1
2 3 4 5
180