Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [neural-networks]

Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network ...

0
votes
1answer
15 views

When the loss value is considered a low value in deep learning? [on hold]

I would like to know when the loss value is considered a low value in a deep learning model?
0
votes
2answers
39 views

Deep Neural Network visualization on multi dimension datasets [on hold]

I have seen the PlayGround of Tensorflow. But it is using only 2D values which means only 2 inputs are taken. I have 7 inputs and the output is only 1. See the datasets sample: ...
0
votes
1answer
13 views

How to deal with unknown classes with a neural network classifier? [on hold]

I have a small RNN with a softmax output, which succesfully classifies sequences within a known set of n classes. Now I have the problem that there might be sequences which do not belong to any of ...
0
votes
0answers
13 views

How can a neural network be used to find distribution parameters?

Let $\{\vec{X}_i, Y_i\}_{i=1}^N$ be a data set of length $N$ where $\vec{X}_i$ is a vector of independent variables and $Y_i$ is a response variable. Assume that each $Y_i$ is drawn from a Beta ...
0
votes
1answer
9 views

ResNet 34 training with custom dataset

I am a beginner in Neural Networks and wanted to implement ResNet34 for a pet project at my workplace. Due to confidentiality issues, I do not want to use ImageNet trained weights. I have a dataset ...
1
vote
0answers
18 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
0
votes
0answers
7 views

Neural networks and Gaussian distribution of the activation

I have noticed that the distributions of the activation of neurons in all my Neural Networks were following normal distributions. It means that when we look at a single neuron for example, and we ...
0
votes
0answers
15 views

Correct tensor sizes for a pixel based classification of multidimensional satellite imagery with neural network

I've successfully worked through a few samples that take an entire image and apply one class to it. I want to classify each pixel in an image. I keep getting hung up when attempting to calculate ...
0
votes
0answers
15 views

Why use absolute value of the gradient for saliency maps?

Simonyan et al. 2014 approximate a neural network locally by a linear function and take the weights of this approximation as a measure for support of a specific class. Can someone explain to me why ...
1
vote
0answers
13 views

What is the difference between Generative Adversarial Networks (GAN) and Generative Antagonistic System (GAS)?

What is the difference between Generative Adversarial Networks (GAN) and Generative Antagonistic System (GAS) in the neural network?
0
votes
1answer
32 views

How to calculate the output from this neural network

bias w0=0.15 and w01=0.5. Assume the intercept of the combination function is 0. Basically, I am studying for my exam and I don't understand how to calculate this question about neural network: 1) ...
0
votes
0answers
25 views

What is the simplest neural network architecture that will perform fairly well on MNIST? [on hold]

I'd like to develop a class project where students implement a neural network from scratch in Python to perform handwritten digit classification using the MNIST dataset for training data. I'd like for ...
0
votes
0answers
19 views

Loss keeps increasing for a copy task involving word embedding

I have a simple model. Inputs are a matrix of integers (0 < x < vocabulary size). Outputs are a matrix of vectors representing the prediction of the input. As you can see from the loss function ...
0
votes
1answer
51 views

Approach to prevent bias/racism in neural network fitting?

I have a dataset comprised of different ethnic groups and I want to build a classification model on this data. When I do this I find that the performance of the algorithm is better on some groups than ...
0
votes
1answer
17 views

Is `sigmoid` required for binary cross entropy?

I have a DNN that has to predict whether an input belongs to a class or not. During training, I use binary cross entropy as a loss function. I noticed that if my ...
1
vote
0answers
24 views

Recommended books about neural networks [closed]

what books do you recommend on neural networks? We can think of a few categories: 1. A python recipe book 2. A general introduction book 3. An Updated book 4. Specialized book on CNNs 5. Specialized ...
0
votes
1answer
27 views

How Probability distribution relates to neural networks?

The concept of random variables and probability distributions are confusing in the context of neural networks. In a neural network, which is the random variable and what is the probability ...
2
votes
0answers
16 views

LSTM : multi-step multidimensional multivariate multi-site timeseries forecasting [closed]

I'm working on a project in which i'm trying to do a pollution forecasting. I googled around and found that LSTM is a good candidate for this task, however, I'm still struggling at how to adapt it to ...
0
votes
0answers
14 views

Performing backpropagation on a 2 layer neural network

I am attempting to construct an NN from scratch without vectorizing. This is the network I'm attemping to model, where h1 and h2 represent the hidden layer nodes (h1 being the top, h2 being the ...
0
votes
0answers
13 views

Why is number of neurons in hidden layers a power of two?

There is a statement in this quora answer: Layer depth is usually a power of 2 because it is convenient for the GPU. Also, in fully connected layers number of ...
0
votes
0answers
14 views

not performing model selection and calculating error estimates simultaneously

I'm trying to develop a prediction model using machine learning algorithms for some specific procedure in my field. So i'm relatively new to this field. I have 125 examples and let's say i want to ...
0
votes
0answers
26 views

Relu6 and vanishing gradients problem

In some recent machine learning papers (e.g. mobileNetV2), ReLU6, defined as $Relu(x)=\min(\max(0,x),6)$ is used instead of regular Relu non-linearities. Doesn't such a function result in the same ...
1
vote
0answers
16 views

Transforming an echo state network to online training

I have recently become interested in echo state networks, I've been using the simple introduction to ESNs here https://mantas.info/wp/wp-content/uploads/simple_esn/minimalESN.py by Dr. Mantas ...
0
votes
0answers
18 views

Steps of mini-batch stochastic gradient descent in an episode [duplicate]

I am using the stochastic gradient descent (SGD) to do the optimization for the deep neural networks (DNN). I know that in one epoch I need to do multiple iterations of mini-batch SGD to make it ...
0
votes
0answers
12 views

Input shape for LSTM to predict customer usage quantity for a given timestep

As per my understanding the Input shape for RNN must be (Number of samples, number of timesteps, number of features). My data has 12 timesteps for each customer and number of features is 15, my ...
1
vote
0answers
24 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
0
votes
0answers
12 views

Multilabel classification problem where a number of output labels are in certain range

I am tackling a multi-label classification task. In this problem, we have 13 classes For one sample $x \in R^{20}$, $2 \leq |h(x)| \leq4$ Here, $h(x)$ means the prediction labels of sample x by ...
0
votes
0answers
21 views

Time series forecasting with decision making

I want to do forecasting for wind power generation but the problem with it is that when the wind speed is below 4 m/s the power output is zero. RNN based models do best when these type of conditions ...
0
votes
0answers
27 views

State prediction of how long someone sleeps using neural nets

I have over hundred thousands of datapoints on how long individual people sleep. I also have information about how soft their beds are, their income, stress levels etc. At first I want to predicted ...
0
votes
1answer
13 views

How is it called when a MLR algorithm predicts a value beyond the range of the training data set and is there a way to avoid this for Neural Networks?

I use two Machine Learning Algorithms to learn how my target variable [0, 8] is affected by four features, each within a scale of [1, 10]. I am using scikit-learn to do this task for me. ...
1
vote
0answers
29 views

Optimize linear regression with data transformation

I am trying to train a linear regression model to predict next value of a signal. The network is very simple: it consists of two LSTM cells and a dense layer composed of a single neuron. The optimizer ...
0
votes
1answer
17 views

Universal approximation [duplicate]

From my understanding, neural networks are universal approximators, which was proven by Cybenko in 1989 for sigmoid activation functions with output units being linear. Castro showed, that this ...
1
vote
1answer
31 views

Feature Scaling in Reinforcement Learning

I am working with RL algorithms like DQN and ActorCritic and I'm curious whether there is a way to correctly scale features which represent state or state/action pair while learning parameters of ...
1
vote
1answer
47 views

What is an ablation study? And is there a systematic way to perform it?

What is an ablation study? And is there a systematic way to perform it? For example, I have $n$ predictors in a linear regression which I will call as my model. How will I perform an ablation study ...
1
vote
0answers
26 views

Universal approximation theorem for whole R^d

The well-known universal approximation theorem states that neural network with one hidden layer can approximate any continuous function on every compact subset of $\mathbb{R}^d$. My question is ...
0
votes
0answers
16 views

Question about PyTorch tutorial

In this PyTorch tutorial the backprop to compute gradients is shown with the following code: ...
8
votes
1answer
237 views

Why can't a single ReLU learn a ReLU?

As a follow-up to My neural network can't even learn Euclidean distance I simplified even more and tried to train a single ReLU (with random weight) to a single ReLU. This is the simplest network ...
0
votes
0answers
17 views

Linear regression performs better than single-hidden-layer NN?

I remember hearing from a professor that there exists a type of dataset for which Linear Regression outperforms single-hidden-layer neural nets. However, I cannot remember what type of dataset he ...
0
votes
1answer
18 views

VAE sampling during test time [duplicate]

On page 11 of this VAE tutorial it is said that new samples of the data distribution X can be found by plugging z ~ N(0, I) into the Decoder P. I don't understand why this is true. During training, ...
0
votes
1answer
20 views

Fitting a NN model on one-to-many function

Given $f(x) = y$ as a surjective (many-to-one) function, we know that $f^{-1} (y) = x$ is a one-to-many mapping for function $f^{-1}$. In my application, $x$ is a spatial data represented by a 2D ...
0
votes
0answers
42 views

Optimization options to minimize mean absolute error when model is a Neural network

Lately I've seen some advantages mostly in model generalization of minimizing an the mean absolute error (or I guess Laplacian MLE would be an equivalent way of saying it). I'm debating first on what ...
0
votes
0answers
7 views

Why does the Conv Neural Net using Tensorflow returns same predictions for all the data points [duplicate]

I am predicting usage quantity for different customers across different categories. Following is the network architecture. ...
3
votes
1answer
41 views

How is an RNN (or any neural network) a parametric model?

I'm going through this paper A Multi-Horizon Quantile Recurrent Forecaster. The authors state that: 3.1. Loss Function In Quantile Regression, models are trained to minimize the total ...
0
votes
0answers
24 views

How is the a function such as ReLU sufficient enough for a neural network to approximate other nonlinear functions? [duplicate]

I'm sort of confused as to how the nonlinearity of an activation function like the ReLu means that relatively complex mappings can all be approximated by a neural network? I guess I am sort of ...
0
votes
0answers
25 views

Understanding and alternative derivation of the back propagation rule for ANN

I'm quite new to machine learning, and as far as I understand the back-propagation rule is an algorithm the allow to compute the gradient of the cost function defined when training a ANN. First ...
0
votes
0answers
8 views

Do we need pretrained blocks for domain adversarial training

I have a question related to the following classic paper on domain-adversarial training: Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M. and Lempitsky, ...
1
vote
1answer
29 views

sklearn.train_test_split truncates my y_train [closed]

I have a standardized dataset which has floats with this (for example, e+01) tail at the end. I know this is a multiplicator to save such a small number without ...
0
votes
0answers
24 views

Is bootstrapping a viable method for augmenting time series data?

I have recently learnt about the bootstrapping method and I am using it in my model tuning phase of my current project. I am working with time series data and therefore have decided to use a ...
0
votes
0answers
9 views

Most suitable comparison between groups

I have a neural network based algorithm which it was trained with 5 different datasets (A, B, C, D and E). For each one, 10 training runs was made {(A1, A2, ..., A10), (B1, ..., B10), ...}. Then, it ...
0
votes
0answers
76 views

Gradient of the cross entropy loss function

I have been puzzled by how to calculate the derivative of the following cross entropy loss function underlying my neural network: CEloss = $\frac{-1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} t_{n,k} \log y_{...