Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

Filter by
Sorted by
Tagged with
215
votes
8answers
144k views

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
45
votes
4answers
17k views

What should I do when my neural network doesn't generalize well?

I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar ...
639
votes
11answers
731k views

How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural ...
279
votes
5answers
235k views

What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ ab = cd $? To ...
141
votes
11answers
100k views

What is the difference between a neural network and a deep neural network, and why do the deep ones work better?

I haven't seen the question stated precisely in these terms, and this is why I make a new question. What I am interested in knowing is not the definition of a neural network, but understanding the ...
106
votes
6answers
39k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
4
votes
1answer
913 views

How can change in cost function be positive?

In chapter 1 of Nielsen's Neural Networks and Deep Learning it says To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good ...
6
votes
1answer
897 views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
107
votes
5answers
38k views

Comprehensive list of activation functions in neural networks with pros/cons

Are there any reference document(s) that give a comprehensive list of activation functions in neural networks along with their pros/cons (and ideally some pointers to publications where they were ...
74
votes
6answers
99k views

What are good initial weights in a neural network?

I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. ...
71
votes
9answers
104k views

How and why do normalization and feature scaling work?

I see that lots of machine learning algorithms work better with mean cancellation and covariance equalization. For example, Neural Networks tend to converge faster, and K-Means generally gives better ...
143
votes
7answers
105k views

What does 1x1 convolution mean in a neural network?

I am currently doing the Udacity Deep Learning Tutorial. In Lesson 3, they talk about a 1x1 convolution. This 1x1 convolution is used in Google Inception Module. I'm having trouble understanding what ...
96
votes
4answers
49k views

Why are neural networks becoming deeper, but not wider?

In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (...
50
votes
3answers
76k views

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only ...
43
votes
5answers
81k views

Why do we use ReLU in neural networks and how do we use it?

Why do we use rectified linear units (ReLU) with neural networks? How does that improve neural network? Why do we say that ReLU is an activation function? Isn't softmax activation function for neural ...
48
votes
8answers
97k views

Data normalization and standardization in neural networks

I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. ...
25
votes
6answers
6k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
167
votes
7answers
164k views

What are the advantages of ReLU over sigmoid function in deep neural networks?

The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages? I know that training a network when ReLU is ...
84
votes
10answers
128k views

Validation Error less than training error?

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
69
votes
3answers
46k views

Proper way of using recurrent neural network for time series analysis

Recurrent neural networks differ from "regular" ones by the fact that they have a "memory" layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not ...
42
votes
5answers
33k views

Cost function of neural network is non-convex?

The cost function of neural network is $J(W,b)$, and it is claimed to be non-convex. I don't quite understand why it's that way, since as I see that it's quite similar to the cost function of logistic ...
73
votes
3answers
13k views

Why do neural network researchers care about epochs?

An epoch in stochastic gradient descent is defined as a single pass through the data. For each SGD minibatch, $k$ samples are drawn, the gradient computed and parameters are updated. In the epoch ...
27
votes
3answers
4k views

Can't deep learning models now be said to be interpretable? Are nodes features?

For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular ...
12
votes
2answers
7k views

How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs? [duplicate]

When used as an activation function in deep neural networks The ReLU function outperforms other non-linear functions like tanh or sigmoid . In my understanding the whole purpose of an activation ...
2
votes
1answer
808 views

Is epoch optimization in CV with constant mini-batch size even possible?

Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of ...
90
votes
1answer
87k views

How to apply Neural Network to time series forecasting?

I'm new to machine learning, and I have been trying to figure out how to apply neural network to time series forecasting. I have found resource related to my query, but I seem to still be a bit lost. ...
95
votes
2answers
112k views

tanh activation function vs sigmoid activation function

The tanh activation function is: $$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Where $\sigma(x)$, the sigmoid function, is defined as: $$\sigma(x) = \frac{e^x}{1 + e^x}$$. ...
28
votes
3answers
9k views

Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
21
votes
5answers
11k views

Reason for not shrinking the bias (intercept) term in regression

For a linear model $y=\beta_0+x\beta+\varepsilon$, the shrinkage term is always $P(\beta) $. What is the reason that we do not shrink the bias (intercept) term $\beta_0$? Should we shrink the bias ...
24
votes
2answers
8k views

Can we use MLE to estimate Neural Network weights?

I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural ...
21
votes
4answers
18k views

How to explain dropout regularization in simple terms?

If you have a half page to explain dropout, how would you proceed? Which is the rationale behind this technique?
15
votes
2answers
9k views

What is the difference between 'regular' linear regression and deep learning linear regression?

I want to know the difference between linear regression in a regular machine learning analysis and linear regression in "deep learning" setting. What algorithms are used for linear regression in deep ...
16
votes
4answers
675 views

What *is* an Artificial Neural Network?

As we delve into Neural Networks literature, we get to identify other methods with neuromorphic topologies ("Neural-Network"-like architectures). And I'm not talking about the Universal Approximation ...
3
votes
1answer
441 views

Neural Networks input data normalization and centering

I'm learning Neural Networks and I grasped the algebra behind them. I'm now interested in understanding how normalization and centering of the input data affect them. In my personal learning project (...
159
votes
2answers
171k views

A list of cost functions used in neural networks, alongside applications

What are common cost functions used in evaluating the performance of neural networks? Details (feel free to skip the rest of this question, my intent here is simply to provide clarification on ...
224
votes
5answers
367k views

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
43
votes
6answers
7k views

Neural network references (textbooks, online courses) for beginners

I want to learn Neural Networks. I am a Computational Linguist. I know statistical machine learning approaches and can code in Python. I am looking to start with its concepts, and know one or two ...
56
votes
4answers
59k views

Why is logistic regression a linear classifier?

Since we are using the logistic function to transform a linear combination of the input into a non-linear output, how can logistic regression be considered a linear classifier? Linear regression is ...
68
votes
4answers
33k views

What're the differences between PCA and autoencoder?

Both PCA and autoencoder can do demension reduction, so what are the difference between them? In what situation I should use one over another?
34
votes
6answers
54k views

What is the difference between logistic regression and neural networks?

How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?
43
votes
4answers
25k views

How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to ...
38
votes
2answers
39k views

Neural Network: For Binary Classification use 1 or 2 output neurons?

Assume I want to do binary classification (something belongs to class A or class B). There are some possibilities to do this in the output layer of a neural network: Use 1 output node. Output 0 (<...
16
votes
2answers
8k views

Getting started with neural networks for forecasting

I need some resources to get started on using neural networks for time series forecasting. I am wary of implementing some paper and then finding out that they have greatly over stated the potential of ...
10
votes
1answer
4k views

Input vector representation vs output vector representation in word2vec

In word2vec's CBOW and skip-gram models, how does choosing word vectors from $W$ (input word matrix) vs. choosing word vectors from $W'$ (output word matrix) impact the quality of the resulting word ...
10
votes
2answers
1k views

How to construct a cross-entropy loss for general regression targets?

It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as simply "cross-entropy." However, this terminology is ambiguous because different probability ...
2
votes
1answer
108 views

Separate Models vs Flags in the same model

I have customer data from 2 brands. The data structure are the same, but I expected the customer behaviour to be different in different brand. So I could train 2 models, 1 for each brand, or I could ...
3
votes
1answer
177 views

Effect of rescaling of inputs on loss for a simple neural network

I've been trying out a simple neural network on the fashion_mnist dataset using keras. Regarding normalization, I've watched this video explaining why it's necessary to normalize input features, but ...
46
votes
5answers
35k views

How does rectilinear activation function solve the vanishing gradient problem in neural networks?

I found rectified linear unit (ReLU) praised at several places as a solution to the vanishing gradient problem for neural networks. That is, one uses max(0,x) as activation function. When the ...
33
votes
6answers
20k views

How to get started with neural networks

I'm completely new to neural networks but highly interested in understanding them. However it's not easy at all to get started. Could anyone recommend a good book or any other kind of resource? Is ...
48
votes
1answer
31k views

How does the Adam method of stochastic gradient descent work?

I'm familiar with basic gradient descent algorithms for training neural networks. I've read the paper proposing Adam: ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. While I've definitely got some ...

1
2 3 4 5
14