Questions tagged [activation-function]
The activation-function tag has no usage guidance.
36 questions
1
vote
0
answers
24
views
Maxout activation function vs ReLU (Number of weights)
From what I understood, Maxout function works quite differently from ReLU.
ReLU function is max(0, x), so the input x is (W_T x + b)
Maxout function has many Ws, and it is max(W1_T x + b1, W2_T x + b2,...
- 11
0
votes
0
answers
31
views
Using a different activation functions within a layer?
As an experiment, one could try to have n different activations/neurons/units in a layer.
One to adapt the automated backpropagation algorithms from deep learning ...
user306937
0
votes
0
answers
48
views
How do B-splines differ from Fourier transforms?
I read that B-splines can be used as activation functions in KAN neural networks, whereas Fourier transforms are not widely used. Can someone please explain the difference between the two in a simple ...
0
votes
0
answers
82
views
Is it bad not to standardize all features (regression)?
I'm working with a neural network with two hidden layers for a regression task. My output values for the training set vary from 0 to 2000 and for the test set from 0 to 600. My main problem is ...
- 1
0
votes
0
answers
23
views
normalized dual activation function for neural tangent kernel
Let $\phi$ be an activation function. In this lecture note, The author assumes that the dual activation function, denoted as $\check{\phi}$ is normalized such that $\check{\phi}(1)=1$. How can it be ...
3
votes
1
answer
73
views
Smoothness of a neural network (specifically second-order)
If we use ReLU activations, then the function which our neural network represents is piecewise linear. It is not smooth and the first derivative doesn't exist everywhere.
However, if we use sigmoid or ...
2
votes
1
answer
45
views
No activation function between two convolutional layers in MUNIT?
I'm reading the code of NVIDIA's MUNIT, the code of the resnet is as follows:
...
- 21
1
vote
0
answers
168
views
Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?
After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
- 395
1
vote
1
answer
414
views
can we use binary cross entropy with labels -1 and 1?
Binary cross entropy is written as follows:
\begin{equation}
\mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right)
\end{equation}
In every reference that I read, when using binary ...
- 137
2
votes
1
answer
25
views
Using threshold and bias at the same time in NN
I'm using NN with sigmoid binary activation. And for threshold I using 0,5. So if output < 0,5, it classified as 0. And if output >= 0,5 it classified as 1. But I'm using bias too at the same ...
- 21
2
votes
0
answers
107
views
With ReLU activation, are we adding if conditions to a neural network's toolset?
I was reading Reverse Engineering a Neural Network's Clever Solution to Binary Addition. Without repeating the whole article, it seems the network figured out addition was already a part of its ...
3
votes
0
answers
83
views
Putting a constraint on the output of the neural network
I am using a neural network to input some complex numbers and to obtain complex numbers. I converted the input complex numbers into real values by stacking the real part and imaginary parts as a ...
0
votes
0
answers
124
views
Neural network activation functions for continuous outcomes
I understand we should always scale/normalize the variables in ANN models (correct me if I am wrong), but still, I was wondering if certain activation functions, like sigmoid, can be used when the ...
- 347
18
votes
1
answer
2k
views
Why isn't (symmetric) log(1+x) used as neural network activation function?
Specifically, I mean
$$
f(x)=
\begin{cases}
-\log(1-x) & x \le 0 \\
\space \space \space \log(1+x) & x \gt 0 \\
\end{cases}
$$
Which is red in the plot:
It behaves similarly to widely used $\...
- 407
0
votes
0
answers
236
views
Should I scale values before using them to train autoencoder?
I have a dataset that contains vectors of shape 1xN where N is the number of features. For each value, there is a float between -4 and 5. For my project I need to make an autoencoder, however, ...
- 11
1
vote
0
answers
69
views
Difference between loss, activation functions lanscapes
I have figures below taken from Diganta Misra's paper
"Mish: A Self Regularized Non-Monotonic Activation Function." They visualized both the loss output landscapes in Fig 4 below:
And they ...
- 249
0
votes
0
answers
122
views
Why unbounded above activation function is important for training
One of the desirable properties of activation functions is to be unbounded above and bounded below. I guess part of the reasons why it should be unbounded above is to avoid vanishing gradient problems ...
- 249
2
votes
1
answer
511
views
Is bias nothing but perceptron threshold value?
I was revisiting neural network basics from this post. The perceptron follows below equation:
$$\begin{align}
y & = 1 & \text{if } \sum_{i=1}^n w_i\times x_i \geq \theta \\
& = 0 & \...
- 151
0
votes
0
answers
177
views
Which activation function should be used for output layer in a regression task, provided that target variable is bound in range (0;1)?
recently i've come across this paper DOI: 10.1371/journal.pone.0061318
In short, authors tried to predict an unbound positive continuous target variable using multi layer perceptron. First, they ...
0
votes
0
answers
66
views
Derivation of (5.76) in "Pattern Recognition and Machine Learning"
The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248
... for softmax outputs we have:
$$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
- 333
0
votes
0
answers
23
views
Mean of $t$ in the logistic regression model
The question comes from a sentence of page 208 of Christopher M. Bishop's "Pattern Recognition and Machine Learning". The sentence is excerpted as follows:
My questions are mainly two:
Why ...
- 333
0
votes
1
answer
68
views
Classifying time horizon of event
Im working on a classification class, where I want to predict the occurrence of a specific event. That is, does the event occur in the next 6 hours, 12 hours or 24 hours? Using softmax here seems not ...
- 1
2
votes
0
answers
31
views
Can predicting negative regression values increase the chance of dying ReLUs?
Since ReLUs have zero gradient around negative values, we know that if a neuron outputs a negative value, the corresponding ReLU activation will cause it to die. Therefore, if I use a neural network ...
- 644
2
votes
0
answers
24
views
How is ReLU used in neural networks if it doesn't squish the weighted sum into the interval of (0, 1)? [duplicate]
So as far as I understand from watching 3Blue1Brown's video on neural networks, all neurons operate on numbers ranging from 0 to 1. Since a weighted sum goes larger than that, a sigmoid function is ...
- 21
1
vote
1
answer
649
views
How are centers in an RBF Network chosen?
I am struggling to understand how RBF (radial basis functions) work. My first question concerns the weights: are the learnable weights the same as the centres? So, is the algorithm essentially ...
- 63
1
vote
0
answers
132
views
4
votes
1
answer
172
views
How do I check if the weights of my perceptron/step activation function are correct
I am new to stack overflow and deep learning so I hope I am doing this the right way. I tried to find the solution myself but it has not been successful so I am seeking some help. This is the ...
- 43
1
vote
0
answers
503
views
Deep learning : Can I make an all positive network (features and weights)?
I am trying to train a deep neural network that will always use positive weights and the user will be obliged to enter a feature vector as input that will always be positive (normalized to 0-1).
In ...
- 231
1
vote
1
answer
39
views
If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?
I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
- 325
0
votes
0
answers
15
views
Which NLP methods use gradient and activation methods?
I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
2
votes
0
answers
103
views
What are the advantages and disadvantages of higher order neuron activation functions?
I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as
...
- 21
3
votes
3
answers
2k
views
What is a sigmoid function and what does it give as output?
I know the equation of the sigmoid function and use it in logistic regression, SVM, etc.
$$
S(x) = \frac{1}{1 + e^{-x}}
$$
In the case of the sigmoid function, What is the exact input and output of ...
- 687
1
vote
0
answers
16
views
What activation function or pre-processing to use for features describing when a certain event occurred in the past?
I have a series of features that describe how long ago a certain event happened and whether it happened at all.
Of course we could break down this features into two, whether it did happen or no, and ...
- 181
10
votes
5
answers
11k
views
Can a neural network work with negative and zero inputs?
As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation ...
- 350
1
vote
1
answer
3k
views
what is the best activation function for binary classification?
i'm beginner in cnn and i want to detect which one is genuine image and which one is spoof image. i got really confused to choose my activation function. for binary classifiers, should i choose ...
- 113
22
votes
5
answers
21k
views
Why is step function not used in activation functions in machine learning?
The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What ...
- 565