Questions tagged [activation-function]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
32 views

No activation function between two convolutional layers in MUNIT?

I'm reading the code of NVIDIA's MUNIT, the code of the resnet is as follows: ...
James's user avatar
  • 21
0 votes
0 answers
14 views

Activation function for performing counting, modified version of leaky ReLU

Is there any known activation function that is like a leaky ReLU but also has its growth capped at the top once it reaches one through linear growth? I mean something like this. I need this for ...
Jose Manuel de Frutos's user avatar
1 vote
0 answers
72 views

Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?

After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
Museful's user avatar
  • 375
1 vote
1 answer
226 views

can we use binary cross entropy with labels -1 and 1?

Binary cross entropy is written as follows: \begin{equation} \mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right) \end{equation} In every reference that I read, when using binary ...
andryan86's user avatar
  • 127
2 votes
1 answer
19 views

Using threshold and bias at the same time in NN

I'm using NN with sigmoid binary activation. And for threshold I using 0,5. So if output < 0,5, it classified as 0. And if output >= 0,5 it classified as 1. But I'm using bias too at the same ...
Arias231's user avatar
2 votes
0 answers
65 views

With ReLU activation, are we adding if conditions to a neural network's toolset?

I was reading Reverse Engineering a Neural Network's Clever Solution to Binary Addition. Without repeating the whole article, it seems the network figured out addition was already a part of its ...
igloo_activation's user avatar
3 votes
0 answers
59 views

Putting a constraint on the output of the neural network

I am using a neural network to input some complex numbers and to obtain complex numbers. I converted the input complex numbers into real values by stacking the real part and imaginary parts as a ...
Prashanth Krishnananthalingam's user avatar
0 votes
0 answers
107 views

Neural network activation functions for continuous outcomes

I understand we should always scale/normalize the variables in ANN models (correct me if I am wrong), but still, I was wondering if certain activation functions, like sigmoid, can be used when the ...
TWest's user avatar
  • 347
17 votes
1 answer
1k views

Why isn't (symmetric) log(1+x) used as neural network activation function?

Specifically, I mean $$ f(x)= \begin{cases} -\log(1-x) & x \le 0 \\ \space \space \space \log(1+x) & x \gt 0 \\ \end{cases} $$ Which is red in the plot: It behaves similarly to widely used $\...
yuri kilochek's user avatar
0 votes
0 answers
147 views

Should I scale values before using them to train autoencoder?

I have a dataset that contains vectors of shape 1xN where N is the number of features. For each value, there is a float between -4 and 5. For my project I need to make an autoencoder, however, ...
sage's user avatar
  • 11
1 vote
0 answers
60 views

Difference between loss, activation functions lanscapes

I have figures below taken from Diganta Misra's paper "Mish: A Self Regularized Non-Monotonic Activation Function." They visualized both the loss output landscapes in Fig 4 below: And they ...
Avv's user avatar
  • 249
0 votes
0 answers
97 views

Why unbounded above activation function is important for training

One of the desirable properties of activation functions is to be unbounded above and bounded below. I guess part of the reasons why it should be unbounded above is to avoid vanishing gradient problems ...
Avv's user avatar
  • 249
2 votes
1 answer
346 views

Is bias nothing but perceptron threshold value?

I was revisiting neural network basics from this post. The perceptron follows below equation: $$\begin{align} y & = 1 & \text{if } \sum_{i=1}^n w_i\times x_i \geq \theta \\ & = 0 & \...
RajS's user avatar
  • 151
0 votes
0 answers
156 views

Which activation function should be used for output layer in a regression task, provided that target variable is bound in range (0;1)?

recently i've come across this paper DOI: 10.1371/journal.pone.0061318 In short, authors tried to predict an unbound positive continuous target variable using multi layer perceptron. First, they ...
user10046685's user avatar
0 votes
0 answers
57 views

Derivation of (5.76) in "Pattern Recognition and Machine Learning"

The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248 ... for softmax outputs we have: $$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
zzzhhh's user avatar
  • 323
0 votes
0 answers
23 views

Mean of $t$ in the logistic regression model

The question comes from a sentence of page 208 of Christopher M. Bishop's "Pattern Recognition and Machine Learning". The sentence is excerpted as follows: My questions are mainly two: Why ...
zzzhhh's user avatar
  • 323
0 votes
1 answer
66 views

Classifying time horizon of event

Im working on a classification class, where I want to predict the occurrence of a specific event. That is, does the event occur in the next 6 hours, 12 hours or 24 hours? Using softmax here seems not ...
f_3464gh's user avatar
2 votes
0 answers
29 views

Can predicting negative regression values increase the chance of dying ReLUs?

Since ReLUs have zero gradient around negative values, we know that if a neuron outputs a negative value, the corresponding ReLU activation will cause it to die. Therefore, if I use a neural network ...
desert_ranger's user avatar
2 votes
0 answers
24 views

How is ReLU used in neural networks if it doesn't squish the weighted sum into the interval of (0, 1)? [duplicate]

So as far as I understand from watching 3Blue1Brown's video on neural networks, all neurons operate on numbers ranging from 0 to 1. Since a weighted sum goes larger than that, a sigmoid function is ...
Yeepsta's user avatar
  • 21
1 vote
1 answer
450 views

How are centers in an RBF Network chosen?

I am struggling to understand how RBF (radial basis functions) work. My first question concerns the weights: are the learnable weights the same as the centres? So, is the algorithm essentially ...
DLTS's user avatar
  • 63
1 vote
0 answers
106 views

When I do not add an activation function to my convolutional layer the model gets quickly stuck in a local optima, why?

I have model A: ...
RogerKint's user avatar
  • 171
3 votes
1 answer
156 views

How do I check if the weights of my perceptron/step activation function are correct

I am new to stack overflow and deep learning so I hope I am doing this the right way. I tried to find the solution myself but it has not been successful so I am seeking some help. This is the ...
Bubo's user avatar
  • 33
1 vote
0 answers
366 views

Deep learning : Can I make an all positive network (features and weights)?

I am trying to train a deep neural network that will always use positive weights and the user will be obliged to enter a feature vector as input that will always be positive (normalized to 0-1). In ...
Dimitris Pappas's user avatar
0 votes
1 answer
32 views

If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?

I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
levitatmas's user avatar
0 votes
0 answers
15 views

Which NLP methods use gradient and activation methods?

I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
Revolucion for Monica's user avatar
2 votes
0 answers
102 views

What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
burn_burn_55's user avatar
1 vote
0 answers
16 views

What activation function or pre-processing to use for features describing when a certain event occurred in the past?

I have a series of features that describe how long ago a certain event happened and whether it happened at all. Of course we could break down this features into two, whether it did happen or no, and ...
Anonymous's user avatar
  • 181
10 votes
5 answers
9k views

Can a neural network work with negative and zero inputs?

As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation ...
spectre's user avatar
  • 350
1 vote
1 answer
3k views

what is the best activation function for binary classification?

i'm beginner in cnn and i want to detect which one is genuine image and which one is spoof image. i got really confused to choose my activation function. for binary classifiers, should i choose ...
Farhan Rabbaanii's user avatar