# Questions tagged [activation-function]

The tag has no usage guidance.

29 questions
Filter by
Sorted by
Tagged with
32 views

### No activation function between two convolutional layers in MUNIT?

I'm reading the code of NVIDIA's MUNIT, the code of the resnet is as follows: ...
• 21
14 views

### Activation function for performing counting, modified version of leaky ReLU

Is there any known activation function that is like a leaky ReLU but also has its growth capped at the top once it reaches one through linear growth? I mean something like this. I need this for ...
1 vote
72 views

### Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?

After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
• 375
1 vote
226 views

### can we use binary cross entropy with labels -1 and 1?

Binary cross entropy is written as follows: $$\mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right)$$ In every reference that I read, when using binary ...
• 127
19 views

### Using threshold and bias at the same time in NN

I'm using NN with sigmoid binary activation. And for threshold I using 0,5. So if output < 0,5, it classified as 0. And if output >= 0,5 it classified as 1. But I'm using bias too at the same ...
• 21
65 views

### With ReLU activation, are we adding if conditions to a neural network's toolset?

I was reading Reverse Engineering a Neural Network's Clever Solution to Binary Addition. Without repeating the whole article, it seems the network figured out addition was already a part of its ...
59 views

### Putting a constraint on the output of the neural network

I am using a neural network to input some complex numbers and to obtain complex numbers. I converted the input complex numbers into real values by stacking the real part and imaginary parts as a ...
107 views

### Neural network activation functions for continuous outcomes

I understand we should always scale/normalize the variables in ANN models (correct me if I am wrong), but still, I was wondering if certain activation functions, like sigmoid, can be used when the ...
• 347
1k views

Specifically, I mean $$f(x)= \begin{cases} -\log(1-x) & x \le 0 \\ \space \space \space \log(1+x) & x \gt 0 \\ \end{cases}$$ Which is red in the plot: It behaves similarly to widely used \... 0 votes 0 answers 147 views ### Should I scale values before using them to train autoencoder? I have a dataset that contains vectors of shape 1xN where N is the number of features. For each value, there is a float between -4 and 5. For my project I need to make an autoencoder, however, ... • 11 1 vote 0 answers 60 views ### Difference between loss, activation functions lanscapes I have figures below taken from Diganta Misra's paper "Mish: A Self Regularized Non-Monotonic Activation Function." They visualized both the loss output landscapes in Fig 4 below: And they ... • 249 0 votes 0 answers 97 views ### Why unbounded above activation function is important for training One of the desirable properties of activation functions is to be unbounded above and bounded below. I guess part of the reasons why it should be unbounded above is to avoid vanishing gradient problems ... • 249 2 votes 1 answer 346 views ### Is bias nothing but perceptron threshold value? I was revisiting neural network basics from this post. The perceptron follows below equation: \begin{align} y & = 1 & \text{if } \sum_{i=1}^n w_i\times x_i \geq \theta \\ & = 0 & \... • 151 0 votes 0 answers 156 views ### Which activation function should be used for output layer in a regression task, provided that target variable is bound in range (0;1)? recently i've come across this paper DOI: 10.1371/journal.pone.0061318 In short, authors tried to predict an unbound positive continuous target variable using multi layer perceptron. First, they ... 0 votes 0 answers 57 views ### Derivation of (5.76) in "Pattern Recognition and Machine Learning" The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248 ... for softmax outputs we have:\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{... • 323 0 votes 0 answers 23 views ### Mean oft\$ in the logistic regression model

The question comes from a sentence of page 208 of Christopher M. Bishop's "Pattern Recognition and Machine Learning". The sentence is excerpted as follows: My questions are mainly two: Why ...
• 323
66 views

### Classifying time horizon of event

Im working on a classification class, where I want to predict the occurrence of a specific event. That is, does the event occur in the next 6 hours, 12 hours or 24 hours? Using softmax here seems not ...
29 views

### Can predicting negative regression values increase the chance of dying ReLUs?

Since ReLUs have zero gradient around negative values, we know that if a neuron outputs a negative value, the corresponding ReLU activation will cause it to die. Therefore, if I use a neural network ...
24 views

### How is ReLU used in neural networks if it doesn't squish the weighted sum into the interval of (0, 1)? [duplicate]

So as far as I understand from watching 3Blue1Brown's video on neural networks, all neurons operate on numbers ranging from 0 to 1. Since a weighted sum goes larger than that, a sigmoid function is ...
• 21
1 vote
450 views

### How are centers in an RBF Network chosen?

I am struggling to understand how RBF (radial basis functions) work. My first question concerns the weights: are the learnable weights the same as the centres? So, is the algorithm essentially ...
• 63
1 vote
106 views

### When I do not add an activation function to my convolutional layer the model gets quickly stuck in a local optima, why?

I have model A: ...
• 171
156 views

### How do I check if the weights of my perceptron/step activation function are correct

I am new to stack overflow and deep learning so I hope I am doing this the right way. I tried to find the solution myself but it has not been successful so I am seeking some help. This is the ...
• 33
1 vote
366 views

### Deep learning : Can I make an all positive network (features and weights)?

I am trying to train a deep neural network that will always use positive weights and the user will be obliged to enter a feature vector as input that will always be positive (normalized to 0-1). In ...
32 views

### If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?

I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
• 165
15 views

### Which NLP methods use gradient and activation methods?

I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
102 views

### What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
1 vote
16 views

### What activation function or pre-processing to use for features describing when a certain event occurred in the past?

I have a series of features that describe how long ago a certain event happened and whether it happened at all. Of course we could break down this features into two, whether it did happen or no, and ...
• 181