Timeline for Batch Normalization or just z-normalization as a Nonlinearity
Current License: CC BY-SA 4.0
16 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Sep 21, 2020 at 19:01 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
May 23, 2020 at 18:04 | comment | added | Aksakal | Wht's the question? you made statements, and but didnt pose a clear question | |
May 23, 2020 at 18:00 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
Jan 19, 2020 at 9:02 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
Sep 17, 2019 at 19:56 | history | reopened | Sycorax♦ neural-networks Users with the neural-networks badge or a synonym can single-handedly close neural-networks questions as duplicates and reopen them as needed. | ||
Sep 17, 2019 at 19:53 | history | edited | RMurphy | CC BY-SA 4.0 |
By request, clarified question. Previously it sounded like a duplicate.
|
Sep 17, 2019 at 19:45 | review | Reopen votes | |||
Sep 17, 2019 at 20:00 | |||||
Sep 17, 2019 at 19:42 | comment | added | RMurphy | @Sycorax, thank you for unduplicating it. I will edit it. | |
Sep 17, 2019 at 19:33 | comment | added | Sycorax♦ | My advice is to use the edit button rewrite your question to clearly articulate what you know, what you would like to know, and where you are stuck. Right now, I can't make heads or tails of what you're trying to ask and what you would like to know. | |
Sep 17, 2019 at 19:28 | history | edited | RMurphy | CC BY-SA 4.0 |
added 94 characters in body
|
Sep 17, 2019 at 19:24 | comment | added | RMurphy | @Sycorax, please, it is not a duplicate question. I understand perfectly well why nonlinear functions are not used, in terms of the answer you have given me above. Also, I do not agree that computing a standard deviation is linear in its arguments. The x you have written will have weights from the previous layer, and we will evaluate a square root of those weights. | |
Sep 17, 2019 at 14:42 | comment | added | Sycorax♦ | A $z$ score is just a linear transformation of the inputs; if this is unclear, note that you can re-write $\frac{x - \mu}{\sigma}$ as $\frac{1}{\sigma}x - \frac{\mu}{\sigma}$. The duplicate question addresses why neural networks use nonlinear activation functions instead of linear functions: linear functions are closed under composition, so a network of linear functions is simply a linear model. | |
Sep 17, 2019 at 14:40 | history | closed | Sycorax♦ neural-networks Users with the neural-networks badge or a synonym can single-handedly close neural-networks questions as duplicates and reopen them as needed. | Duplicate of ReLu vs a linear activation function | |
Sep 17, 2019 at 13:59 | comment | added | RMurphy | @Sycorax. In some sense, that's the point of my question. I understand the original motivation behind batch norm, but couldn't it also double as a nonlinearity? I mean, why do we need to compose something like a sigmoid, a relu, an elu etc. etc. with a z-standardization or its glorified cousin batch norm? | |
Sep 16, 2019 at 20:04 | comment | added | Sycorax♦ | I'm not sure what information you're seeking. My best guess is that your question is premised on a misunderstanding of how batch norm works. The point of batch norm is to use running mean and running standard deviation estimates; these estimates are used to compensate for the shifting means and standard deviations of inputs to the norm layer which occur because the network is training. Does this answer your question, or do you need clarification about a different component of batch norm? (What component do you wish to understand in more detail?) | |
Sep 16, 2019 at 19:58 | history | asked | RMurphy | CC BY-SA 4.0 |