3

First of all, not only neural networks are universal approximators. There is nothing special about them, they just proved to work quite well for a class of problems. Kernel based methods generally don't scale well. Neural networks gained popularity in the time when we (a) improved our computers, so we were able to run bigger neural networks and do this in ...


3

Your question is more theoretically founded, but goes in the same direction as this one. You might want to check the answers there (disclaimer: one of them is mine). In order to answer your question from the last paragraph, I believe it is useful to identify conceptual basic blocks in machine learning: feature selection feature transformation parameter ...


2

Take as an example the simple neural network diagram from Wikipedia. Each arrow on the diagram shows the weight of the model, biases are not shown. With the simple model as linear regression to judge its complexity, you would just count the parameters. Here notice that the parameters on the second layer depend on the parameters of the first layer. How ...


2

Different combinations can give the same output value! Consider a simple neural network with one feature, a hidden layer with two neurons (ReLU activation), and an output neuron with an identity activation. $$ \widehat{y_i} = \widehat{b_{1,2}} + \widehat{w_{1,2}}ReLU\bigg( \widehat{b_{1,1}}+\widehat{w_{1,1}}x \bigg) + \widehat{w_{2,2}}ReLU\bigg( \widehat{b_{...


1

Typically k-fold is not used in neural networks because it's more expensive. When dataset is large enough, one validation set is more economic, and you end up with a single final model. The test set (i.e. 10K samples) is not to be touched until your model is ready. So, you'll use your 60K samples for train+validation. A 15-20 % of the 60K samples can be ...


1

If the voting ensemble does not work out with the two models, you can either enlarge the ensemble (e.g. see Risch/Krestel reporting "drastically increases when ensembling up to 15 BERT models") stack the two models by training a meta-model (e.g. see Mohammed/Kora for more details and references in the context of text classification)


1

For example in anomaly detection, is it not more accurate to say that it is learning $p(Y|X)$ where Y = 1 if the data is an anomaly and 0 otherwise? If that was the case, how would this differ from classification? In anomaly detection you don't have labels, or have insufficient labels to treat this as classification. In such a case, you don't observe $Y$ so ...


1

An intriguing question, lead me to check a recent Karpathy presentation https://www.youtube.com/watch?v=FwT4TSRsiVw. This is a crazily difficult task, and the presenter gives some great insights on how to make things work! You may see from there the video, if one would try to do everything end-to-end (from image(s) to actions) one would face several more ...


Only top voted, non community-wiki answers of a minimum length are eligible