27

This will probably be closed quickly as opinion-based, but here is a point you may want to consider. 200 features is a lot, and 30k rows is less than it sounds like. A "fishing expedition" to find relevant features is quite likely to overfit and select spurious features. The danger is that when you go to your domain experts with these features you &...


27

One limitation of random search is that searching over a large space is extremely challenging; even a small difference can spoil the result. Émile Borel's 1913 article "Mécanique Statistique et Irréversibilité" stated if a million monkeys spent ten hours a day at a typewriter, it's extremely unlikely that the quality of their writing would equal a ...


20

John Elder in 2005 gave a (now classic) presentation called: "Top 10 Data Mining Mistakes". Number 4 in that list is: Listen (only) to the data. Specifically for business environments where it is almost certain that we act using incomplete information (e.g. client priorities, financial and physical resources, legal framework, etc.) which affects ...


19

Consider a neural network model with 100 weights. If we think only about getting the sign of the weights right and don't worry for the moment about their magnitude. There are 2^100 combinations of the signs of these weights, which is a very large number. If we sample 60 random weight vectors, we will have seen only an minuscule proportion of that space, ...


13

Gradient-free learning is in the mainstream very heavily, but not used heavily in deep learning. Methods used for training neural networks that don't involve derivatives are typically called "metaheuristics." In computer science and pattern recognition (which largely originated in electrical engineering), metaheuristics are the go-to for NP-hard ...


12

Suppose we want to answer your question with a 1000 character answer. One approach could be to sample 60 1000-tuples of characters, punctuation marks, and whitespace. With 95% probability, one of these will be within the most useful 5% of all possible Stack Exchange answers within this character limit. Basically the problem as you point out is that being ...


11

Great question! To put it briefly, "Gradient Free Learning" (i.e. "metaheuristics", as pointed out by @user0123456789) is usually used when the "gradient" (i.e. derivative) of the loss function can not be evaluated. This can occur in instances such as : The derivative of the loss function does not exist (e.g. contains "...


10

There's a mathematical result in optimisation, less interesting than it first sounds, called the "No Free Lunch Theorem". It says that for a discrete problem (like @JonnyLomond's answer), no algorithm can beat random search when its performance is averaged over all possible functions to be optimised. That is, you have a function $f:\Omega\to L$ ...


6

The problem you are dealing with is a selection of variables problem, and so standard principles and methods apply. In particular, if you have a large number of initial variables/features to select from, there is a danger of overfitting if you fail to adopt appropriate methods that account for multiple comparisons. In the case where you have an exogenous ...


5

The reason we don't use gradient-free methods for training neural nets is simple: gradient-free methods don't work as well as gradient-based methods. Gradient-based methods converge faster, to better solutions. Gradient-free methods tend to scale poorly (for instance, one of the papers you cite only tests on MNIST, which is a tiny dataset and task; the ...


4

In my understanding it's a consequence of the high number of variables that neural networks tends to require when tackling interesting problems. For simple tasks gradient-free methods work very well and are quite capable of beating gradient-based methods, as many of them deal with non-convex functions/local optima better than the grad-based methods and that ...


4

No, it is not safe to assume that all loss functions are quadratics. One of the most common cost functions is the binomial cross-entropy $$ L(p) = y\log p +(1-y)\log(1-p) $$ where $0 \le p \le 1$ and $y \in \{0,1\}$. The function $L$ is not a quadratic because it is not a polynomial of degree 2.


3

You can use soft targets, but you would need to have them. The soft target for classification would be the probability that the sample belongs to a particular class, this is usually not something that you could observe. It is even available in the R’s implementation of logistic regression. The scenario where you would have the soft targets is when using ...


3

There are two aspects here causal inference and explainability. From a causal inference perspective, domain expertise should guide the process of building relevant factors on a given purpose, targets, that are really linked and not just correlations explored or discovered by data scientists. Inference and Intervention: Causal Models for Business, by Ryall-...


3

The centroid is referring to the central point of a k-means cluster. Where does k-means come in? At the end of p.2 they say: Instead of choosing priors by hand, we run k-means clustering on the training set bounding boxes to automatically find good priors. Also see Figure 2.


3

Your question is more theoretically founded, but goes in the same direction as this one. You might want to check the answers there (disclaimer: one of them is mine). In order to answer your question from the last paragraph, I believe it is useful to identify conceptual basic blocks in machine learning: feature selection feature transformation parameter ...


3

First of all, not only neural networks are universal approximators. There is nothing special about them, they just proved to work quite well for a class of problems. Kernel based methods generally don't scale well. Neural networks gained popularity in the time when we (a) improved our computers, so we were able to run bigger neural networks and do this in ...


3

There are many wrong or slightly off assumptions in this question, I will try to work through them. Minor point: Neural networks may or may not be statistical models (many of them are not). I would say that you need a likelihood function or at least a generative model of the data for a mathematical/computational model to be called a statistical model. ...


3

regardless of how many dimensions your function has, there is a 95% probability that only 60 iterations are needed to obtain an answer in the top 5% of all possible solutions! Finding a 95th-percentile solution is no guarantee of finding a good solution. The nature of the curse of dimensionality is that your "optimization distribution" becomes ...


3

You don’t. It uses hashing trick, so the data is passed through a hashing function that maps the data to codes. The function can and will map different values to same codes, because in general it is used to reduce cardinality of your data. To learn what values were mapped to what codes, you need to create a dictionary by looping over your data, transforming ...


2

Different problems that you can can tackle using a graph representation. A graph is defined as a set of <V,E> (nodes , links). Think of 2 examples. atoms and their bonds (making together a molecule). in this case V= atoms, E= chmical bonds, graph is molecule users in social network being connected if they are friends (making together a social network....


2

If I understand correctly, what you have done is feature extraction, where the weights are not derived from your dataset. To improve performance, you want the model to learn parameters suitable to your dataset. Have you tried retraining the pre-trained ResNet-50 with your dataset? Here you can first try without freezing any layer. This would mean that all ...


2

Different combinations can give the same output value! Consider a simple neural network with one feature, a hidden layer with two neurons (ReLU activation), and an output neuron with an identity activation. $$ \widehat{y_i} = \widehat{b_{1,2}} + \widehat{w_{1,2}}ReLU\bigg( \widehat{b_{1,1}}+\widehat{w_{1,1}}x \bigg) + \widehat{w_{2,2}}ReLU\bigg( \widehat{b_{...


2

The best explanation is given in A Tutorial on Energy-Based Learning by LeCun et al, concretely in section 5. Also, Learning a Similarity Metric Discriminatively, with Application to Face Verification by Chopra et al. provides a detailed analysis for the case of face verification. The motivation for introducing the margin is to avoid a collapsed solution (...


2

I have my doubts that you will be able to find anything of value from this data, because it involves market data, which is very chaotic and involves... people, which all have unknown motivations. However you should still try, you never know. To summarize your question and comments: end companies will need some amount of a certain product brokers predict how ...


2

Take as an example the simple neural network diagram from Wikipedia. Each arrow on the diagram shows the weight of the model, biases are not shown. With the simple model as linear regression to judge its complexity, you would just count the parameters. Here notice that the parameters on the second layer depend on the parameters of the first layer. How ...


2

There is no single optimizer that beats all the others. If you look at the published papers, you would see different optimizes used. People often still use stochastic gradient descent, you can find nice discussion on this on Quora. There are results suggesting that the basic SGD may generalize better (Hardt, Recht, & singer, 2016). So neither deep ...


2

Why do we use Gradient Descent instead of Random Search for optimizing the loss functions in Neural Networks? We do use both at the same time currently. Meaning that, there is already a degree of random search even if we use stochastic gradient decent in training neural networks, i.e., random initialisation and in reinforcement learning via random search in ...


2

The only reason that I can think of, is that if the ranked distribution of the optimization values are "heavily negative skewed" Sort of. There is a compounding that occurs when you add dimensions that is similar to what you get when you add more randomly sampled models, except that it works against you rather than for you. As you add more ...


2

This thought has appeared in some of the answers, but I would like to say, that being in the 5% of the best solutions may still produce a solution of very poor quality. Consider classification problem on ImageNet and some large networks with millions of parameters. Doing a random search in the space of parameters, you can find a network, that is in the top 5%...


Only top voted, non community-wiki answers of a minimum length are eligible