New answers tagged

1

The equation is parametrizing a hyperplane. You could set your decision boundary to be any parametrized by say Wp + b' = 1 if you wanted to. However, the results would be the same. Since this would be equal to saying the decision boundary is Wp + b'-1 = 0, and then we can define b=b'-1 and estimate b instead of b'


0

tuneThreshold() simply determines the optimal threshold for predicting one class vs another given a set of predictions. That is, the only thing that is changed is the probability threshold that separates the classes; the predictions of the probability for each example are the same. In your first example, you're also tuning a hyperparameter of the learner. ...


1

Assuming that by binary encoding you mean the one explained here, I would advice against using it. Seems an ill-advised idea, I will explain why. First explaining shortly the idea: Suppose (only for simplicity) your categorical variable have $p=2^q$ levels, for the example I take $q=3$. Then code the levels with the binary numbers $0=000_2, 1=001_2, 2=...


1

Binning a continuous IV is almost always a mistake. See Frank Harrell's book Regression Modeling Strategies where he lists 11 problems with this and sums up "Nothing could be more disastrous". Leave the IV continuous and then you can try using a spline of it as a predictor.


1

Over-fitting, most probably. Your model might fit this particular set of data very well, but it could be unlikely to generalize well to another sample of data from the same population. With 150 patients total and a 2:1 class ratio, you only have about 50 in the minority class. With biomedical data such as you have, an unpenalized model is likely to be ...


0

Yes. Unrotated PCs are orthogonal, so Sy is diagonal, with declining variances on the diagonal.


0

Why does a decision tree have low bias & high variance? Does it depend on whether the tree is shallow or deep? Or can we say this irrespective of the depth/levels of the tree? Why is bias low & variance high? Please explain intuitively and mathematically. Bias vs Variance More Bias = error from the model being more simpler (does not fit the ...


1

The method that you use can benefit from the fact that these 128 numbers have temporal relation. One common way to achieve this is to use discrete cosine transform (DCT)as a pre processing step. After DCT you can use your classifier of choice. In this way given the proper use of model complexity control tools you can achieve a more compact model with a ...


0

I'm not familiar with Microsoft ML but for time sequence data LSTMs or GRUs are widely applied. When you say '128 points', is that the number of features per time step or the length of each sequence to be classified?


3

It's a MAP decision rule because it respects the class priors and decides based on the posterior, not the likelihood only, e.g. $$P(C=c|X)\propto \underbrace{P(X|C=c)}_{\text{likelihood}}\underbrace{P(C=c)}_{\text{class prior}}$$ MAP rules respect posteriors, and ML rules respect likelihood instead.


0

After long time, I would conclude the answer is no, based on some quite solid theoretical basis https://en.wikipedia.org/wiki/Data_processing_inequality


2

Is there an ExtraTreesClassifier-like classifier that has decision boundary function like SVM? The SVM boundary arises directly from how SVMs are defined. It's not something that can be "tacked on" to an arbitrary classifier because non-SVM classifiers are not defined in the same way. There are alternative ways for decision trees to make predictions, but I ...


2

This is often called algorithmic design or optimal design. The last name hints on some optimality criteria, and there are many to choose from! Much used is D-optimality. Some related posts here is Motivations for experiment design in statistical learning? and Is DoE applicable to collect data for machine learning model?, look at the links and references in ...


1

Update: I have learned that this is possible. In keras, the parameters for the LSTM layer go in this order: batch, sequence_length, num_features


1

A key idea to remember about a test or validation set in machine learning is that you use it to mimic the real application of using your model to make predictions on data where you truly don’t know the answer. Do your split; then ignore the holdout data for the remainder of the training. In your case, this is exactly what Cagdas Ozgenc described: diagonalize ...


2

when challenged, just say "the problem is highly non-convex and optimization is hard" However handwavy that may sound, that is actually very precise description of the problem. When facing such a problem, you have two options: Try hard to come up with some convex relaxation to (at least a part of) the problem where you can guarantee desirable properties ...


1

This is not about data streams or incomplete / yet-unavailable new data. Your data needs to be on the same scale in all data sets - even if the scale is off, as long as it's the same - for the model to work as planned, which is why you only use your training parameters to transform all the other data sets, since you used these parameters to transform your ...


1

You can perform dimensionality reduction in an unsupervised manner such as using PCA. You can then perform clustering in as many components as you need. Be cautious of the information gain for each PCA component. Since it drops drastically as you increase the number of components and you have added noise. example:


1

It is common to have to design a "fusion" module between two inputs (say an image, along with some scalar input). The "two-stream then fuse" approach you've suggested is pretty common. I don't know that there's an actual name for your "problem" (typically the concern is not that it's difficult to add one more input to an FC input layer, but rather that you ...


0

I'm not sure what your question is, because the paper is quite straightforward in explaining how this is accomplished. Nonetheless I'll summarize. A single neural network layer computes $z' = f(x,z; \theta)$, where $x$ is the input, $z$ is the output of the last layer, and $\theta$ are the network weights. A fixed point $z^*$ of this function is one where $...


1

For prediction, we are interested in the best outcome and want to include as much information as possible in the model to explain the response, but still without over fitting (don't capture the noise) as we want our model to generalize well to new data. Generally, lower values of the LASSO tuning parameter are needed for prediction. When a group of ...


4

TL;DR: For time series and density modeling, neural ODEs offer some benefits that we don't know how to get otherwise. For plain supervised learning, there are potential computational benefits, but for practical purposes they probably aren't worth using yet in that setting. To answer your first question: Is there something NeuralODEs do that "...


1

It depends what you mean by machine learning. Smoothing splines are used to model non-linear effects and the properties of the regression function (e.g., the smoothness of the regression function) are often determined through cross validation. I would consider that falling into the same category as what many people mean when they refer to machine learning.


0

Have you found any solutions to this problem? regards Anatoliy


0

Certainly this statement is not always true strictly as written: letting $\sigma$ be the logistic function $\sigma(x) = 1 / (1 + \exp(-x))$, consider a (very simple) network of the form $$ f_w(x) = \begin{cases} \sigma\left(\frac{x}{w}\right) & w \ne 0 \\ \mathrm{sgn}(x) & w = 0 \end{cases} .$$ Letting $\mathcal W = [-1, 1]$, $\mathcal W$ is compact, ...


1

This is just the result of a bit of tedious algebra. From the other definitions in the article, we have the following identities: $$w^* = A^{-1}b$$ $$w^k = A^{-1}b + Qx^k$$ One identity they assume you know is that $Q^TQ = I$, which implies: $$ Q^T A Q = Q^T Q \text{diag}[\lambda_1,\dots,\lambda_n]Q^T Q = \text{diag}[\lambda_1,\dots,\lambda_n] $$ The next ...


0

You'll always use your unseen test data to evaluate your model in order to be fair and reduce bias in your results.


0

Ad different sizes: The rule of thumb I observed is that shallow networks (https://www.aclweb.org/anthology/D14-1181.pdf) use multiple kernel widths, deep networks (https://arxiv.org/pdf/1705.03122.pdf, https://arxiv.org/pdf/1711.04352.pdf) use typically kernel width 3. I guess the reason is that with a deeper network, you get a broader receptive field by ...


1

The feature space of the RBF Kernel actually has infinite dimension, so you cannot write it in the form of a fully expanded representation. However, you can write it as a sum. I will link to some course notes: Source: https://arxiv.org/pdf/0904.3664v1.pdf


0

I would guess not, because the hyperparameter ‘mtry’ still injects randomness into the features that are included during a particular run, not to mention the randomness in the cases selected during cross validation. If anything I would suspect it becomes ‘more deterministic’ as mtry approaches # of features.


0

Does it make sense to include the team ID as a feature in the model? In most case yes. To your point, about a team's performance being correlated with rival teams (or with the performance of a team last year from which a star player had been transferred into the new team), the only the model can pick it up is if it is able to keep track of which data comes ...


2

I don't think this is a useful viewpoint. Computation with real numbers is, in a technical sense, impossible (most real numbers are uncomputable). However most computer programs use and manipulate (representations of) real numbers. In most cases, these are imperfect floating point numbers. Therefore is all the success of computing derived from the fact ...


1

Spectral embedding, the dimensional reduction part of spectral clustering, preserves topological features of the adjacency graph, not metric properties of the embedded data set. Your $W$ is an affinity matrix - either a nearest-neighbors adjacency matrix, or a dissimilarity matrix from a Gaussian radial basis kernel, etc. The spectral embedding algorithm can ...


0

In this paper, the authors demonstrate that if you work to find good initializations, you can train a 10,000-layer CNN. Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks." 2018. In recent years, ...


0

"Conventional" neural networks often don't go much deeper than a few hundred layers. In actuality, with the prominent use of skip-connections, the "effective" depth of the network is often one layer deep, it's just that that super layer becomes really expressive (one example of that is the U-net for image segmentation, which has several lower resolution ...


1

Genetic algorithms are best when many processors can be used in parallel. and when the object function has a high modality (many local optima). Also, for multi-objective optimization, there are multi-objective genetic algorithms, MOGA. However, I think Genetic algorithms are overrated. A lot of the popularity probably comes from the fact that they are ...


0

I think it is too little data (features) to make any ML algorithm perform well. Do you have access to any other features? Such as movie genre, cast and crew, movie length, etc? If you have those data then you can start with a simple neural network. This paper might give you some ideas. Also, before thinking of any ML algorithm, you can think of what are the ...


0

Another popular question of mine. Well here's to self-help, starting with a sketch illustrating the 3 paths in the $N = 4, k =2$ situation: The number of ways to arrange the 2 test sets to occur in 4 time periods is ${4 \choose 2} = 6$, and $\frac{k}{N} = .5$ is the fraction of the combinations that will start with a test set. Since a "path" is a continuous ...


2

The Gaussian RBF kernel, also known as the squared exponential or exponentiated quadratic kernel, is $$ k(x, y) = \exp\left( - \frac{\lVert x - y \rVert^2}{2 \ell^2} \right) ,$$ where $\ell$ is often called the lengthscale. Remember that for $f \sim \mathcal{GP}(0, k)$, the correlation between $f(x)$ and $f(y)$ is exactly $k(x, y)$. So with a Gaussian RBF ...


1

For me, the notion of what i.i.d really is and why it is, in many cases, a necessary assumption makes more sense from the Bayesian perspective. Here, instead of data being thought of as i.i.d in an absolute sense, they are though of as conditionally i.i.d. given model parameters. For instance, consider a normal model from the Bayesian perspective. We ...


1

Yes. ReLU is a nice well-behaved function and neural networks are universal function approximaters, so they can certainly learn ReLU. For example: https://arxiv.org/abs/1906.09529


11

The operational meaning of the IID condition is given by the celebrated "representation theorem" of Bruno de Finetti (which, in my humble opinion, is one of the greatest innovations of probability theory ever discovered). According to this brilliant theorem, if we have a sequence $\mathbf{X}=(X_1,X_2,X_3,...)$ with empirical distribution $F_\mathbf{x}$, if ...


4

Yes, samples in the dataset may not be completely iid, but the assumption is present to ease the modelling. To maximize the data likelihood (in almost all models this is explicitly or implicitly part of the optimization), i.e. $P(\mathcal{D}|\theta)$, without the iid assumption, we'd have to model the dependence between the data samples, i.e. the joint ...


2

No, similar distribution of Y's does not entail poor classification results. See this for example, two features X1, X2 and binary response variable(colored green and red). Clustering results are apparent and a simple heuristic based classifier would work fine. ##Generate cluster based data cl_size = 100 c1 = [(x, y) for x, y in zip(list(np.random.uniform(...


3

$w^T\Sigma w$ is convex function, you're right. As far as I see, whuber's answer defines concave functions as what we usually know as convex functions. It's also pointed out in the comments. Take two points on the unit sphere, and connect them with a line. Is the line (all of it) inside the domain? No, then the domain is not convex. Because, you maximize ...


0

An MDP (markov decision process) or POMDP (partially observable MDP) are two popular mathematical abstractions which are used to describe how an agent interacts with the environment. Almost all games or tasks can be formulated as either an MDP or a POMDP. Therefore, when people develop RL algorithms, they don't design them for a specific game or ...


1

Without choosing a loss function, you cannot construct the SVM because you won't have an optimization objective to work on. So, you first choose your loss function and construct your model. In the end, there will be two different models to compare.


0

You appear to have one independent variable (depth) (but maybe also actual latitude) and one dependent variable (transmitted latitude) and maybe repeats. But there are a bunch of issues: Transmitted latitude is constant at each depth, so the repeats are problematic. Transmitted latitude is the same for three of the four depths, so depth is problematic. ...


0

Since your data has generated among two variables: setups and trials, and for every of their combination (setups*trials) you have just one value; I think you should use Two-Way ANOVA.


0

Euclidean distances are not a very good metric for quantifying n-space distances. For example, consider this work: "To avoid the undesired effects of distance concentration in high-dimensional spaces, previous work has already advocated the use of fractional ℓp norms instead of the ubiquitous Euclidean norm. Closely related to concentration is the emergence ...


Top 50 recent answers are included