New answers tagged machine-learning
1
vote
How does a single layer/single unit with Adam optimizer network work?
I will answer your first question. Suppose you have observations $(X_{i}, Y_{i})_{i=1}^{n} \in \mathbb{R}^{p} \times \mathbb{R}$. Then the neural network you have in mind can be represented as the ...
- 151
5
votes
Building a Statistically Sound ML Model
I suspect you would be a fan of Frank Harrell's Regression Modeling Strategies textbook. He discusses pieces of that book on his Statistical Thinking blog. I also like his Controversies in Predictive ...
- 67.1k
6
votes
Building a Statistically Sound ML Model
This is a very broad question, and any specific answer will depend heavily on your use case. However, I believe some general points can be made.
Acquire subject knowledge. Talk to experts in the ...
- 131k
1
vote
Accepted
Handling a very informative feature with significant missing values
Indeed even in regression methods, this is a classic case of dealing with an informative but missing-not-at-random (MNAR) feature in a regression setting. The key challenge is to balance the richness ...
- 1,245
1
vote
How does a single layer/single unit with Adam optimizer network work?
In general momentum-based optimizers such as Adam and RMSProp rely on adaptive estimates of the first and second moments ($m_t,v_t$) of the gradient. Therefore when the scale of the label is large or ...
- 1,245
0
votes
Does it make sense to regularize the loss function for binary/multi-class classification?
A lot of times in NN training, we put L2 regularization in the optimizer (called weight decay) instead of on loss function
1
vote
Data leakage when using walk forward optimization
From my perspective, it depends.
If you use what the diagram calls "testing set" for hyperparameter tuning, there is a risk of data leakage.
If you use what the diagram calls "testing ...
- 11
5
votes
Why the loss is not considered as a "supervisory signal" in unsupervised learning?
You're right in noting that both supervised and unsupervised learning usually involve loss functions guiding the optimization process. However, in unsupervised learning there are no explicit ...
- 1,245
0
votes
Categorical Dependent Variable
How to approach this? I'd say "very, very cautiously, especially if you don't have a huge sample."
First, I'd think very carefully about whether and why you need 60 levels of the dependent ...
- 128k
1
vote
Is the mean of samples still a valid sample?
It depends on $n$. In the case where $n = 1$, the sample mean $\bar{x}=x_1$ and trivially matches $p_\theta(x)$ as it is just the single sample.
For specific degenerate distributions, the mean can ...
- 11.5k
0
votes
Signal-to-noise ratio in predictive modeling and machine learning
Probably the easiest to digest example is a simple detection task.
Say you make a measurement $X_m$ of some system and you want to detect whether or not a DC (constant) signal with amplitude $\mu$ is ...
- 171
0
votes
Explain "Curse of dimensionality" to a child
My understanding might be a little skewed or, even, childish itself, but I will have a go at this too.
The curse of dimensionality is basically "as dimensions increase, there's more space for ...
- 61
2
votes
Accepted
How do machine learning topics fit into a traditional undergraduate statistics course on estimation?
To integrate those ML topics into undergraduate traditional statistics topics of estimation, begin with statistical estimation theory (MLE, MAP, MME) to set the foundational stage and then apply it to ...
- 1,245
2
votes
Model intercomparison with no ground-truth labels
Is Model intercomparison with no ground-truth labels a good idea?
No. Let me give a counterexample.
Given you search models giving directions.
Now you have (hidden) ground truth A giving direction (0,...
- 1,783
1
vote
Bayes Predictor for linear regression with square loss and expected value properties
For the squared error loss $l(\theta, a)=(\theta-a)^2$, the Bayes estimate of $\theta$ after $X=x$ is observed is given by the value of $a$ which minimises the expected squared error loss $\mathbb{E}[(...
- 51
1
vote
Classification ML model: probability of positive label knowing the model score
Probably too late to be useful, but...
Without knowing anything else about it, my best guess for x belonging
to the positive class is the frequency of the positive labels in the
known population ... ...
- 56.2k
0
votes
Is it possible to train Neural networks for time series forecasting using elastic distances (such as dtw) as a loss function?
My answer is certainly not covering all aspects of your question, but here are some thoughts about it.
i) DTW is not differentiable as is, due to the MIN operator.
ii) you could have a look as softDTW ...
2
votes
Reasons and potential solutions for poor performance of elastic net penalized quantile regression
To my mind, at least, if you want "to identify metabolites most strongly associated with PM2.5," the most logical way is to use PM2.5 as a predictor variable in a multivariate (multiple-...
- 101k
0
votes
Predicting the probability distribution of a deterministic dataset
In this case the dependence is simply a function $$y=f(x).$$ We can give it a formally probabilistic form by writing
$$p(y|x) = \delta\left(y-f(x)\right).$$
The error in inferring the function may be ...
- 4,487
2
votes
Predicting the probability distribution of a deterministic dataset
To be honest, I somewhat fail to see the problem.
Until we arrive at quantum levels, pretty much everything has a "theoretically knowable ground truth". If you know the position and current ...
- 131k
3
votes
Accepted
Questions on backpropagation in a neural net
The answers to both of your questions are "because of the chain rule". This will be more clear with examples.
Backprop doesn't care about batches
In other words, if you were to implement ...
- 1,625
4
votes
Model intercomparison with no ground-truth labels
Suppose you have three measures, A, B, and C. Measure B is "mostly correct", and measure A is highly correlated with it. Measure C is "exactly correct", and has lower correlation ...
- 10.4k
0
votes
Using statistical sampling and machine learning together?
In the sklearn implementation, there is the argument class_weight which does not do a balanced sampling as you propose but achieves the same effect by weighting a ...
- 1
2
votes
Separate Test Set for Cross-Validation for Small Sample (n=140)
Although a model developed on 126 cases might not be much different from one developed on all 140, the remaining 14 cases will be too few to test the model adequately (as you sense). This is even more ...
- 101k
0
votes
Confusion regarding the criteria for defining a ML model as a linear model
This is my current understanding, adding it here incase of future reference. Whosoever is reading this - if my understanding is incorrect, please do correct me. Any feedback would always be welcome!
...
- 53
11
votes
Accepted
Is my evaluation for this multiple linear regression correct?
You should be using the root MSE, not the MSE, as a measure of expected error; that is, the error is on the order of \$6K, not $36M. (The units of MSE are the square of the units of the response ...
- 47.3k
9
votes
Is my evaluation for this multiple linear regression correct?
I do not know this particular dataset. However:
You are looking at the Mean Squared Error, which is about 34,510,000$^2. (It's a squared error, so its units are of course squared dollars.) This is an ...
- 131k
0
votes
Similarity metrics for more than two vectors?
Gini Coefficient as a Unified Metric for Evaluating Many-versus-Many Similarity in Vector Spaces https://arxiv.org/abs/2411.07983
The Gini coefficient can be used as a unified singular metric to ...
1
vote
What is the weak side of decision trees?
Some weakness (which is surprisingly rarely mentioned) of tree-based algorithms for regression is: They are by design unable to predict values outside the seen value range. So they can not extrapolate....
- 111
0
votes
Filter methods for feature selection are often univariate. What multivariate filter methods exist, and what are their dis/advantages?
Yes, multivariate filter methods do exist. One example would be Joint Mutual Information Maximisation (JMIM), which has been shown to outperform other multivariate filter methods on classification ...
- 171
0
votes
Choose a good estimator in a candidate set
Write $I$ for the chosen index. An apparently trivial bound is that the error for $A_I$ is no more than the sum of the errors for all the $A_i$. If the number of candidate estimators $p$ is small, or ...
- 46.8k
2
votes
Accepted
Custom Loss function Overfits to the Wrong Output but MSE Doesn't
The problem is that the loss function only has 0 gradient when the neural network output is 0. This is not desirable! Instead, we want a loss function that has a 0 gradient when the output matches the ...
- 94k
0
votes
Accepted
GNNs with higher order adjacency matrices
One can consider a Laplacian Polynomial, see this post for details..
The polynomial has the form:
$$X'_u=\sum_{d=0}^{d=i} w_i L_u^d X$$
where $X$ are the nodes' features, $L$ the Laplacian $L=D-M$, $M$...
4
votes
Analyses of Associations and Predictive Models in Random Forest
If I want to create a predictive model, then I understand that I should use a training set (for example, 70%) and a test set (30%). On the other hand, if I only want to analyze the main variables ...
- 67.1k
2
votes
Does SVM suffer from curse of high dimensionality? If no, Why?
The reason that the support vector machine is somewhat resistant to the curse of dimensionality is that it is an approximate implementation of a bound on generalisation performance that is independent ...
- 56.2k
0
votes
Is there anything like spaced repetition learning for machine learning models?
Obviously, machine learning models once trained and fixed do not "forget" anything. However, there could be similarities in how they are trained. And, just about anything to do with human ...
- 35.2k
2
votes
Multicollinearity (collinear predictors) checking is needed for non-linear regression (Poisson, negative binomial) and machine learning algorithms?
The type of regression does not matter; the use to which result will be put does matter.
Collinearity is not really a problem if your only goal is prediction and you will not be using the parameter ...
- 128k
3
votes
Signal-to-noise ratio in predictive modeling and machine learning
SNR is defined as a precise measure from the field of signal processing which is ultimately related to the variance of both signal and noise random variables. For ML predictive modeling we only have ...
- 1,245
6
votes
Signal-to-noise ratio in predictive modeling and machine learning
I don't think I have ever come across a rigorous, formal and commonly accepted definition of "signal", "noise" or the "signal to noise ratio". Even in the original thread,...
- 131k
1
vote
Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)
I started looking at the eigenvalues of the normalized confusion matrix. In this case, the first eigenvalue is 1 and the eigenvector gives the stable point. I like to call this intrinsic prevalence, ...
- 11
7
votes
Signal-to-noise ratio in predictive modeling and machine learning
I think the following examples bring out several important points.
First, consider how much lower than $R^2$ is $R^{2}_\text{adj}$ when $R^2$ is large compared to when it's small. There is much more ...
- 98.4k
1
vote
Should normalization be applied on interaction feature
At least for gradient boosted trees, there is no need either to standardize or to pre-define interaction terms (except if there are numerical problems due to measurement scales of variables). At each ...
- 101k
0
votes
Generalization error as U shape curve with respect to model complexity (bias variance tradeoff))
Indeed the usual empirical risk minimization principle cannot directly show the tradeoff's U shape curve, but if you follow its regularized version aka structural risk minimization (SRM) then it'll be ...
- 1,245
17
votes
Why doesn't ML suffer from curse of dimensionality?
You're absolutely correct about the curse of dimensionality in traditional density estimation which seems impossible to estimate in very high dimensions like images. However, modern deep learning ...
- 1,245
15
votes
Why doesn't ML suffer from curse of dimensionality?
ML has all the problems of statistical models and more with respect to the curse of dimensionality. Think of it this way. A standard statistical model assumes additivity, and if it assumes assumes ...
- 98.4k
0
votes
Alternatives for RMSE to Evaluate Goodness of Fit for Stable Distribution Parameters
You don't get to know how good your empirical estimates are. This is why we develop estimators with desirable theoretical properties, so we have some reasonable expectation of acceptable performance ...
- 67.1k
0
votes
Best way to remove multicollinearity and feature selection for binary classification problem?
My recommendation is to use the variance inflation factor, which is the reciprocal of 1.0 less the r-squared predicting that variable by all the other predictors. There is a nice algorithm for the ...
- 11
Top 50 recent answers are included