Questions tagged [machine-learning]
Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.
2,304
questions
12
votes
1
answer
10k
views
Why L1 regularization can "zero out the weights" and therefore leads to sparse models? [duplicate]
I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1].
To better understand it I'm reading Google's ...
12
votes
1
answer
4k
views
Question about subtracting mean on train/valid/test set
I'm doing data preprocessing and going to build a Convonets on my data after.
My question is:
Say I have a total data sets with 100 images, I was calculating mean for each one of the 100 images and ...
12
votes
1
answer
3k
views
Stratified classification with random forests (or another classifier)
So, I've got a matrix of about 60 x 1000. I'm looking at it as 60 objects with 1000 features; the 60 objects are grouped into 3 classes (a,b,c). 20 objects in each class, and we know the true ...
12
votes
1
answer
9k
views
Reconciling boosted regression trees (BRT), generalized boosted models (GBM), and gradient boosting machine (GBM)
Questions:
What is the difference(s) between boosted regression trees (BRT) and
generalized boosted models (GBM)? Can they be used interchangeably?
Is one a specific form of the other?
Why did ...
12
votes
2
answers
13k
views
Imputing missing values on a testing set
I'm a newbie to machine learning so forgive me if the answer to this question is obvious.
I have been working on a binary prediction problem using logistic regression. Using a selection of categorical ...
12
votes
1
answer
2k
views
Model selection in offline vs. online learning
I've been trying to learn more about online learning lately (it's absolutely fascinating!), and one theme that I haven't been able to get a good grasp on is how to think about model selection in ...
12
votes
2
answers
8k
views
Why is bias equal to zero for OLS estimator with respect to linear regression?
I understand the concept of bias-variance tradeoff. Bias based on my understanding, represents the error because of using a simple classifer(eg: linear) to capture a complex non-linear decision ...
12
votes
1
answer
1k
views
Preventing Pareto smoothed importance sampling (PSIS-LOO) from failing
I recently started using Pareto smoothed importance sampling leave-one-out cross-validation (PSIS-LOO), described in these papers:
Vehtari, A., & Gelman, A. (2015). Pareto smoothed importance ...
12
votes
5
answers
19k
views
Mathematical background for neural networks
Not sure if this is appropriate for this site, but I'm beginning my MSE in computer science (BS in applied mathematics) and want to get a strong background in machine learning (I'm most likely going ...
12
votes
1
answer
9k
views
Understanding the use of logarithms in the TF-IDF logarithm
I was reading:
https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition
But I cannot seem to understand exactly why the formula was constructed the way it is.
What I do Understand:
iDF should at ...
12
votes
1
answer
33k
views
Binary Encoding vs One-hot Encoding
What is the difference between binary encoding and one-hot for categorical input variables for English Text and their impact on the neural network?
Can anyone help me to find a scientific paper about ...
12
votes
3
answers
24k
views
What is the difference between logistic regression and bayesian logistic regression?
I'm a bit confused whether these two are the same concept. If they are different what's the difference?
Thanks!
12
votes
1
answer
701
views
What are $\ell_p$ norms and how are they relevant to regularization?
I have been seeing a lot of papers on sparse representations lately, and most of them use the $\ell_p$ norm and do some minimization. My question is, what is the $\ell_p$ norm, and the $\ell_{p, q}$ ...
12
votes
2
answers
4k
views
Foundation models : Is it a new paradigm for statistics and machine learning?
A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any ...
11
votes
6
answers
40k
views
Compare R-squared from two different Random Forest models
I'm using the randomForest package in R to develop a random forest model to try to explain a continuous outcome in a "wide" dataset with more predictors than samples.
Specifically, I'm ...
11
votes
2
answers
13k
views
Vectorization of Cross Entropy Loss
I am dealing with a problem related to finding the gradient of the Cross entropy loss function w.r.t. the parameter $\theta$ where:
$CE(\theta) = -\sum\nolimits_{i}{y_i*log({\hat{y}_{i}})}$
Where, $\...
11
votes
1
answer
5k
views
How to understand the log marginal likelihood of a Gaussian Process?
I'm trying to understand Gaussian Processes. Could anyone tell me:
Why we need to use the log marginal likelihood?
Why using log, the marginal likelihood can be decomposed to 3 terms (including a ...
11
votes
2
answers
8k
views
Anomaly detection: what algorithm to use?
Context: I'm developing a system that analyzes clinical data to filter out implausible data that might be typos.
What I did so far:
To quantify the plausibility, my attempt so far was to normalize ...
11
votes
1
answer
3k
views
How are piecewise cubic spline bases constructed?
There are words from the The Elements of Statistical Learning on page 119:
It is not hard to show that the following basis represents a cubic spline with knots at $\xi_1$ and $\xi_2$:
$h_1(X)=...
11
votes
1
answer
8k
views
What's the physical meaning of the eigenvectors of the Gram/Kernel matrix?
If we have some centered dataset $X$ then the eigenvectors of $X^TX$ represent the principal components of the dataset, and their physical meaning is the directions that data follow in the original ...
11
votes
2
answers
13k
views
SVM rbf kernel - heuristic method for estimating gamma
I read on this exchange a heuristic method of estimating gamma for the rbf kernel in SVMs. I was wondering if someone might be able to explain it to me in a little more detail? I believe you select ...
11
votes
1
answer
3k
views
Training a convolution neural network
I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
11
votes
7
answers
3k
views
Why do we use Linear Models when tree based models often work better than linear models?
In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
11
votes
2
answers
3k
views
Are XGBoost probabilities well-calibrated?
In general, can you say anything about how well are the probabilities returned by XGBoost are calibrated? Is it true that, because XGBoost directly optimizes log-loss, probabilities are generally well-...
11
votes
2
answers
4k
views
Multivariate time series clustering
I am collecting a group of multivariate time sequences. For example, there are 2000 time series. Each time series is of 12 dimensions.
Are there any systematic models/algorithms that can cluster ...
11
votes
2
answers
2k
views
Predictive maintenace model to identify indication of failure before it happens
Situation
I'm working on a problem where I'm using sensor data to predict machine failure before the failure happens and I need some advice on which methods to explore.
Specifically, I want to ...
11
votes
1
answer
10k
views
Extending 2-class models to multi-class problems
This paper on Adaboost gives some suggestions and code (page 17) for extending 2-class models to K-class problems. I would like to generalize this code, such that I can easily plug in different 2-...
11
votes
3
answers
16k
views
Regarding using bigram (N-gram) model to build feature vector for text document
A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
11
votes
1
answer
12k
views
How can we explain the fact that "Bagging reduces the variance while retaining the bias" mathematically?
I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias".
What is the mathematically principle behind this intution? I checked with few experts ...
11
votes
2
answers
3k
views
Machine learning applications in number theory
Is there any research into or applications of machine learning in number theory?
I am also looking for (leading examples of) statistical/empirical analysis of number theory questions. Also wondering ...
11
votes
1
answer
1k
views
How to calibrate models if we don't have enough data?
I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23
I was reading about calibration of models (binary ...
11
votes
1
answer
586
views
Recreating figure 3.6 from Elements of Statistical Learning
I am trying to recreate FIGURE 3.6 from Elements of Statistical Learning. The only information about the figure is included in the caption.
To recreate the forward stepwise line my process is as ...
11
votes
2
answers
3k
views
Bonferroni Correction & machine learning
In psychology studies I learned that we should use the Bonferroni method to adjust the significance level when testing several hypothesis on a single dataset.
Currently I am working with machine ...
11
votes
1
answer
2k
views
Motivation behind random forest algorithm steps
The method that I'm familiar with for constructing a random forest is as follows:
(from http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm)
To build a tree in the forest we:
Bootstrap a ...
11
votes
2
answers
2k
views
How to prove that the manifold assumption is correct?
In machine learning, it is often assumed that a data set lies on a smooth low-dimensional manifold (the manifold assumption), but is there any way to prove that assuming certain conditions are ...
11
votes
1
answer
867
views
How does class balancing via reweighting affect logistic regression?
When developing machine learning classifiers, some people upsample or upweight the minority class to achieve a 50-50 balance, claiming that this improves performance. Some statisticians have ...
11
votes
3
answers
3k
views
Why do people like smooth data?
I am to use the Squared Exponential kernel (SE) for Gaussian Process Regression. The advantages of this kernel are: 1) simple: only 3 hyperparameters; 2) smooth: this kernel is Gaussian.
Why do ...
11
votes
1
answer
21k
views
How to use decision stump as weak learner in Adaboost?
I want to implement Adaboost using Decision Stump. Is it correct to make as many decision stump as our data set's features in each iteration of Adaboost?
For example, if I have a data set with 24 ...
11
votes
2
answers
4k
views
Gaussian process and Correlation
I am have been wondering why people use Gaussian processes (GP) to model an unknown (sometimes deterministic) function. For instance consider an unknown function $y=f(x)$. We have three independent ...
11
votes
1
answer
7k
views
Classification on variable-length time series
I have a series of transactions like the following:
[0, 2, 2, 3, 1, 0, 0, 0, 1]
[1, 0, 0]
[3, 3, 1, 1]
I would like to classify each transaction as being part of one of two categories: class A or ...
11
votes
3
answers
5k
views
Is the Gaussian Kernel still a valid Kernel when taking the negative of the inner function?
In support vector machines (SVMs) and other Kernel based methods, like Gaussian processes, the Kernel replaces the inner product of two feature vectors $k(x_n,x_m)=x_n^Tx_m$. The Gaussian kernel
$$k(...
11
votes
2
answers
4k
views
What is a log-odds distribution?
I am reading a textbook on machine learning (Data Mining by Witten, et al., 2011) and came across this passage:
... Moreover, different distributions can be used. Although the normal
distribution ...
11
votes
2
answers
10k
views
gradient versus partial derivatives
how exactly is partial derivative different from gradient of a function?
In both the case, we are computing the rate of change of a function with respect to some independent variable. While I was ...
10
votes
2
answers
36k
views
What is the "binary:logistic" objective function in XGBoost?
I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised.
$$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + ...
10
votes
1
answer
3k
views
Connection between filters and feature map in CNN
I am learning CNN with TensorFlow and Python.
I do not understand the connection between layer $\ell$ and layer $\ell+1$. For example, for the input image and the first layer, it is easy as there is ...
10
votes
2
answers
2k
views
Why are my steps getting smaller when using fixed step size in gradient descent?
Suppose we are doing a toy example on gradient decent, minimizing a quadratic function $x^TAx$, using fixed step size $\alpha=0.03$. ($A=[10, 2; 2, 3]$)
If we plot the trace of $x$ in each iteration, ...
10
votes
1
answer
8k
views
Connection between loss and likelihood function
Simple question: Can we generally think of the loss function as the negative of the likelihood function?
For instance with regards to logistic regression, the likelihood function in a binary setting ...
10
votes
4
answers
3k
views
Why is KNN not "model-based"?
ESL chapter 2.4 seems to classify linear regression as "model-based", because it assumes $f(x) \approx x\cdot\beta$, whereas no similar approximation is stated for k-nearest neighbors. But aren't both ...
10
votes
5
answers
18k
views
Why using cross validation is not a good option for Lasso regression?
I watched the lecture about Lasso and at the end of this module (between 00:40 and 01:25) she explains how to choose the regularization parameter Lambda and it sounds like using (K-fold)Cross ...
10
votes
3
answers
12k
views
XGBoost vs Gradient Boosting Machines
For a classification problem (assume that the loss function is the negative binomial likelihood), the gradient boosting (GBM) algorithm computes the residuals (negative gradient) and then fit them by ...