New answers tagged machine-learning
1
vote
R squared of subgroups
How you perform and calculate such a statistic depends on what you want to learn by from it.
My belief about $R^2$ is that it is a comparison of how your model performs (in terms of square loss) vs ...
- 46.6k
1
vote
Why can LASSO MAE be worse than individual feature linear regression MAE?
why did LASSO chose multiple features that gives higher error rather than only keeping a single or fewer features that gives a lower error?
You told the regression to minimize the LASSO loss and then ...
- 46.6k
0
votes
Neural network parameters dependency vs gradient descent
From my experience, there is not a simple solution to this issue. A reasonable approach is to use the state-of-the-art parameters in your field, like computer vision and so on. For example, the choice ...
- 19
2
votes
prior & posterior probability in Bayesian Decision Theory
Let us give an example that makes this as simple as possible.
Suppose your samples are all taken from a Bernoulli distribution, i.e. these are ``binary samples'', these are just $1$'s and $0$'s. Let ...
- 1,725
0
votes
prior & posterior probability in Bayesian Decision Theory
In a very simplified way:
Posterior probability = $P(\gamma | D)$ = probability that your parameter (or vector of parameters) $\gamma$ is equal to the value you've sampled given your dataset $D$.
...
- 145
3
votes
prior & posterior probability in Bayesian Decision Theory
$x$ and $c$ in Bayes theorem are random variables. Any random variables. Bayes's theorem is about being able to flip sides of the conditional distribution from $P(x|c)$ to $P(c|x)$ or the other way ...
Tim♦
- 128k
0
votes
Combining image and scalar inputs into a neural network
The features vector can be combined to an image by -
Adjusting the features shape by using tf.reshape and tf.tile
Combining the features and image by performing concatenation, add (as described in ...
- 101
2
votes
I need to get 100% accuracy on my training data
What you are describing is that you need to losslessly compress and retrieve the data. Did you consider any off-the-shelf caching solution? Since you care about “predicting” the seen data, it's about ...
Tim♦
- 128k
1
vote
Why do the error derivatives become small if we start with a large learning rate?
So in the case of sigmoid neurons, having large weights means the hidden unit output saturates, so then changes in the weights have minimal effect on the hidden unit output, and therefore the error ...
- 5,384
5
votes
Accepted
Is the iid assumption in Linear Regression necessary?
If you are asking about the i.i.d. assumption in machine learning in general, we already have that question answered in the On the importance of the i.i.d. assumption in statistical learning question.
...
Tim♦
- 128k
0
votes
What are the methods to increase the dimension of a feature space?
There are lots of different ways. A classic example is polynomial expansions - you take all powers of your input variables (https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem from 1885)
...
1
vote
Accepted
Calculating KL divergence with entropy and cross entropy for VAEs
The Kullback–Leibler divergence is also called relative entropy.
It is easy to see that:
\begin{align}
D_{KL}(P\|Q)
&=\int p(x)\log\left(\frac{p(x)}{q(x)}\right)dx\\
&=\underbrace{\int p(x)\...
- 17k
0
votes
What are the methods to increase the dimension of a feature space?
A feature transformation can be "learned" by fitting a neural network. For example, if the original features $x \in \mathbb{R}^d$ are mapped to one of $k$ classes, the mapping may be modeled ...
7
votes
Practical usefulness of PCA
I use three examples in my lectures to illustrate what PCA can do (click the links for pointers to the slides). They're chosen to show how useful it is in general data science practice and how ...
4
votes
What are the methods to increase the dimension of a feature space?
Square them.
Log them.
Multiply them.
Multiply their logs.
Log their products.
Take Fourier or wavelet transforms of time series data.
Any function of your set of features is a possible additional ...
1
vote
Practical usefulness of PCA
Lots of what we're trying to get at with dimensionality reduction, whether linear or nonlinear, is abstraction of the data away from a raw space and towards a manifold or embedding in which complex ...
0
votes
How to handle machine learning inputs that should be considered as a set of vectors, but whose interpretation is order invariant (order agnostic set)
This is "multiple instance learning"; the wikipedia page has a good introduction.
Your examples sound like they don't conform to the standard assumption, so you'll need some "metadata-...
- 3,715
4
votes
Practical usefulness of PCA
It is absolutely not clear, obvious, or agreed-upon what "intelligence" is in humans (or, for that matter, in nonhuman animals). What is clear is that people's performance on a variety of ...
0
votes
Can I use K-Means to group customers based on a single variable?
If you want a data driven clustering, k-means looks promising in the sense that it will produce clusters with similar within-cluster variance, which may make sense in your application. The problem of ...
- 16.7k
3
votes
Practical usefulness of PCA
PCA just comes down to using the eigendecomposition of the (empirical) covariance matrix of the data.
The full eigendecomposition of the covariance matrix results in a set of eigenvectors and ...
3
votes
Practical usefulness of PCA
I've used PCA in facial motion capture for real time animatronic control of the 'lots of dots on a face' variety.
I was able to find out which dots - which is to say regions of the face - encoded the ...
7
votes
Practical usefulness of PCA
First, from the perspective of education, PCA is a good entryway to the world of dimension-reduction techniques and associated methods. Whether we're talking ICA, non-negative matrix factorization, ...
1
vote
Can I use K-Means to group customers based on a single variable?
Given your description, I would just assign cutoff points at percentiles of the distribution of total spend. With five categories, equal intervals (same number of observations in each category) would ...
- 21
1
vote
Does feature selection and model testing have to be coupled in each fold of the cross-validation?
Yes, you should run the entire pipeline for each fold
You are right to say that using feature selection on all samples would mean you have data leakage in the following classification cross-validation ...
- 405
18
votes
Practical usefulness of PCA
One important use of PCA is in analysis of electroencephalography (EEG) data. To measure an EEG, dozens of electrodes are attached to your scalp and measure electric currents in your brain, either at ...
5
votes
Practical usefulness of PCA
One application of PCA that I have used a few times is the construction of social indicators. We use the projection of each observation (usually households) over a component axis (usually the first), ...
5
votes
What is the algebra showing the logistic and log loss to be equivalent?
Given $y_i\in\{-1,+1\}$, $z_i\in\{0,1\}$ and $z_i = (y_i+1)/2 \iff y_i = 2z_i-1$.
Also,
$$p(y_i\equiv1) = p(z_i\equiv1)= \left(1+\exp(-w^Tx_i)\right)^{-1}\\
p(y_i\equiv-1) = p(z_i\equiv0)= \left(1 + \...
- 17k
0
votes
Why does a neural network perform poorly in case of small loss?
As you get out past iteration $6000$, it seems like the train and test loss values are stable, as is the distance between them, which looks small. However, both values are tiny compared to the ...
- 46.6k
0
votes
Why does a neural network have the same output for every item in a batch?
I met the same issue as you. I tried to fine-tune a large language model with more than millions of parameters but it outputs exactly the same for each batch. Finally, I figured out that I used a too-...
- 21
1
vote
Accepted
Is the adjusted R square good for the interpretation prediction accuracy of machine learning regression model and how to interpret it?
An adjusted $R^2$ is problematic in practice for many machine learning models, since the degrees of freedom range from complicated to calculate to totally unclear. Combine that with the fact that many ...
- 46.6k
1
vote
How do you deal with imbalanced data when you're doing regression?
Your model is kind of behaving how it should. It knows that values are likely to be at the low end of the range of observed values. Can you fault the model for making predictions that reflect this ...
- 46.6k
0
votes
Why does sklearn list weighted precision and micro precision separately if they are the same thing?
I assume you mean why "macro" and "weighted" precision are same in the above example. "weighted" precision is actually a weighted version of "macro" precision. ...
2
votes
What are some good calculus resources relevant for Machine learning researcher aspirant?
It's not a book and not addressing optimization, but one of the best resources to self-learn calculus are the lectures by Gilbert Strang that were recorded and are available on YouTube. He also wrote ...
Tim♦
- 128k
12
votes
What is the algebra showing the logistic and log loss to be equivalent?
Consider the case when $y_i = -1$ in the logistic loss and $y_i = 0$ in the log loss. The summand in the logistic loss becomes $$\log\left(1 + \exp(w^Tx_i)\right)$$ and the summand in the log loss ...
- 3,657
0
votes
How to deal with training and test data that have different imbalances?
It is not so unusual for in-sample and out-of-sample data to have differences in the class ratios, just by flukes of randomly sampling to allocate observations to the in-sample and out-of-sample data (...
- 46.6k
2
votes
Two definitions of logistic loss function?
Quick Take: it turns out that the two are equivalent, so it does not matter which you use as long as you are clear about what the terms mean and what numbers you input into the equations.
Let's break ...
- 46.6k
1
vote
Accepted
HiClass: Modelling a Hierarchical Classifier
I am the main developer of HiClass.
Apparently, from the URL you linked you are using a third-party package from globality-corp, which is developed by someone else. This is the correct link for ...
- 26
3
votes
How can a statistician be relevant today?
The role of the statistician continues to be essential. I would say that there is a shortage of people with good statistical thinking. Testing hypotheses is required in any scientific rigorous ...
7
votes
How can a statistician be relevant today?
Statisticians often work in some form of consultation. As you said, many people need to validate a certain hypothesis, in science, medicine, etc. A statistician can analyze the data, but more ...
5
votes
What is the roadmap to self-taught probability and statistics for artificial intelligence?
If you were an academic, one must assume you already have a good reference for multivariable calculus, linear algebra, and differential equations – these are not optional. I personally heard from ...
3
votes
Are we estimating the Bernoulli parameter in Logistic Regression?
Logistic regression tries to fit a model such as $$p(x_i)=\frac{1}{1+e^{-(\beta_0+\beta_1 x_i)}}$$ or equivalently with the log-odds
$$\log_e\left(\frac{p(x_i)}{1-p(x_i)}\right)=\beta_0+\beta_1 x_i$$ ...
- 34.6k
7
votes
Accepted
Are we estimating the Bernoulli parameter in Logistic Regression?
The logistic regression model is a kind of generalized linear model, so it consists of the linear predictor
$$
\eta = \boldsymbol{\beta}X
$$
we pass it through the inverse of the link function $g$ (...
Tim♦
- 128k
1
vote
What does the error in artificial neural network stand for, is the same with mean square error (MSE)
It depends on what loss or error function you use when you code the network.
If your loss is the sum of squared errors (SSE), then divide this value by the number of predictions being made.
If your ...
- 46.6k
2
votes
Machine learning method(s) to compare probability of success of two groups
You are looking for a probabilistic classifier. You can feed in your predictor data for the two groups (so the predictors will presumably only differ in the group membership) and get the success ...
- 109k
0
votes
How to predict both category and sub category in machine learning classification?
With neural networks, you would do this by having multiple outputs:
the (logit)-probability of being in each category from a to d (that's the easy bit with not too many choices)
AND one of the ...
- 27.2k
0
votes
Accepted
Meaning of "reconstruction error" in PCA and LDA
It seems like there are three valid definitions in the case of PCA:
Matrix-wise L2: Whole matrix $\bf{X}$
$ || \bf{X} - \bf{X_r} ||$
Row-wise L2: Feature vectors on dataset $\bf{Y}$:
$ \sum_i{|| \...
- 316
0
votes
Least-square fit with uneven distribution of data
For this problem, I found this article to be of great interest.
For what I understood, there is usually 2 ways to deal with such non well distributed dataset :
either we resample the dataset, usually ...
- 1
3
votes
Accepted
LGBM fails to overfit
The problem is that you did not actually create any trees, because by default you need in each branch at least 20 records (but you only have 32 records in total) so that no branching happens. It helps ...
- 27.2k
5
votes
Is more data really always better in machine learning?
You are right, it is not only about the size of the dataset. As two other answers pointed out, having more data (vs very little) is desired, as even in a noiseless scenario it may help you to get a ...
Tim♦
- 128k
5
votes
Is more data really always better in machine learning?
My intuition is that, given $(x_{i},y_{i})_{i=1}^n$ and $(x_{i},y_{i})_{i=1}^N$ have the same "information" (I know this is a fuzzy term), using $(x_{i},y_{i})_{i=1}^N$ should not better the ...
- 61.2k
Top 50 recent answers are included