72 votes
Accepted

What is "reduced-rank regression" all about?

1. What is reduced-rank regression (RRR)? Consider multivariate multiple linear regression, i.e. regression with $p$ independent variables and $q$ dependent variables. Let $\mathbf X$ and $\mathbf Y$ ...
amoeba's user avatar
  • 104k
45 votes
Accepted

How to perform isometric log-ratio transformation

The ILR (Isometric Log-Ratio) transformation is used in the analysis of compositional data. Any given observation is a set of positive values summing to unity, such as the proportions of chemicals in ...
whuber's user avatar
  • 321k
39 votes
Accepted

What's in a name: Precision (inverse of variance)

Precision is often used in Bayesian software by convention. It gained popularity because gamma distribution can be used as a conjugate prior for precision. Some say that precision is more "...
Tim's user avatar
  • 138k
38 votes
Accepted

What test can I use to compare slopes from two or more regression models?

To answer these questions with R code, use the following: 1. How can I test the difference between slopes? Answer: Examine the ANOVA p-value from the interaction of Petal.Width by Species, then ...
Kayle Sawyer's user avatar
36 votes
Accepted

What is the difference between univariate and multivariate time series?

Univariate time series: Only one variable is varying over time. For example, data collected from a sensor measuring the temperature of a room every second. Therefore, each second, you will only have a ...
Renan Fonteles's user avatar
26 votes
Accepted

Correlated Bernoulli trials, multivariate Bernoulli distribution?

No, this is impossible whenever you have three or more coins. The case of two coins Let us first see why it works for two coins as this provides some intuition about what breaks down in the case of ...
fuglede's user avatar
  • 378
25 votes
Accepted

categorizing a variable turns it from insignificant to significant

One possible explanation would be nonlinearities in the relationship between your outcome and the predictor. Here is a little example. We use a predictor that is uniform on $[-1,1]$. The outcome, ...
Stephan Kolassa's user avatar
25 votes

Why is correlation only defined between two variables?

Pearson correlation is defined as a measure of the linear relationship between two variables. For other relationships, like multidimensional relationships, we use other names. For instance: one could ...
Sextus Empiricus's user avatar
24 votes
Accepted

Why does component-wise median not make sense in higher dimensions?

The underlying concept is that a median splits the data (or a distribution) into two halves with equal amounts in each half (by count or probability). Even in one dimension the median is problematic. ...
whuber's user avatar
  • 321k
19 votes
Accepted

Is there a multivariate two-sample Kolmogorov–Smirnov test?

A 2004 article On a new multivariate two-sample test by Baringhaus and Franz maybe helpful, they provided a brief literature review on the two-sample multivariate GoF tests and then a R package ...
Francis's user avatar
  • 3,150
19 votes

Why is correlation only defined between two variables?

Correlation is defined between more than two variables, through a correlation matrix. This is not a single number of course, but that is only natural given that it is describing correlation between ...
Ben's user avatar
  • 123k
18 votes
Accepted

Multivariate vs Multiple time series

Multiple time series is just that: Multiple series instead of a single series. Multivariate time series is usually contrasted with univariate time series, where each observation at a time $t$ is a ...
Skander H.'s user avatar
  • 11.8k
17 votes

What's in a name: Precision (inverse of variance)

Precision is one of the two natural parameters of the normal distribution. That means that if you want to combine two independent predictive distributions (as in a Generalized Linear Model), you add ...
Neil G's user avatar
  • 15.1k
16 votes
Accepted

Discriminant analysis vs logistic regression

When the classes are well-separated, the parameter estimates for logistic regression are surprisingly unstable. Coefficients may go to infinity. LDA doesn't suffer from this problem. If there are ...
JohnK's user avatar
  • 20.2k
15 votes
Accepted

Multivariate log-normal probabiltiy density function (PDF)

Just for the sake of completeness, I'll provide an answer here. This is a simple application of the multivariate change of variables theorem: say $Y = \Phi(X)$ where $\Phi$ is a smooth bijective ...
icurays1's user avatar
  • 492
15 votes
Accepted

Robust covariance and OGK outlier detection

To justify the OGK, you need to see the following chain: Outlier detection and robust estimation are essentially equivalent problems, At its core, outlier detection is related to the problem of (...
user603's user avatar
  • 22.5k
14 votes

In cluster analysis should I scale (standardize) my data if variables are in the same units?

Some simple and obviuos, universal considerations for multivariate analysis, including clustering. Case 1. Incomparable units. Height vs weight. You cannot compare, so the default decision is to ...
ttnphns's user avatar
  • 57.2k
13 votes

What are variable importance rankings useful for?

This is completely anecdotal, but I've found variable importance useful in identifying mistakes or weaknesses in GBMs. Variable importance gives you a kind of huge cross-sectional overview of the ...
Dex Groves's user avatar
  • 1,663
13 votes

Why is correlation only defined between two variables?

Such a statistic would be hard to define and interpret. Say you have the variables $A$, $B$, and $C$. The pairwise correlation between $A$ and $B$ is close to $+1$ and the pairwise correlation between ...
Tim's user avatar
  • 138k
13 votes

Why is correlation only defined between two variables?

why do we only evaluate the correlation between two variables and not more than two variables? It can be more than 2 variables. Three point correlation function (3PC) is used in cosmology, The Three-...
patagonicus's user avatar
  • 2,520
12 votes

Two-dimensional Kolmogorov–Smirnov

Python implementation I have written a python implementation using numpy. You can find the code here, you may find more infomation in the docstring in the code. And here's another one (not by me). ...
Syrtis Major's user avatar
12 votes
Accepted

Restricted Maximum Likelihood (REML) Estimate of Variance Component

NB. I simplify notation somewhat and do not use bold typesetting. The following rules for matrix differentials are useful: \begin{align} d\log \vert A\vert &= \mathrm{tr}(A^{-1}dA) \\ dA^{-1} &...
KOE's user avatar
  • 4,521
12 votes
Accepted

Correction for multiple testing in Multiple regression analysis

(NB: I started to write this before the question was edited; I don't comment here on the "empirical aspect", i.e., what is done in the literature, but rather on what "should" be done.) This is a good ...
Christian Hennig's user avatar
12 votes
Accepted

How to determine if a sample is within a standard deviation of a multivariate normal distribution

The usual metric to use here is the scaled Mahalanobis distance: $$S(\mathbf{x}) \equiv \frac{D(\mathbf{x})}{\sqrt{n}} = \sqrt{\frac{(\mathbf{x} - \boldsymbol{\mu})^\text{T} \mathbf{\Sigma}^{-1} (\...
Ben's user avatar
  • 123k
11 votes

How to construct a multivariate Beta distribution?

It is natural to use a Gaussian copula for this construction. This amounts to transforming the marginal distributions of a $d$-dimensional Gaussian random variable into specified Beta marginals. The ...
whuber's user avatar
  • 321k
11 votes
Accepted

sequential/recursive/online calculation of sample covariance matrix

It's easy if you write $$ \hat\Sigma_n= \frac{1}{n-1}\sum_{i=1}^nX_i X_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T. $$ Split up the sum over $n$ elements into two parts. One will involve the first $n-...
Taylor's user avatar
  • 20.4k
11 votes
Accepted

Is there a mathematical definition of non-collapsibility?

Yes, there is, for instance, you can find one in Pearl, definition 6.5.1. Consider any functional $g[P(x, y)]$ of the joint distribution of $X$ and $Y$. We say $g$ is collapsible on $Z$ if: $$E_z\...
Carlos Cinelli's user avatar
11 votes
Accepted

Confidence regions on bivariate normal distributions using $\hat{\Sigma}_{MLE}$ or $\mathbf{S}$

Assume first the the parameters $\boldsymbol\mu$ and $\boldsymbol\Sigma$ are known. Just as $\frac{x-\mu}\sigma$ is standard normal and $\frac{(x-\mu)^2}{\sigma^2}$ chi-square with 1 degree of ...
Jarle Tufto's user avatar
  • 10.7k
11 votes
Accepted

Name of classification algorithm based on gaussian distributions estimated from data?

Probably Quadratic Discriminant Analysis. There are also names for different constraints you could make: Covariance matrices of both classes are equal - Linear Discriminant Analysis. Only diagonal ...
Karolis Koncevičius's user avatar
11 votes
Accepted

Why is it bad if the estimates vary greatly depending on whether we divide by N or (N - 1) in multivariate analysis?

The comment seems to be a way of saying that we like large sample sizes in machine learning. The numerator is the numerator, whether you divide by $N$ or $N-1$, so all that matters to our discussion ...
Dave's user avatar
  • 61k

Only top scored, non community-wiki answers of a minimum length are eligible