Return to Answer

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of the $ k^{th}$ row of Pascal's triangle is $ 2^k$.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 = AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension that we eliminate this means that the mean loses a row and the covariance loses both a row and a column. We can state this as

$ k \left( d\right)= d^2+d$.

of

$k \left( d+1 \right) - k \left( d\right) = 2 d + 2$

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

so we have:

$ n_1 A +2(k_2+2d+2) = n_2 A +2k_2 $

solving for the increase in samples with dimension gives:

$ n_2- n_1 = (2(k_2+2d+2) - 2k_2) A^{-1} = (4 d+4 ) \cdot A^{-1} $

So what is the scaling function? Lets assume that for a 2-dimensional multivariate Gaussian the number of samples required is 15 per parameter. There are 2 means and 4 elements of the covariance therefore 6 parameters or 90 samples. The difference is 60 samples, the value of $ A^{-1} = 5$.

At this point I would say that the heuristic starts a little low but ends up being about 2x the number of samples required. Its range of best utility, in my personal opinion, is around 4 dimensions or so.

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of the $ k^{th}$ row of Pascal's triangle is $ 2^k$.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 = AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension that we eliminate this means that the mean loses a row and the covariance loses both a row and a column. We can state this as

$ k \left( d\right)= d^2+d$.

of

$k \left( d+1 \right) - k \left( d\right) = 2 d + 2$

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

so we have:

$ n_1 A +2(k_2+2d+2) = n_2 A +2k_2 $

solving for the increase in samples with dimension gives:

$ n_2- n_1 = (2(k_2+2d+2) - 2k_2) A^{-1} = (4 d+4 ) \cdot A^{-1} $

So what is the scaling function? Lets assume that for a 2-dimensional multivariate Gaussian the number of samples required is 15 per parameter. There are 2 means and 4 elements of the covariance therefore 6 parameters or 90 samples. The difference is 60 samples, the value of $ A^{-1} = 5$.

At this point I would say that the heuristic starts a little low but ends up being about 2x the number of samples required. Its range of best utility, in my personal opinion, is around 4 dimensions or so.

EDIT:

So I have read the answer of @whuber and I like it. It is empirical, and in this case that is authoritative. I voted for his answer.

In the following I am attempting to discuss and hoping to be able to use more than ~300 characters, and I am hoping to be able to embed pictures. I am therefore discussing within the bounds of the answer. I hope this is okay.

I am at this point not convinced that use of AIC for this, or how sample size and parameter sizes were used was incorrect.

Next steps:

• replicate @whuber's results, confirm them empirically
• Test the AIC, at least in some ensemble sense, to confirm whether it is appropriate
• If AIC is appropriate, then try to use empirical methods to chase down flaw in reasoning.

Comments and suggestions welcome.

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of the $ k^{th}$ row of Pascal's triangle is $ 2^k$.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 = AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension that we eliminate this means that the mean loses a row and the covariance loses both a row and a column. We can state this as

$ k \left( d\right)= d^2+d$.

of

$k \left( d+1 \right) - k \left( d\right) = 2 d + 2$

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

so we have:

$ n_1 A +2(k_2+2d+2) = n_2 A +2k_2 $

solving for the increase in samples with dimension gives:

$ n_2- n_1 = (2(k_2+2d+2) - 2k_2) A^{-1} = (4 d+4 ) \cdot A^{-1} $

So what is the scaling function? Lets assume that for a 2-dimensional multivariate Gaussian the number of samples required is 15 per parameter. There are 2 means and 4 elements of the covariance therefore 6 parameters or 90 samples. The difference is 60 samples, the value of $ A^{-1} = 5$.

At this point I would say that the heuristic starts a little low but ends up being about 2x the number of samples required. Its range of best utility, in my personal opinion, is around 4 dimensions or so.

If I have to edit this answer again, I'm going to delete it.

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of the $ k^{th}$ row of Pascal's triangle is $ 2^k$.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 = AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension that we eliminate this means that the mean loses a row and the covariance loses both a row and a column. We can state this as

$ k \left( d\right)= d^2+d$.

of

$k \left( d+1 \right) - k \left( d\right) = 2 d + 2$

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

so we have:

$ n_1 A +2(k_2+2d+2) = n_2 A +2k_2 $

solving for the increase in samples with dimension gives:

$ n_2- n_1 = (2(k_2+2d+2) - 2k_2) A^{-1} = (4 d+4 ) \cdot A^{-1} $

So what is the scaling function? Lets assume that for a 2-dimensional multivariate Gaussian the number of samples required is 15 per parameter. There are 2 means and 4 elements of the covariance therefore 6 parameters or 90 samples. The difference is 60 samples, the value of $ A^{-1} = 5$.

At this point I would say that the heuristic starts a little low but ends up being about 2x the number of samples required. Its range of best utility, in my personal opinion, is around 4 dimensions or so.

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of that row of Pascal's triangle is 2^k.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 := AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension we eliminate the mean loses a row and the covariance loses both a row and a column. $2 k_2 + 2= k_1$.

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

Some algebra yields:

$ n_2 - n_1 = (2 k_{lower} +4) \cdot A^{-1} $

The only assumption is that one case has fewer dimensions than the other and that the error is consistent over samples.

If we assume that for a univariate Gaussian the number of samples is 30 then we can start populating the table.

Some brief numerics gives the following error distributions for the fit of 30 samples created from a standard normal distribution then fit to a univariate Gaussian.

The quartiles are indicated. It is assumed that this level of variation is desired in the multi-dimensional case.

I don't have the time to beat up MatLab to get the total result, so I will share my "rule of thumb". The 30 is provided as a rule of thumb, or heuristic so it is assumed that heuristics are not unacceptable.

My heuristic is to use Pascal's triangle multiplied by the univariate case.

If I am using 2d data then I go to the 2nd row and sum it to get 2x the number of samples, or 60 samples. For 3d data I go to the 3rd row and sum it to get 4x the number of samples or 120 samples. For 5d data I go to the 5th row and sum it to get 16x the number of samples, or 480 samples.

Best of luck.

EDIT:

It was intuitive, but everything has to be defended in math. I can't just take leaps from formulation of polynomial forms from Finite Elements with experience to get a ballpark.

The equation for the sum of the $ k^{th}$ row of Pascal's triangle is $ 2^k$.

My idea for the approach here is to equate the AIC of a higher-dimensional distribution with more samples to a reduced dimensional distribution with fewer samples.

The Akaike Information Criterion (AIC) is defined as $ AIC = n \log( \frac {RSS}{n}) + 2*k$ where $ RSS$ is residual sum of squares, $ n$ is sample count, and $ k$ is parameter count for the model.

$ AIC_1 = AIC_2$

$ n_1 \log(\frac {RSS_1}{n_1}) +2k_1 = n_2 \log(\frac {RSS_2}{n_2}) +2k_2$

For each dimension that we eliminate this means that the mean loses a row and the covariance loses both a row and a column. We can state this as

$ k \left( d\right)= d^2+d$.

of

$k \left( d+1 \right) - k \left( d\right) = 2 d + 2$

Assuming the error per sample point is constant relates the residual sum of squares to the sample count, and the term in the logarithm stays constant. The difference in sample count becomes a scaling constant.

so we have:

$ n_1 A +2(k_2+2d+2) = n_2 A +2k_2 $

solving for the increase in samples with dimension gives:

$ n_2- n_1 = (2(k_2+2d+2) - 2k_2) A^{-1} = (4 d+4 ) \cdot A^{-1} $

So what is the scaling function? Lets assume that for a 2-dimensional multivariate Gaussian the number of samples required is 15 per parameter. There are 2 means and 4 elements of the covariance therefore 6 parameters or 90 samples. The difference is 60 samples, the value of $ A^{-1} = 5$.

At this point I would say that the heuristic starts a little low but ends up being about 2x the number of samples required. Its range of best utility, in my personal opinion, is around 4 dimensions or so.

If I have to edit this answer again, I'm going to delete it.