Questions tagged [underdetermined]

Analyses are underdetermined when the number of parameters to be estimated is greater than the number of data. This problem is also referred to as 'p >> n'.

Filter by
Sorted by
Tagged with
3 votes
0 answers
35 views

Gauss-Markov with $p>n$

Let $p$ be the number of parameters in a linear regression model, let $n$ be the number of observations, and let $p>n$. $$\mathbb E[Y\vert X] = \beta_0 +\beta_1X_1 +...+\beta_pX_p$$ Does the Gauss-...
user avatar
  • 31.3k
0 votes
0 answers
15 views

Underdetermined system is fit with constant residuals

I am observing a strange (constant) residual error in my linear model. PROBLEM: I fitted a linear model (LM) $y=Ax$, where $A \in \mathbb{R}^{n \times p}$, with $n < p$. Hence, $A$ is ...
user avatar
  • 123
1 vote
1 answer
77 views

Quantifying how underdetermined a system of equations or optimization problem is

In linear systems we have an exact solution when we have as many equations as unknowns and the equations are linearly independent, e.g., $$ x_0 + 2 x_1 = 5 \\ x_0 + 3 x_1 = 7 \\ $$ has the unique ...
user avatar
0 votes
0 answers
19 views

Bayesian Approach for Underdetermined Datasets

If Bayesian Linear Regression with Gaussian prior produces L2 norm and Laplacian Prior produces L1 norm, is it fair to say that handling of underdetermined data sets (where number of columns > ...
user avatar
4 votes
0 answers
43 views

What is scikit-learn's LinearRegression doing when there are more features than observations? [duplicate]

I'm trying to understand what sklearn's LinearRegression (which should be using ordinary least squares) is doing when there are more features than observations. ...
user avatar
  • 41
1 vote
1 answer
123 views

How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
user avatar
  • 153
1 vote
0 answers
166 views

Appropriate dimensionality reduction technique for a small, but high-dimensional sample

I am attempting to conduct some multivariate analysis on a dataset I've been given with a sample size (n) of 23 and a feature number (p) of ~800. I would like to use dimensionality reduction, but ...
user avatar
0 votes
1 answer
170 views

Best estimate of underdetermined system using prior

I have measured two variables which depend on the same set of four parameters. I want to know the parameters which best explain my measurements. Of course, I cannot solve for four unknowns from just ...
user avatar
  • 103
3 votes
1 answer
169 views

Linear discriminant analysis with $p\gg n$

I am studying Linear Discriminant Analysis (LDA). According to the formula for LDA, we are supposed to get the inverse of within group covariance. However, if $p\gg n$ (i.e., the dimension is much ...
user avatar
  • 235
14 votes
2 answers
4k views

Can one (theoretically) train a neural network with fewer training samples than weights?

First of all: I know, there is no general number of sample size required to train a neural network. It depends on way too many factors like complexity of the task, noise in the data and so on. And the ...
user avatar
  • 165
5 votes
3 answers
2k views

Why is $n < p$ a problem for OLS regression?

I realize I can't invert the $X'X$ matrix but I can use gradient descent on the quadratic loss function and get a solution. I can then use those estimates to calculate standard errors and residuals. ...
user avatar
  • 1,669
4 votes
1 answer
205 views

Fitting least squares when number of predictors are larger than instances

A statement from the book Introduction to Statistical learning with applications in R, didn't quite make sense to me. It says, "In cases when number of predictors are greater than the instances we ...
user avatar
  • 622
0 votes
0 answers
165 views

tall, wide, big, small - qualitative descriptions

I was reading this link (link) where the authors say For "tall" data we cover a wide range of... and For "wide" data we discuss various forms... There is of course the trough-of-...
user avatar
  • 8,280
5 votes
0 answers
1k views

How to identify a SEM with formative dependent variable (with R's lavaan package)?

I have a formative construct in a structural equation model (SEM) which I would like to estimate with the function sem in the ...
user avatar
  • 305
0 votes
1 answer
1k views

Do I have too many variables and not enough data points for cluster analysis?

I have 75 observations and 152 variables. I want to perform cluster analysis. If I perform cluster analysis and this data will the results be meaningful? Do I need to reduce the number of variables ...
user avatar
  • 43
3 votes
0 answers
412 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
user avatar
  • 31
6 votes
4 answers
2k views

Solving a practical machine learning problem

I am currently doing my Phd in computational biology at Stanford. I get the data I need to answer the questions I am interested in. The data sets are sometimes "large" and these large problems take ...
user avatar
  • 2,519
2 votes
1 answer
207 views

Dataset for Least Angle Regression

I have read that least angle regression is good for high dimensional data. I didn't actually understand the meaning of high dimensional data, so does this mean $p>>n$ case? And does anyone know ...
user avatar
  • 123
13 votes
1 answer
4k views

Applying ridge regression for an underdetermined system of equations?

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$ can be written as \begin{equation} \begin{array} &\operatorname{min}\ \| y - ...
user avatar
  • 779
1 vote
0 answers
248 views

Interpretation of regression coefficients obtained from applying left inverse of regressor matrix in an underdetermined system?

If $X^\dagger$ is the pseudo-inverse of $X$, $\beta = X^\dagger y$ is the least squares solution for $\beta$ when $y=X\beta$. In the overdetermined case, applying $X^{\dagger,L} = (X^TX)^{-1}X^T$ ...
user avatar
  • 779
2 votes
1 answer
250 views

Dealing with underdetermination in Bayesian models

Bayesian models are supposedly well equipped to deal with high-dimensionality problems, and can handle sparse data well, too. But suppose I've created a model that estimate more parameters than there ...
user avatar
31 votes
1 answer
22k views

Feature selection & model with glmnet on Methylation data (p>>N)

I would like to use GLM and Elastic Net to select those relevant features + build a linear regression model (i.e., both prediction and understanding, so it would be better to be left with relatively ...
user avatar
  • 565
5 votes
3 answers
4k views

SVM has relatively low classification rate for high-dimensional data even though 2-D projections show they are separable

I have another problem with 14000 features and 500 training samples. It is a binary classification problem and approximately in the form of an ellipse. My classification accuracy using the 2nd degree ...
user avatar
  • 149
5 votes
2 answers
1k views

Why is it bad if number of dimensions / factors > sample size?

I've been told (read) this many times, but I never understood why it's bad for the number of dimensions in your data, or the number of explanatory variables in your model to be higher than your ...
user avatar
  • 869
31 votes
5 answers
12k views

Detecting significant predictors out of many independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...
user avatar
  • 2,423