-1
$\begingroup$

I have a dataset which is a size of 50k x 280 i want to find linear dependent column indices so that i can eliminate them, rank of my matrix is 249 so what i understand that there are 31 columns are linearly dependent, i know a method to get directlty indices pf LI columns by sympy's rref() method but it is taking so long to get indices, as i know that we can also get LI columns by pearson coefficient but how to choose which columns are LI by looking at matrix of size 280 X 280 of P values.

$\endgroup$
0
0
$\begingroup$

Try sympy's rref() on a simple random sample of 1000 observations. This should speed up the process by 500x. The downside is that you may get some false positives, but I think that's pretty unlikely. If you do see 32 or 33 free variables, you can just try a different random sample and take the intersection.

You can't rely on pearson correlation coefficients since they'll only find pairwise relationships. If, for instance, column_1 = 3*column_2 + 4*column_3, you won't see that by looking at pearson correlation coefficients.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.