@AryaMcCarthy do you have any source for the expected classification accuracy? Internet search left me empty handed.

Don't scatterplot 40K points just like that - they are drawn on top of each other, and you don't know if you have a lot of points in some region, or very few. Do something else. One simple trick is to set alpha=0.1. If it is still not clear, then switch to a different visualization, like seaborn.pydata.org/examples/hexbin_marginals.html.

In that case cross-validation is the way to go 👍

Lasso is more subtle than just normal regression + removing coefficients under a certain value. I find it surprising, that you get the same result. My advice would be to double check the code to make sure you don't have any bugs, and if not, post a new question.

This is a good answer (+1), but the core idea is delayed until the the fourth paragraph. Maybe it would be better to open the answer with "If the distribution family of your sample is unknown, use a density estimator".

Hi scikitnoob and welcome! Your question is a very well known one, and goes by the name of "variable selection". It has been discussed on this site multiple times already, Check out this one stats.stackexchange.com/questions/122931/…. It has a recommendation to use lasso as an answer and some more links in the comment to the question.

The second derivative here is $2$, not $2c$ :)

@AhmMMM I don't think your independent variables matter. You want a Bernoulli, which is logistic regression, which is family=binomial(link="logit") in glm in R.

$X_i$ are independent right? And also, is this a self-study question? If so, please add a tag.