New answers tagged

1

So I have this same question coming up for myself. I don't have a good answer, but I'd like to throw up a couple possibilities to see how people shoot them down. 1.) Process in blocks. Take the regression for each block, doing some kind of weighted average at the end. 2.) Take advantage of svd of the covariance matrix of A having some relationship to the ...


3

The p-values are what they are. Your large sample size, however, will (correctly) flag tiny differences as statistically significant, even though the difference might be too small to be interesting. The lesson here is that p-values should not be used to guide you toward differences that are large; that would be effect size. What p-values do is suggest that ...


0

What test(s) can I use to determine whether this is a valid p-value or whether this happened because the sample size was large? Nothing. There are investigations you may be able to perform to determine if your model is bad, but not if the significance is significant. As sample sizes become very large, Frequentist tests become very sensitive to small ...


1

At $\alpha$ significance level, an irrelevant variable should only be significant in a fraction $\alpha$ of random samples, regardless of the sample size. I think you were simply unlucky with your random variable generation. If your repeated it 100 times, you should only find about 5 instances where the variable is significant at 5% level. I indeed ran it ...


1

SQL is a query language used by many different databases. “Big data“ is a broad, catch-all, term for many different, pretty diverse, technologies. It’s like asking if it is better to learn Spanish, or “Asian languages”. Moreover, many of the query languages that can be used with big data databases are based on SQL, do you need to learn it first. I don’t ...


Top 50 recent answers are included