2
$\begingroup$

Is there any number that we can land on for our regression model to predict with high accuracy? (accuracy metrics I have in mind at RMSE or R-squared). Also high accuracy may mean something above 88% to 90% accuracy with 95% confidence interval (I am not glued to any particular number)

In my current setup, we have to run some tests, but the issue is that we cannot collect a very large amount of dataset due to the limitation of time in running a single test. We have concluded that we will run around 80 tests (by calculating the avg. amount of time each test would take, plus the time needed to change the configuration etc.).

However, I am not convinced that this much amount of data points would sufficiently give us an accurate model.

Plus, for each test, we are running the test once only: My concern with this that the outcome we may have may be fluke. How many runs per test would nullify the chances of fluke statistically speaking?

$\endgroup$
10
  • 2
    $\begingroup$ What is your criterion of accuracy any way? For example, if the criterion is predict values exactly, most regression-type models fail absolutely. $\endgroup$
    – Nick Cox
    Nov 23 at 15:37
  • 2
    $\begingroup$ OK, but RMSE 90% of SD(y) and R-square of 90% are still different criteria. $\endgroup$
    – Nick Cox
    Nov 23 at 15:55
  • 1
    $\begingroup$ Two could work -- and millions might fail to be sufficient. At the very least you need to indicate how the prediction relates to the data, what the model is, how you measure accuracy, and what the standard errors are in your model. $\endgroup$
    – whuber
    Nov 23 at 19:03
  • 1
    $\begingroup$ How well-behaved is your data? If it's extremely noisy, you might need thousands of data points; if it's smooth, you could get away with two. $\endgroup$
    – Mark
    Nov 24 at 0:58
  • 1
    $\begingroup$ R^2 is a measure of noise, mostly. Accordingly, if your goal is to get a high R^2 for whatever unholy reason (please don't do that), you can estimate how the noise in your signal behaves and collect data accordingly. If the model is a poor fit for the data overall, it won't help, and if you would insist on pushing R^2 higher, overfit is all you will get. Seriously, stop chasing metrics senselessly, figure out what is the actual problem you are trying to solve. Stephen links an excellent community wiki in their answer: some problems are outright ill-defined and/or hopeless. $\endgroup$
    – Lodinn
    2 days ago

2 Answers 2

17
$\begingroup$

We can't tell you. It depends on your situation and how easy prediction is in your situation.

How many coin tosses do you need to observe before you can predict the next one with 90% accuracy?

Related: How to know that your machine learning problem is hopeless?

And of course, in many situations you can predict with "better than 90% accuracy" without learning at all, namely when one outcome occurs in more than 90% of cases - then just always predict that. For instance: always classify a credit card transaction as non-fraudulent. Most CC transactions are non-fraudulent, so such a useless prediction will look very good in terms of accuracy... because accuracy is not a good evaluation measure.

$\endgroup$
2
$\begingroup$

I can do this with zero data points. My estimate is 42. Say the true distribution is around 0 with a deviation of 10 (but it can be different as well). Given that, if you define the accuracy as 90%, then this estimate satisfies the condition.

example

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.