0
$\begingroup$

I am a bit confused on how to calculate type 2 error to check whether the sample data I am using is sufficient. I have a data file that I have used to build a machine learning model. This data file consists of 500 entries describing information about entities and the funding these entities have received. The ML model uses these information to predict whether an entity will receive funding in the future.

Now I want to calculate type 2 error. In all the courses and tutorials I have read online they talked about two hypothesis. Usually they are two experiments or tests done over a different period of time or in different setups. But I was asked in this exercise to calculate type 2 error using one data file. I manage to calculate the lower and upper bounds of the null hypothesis; but then I stopped because I don't have the expected mean of the second hypothesis. I actually don't know what should be the alternative hypothesis in this case.

My questions are: is it possible to calculate type 2 error in this case? can I use the 500 entries to build the ML model and consider this the null hypothesis; then get another dataset that contains the same information but describing different entities and use it as the alternative hypothesis (this dataset will not necessarily be different in terms if time or any other parameter)?

$\endgroup$
0
$\begingroup$

Type 2 error in the context of science refers to incorrectly deciding that some data provides good evidence for a hypothesis that is not actually true. This really only makes sense if you have some hypothesis. Think about this as a "False Negative" in the context of science. A "Positive" is when we flag a hypothesis as untrue, and a "Negative" is when we decide there is nothing interesting to see here (broadly speaking). So a "False Negative" is when we incorrectly decide there is nothing interesting going on here and retain our null hypothesis.

(The null hypothesis typically states "There is no difference between these groups" or "This drug doesn't work".)

However, in the context you're talking about, you don't have an explicit hypothesis. Instead, you will want to use the same general concept to evaluate the performance of your model. This can be done by producing something called a confusion matrix on the results of the second dataset (you would indeed have to get some new values, or wait for these same groups to receive funding to check your predictions).

This Wikipedia page explains it quite well, and you can see from the Confusion Matrix table that it has labelled 'Type 2 error' which is what you are looking for. I am fairly new to machine learning, but I believe that it's a perfectly reasonable metric for estimating how well your model performs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.