0
$\begingroup$

I have selected SMS Spam Collection as my dataset for natural language processing task. I have done many pre-processing tasks on dataset such as removing punctuations, spell correction, stemming, and then I trained the model using the Naive Bayes classifier from sklearn library. The precision was 98 percent.
Training the model on messy data also had a precision of 98 percent. why is that happening? What is wrong with my task?
I have added my Jupyter Notebook file here.

$\endgroup$
0
$\begingroup$

Firstly, after quickly looking to your notebook I don't see there exactly results that you mentioned - comparison of results of the model trained on messy data and cleaned. What I saw in the notebook is that you checked the model trained on whole corpus on the same corpus and then created 80/20 train/test split and received the same precision 0,98 (but with some details in detailed classification report).

But even if you calculate precision more precisely it would be:

0,9826812634601579325197415649677

and

0,97921076233183856502242152466368

Secondly, in such unbalanced datasets, it's common to have high precision so when calculated with only two digits it's expected that it might be the same for two approaches.

Moreover, precision alone is not the best measure to compare two models. It's a topic longer than a SE answer but I would start with accuracy and then with cross validation (and then comparing two sets of results of CV leads to statistical testing / hypothesis).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy