The short answer is yes, you need to assess your model's performance on data not used in training.
Modern model building techniques are extremely good at fitting data arbitrarily well and can easily find signal in noise. Thus a model's performance on training data is almost always biased.
It is worth your time to explore the topic of cross validation (even if you are not tuning hyperparameters) to gain a better understanding of why we hold out data, when it works, what assumptions are involved, etc. One of my favorite papers is:
No unbiased estimator of the variance of k-fold cross-validationNo unbiased estimator of the variance of k-fold cross-validation