The Kolmogorov-Smirnov (K-S) test can be used to compare other distributions, so if you have enough data in the fourth category, I think you should be able to compare its distribution to the distribution(s) of the data in the other categories. It seems unlikely that you will have enough power to reject the null with only four data points in the fourth category's distribution though. I doubt this would provide sufficient power for most NHSTs, but there may be exceptions.
I think you should probably know already if it's safe to assume your four data points in the fourth category were sampled randomly from a normal distribution. That is, if you don't know that it's safe to assume this for prior theoretical and methodological reasons, it probably isn't safe to assume. However, the degree and manner of deviation from a normal distribution is more important in most cases than the simple, binary outcome of a NHST of your distribution's normality. You may want to consider a related, popular question here on Cross Validated: Is normality testing 'essentially useless'?
Here's a useful quote I found in an answer to that question:
As a rule of thumb (not a law of nature), inference about means is sensitive to skewness and inference about variances is sensitive to kurtosis.
This suggests you might at least wish to test the skewness or kurtosis separately, focus on whichever is more relevant to the subsequent analysis you have in mind, and pay attention to the value of the skewness or kurtosis statistic itself, not just the $p$ value of whatever NHST to which you might've felt obligated to cede inferential authority.
Nonparametric analyses do not assume non-normal distributions. Most also possess power and reliability that match equivalent parametric tests. Therefore you probably haven't got much to lose by just defaulting to the nonparametric alternative and skipping the normality test entirely. The main tradeoffs are usually computational complexity and the need to provide references due to methodological unconventionality, but all this necessitates are a modern computer and a little extra reading and writing, so it's probably generally advisable to just go for the nonparametric alternative. This depends on what analysis you have in mind though, so you may want to look into the drawbacks of the particular nonparametric alternative you have in mind for yourself.
Last, I don't see how you'd do any additional harm in switching to a Shapiro-Wilk (S-W) test, but if you want to compare its result to those of the K-S tests, I'd say you should also perform S-W tests on those distributions so you have some idea of how the method variance affects your test statistics within each sample distribution. Also, bear in mind that the S-W test is directly contraindicated by many of the answers to the aforementioned question...but I assume many of those objections would apply similarly to a K-S test that references a normal distribution.
