PCA is IMHO not a suitable technique for your task - at least not only PCA, and not PCA followed by an arbitrary classifier. You basically ask for a classifier that focuses only on the class boundary and not on higher concentrations and is yet not too noisy at the boundary.
Your neurotransmitters are substances that come at a concentration and you expect the signal to be a function of that concentration. This puts your data primarily into a regression context, even if your ultimate goal is "only" qualitative analysis. Regression models will be able to make use of higher concentrations to get a better estimate of the component spectra (assuming the concentrations aren't too different - the chemical species of your neurotransmitter should be the one you have at your LOD).
Also consider whether whether the almost-classical characterization of your SERS method with multivariate limits of detection and quantitation (LOD and LOQ) would be more appropriate than a direct classification. After all, you ask about detection of substances.
Multivariate LOD and LOQ are far from straightforward or standard approaches (though analytical expressions exist under certain assumptions).
But you can always go back to Kaiser's basic definitions of the LOD (concentration at which you correctly detect presence in 95% of the cases) - this allows for straightforward testing, regardless of whether you can use any of the analytical expressions.
Even for SERS as first approximation you can probably assume increasing signal with slightly increasing concentration and approximate this by a linear model. Thus, you may want to look into PCR (principal components regression) or PLSR (partial least squares regression). These would allow you to establish a threshold signals for clear absence, unclear and clear presence of your sample.
You'll also need to decide whether you want to set up distinct models for each neurotransmitter or one model that covers all your neurotransmitters. Which approach is more sensible depends on your application: can/will all neurotransmitters appear in the same sample, or do you know beforehand that only one or a certain selection of them can possibly be present?
My problem is that characteristic neurotransmitter peaks begin to disappear at lower concentrations; some peaks disappear while other peaks are very weak relative to higher concentrations.
Is this a signal-to-noise ratio problem or a chemical problem (e.g. adsorption equilibrium)?
If it is "only" the signals vanishing under noise, that is what is expected and what e.g. regression + LOD determination are exactly about: which concentration do I need to reliably detect the analyte signal?
If it is chemical, it will boil down again to LOD, but now it is not only "pure" (linear) signal to noise but a question of at which concentration do I get enough adsorption to detect a SERS signal?
If it is chemical in that at low concentrations you actually have a different chemical species in the vicinity of your surface, then you may want to consider capturing this species as well in your model.