2
$\begingroup$

I'm working on a classification problem in which the underlying signal to identify is very hard to find.

I suppose that this is because the signal/noise ratio is very low.

My questions are fundamentally linked to my precedent question here:

1) Is there a way to calculate\estimate the signal/noise ratio? Given a set of features, and classes, for example?

2) If my signal/noise is very low, will increasing my training set size augment the signal/noise ratio? Could be also that the difficulties are bound to the features and to the type of problem? In which case is useless to increase the training set?

3) Concerning my precedent question on the threshold used for classify in neural network, a user (elkoul) said:

Usually lowering the threshold takes place when you care a lot about the metric called recall. For instance you develop an algorithm to detect terrorists in an airport and you want to find them all, even though some times you might identify normal people as terrorists. The metric that you are interested in this case is recall. On the other hand, increasing the threshold takes place when you care a lot about precision.

In my specific case, lowering the threshold increase the recall, but on the opposite, increasing the threshold don't increase the precision, leading to near or totally random results. (my false positive rate is not improving\lowering)

My AUROC(area under the Receiver Operating Characteristic Curve) on the problem is between 0.5 to 0.57 using different nets from a grid search. If my precision is not increasing with the threshold shifting, does this mean that I'm not learning anything?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

1) You can always put a lower bound on Signal/Noise by getting yourself something like a pseudo-R^2 from a model (using out-of-sample predictions). Since there may be a better model, you can't know that that is your S/N, but the S/N cannot be lower (ignoring some small degree of sampling variation -- which is not super relevant unless you want to do actual inference on your S/N).

2) S/N is probably best viewed as a fundamental aspect of the DGP. The data is generated with a certain amount of intrinsic noise. Changing the size of your dataset won't affect that. However, holding S/N constant, larger datasets would let you find smaller signals. So more data is useful -- but it doesn't change the S/N, rather it affects your ability to deal with the S/N you are given.

3) I don't know the context of your prior question. In general, I don't think the lack of precision changes definitively implies that you aren't learning anything -- there are funky DGP/model combinations that could potentially lead to that. Having said that -- it is probably good evidence that you haven't learned much.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What do you mean as pseudo-R^2 from a model? $\endgroup$
    – Nikaido
    Commented Mar 7, 2018 at 11:24
  • $\begingroup$ I don't know that it matters which pseudo-R^2 you pick if that's what you're asking. My thoughts specifically are below: R^2 is a measure of signal to noise, which is made by looking at how much your predictions improve relative to a prediction of just the mean. However, the definition of R^2 leans heavily on a linear regression setup. A Pseudo-R^2 (in my mind) would compare the variance of the (again-- ideally out of sample) errors to the variance overall -- giving a measure of S/N, but one not limited to a linear regression. $\endgroup$
    – user5957401
    Commented Mar 7, 2018 at 17:56
  • $\begingroup$ If you can link me some reference, I'm going to accept your answer :) $\endgroup$
    – Nikaido
    Commented Mar 14, 2018 at 15:22
0
$\begingroup$

SNR = μ/σ

Where, μ - Mean, σ - Standard Deviation, SNR - Signal to Noise Ratio.

If given a set of features, I would get the fraction of explained variance by interested feature to use it as variance in the formula with mean across this feature. Formalar really resembles just normalization of the mean.

If you are interested in the mean signal overall, the situation can be complicated if features interaction occurs - GWR (geographically weighted regression) could be needed with further approximation to define mean over all variability/variance)

p.s. concerning PR-ratio & threshold behaviour - see your PR-curve. If it has strange shape - you have either not enough data or not correct model. If the shape is OK - your threshold moving on it will act logically

Improve this answer
$\endgroup$
2
  • $\begingroup$ Another equation seen in lit: SNR=Variance of Signal/Variancs of Noise . With "A high SNR (>> 1) indicates a high precision measurement, while a low SNR indicates very noisy data." $\endgroup$
    – JeeyCi
    Commented Aug 17, 2022 at 3:44
  • $\begingroup$ ... ref.lit link about PCA, here as well assumptions for PCA. And here shown Orthogonal Rotation while PCA. I Consider this concept to be the main direction of SNR use $\endgroup$
    – JeeyCi
    Commented Aug 17, 2022 at 4:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.