1
$\begingroup$

I'm working on a classification problem in which the underlying signal to identify is very hard to find.

I suppose that this is because the signal/noise ratio is very low.

My questions are fundamentally linked to my precedent question here:

1) Is there a way to calculate\estimate the signal/noise ratio? Given a set of features, and classes, for example?

2) If my signal/noise is very low, will increasing my training set size augment the signal/noise ratio? Could be also that the difficulties are bound to the features and to the type of problem? In which case is useless to increase the training set?

3) Concerning my precedent question on the threshold used for classify in neural network, a user (elkoul) said:

Usually lowering the threshold takes place when you care a lot about the metric called recall. For instance you develop an algorithm to detect terrorists in an airport and you want to find them all, even though some times you might identify normal people as terrorists. The metric that you are interested in this case is recall. On the other hand, increasing the threshold takes place when you care a lot about precision.

In my specific case, lowering the threshold increase the recall, but on the opposite, increasing the threshold don't increase the precision, leading to near or totally random results. (my false positive rate is not improving\lowering)

My AUROC(area under the Receiver Operating Characteristic Curve) on the problem is between 0.5 to 0.57 using different nets from a grid search. If my precision is not increasing with the threshold shifting, does this mean that I'm not learning anything?

$\endgroup$
1
$\begingroup$

1) You can always put a lower bound on Signal/Noise by getting yourself something like a pseudo-R^2 from a model (using out-of-sample predictions). Since there may be a better model, you can't know that that is your S/N, but the S/N cannot be lower (ignoring some small degree of sampling variation -- which is not super relevant unless you want to do actual inference on your S/N).

2) S/N is probably best viewed as a fundamental aspect of the DGP. The data is generated with a certain amount of intrinsic noise. Changing the size of your dataset won't affect that. However, holding S/N constant, larger datasets would let you find smaller signals. So more data is useful -- but it doesn't change the S/N, rather it affects your ability to deal with the S/N you are given.

3) I don't know the context of your prior question. In general, I don't think the lack of precision changes definitively implies that you aren't learning anything -- there are funky DGP/model combinations that could potentially lead to that. Having said that -- it is probably good evidence that you haven't learned much.

$\endgroup$
  • $\begingroup$ What do you mean as pseudo-R^2 from a model? $\endgroup$ – Nikaidoh Mar 7 '18 at 11:24
  • $\begingroup$ I don't know that it matters which pseudo-R^2 you pick if that's what you're asking. My thoughts specifically are below: R^2 is a measure of signal to noise, which is made by looking at how much your predictions improve relative to a prediction of just the mean. However, the definition of R^2 leans heavily on a linear regression setup. A Pseudo-R^2 (in my mind) would compare the variance of the (again-- ideally out of sample) errors to the variance overall -- giving a measure of S/N, but one not limited to a linear regression. $\endgroup$ – user5957401 Mar 7 '18 at 17:56
  • $\begingroup$ If you can link me some reference, I'm going to accept your answer :) $\endgroup$ – Nikaidoh Mar 14 '18 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.