0
$\begingroup$

I'm running a grid search, in order to fine-tune a NN hyper parameters. the question is: the MAE values I get from the trainings are too close. since I have the statistical attributes of the target values, is there a way to somehow come up with a starting value for MAE, where the worst regression can achieve.

I think the question is not clear. the value I'm looking for, is analogous to a probability for classification problems. (example: an neural network which is supposed to classify the inputs into 8 possible classes, will have accuracy of 0.125 just by random classification of inputs. so a 12.5% accuracy is the measure for such classifier) now, I have the Mean, and StdDev (and other stats if need be) of the target values of my samples. how can I calculate a measure to judge my MAE (and not by comparing different MAE values from different trainings)?

$\endgroup$
  • $\begingroup$ yes, MAE:mean absolute error. I'm tryna come up with a value for MAE, that even a non trained NN can achieve. (like the worst MAE. I know it's worst possible value is +inf. but wht I'm looking for is a value like E where I can validly say: "my NN should have MAE values of less than E. because even a untrained NN can achieve a MAE of E or less.") $\endgroup$ – Phisher May 9 at 7:42
  • $\begingroup$ I tried to explain the thing I'm looking for, by exemplifying the classification analogue. (an 8-class NN classifier, should do better than %12.5 accuracy. cuz any damn classifier can achive that 12.5 accuracy just by randomly choosing a class. (of course, assuming the class probability distribution is uniform)) $\endgroup$ – Phisher May 9 at 7:45
  • $\begingroup$ In a classification problem, the baseline scenario is not to emit randomly, but to emit default class distributions. For example if the distribution of classes in training data is 70% class A and 30% class B, then you output these figures not 50/50. In the regression case the baseline scenario is the mean only model for Squared Error, or median only model for Absolute Error. For this reason the most basic model always emits the median of the training values (median of the targets, not inputs), and you can calculate MAE from this value. $\endgroup$ – Cagdas Ozgenc May 9 at 8:50
  • $\begingroup$ THANK YOU. exactly it is. median for MAE. I suggest this should better be posted as answer than a comment, as it is the exact answer for the question. I calculated the median of targets array and MAE (assuming all predictions are median value) and it gives a values close to MAE of my untrained NN. so this is correct. $\endgroup$ – Phisher May 9 at 8:58
0
$\begingroup$

I think what you’re looking for is the median of the pooled data.

In a classification problem, the most naive way to find class probabilities is to use your training data. In MNIST, you have a 10% chance of each number. If you’re a doctor specializing in disease X than affects 7% of people, say that everyone has a 7% chance of having disease X.

The point of regression is to tighten up that estimate for a subject with some other values measured. Perhaps the doctor knows that 7% of people have the disease but that 70% of people who work around asbestos have the disease. If a patient works around asbestos, then 7% probably is the wrong probability to assume. Maybe 70% is wrong, but it’s probably more along the lines of the right answer than 7%. (Or maybe not. Maybe there’s some other covariate like smoking status or genetics that should be included in the model.)

When you do a regression with a numeric response variable, the naive way to guess the conditional parameter of interest (often the conditional mean, given covariates like asbestos), is to guess the estimate from the pooled distribution of all training data. This is why r-squared compares to the sum of deviations from the mean of the response variable, as that would be the most naive sensible way to guess. If your regression predicts median instead of mean, then your first guess would be the median of the pooled data. Ditto for other quantile regressions (e.g. 0.25 quantile).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.