Skip to main content
5 votes
Accepted

What is the best epoch to evaluate the test images?

Surrogate losses are used precisely because measures like accuracy are flawed. You want to use a score that is minimized when the predicted class probabilities match the true class probabilities (i.e.,...
Frans Rodenburg's user avatar
3 votes
Accepted

How to generate 95% prediction interval around predictions from ML model?

THIS SEEMS TO BE AN OPEN PROBLEM. Let's look at some possible solutions and their drawbacks. First, you propose this Yhat +- 1.96 * std(residuals). Let's put that ...
Dave's user avatar
  • 65k
2 votes
Accepted

Efficient prediction using Lightgbm/XGBoost when varying single feature keeping the remaining constant

The problem is that in principle, boosting can model any kind of interaction between your fixed $C$ and your varying $z$, so even if $C$ is fixed, its particular setting may have any impact on the ...
Stephan Kolassa's user avatar
2 votes
Accepted

Forecasting Survival Analysis

A Kaplan-Meier curve displays the survival data available to date, but you need to make assumptions to extrapolate. If you are willing to assume a particular parametric distribution, like the Weibull ...
EdM's user avatar
  • 95.7k
2 votes
Accepted

Is Mean Square Prediction Error acceptable to use if predicted values are continuous but actual observed values are discrete?

Absolutely yes. The (R)MSE elicits (=rewards) unbiased expectation forecasts. Expected sales could definitely be fractional, even if observed sales are integer. The simplest example would be if sales ...
Stephan Kolassa's user avatar
1 vote

How to generate 95% prediction interval around predictions from ML model?

Simple linear regression is parametric, since you model $Y = \beta X + \epsilon$ with the assumption of $\epsilon \sim N(0, \sigma^2)$. The prediction interval for a new $\hat y_h$ is bigger than the ...
qwr's user avatar
  • 548
1 vote
Accepted

Evaluating estimator of expected value plus variation

What you are looking for is probably loss/scoring functions for quantiles forecasts. Let me rephrase your statement "How long do you need such that there's an 80% chance the project is done in ...
picky_porpoise's user avatar
1 vote

Regression analysis with max value

If this is your actual data set, then you do not have enough to fit more than a linear or maybe a quadratic. Then you have the problem of a few points that are identical, and an outlier. If you cannot ...
Peter Flom's user avatar
  • 125k
1 vote

Surival Prediction - Train/Test data vs Production data

It isn't quite correct to say "these right-censored customers [are] the ones that I'm interested in generating a prediction for." You want to generate predictions for all individuals like ...
EdM's user avatar
  • 95.7k
1 vote
Accepted

Predictive Maintenance of factory parts

Given that you know the periodicity, I would suggest that you aim to make a model to predict the 24-hour sliding average, and another to predict the daily variation. For the latter, the training set ...
chrishmorris's user avatar
  • 1,855

Only top scored, non community-wiki answers of a minimum length are eligible