New answers tagged predictive-models
2
The GLMsData package contains the datasets used as examples in the book by Peter Dunn and myself (Dunn and Smyth, 2018).
The lime dataset is used in the book as an example of gamma regression. The main model used a log-link with Foliage as the reponse and a linear model that included a linear regression on log(DPH) for each level of Origin. The book included ...
3
The error says that glmnet doesn't have a vcov method so it can't compute confidence intervals. Not that you would want to anyway, the (intentional) bias would mean the confidence intervals would likely not have the appropriate coverage probability (if they covered the estimand at all).
2
The data (dry foliage biomass in kg) are not count data, so you shouldn't use either Poisson or quasi-Poisson responses.
The first thing to try is usually a regular linear model:
m1 <- lm(Foliage ~ DBH+Age+Origin, data = lime)
library(performance)
check_model(m1)
This is pretty terrible (if the first row [nonlinearity and heteroscedasticity] is bad, you ...
3
Replacing parameter values with their best estimates produces a plug-in estimated sampling distribution. This is your estimate of the true data generative process. This can be used for producing point estimates of population percentiles. A tolerance interval is simply a confidence interval for a population percentile. Think of the confidence intervals ...
0
Remember the definition of $R^2$.
$$
R^2 = 1 - \dfrac{\sum\big(y_i - \hat y_i\big)^2}{\sum\big(y_i - \bar y\big)^2}
$$
The numerator of that fraction is proportional to the $MSE$, and the denominator is proportional to the variance of the $y$ observations.
Your objection is to getting an $R^2$ that looks good yet also getting an $MSE$ that looks big.
If that ...
0
I focus on the ROC as an metric for discrimination ability of prediction model.
In the general ROC estimation, we should build a logistic regression model and estimate unconditional probability of a positive response for each observation. For example:
set.seed(63126)
n <- 1000
x <- rnorm(n)
pr <- exp(x)/(1+exp(x))
y <- 1*(runif(n) < pr)
mod &...
0
Suppose $x_t$ follows a stationary AR(1) process,
$$
x_t=\varphi x_{t-1}+\varepsilon_t.
$$
An infeasible optimal* forecast of $x_{t+1}$ is $\hat{x}_{t+1|t}=\varphi x_t$. If we can estimate $\varphi$ with sufficient precision (which we often can, given a sufficiently large sample), this forecast will be better than a forecast that only depends on $\bar{x}$ ...
0
You could also create further categorical features from the numerical BMI like 'very underweight' up to 'highly overweight'. It might make it easier for your model to get improved predictions.
It might also make sense to change the algorithm. Why do you want to rule out algorithms like SVM and XG Boost before trying to tune them? These algorithms might get ...
0
Your analysis should always be guided by existing knowledge and theory. Don't blindly run many models and pick the "best" one, even if you cross-validate, because you may simply be overfitting to the test set.
In my understanding, there is a reasonable assumption that there may be an interaction between smoking and BMI, so it would make sense to ...
2
The Weibull model is unique in that it satisfies both the accelerated failure time assumption (multiplicative survival curves) as well as the proportional hazards assumption (multiplicative hazard curves). This should explain why the difference in performance between these models is essentially indiscernible. Can you elaborate on the x-axis in each of your ...
3
That’s fine. A standard example of this is the Shapiro-Wilk test, which has a null hypothesis that the data come from a normal distribution and an alternative hypothesis that the data do not come from a normal distribution.
Keep in mind, however, that hypothesis testing is literal, and, given a large sample size, will (correctly) indicate statistical ...
3
There's a simpler way to tackle it, using the Law of Iterated Expectations. For notational clarity, I will define $y$ as "future $x$", so your first line would be $\mathbb{E}(y|x) = \dots$. Writing out the Law as applied to this case gives us:
$$\mathbb{E}[y|x] = \mathbb{E}_{\theta | x} \mathbb{E}[y|\theta]$$
We have $\mathbb{E}[y|\theta] = n\...
0
In comments, OP writes
I was thinking I'd try to generate new cards for existing trading card games, for example. The data set is all the pre-existing cards that have been printed. I would like to be able to specify some parameters and have the model generate the rest based on the data set
This is crucial information! People have used RNNs to generate ...
0
There are many ways of representing a 3D rotation as some vector in $\mathbb{R}^n$. However, many of these representations have the disadvantage that arbitrarily small rotations / change in rotation can lead to large differences the vector representation. For example, with euler angles, a small movement near the poles might be represented as a 180-degree ...
0
I think it would be sufficient to model the probability of passing through the funnel conditional on having passed through the previous step (e.g. probability of placing an item in the cart conditioned on you coming to the site, or the probability of purchasing conditional on having put something in your cart). In principle, this should mean running a ...
5
A useful rule of thumb for logistic regression is to limit yourself to about 1 unpenalized predictor per 15 cases of the minority class. See Section 4.4 of Frank Harrell's course notes, for example. That's when you have a
typical problem in medicine, epidemiology, and the social sciences in which the signal:noise ratio is small.
See for example, this page ...
0
To me it seems to be a data problem. You are splitting the data 70:30, but are all data from the 70% prior to data from the 30% set?
It can be a problem if you mix older and newer data in the training set. If time is involved in the generation of data, which seems to be the case as you have live data, test set should never contain data that were generated ...
0
I second Stephan's answer that the likely culprit is overfitting the entire dataset. That said, another thing to validate is that there are no differences between data processing pipelines in your training vs. production code. E.g. are you normalizing the features before training? If so, do you record the means and standard deviations and apply the same ...
1
I'm not familliar with the Pearson Residual Test so I will exclude it.
It has been recommended here that Hosmer-Lemeshow not be used assess calibration of logistic regression or other probabilistic models due to low statistical power and uninformative p value with respect to the type/extent of miscalibration. A better way to assess calibration would be to ...
11
It's hard to say without digging deeply into your model and your data. However, it seems like you have been doing a lot of cross-validation, model tuning, cross-validation, model tuning and so forth. That, together with bad out-of-sample performance, suggests that you are overfitting to your test set. That is harder than overfitting in-sample (which is easy ...
0
Your sample size is very small, I would not be surprised to see such a wide CI. E.g. if you have 10x the size with the same proportions of smoking/disease and change your code accordingly, your ORs will be 11 (95%CI 5.32-22.76). You should probably do some power calculations for your study to check if it is enough to make any conclusions.
1
Without seeing your actual data and especially what tool you are using, it is very hard to give a conclusive answer. Note that calculating confidence intervals for odds ratios is not straightforward and usually involves some large-sample approximations. These, in turn, are likely not warranted with your small dataset, so your tool may be doing something else,...
1
$p(x_{new}|x)$ is the posterior predictive distribution. It can be computed as an integral over the parameter $\theta$ as stated in your question, in fact this is just an application of marginal probability density functions (https://en.wikipedia.org/wiki/Marginal_distribution): $p(x_{new}|x)=\int_\theta p(x_{new},\theta|x) d\theta$.
$p(x_{new})$ is the ...
0
You may want to double-check this is correct... You could exract the means and CI of each group at a reference grid and plot the results. The package emmeans should do most of the job.
I use here the Orthodont dataset with a model similar to yours:
library(lme4)
library(ggplot2)
library(emmeans)
data(Orthodont,package="nlme")
fit <- lmer(...
0
Inclusion of predictors because of "statistical significance" is an invalid approach, and changes in c-index (concordance probability; AUROC) is a poor choice of measures because of lack of sensitivity/statistical power. Pre-specification of models is a better idea. As for indexes of predictive performance look at this.
0
You can use a squared error loss function for XGBRegressor, which is the default often. A squared error will weigh heavily any outliers in your dataset. This is also the reason people will typically try to get rid of the outliers, because they such a big influence on the final result. If you want to emphasize them even more I guess you can go with a cubic ...
2
700K and 1.3M is pretty balanced to me, usually people are talking about 1:1000 or even worse as imbalanced data. For example, fraud detection, it can be 1:100000.
In addition, Decision tree (and most machine learning model, such as logistic regression) works fine with the imbalanced data set. Just pay attention to evaluation metrics, and use precision and ...
1
Engineering failure-time analysis is very similar to demographic analysis of human survival. In both cases there is typically an early failure rate (infant mortality) and an aging process. In engineering circles the aging process is typically fit to a Weibull distribution, and for analysis of units in service it would be the aging portion of the data that ...
1
This sounds like a Poisson or negative binomial process. You could model this using a generalized linear model accounting for the exposure of each part. Newly added parts would have shorter exposure, older parts would have longer exposure. To investigate a time trend you could include time as a factor in the model. This could be performed in SAS Proc ...
1
I wouldn't filter out the data.
Reason being is that you will make your model biased towards predicting the optimum. Look into the bias-variance trade-off. Hence, I would drop the filtering option.
Your last reasoning however seems good.
However, in the beginning, you mentioned that you are interested in predicting a continuous variable. But you are ...
10
Consider a population $Y|X$ that follows some distribution according to a true model, and you have a set of trained models $f(X,\theta)$ that make predictions of $Y$ given $X$ and are parameterized by $\theta$.
The goal is to find out what the error of the models is, in making predictions about samples from the population, as function of the parameter $\...
0
The typical rule of thumb is to use modes (principal components, "PCs") if their associated eigenvalue is greater than one. However, it might be better to use an eigenvalue threshold based on the Marcenko-Pastur law:
\begin{equation}
\lambda^+=\left(1+ \sqrt{\frac{p}{n}} \right )^2
\end{equation}
where $p$ is the number of variables and $n$ is the ...
Top 50 recent answers are included