19,670 reputation
12876
bio website biostat.mc.vanderbilt.edu/…
location Nashville, TN
age
visits member for 3 years, 5 months
seen 45 mins ago

I am Professor of Biostatistics and Chairman of the Department of Biostatistics at Vanderbilt University School of Medicine, Nashville TN USA. I am Associate Editor of Statistics in Medicine, a member of the Faculty of 1000 Medicine, and a member of the policy advisory board for the Journal of Clinical Epidemiology. I am a Fellow of the American Statistical Association. I am author of Regression Modeling Strategies (Springer, 2001). My specialties are development and validation of predictive models, clinical trials, observational clinical research, cardiovascular research, technology evaluation, clinical epidemiology, medical diagnostic accuracy, biomarker research, pharmaceutical safety, Bayesian methods, quantifying predictive accuracy, missing data imputation, and statistical graphics and reporting. I am a long-time user of R. In August 2014 I was given the WJ Dixon Award for Excellence in Statistical Consulting by the American Statistical Association. Among many other things, Dr Dixon was the lead developer of the first general-purpose statistical software package, BMD.


14h
answered Multiple imputation for missing values
19h
answered Graph with 2 interacted continuous predictor vatiable
1d
revised Adjust for everything you have in propensity score?
added comment about matching
1d
revised Adjust for everything you have in propensity score?
edited tags
1d
answered Adjust for everything you have in propensity score?
1d
awarded  Nice Answer
2d
comment Should train and test datasets have similar variance?
There are papers by Ewout Steyerberg on this topic. In my experience for a binary outcome you need more than 15,000 subjects for split-sample validation to work satisfactorily. The bootstrap and cv work well for any size dataset, and you don't also need an independent sample when you use them unless you think the data collection process or the type of observation has changed. If the datasets are really large the confidence interval for the accuracy score computed in the independent sample is narrow enough to make the point estimate trustworthy.
2d
comment Training, testing, validating in a survival analysis problem
I addressed that in the post you just put on another topic page.
2d
comment Should train and test datasets have similar variance?
Unless both training and independent test sets are huge, resampling performs better. You just have to make absolutely sure that the resampling procedure repeats all steps that utilized $Y$, afresh for each resample.
Sep
12
comment Should train and test datasets have similar variance?
If you use a proper or semi-proper accuracy scoring rule (e.g., you don't use the ill-advised proportion classified correctly) the .632 bootstrap (which is implemented in R rms) is not needed and you can use the ordinary optimism bootstrap. If you did not do variable selection then why did you mention 'model selection criteria'?
Sep
12
comment Should train and test datasets have similar variance?
You have uncovered a big problem with cross-validation: the choice of the fold sizes. Use the bootstrap instead, which requires only the choice of the number of resamples (I suggest about 500 here). You didn' explain why you need variable selection. That is causing much of your problem. Variable selection is arbitrary and does not usually help with overfitting. For your sample size you badly need a penalized MLE method.
Sep
12
comment Should train and test datasets have similar variance?
Cross-validation is validating the model. Don't use a single test set. If you need to tweak the model in a way that requires cross-validation to determine the tweak (not usually recommended) then you will need nest cross-validation or double bootstrap.
Sep
12
revised What is the difference between GLM and splines?
edited tags
Sep
12
answered What is the difference between GLM and splines?
Sep
12
answered Should train and test datasets have similar variance?
Sep
12
comment why is there a huge difference existed in coefficient of determination obtained from 10-fold cross validation?
Provide the sample size, distribution of $Y$, and total number of parameters being fitted or entertained.
Sep
11
comment When to report proportions and when to report median + IQR for ordinal variable
A good question. Probably a histogram. Regarding your first question, the discrete variable has to not only have a lot of levels overall but to have a lot of levels in the vicinity of the quantile you are estimating.
Sep
11
answered Use of data from ROC curve
Sep
10
comment How do you validate your machine learning models?
I would never dream of not using probabilities. They capture close calls and minor errors, as opposed to the all-or-nothing classification approach. They also lead us to consider proper scoring rules so that one optimizes the right criterion. I don't have a single best reference on the subject - perhaps an article in JASA on proper scoring rules - but this is covered in a multitude of places.
Sep
10
revised When to report proportions and when to report median + IQR for ordinal variable
edited tags