Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

How important is out of sample validation for explanatory models

The majority of out of sample / cross validation techniques such as training and test splits I see examples of online are when the goal is prediction. How important is validating a model via ...
0
votes
1answer
10 views

Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
0
votes
2answers
28 views

What validation if KFold scores differ a lot? Repeated KFold, LOO or Holdout?

Suppose you are given a medium-sized dataset and you did a KFold validation once. You notice that scores on each old differ noticeably. Which validation type is the most practical? I thought about ...
0
votes
0answers
11 views

Is there a difference between these two validation measurements?

Say I have one large data set and 10 smaller data sets. What is the difference between the following two validation approaches, if any: Fit the model on the large data set once, and then test it on ...
0
votes
0answers
12 views

Decision tree learning curve - training accuracy does not decrease with increase in training set size

I am training a Decision Tree classifier. I was initially happy with the performance metrics of the model, but I wanted to plot the learning curve to get a better understanding of how my model is ...
0
votes
1answer
32 views

Selecting a model for linear regression: adjusted metrics (BIC, AIC, adjusted R2 etc…) on training set or validation/crossvalidation using test set?

Linear regression has model hyperparameters such as number of predictors. For example in a autoregressive time series model AR(p), p is the number of predictors. To find which value of p to find we ...
0
votes
3answers
45 views

train acc vs validation acc

i'm working on image segmentation problem in pytorch. using images with two classes ( background and the object to segment). So I divided the dataset into 3 parts: train 50% (199 images) val 25% (99 ...
1
vote
2answers
76 views

Comparison of Bayesian and Classical estimates

Is it correct to compare Bayesian and Classical estimates using Mean Squared Error (MSE)? MSE is a criterion that is used in the classical paradigm. For example: I am comparing the performance of ...
0
votes
0answers
20 views

Explaining and Addressing the Bias-Variance Tradeoff

Let's say we have a model whose training error is 7% and the validation error is 10%. What does this mean in terms of the bias-variance tradeoff? I know that high validation error and low training ...
5
votes
2answers
26 views

Internal validation steps

I am currently developing models for a prediction analysis. I have read through much of Harrell's Regression Modeling Strategies text and am confused on one point regarding internal validation. ...
0
votes
0answers
9 views

Is there a benefit to using batches in the validation and testing step?

Assuming that one is able to fit all samples and labels into memory at once, is there a benefit to using batches for the validation and testing steps? I know the proven benefit of batches at training ...
0
votes
0answers
3 views

Validation for time-series with limited history

I'm forecasting a number of macroeconomic time series with limited history (~20 years of month data). The standard approach to out-of-sample testing for time-series is to reserve the latter part of ...
0
votes
0answers
17 views

Learning method selection and train/validation/test

While there are many answers that explain the reasoning behind train/validation/split, these are usually concerned with one learning method. So there are two parts to this question: Part 1: Suppose I ...
3
votes
0answers
32 views

Official name of a common type of Bayesian simulation study

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model: For independent replication $i = 1, ..., n$: Draw a set of "true" parameters ...
1
vote
0answers
7 views

Resampling Out-of Time Sample makes sense?

I have independent training, validation and test sets, all from different years. So far i have trained my models and compared their results on the validation set. Im wondering if it makes sense to ...
2
votes
1answer
38 views

Demand forecasting: training a model using falsified historical sales as a target instead of actual sales

The framework: We have different products being sold for a short period of time (maximum 3 weeks) once a year. Some products may have no sales history - new products. Since this is not a classical ...
1
vote
1answer
55 views

Demand forecasting: what is the proper validation strategy to objectively compare a new algorithm with the one already used in the system?

Although the question is related to any problem related to demand forecasting, I would like to describe our framework: The framework: We have different products being sold for a short period of time (...
0
votes
0answers
13 views

Reporting cross-validated score and test score

Is this the right way to do cross-validation and testing of a machine learning model: Split my data into train (85%) and test (15%) Do hyperparameter tuning based on the train set Do 10-fold cross-...
0
votes
1answer
24 views

Are Jaccard score and F1 score monotonically related?

I have compared the rankings obtained by comparing 10+ classifiers with this two metrics: Jaccard score F1 score They show a perfect correlation. This results holds on 50+ datasets. When comparing ...
2
votes
1answer
17 views

Looking for scientific validation for a heuristic model

The problem: infer the nationality of a person from a limited number of features (name, email, ...). I do not have enough "ground truth" to use ML techniques, I'd like to try what for a ...
0
votes
0answers
20 views

How to validate hierarchical clusters in r

I have a dataset in which each row corresponds to a dive carried out by a whale ('id' in table below) and the columns to the variables calculated for each dive (maximum depth, duration, speed, etc.). <...
0
votes
1answer
20 views

Validation on independent biological dataset

I am working in the space of cancer statistics. I am looking for reasons why it is important to validate statistical observations in an independent biological dataset. Does anyone have a list or a ...
2
votes
1answer
29 views

How do I validate a regression model's inferences and not its predictions?

Suppose over many years I collect data $X$ on a quantity of interest $y$ and some control variables $Z$. I fit an OLS model $$y = \beta X + \delta Z + \epsilon $$ and use the coefficients $\beta$ and ...
0
votes
0answers
8 views

Selecting Binary Classification Probability Threshold [duplicate]

I have a binary classification problem I have modeled and I'm trying to determine the best way to select my probability threshold. Here was my modeling approach: Create a training and testing set. My ...
0
votes
1answer
16 views

What are some of the disadvantages of non-random split for external validation?

Validation procedures on non-random split samples (i.e., when samples are split by centre, region, location) can be considered as external validation. However, what are some of the drawbacks of using ...
1
vote
1answer
23 views

2-fold cv vs. validation set

Will 2-fold CV be the same as the validation set approach? To me it seems that the answer is no, since a 2-fold CV will go through the data set 2 times, while the validation set only will do the ...
6
votes
4answers
341 views

Train and Validation vs. Train, Test, and Validation

I am embarking on a new job that will give me the opportunity to do some cool machine learning stuff. I haven't touched this stuff on a deeper level since graduate school and I wanted to get some ...
0
votes
0answers
8 views

Validation dataset for a Gaussian process in a problem of time series forecasting

I am relatively new to Machine Learning, which I am trying to study to define a Gaussian process in a problem of time series forecasting. Then, for Gaussian processes, we wish to select a mean ...
0
votes
1answer
28 views

Is it necessary to retrain a random forest instead of removing trees when comparing accuracy between different numbers of trees?

I have a train data set and a validation set using which I wish to optimize the hyperparameter that is the number of trees in a binary classification random forest (scikit-learn). (As Sycorax ...
0
votes
0answers
23 views

Similarity Metric Validation

I want to score a number of similarity metrics, i.e. given a function s(x,y) which returns a number that is higher the more similar x and y are. I'm want to objectively score a number of different ...
0
votes
0answers
50 views

Model evaluation from a comparison between simulated density data and individual presence data

I built a mathematical model of range expansion for a mosquito species. The model output is a map (raster format) of simulated mosquito density (mosquitoes/km²) in a study area. Here is an overview ...
0
votes
0answers
15 views

Why my validation accuracy is 1, but training accuracy is low?

I use U-net model and input .nii File type My data set have 103 totally validation_split is 0.2 It means Train on 242 samples, validate on 61 samples batch-size is 6, and epoch is 300 But my accuracy: ...
0
votes
0answers
39 views

Name / packages for using bootstrap samples to validate a statistical formula

I am looking for the name for a statistical validation process where I can quickly validate a theoretical formula, which describes a quantity of interest over a large parameter space, using simulated ...
0
votes
0answers
5 views

Translation issue on validation study

I am conducting a translation and validation on an assessment tool. There are few questions that would like to ask for suggestions and sharings I prefer using forward translation with testing (...
1
vote
1answer
18 views

Is it a good way to present updated model coefficients by refitting the model using both derivation and validation data?

I am conducting a research on the development and external validation of a prediction model. I have developed a regression-based prediction model using derivation set data and externally validated it ...
0
votes
0answers
7 views

Combining the discovery and replication data sets to calculate final statistics?

I have a question about calculation and reporting of statistics combining data from replication studies. Our techniques don't involve ML, but ML techniques have me thinking about this. Often in ML ...
2
votes
1answer
61 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
0
votes
0answers
11 views

Is there a principled method for preventing overfitting of a model to the validation set?

Overfitting almost always implicitly refers to overfitting onto the training set, which could occur, for instance, when a model is trained for too long, where we see a dip in the validation accuracy ...
0
votes
0answers
22 views

Dice score Validation accuracy higher than training accuracy

I am using the unet model for fascia segmentation in tensorflow as: ...
1
vote
1answer
66 views

ARIMA Cross Validation

I work with R and have got some questions regarding my ARIMA model. In specific, I have yearly data ranging from 1946 to 2019 and would like to do a basic two-step ahead ARIMA forecast for 2020 and ...
0
votes
0answers
54 views

Large Mismatch between Validation and Test Set Performance

I have a time-series dataset, which is split into the following sections: First 50% purely for training Next 40% is for validation, split into 20-folds Last 10% is purely for testing This is a ...
0
votes
0answers
13 views

What would be the best interpretation of my loss function plot?

I am predicting the results of football games. I trained a 100 cycles. The code is available here. My interpretation is that the validation loss is at its is lowest already in the start, it only gets ...
1
vote
1answer
34 views

Assessing Logistic Regression and Determining if Splines Are Appropriate

I'm working on building a logistic model which will be used to estimate the probability that an account will skip on their monthly payment. My dataset roughly includes 50,000 observations with 15% of ...
0
votes
0answers
19 views

Uncertainty in categorical statistics such as accuracy, sensitivity, specificity

The samples I've drawn (from a larger population) can be categorised into several classes (false positive, false negative, true positive, true negative). I'm interested in inferring various statistics ...
0
votes
0answers
12 views

Multidimensional scaling/perceptual mapping validation question

Basically, I have this map that shows a collection of beer brands on a two dimensional map. One dimension represents how well-known the brand is and the other dimension represents the price. We have ...
3
votes
1answer
163 views

External validation of a published Cox PH model

My aim is to externally validate a risk prediction model published in the medical literature that is based on a Cox regression model. I have a dataset with all the variables from the score. I read ...
0
votes
0answers
15 views

Attribution modelling: Baseline estimation

My thesis is about measuring the impact of TV ads on online search volume. I have the schedule of tv ads, and the data of search volume represented in the picture by the blue plot. I need to calculate ...
0
votes
1answer
86 views

Why is my training curve and validation curve on the same level and how to find a proper architecture for an ANN?

I have build a neural network model. All in all it doesn't work properly, but since it is part of my Master Thesis I've to evaluate it and find some explanations. The learning curve looks like this: ...
0
votes
1answer
83 views

Why is my learning curve always at 100% while my validation curve increases with training samples?

I am relatively new to machine learning and have the following problem: I have built a random forest model which works relatively well and now I am trying to interpret the results. The learning curve ...
0
votes
0answers
19 views

Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more ...

1
2 3 4 5
15