Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Logistic model - differences between development and validation

I have some data (eg. Titanic) and I want to use logistic regression to predict probability of survive. I have problem to understand the difference between model development and model validation. ...
0
votes
1answer
80 views

Validation of inverse problem solution based on Bayesian method

Recently, I read a paper about the inverse problem and parameter estimation. The main approach of the paper is based on the Bayesian method. The answer in this method is a posterior probability ...
0
votes
0answers
9 views

Model Validation for Aggregate Loss

I have loss data that consists of 134 severity (for 54 years, already adjusted for inflation). I want to create aggregate loss model from this data. I can sum up the severity (loss sizes) in each year ...
1
vote
1answer
30 views

Model evaluation, validation and verification

Reading scientific articles that have a statistical model, I sometimes find these three terms: Model evaluation Model validation Model verification From a statistical point of view, are all these ...
0
votes
0answers
11 views

Valid cases for time-based splitting except for time series forecasting

Let's say we are trying to train a propensity model that predicts churn or conversion (the probability to stop using a service or subscribe). For both models, we have a dilemma. On the one hand, for ...
0
votes
0answers
19 views

How to assess the performances of a Kalman Filter?

I have a Kalman Filter working in a simulation environment, therefore allowing me to plot the estimates against the ground truth. One way I assess the performances of my filter is to ensure that the ...
0
votes
2answers
11 views

training validation and test example

I have never used a validation set (I have used test sets, or holdouts). I THINK I understand them, but I have looked EVERYWHERE, and simply cannot find an example with numbers of how the validation ...
0
votes
0answers
27 views

How can I use validation set for hyper parameter tuning in a direct forecast time series problem?

I have a time series dataset with features corresponding to the same dates as the target. I'm using these features, along with date features such as month, week number etc to build a multivariate ...
0
votes
1answer
22 views

Definition of extra sample observation

I have been seeing this term used in a few books in machine learning, however, the authors never care to explain what that actually means, they just use it. I hope to understand this concept as it ...
0
votes
0answers
13 views

How important is out of sample validation for explanatory models

The majority of out of sample / cross validation techniques such as training and test splits I see examples of online are when the goal is prediction. How important is validating a model via ...
1
vote
1answer
24 views

Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
0
votes
2answers
108 views

What validation if KFold scores differ a lot? Repeated KFold, LOO or Holdout?

Suppose you are given a medium-sized dataset and you did a KFold validation once. You notice that scores on each old differ noticeably. Which validation type is the most practical? I thought about ...
0
votes
0answers
11 views

Is there a difference between these two validation measurements?

Say I have one large data set and 10 smaller data sets. What is the difference between the following two validation approaches, if any: Fit the model on the large data set once, and then test it on ...
0
votes
0answers
38 views

Decision tree learning curve - training accuracy does not decrease with increase in training set size

I am training a Decision Tree classifier. I was initially happy with the performance metrics of the model, but I wanted to plot the learning curve to get a better understanding of how my model is ...
0
votes
1answer
42 views

Selecting a model for linear regression: adjusted metrics (BIC, AIC, adjusted R2 etc...) on training set or validation/crossvalidation using test set?

Linear regression has model hyperparameters such as number of predictors. For example in a autoregressive time series model AR(p), p is the number of predictors. To find which value of p to find we ...
0
votes
3answers
212 views

train acc vs validation acc

i'm working on image segmentation problem in pytorch. using images with two classes ( background and the object to segment). So I divided the dataset into 3 parts: train 50% (199 images) val 25% (99 ...
1
vote
2answers
115 views

Comparison of Bayesian and Classical estimates

Is it correct to compare Bayesian and Classical estimates using Mean Squared Error (MSE)? MSE is a criterion that is used in the classical paradigm. For example: I am comparing the performance of ...
0
votes
0answers
20 views

Explaining and Addressing the Bias-Variance Tradeoff

Let's say we have a model whose training error is 7% and the validation error is 10%. What does this mean in terms of the bias-variance tradeoff? I know that high validation error and low training ...
5
votes
2answers
41 views

Internal validation steps

I am currently developing models for a prediction analysis. I have read through much of Harrell's Regression Modeling Strategies text and am confused on one point regarding internal validation. ...
0
votes
0answers
9 views

Is there a benefit to using batches in the validation and testing step?

Assuming that one is able to fit all samples and labels into memory at once, is there a benefit to using batches for the validation and testing steps? I know the proven benefit of batches at training ...
0
votes
0answers
3 views

Validation for time-series with limited history

I'm forecasting a number of macroeconomic time series with limited history (~20 years of month data). The standard approach to out-of-sample testing for time-series is to reserve the latter part of ...
0
votes
0answers
18 views

Learning method selection and train/validation/test

While there are many answers that explain the reasoning behind train/validation/split, these are usually concerned with one learning method. So there are two parts to this question: Part 1: Suppose I ...
5
votes
0answers
87 views

Official name of a common type of Bayesian simulation study

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model: For independent replication $i = 1, ..., n$: Draw a set of "true" parameters ...
1
vote
0answers
7 views

Resampling Out-of Time Sample makes sense?

I have independent training, validation and test sets, all from different years. So far i have trained my models and compared their results on the validation set. Im wondering if it makes sense to ...
2
votes
1answer
46 views

Demand forecasting: training a model using falsified historical sales as a target instead of actual sales

The framework: We have different products being sold for a short period of time (maximum 3 weeks) once a year. Some products may have no sales history - new products. Since this is not a classical ...
1
vote
1answer
92 views

Demand forecasting: what is the proper validation strategy to objectively compare a new algorithm with the one already used in the system?

Although the question is related to any problem related to demand forecasting, I would like to describe our framework: The framework: We have different products being sold for a short period of time (...
0
votes
0answers
17 views

Reporting cross-validated score and test score

Is this the right way to do cross-validation and testing of a machine learning model: Split my data into train (85%) and test (15%) Do hyperparameter tuning based on the train set Do 10-fold cross-...
0
votes
1answer
84 views

Are Jaccard score and F1 score monotonically related?

I have compared the rankings obtained by comparing 10+ classifiers with this two metrics: Jaccard score F1 score They show a perfect correlation. This results holds on 50+ datasets. When comparing ...
2
votes
1answer
18 views

Looking for scientific validation for a heuristic model

The problem: infer the nationality of a person from a limited number of features (name, email, ...). I do not have enough "ground truth" to use ML techniques, I'd like to try what for a ...
0
votes
0answers
33 views

How to validate hierarchical clusters in r

I have a dataset in which each row corresponds to a dive carried out by a whale ('id' in table below) and the columns to the variables calculated for each dive (maximum depth, duration, speed, etc.). <...
0
votes
1answer
39 views

Validation on independent biological dataset

I am working in the space of cancer statistics. I am looking for reasons why it is important to validate statistical observations in an independent biological dataset. Does anyone have a list or a ...
2
votes
1answer
31 views

How do I validate a regression model's inferences and not its predictions?

Suppose over many years I collect data $X$ on a quantity of interest $y$ and some control variables $Z$. I fit an OLS model $$y = \beta X + \delta Z + \epsilon $$ and use the coefficients $\beta$ and ...
0
votes
0answers
55 views

Selecting Binary Classification Probability Threshold [duplicate]

I have a binary classification problem I have modeled and I'm trying to determine the best way to select my probability threshold. Here was my modeling approach: Create a training and testing set. My ...
0
votes
1answer
57 views

What are some of the disadvantages of non-random split for external validation?

Validation procedures on non-random split samples (i.e., when samples are split by centre, region, location) can be considered as external validation. However, what are some of the drawbacks of using ...
1
vote
1answer
37 views

2-fold cv vs. validation set

Will 2-fold CV be the same as the validation set approach? To me it seems that the answer is no, since a 2-fold CV will go through the data set 2 times, while the validation set only will do the ...
6
votes
4answers
738 views

Train and Validation vs. Train, Test, and Validation

I am embarking on a new job that will give me the opportunity to do some cool machine learning stuff. I haven't touched this stuff on a deeper level since graduate school and I wanted to get some ...
0
votes
0answers
20 views

Validation dataset for a Gaussian process in a problem of time series forecasting

I am relatively new to Machine Learning, which I am trying to study to define a Gaussian process in a problem of time series forecasting. Then, for Gaussian processes, we wish to select a mean ...
0
votes
1answer
38 views

Is it necessary to retrain a random forest instead of removing trees when comparing accuracy between different numbers of trees?

I have a train data set and a validation set using which I wish to optimize the hyperparameter that is the number of trees in a binary classification random forest (scikit-learn). (As Sycorax ...
0
votes
0answers
48 views

Similarity Metric Validation

I want to score a number of similarity metrics, i.e. given a function s(x,y) which returns a number that is higher the more similar x and y are. I'm want to objectively score a number of different ...
0
votes
0answers
51 views

Model evaluation from a comparison between simulated density data and individual presence data

I built a mathematical model of range expansion for a mosquito species. The model output is a map (raster format) of simulated mosquito density (mosquitoes/km²) in a study area. Here is an overview ...
0
votes
0answers
20 views

Why my validation accuracy is 1, but training accuracy is low?

I use U-net model and input .nii File type My data set have 103 totally validation_split is 0.2 It means Train on 242 samples, validate on 61 samples batch-size is 6, and epoch is 300 But my accuracy: ...
0
votes
0answers
39 views

Name / packages for using bootstrap samples to validate a statistical formula

I am looking for the name for a statistical validation process where I can quickly validate a theoretical formula, which describes a quantity of interest over a large parameter space, using simulated ...
0
votes
0answers
5 views

Translation issue on validation study

I am conducting a translation and validation on an assessment tool. There are few questions that would like to ask for suggestions and sharings I prefer using forward translation with testing (...
1
vote
1answer
19 views

Is it a good way to present updated model coefficients by refitting the model using both derivation and validation data?

I am conducting a research on the development and external validation of a prediction model. I have developed a regression-based prediction model using derivation set data and externally validated it ...
0
votes
0answers
29 views

Combining the discovery and replication data sets to calculate final statistics?

I have a question about calculation and reporting of statistics combining data from replication studies. Our techniques don't involve ML, but ML techniques have me thinking about this. Often in ML ...
2
votes
1answer
153 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
0
votes
0answers
21 views

Is there a principled method for preventing overfitting of a model to the validation set?

Overfitting almost always implicitly refers to overfitting onto the training set, which could occur, for instance, when a model is trained for too long, where we see a dip in the validation accuracy ...
0
votes
0answers
61 views

Dice score Validation accuracy higher than training accuracy

I am using the unet model for fascia segmentation in tensorflow as: ...
1
vote
1answer
327 views

ARIMA Cross Validation

I work with R and have got some questions regarding my ARIMA model. In specific, I have yearly data ranging from 1946 to 2019 and would like to do a basic two-step ahead ARIMA forecast for 2020 and ...
1
vote
0answers
154 views

Large Mismatch between Validation and Test Set Performance

I have a time-series dataset, which is split into the following sections: First 50% purely for training Next 40% is for validation, split into 20-folds Last 10% is purely for testing This is a ...

1
2
3 4 5
16