Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
20 views

Measure that says how good an output is at classifying subjects

I have a function that produces a 1-dimension output that is used to classify subjects into two categories, A and B. It ranges from -4 to 4, and ideally, a threshold can be chosen so that subjects ...
0
votes
0answers
14 views

Interpreting Validation Curves to Fix Overfitting for Random Forests

i'm curently struggeling to interpret validation curves for my model. See the image. My training data set size is around 1200. I'm using 10-fold cv and the Random Forest Algorithm. Before i took a ...
0
votes
0answers
6 views

Strange learning and validation curve by training a CNN

by training a cnn and plotting the learning and validation curve as well as the training loss and validation loss I get strange results. What could be the reason for such a break-in?
0
votes
0answers
21 views

Pseudo $R^2$ low from val.prob calculation

I have just created a logistic model, and now I want to validate it. Therefore I am using the val.prob function from the rms ...
0
votes
0answers
14 views

Siamese Network: validation accuracy highly fluctuating

I am trying to train a Siamese Network with 1D CNN, where I calculate the absolute difference between the two latent vectors, and then pass it to a sigmoid neuron to determine if the two inputs belong ...
1
vote
0answers
20 views

Evaluating model that is in production without access to new, labeled data

I have a deep learning image classification model that is in production. I am trying to get a sense of model drift by calculating distribution metrics (e.g. mean predicted probability per label) over ...
0
votes
0answers
12 views

Cross-validation with binary response

I have a dataset (size = 36234) with 9 response variables, all of them binary, along with 3 continuous predictor variables. Since the response variables might be correlated, I'm using Ising to fit the ...
1
vote
1answer
51 views

About 10-fold cross validation train/test split

So, I want to do 10 fold CV. After I googled it, all of the websites I've found told me that to do the split, take 1 fold as test and the rest as train. But my professor told me another way. She told ...
0
votes
0answers
34 views

How to bootstrap a logistic regression model? [duplicate]

Currently I have a dataset with 4712 records focused on binary classification. I have divided my data into train(70%) and test (30%) split. I run a gridsearchcv on my train data and choose the best ...
0
votes
0answers
38 views

What is the relation between MSE of K-NN for a regression problem and LOOCV?

I trying to answer this question: Denote the MSE of K-NN for a regression problem: $𝐸_{𝑖𝑛} =\frac{1}{𝑛}\sum_{i=1}^n(𝑦_𝑖 − \frac{1}{𝑘}\sum_{j=1}^k𝑦_𝑗)^2$ , where for each $𝑦_𝑖$, the ...
0
votes
0answers
26 views

In feature selection, what the size of the data set is considered as too small? Is this an appropriate use of machine learning?

I am in a non-computer science field, and machine learning is being blatantly misused in my field. I recently got a journal paper to review, where the researchers used machine learning to develop a ...
0
votes
0answers
15 views

What's the good index to choose number of clusters so that obtained clusters are homogeneous?

I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster ...
1
vote
1answer
22 views

how to interpret the case when the cross validation accuracy is more than the model accuracy

I've trained a ANN model which resulted in 94.62%, but when I do a 5 fold cross validation the mean accuracy is 94.75%. Also 4 out of 5 cross validated models accuracy is more than 94.62%. How to ...
0
votes
0answers
9 views

How to validate Generalized Least Square model for longitudinal response

I have a dataset with body weights before and in the follow-up visits after surgery, for a group of patients with obesity. Our goal is to fit a model to predict weight loss throughout the follow-up. ...
1
vote
1answer
16 views

Should I use a validation set when not cross validating?

I have 3 kinds of models (Lineair regression, random forest regression and the gradient boosted forest regression). Normally I would apply CV to all 3 of them and use a validation set for that and use ...
0
votes
0answers
19 views

Validation loss is decreasing, accuracy is decreasing too

So, I have the following charts from my experience.Can any one explain why accuracy is decreasing while the loss in train and validation is decreasing? The point is that i can't early stop too in the ...
0
votes
0answers
14 views

Multi-objective optimization for finding regularization weight

Usually, we do k-fold cross-validation for tuning the regularization weight in an ML task. However, I have seen a couple of works on multi-objective optimization that they claim their algorithm can be ...
0
votes
1answer
25 views

is it okay to include the prediction data from k-fold cross validation for machine learning training?

Currently, I started to study machine learning to write an academic paper. Lets say I have 1000 data, and I split to 70:30 for training:testing. While training the machine learning (assume binary ...
1
vote
1answer
15 views

How to split training data when learning DNN for unknown test data?

I'm designing a CNN model for a data mining competition in which we are provided with N sample of training data. We do not know the test size, but presumably it is from the same distribution as ...
0
votes
1answer
51 views

Why does my OLS model produce high error on test set?

To create the feel of using my model to predict unknown data, I split the dataset I have into training set and test set. The $R^2$ value of my OLS regression model in the training set is decent (89.16%...
18
votes
4answers
2k views

Is leave-one-out cross validation (LOOCV) known to systematically overestimate error?

Let's assume that we want to build a regression model that needs to predict the temperature in a build. We start from a very simple model in which we assume that the temperature only depends on ...
0
votes
0answers
13 views

Unsure how to continuously train a churn model after my model has gone live

I'm having trouble describing this properly so I'll provide as many details as possible. First, here are a few details on the model that I am building: I have built a classification model that ...
1
vote
1answer
53 views

Common practice for validating classifiers in medical statistics

We are writing a paper, where we suggest a new classifier. We have a data base with about 1400 medical cases. Would it be sufficient just to divide the dataset into training 70% and test 30%? Or ...
0
votes
1answer
24 views

Estimate of an AR model

I have this part of a project which states this: Once you choose the best three models for each series (according to AIC, the PACF and ACF, and "from general to specific), the next step is to estimate ...
0
votes
1answer
41 views

Loss and Accuracy computation

I have a doubt about the computation of the loss and accuracy for the training / validation / test. I split my training set into batches, and I'm training my network with them for N epochs. My idea ...
14
votes
5answers
2k views

Can increasing the amount of training data make overfitting worse?

Suppose I train a neural network on dataset A and evaluate on dataset B (that has a different feature distribution than dataset A). If I increase the amount of data in dataset A by a factor of 10, is ...
0
votes
1answer
13 views

Sample Size to Validate if Report is Pulling Correct Information

I am having a report written on very complex EHR data where the data needed comes from different formats, from different locations. I need to choose a sample from about 20,000 accounts to manually ...
0
votes
0answers
5 views

Difficulty sampling for the Validation Dataset - Independent and Identically Distributed

I have data on horse races. This is for a project (I do not intend to gamble). Each record in my dataset is for horse/race combination. So if there are say 7 horses in a race I get seven records. ...
1
vote
2answers
58 views

What is the most appropriate way to validate prediction models with clustered data?

I am attempting to develop and validate a multivariable classification model using data from 10 clinical trials. I would like guidance on the most appropriate way to validate (internally and ...
-1
votes
1answer
17 views

Rather use a sample of a population or analyze a whole population but smaller?

I am validating an automated surveillance program. It is about infection incidence on a hospital ward. I could easily check if the small list of infections produced by the automated surveillance ...
1
vote
1answer
32 views

Detecting overfitting on multi-class classification model

I have seen this question asked in one flavor or another, but I'm looking for clarity on a more specific piece. I have two text classification models: Model A: train score=88%, test score=76% Model ...
8
votes
2answers
188 views

When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
0
votes
0answers
10 views

Exact interpretation of $\beta$ $F_{\beta}$ Score

The $F_{\beta}$ is defined as $F_{\beta} = \frac{(1 + \beta ^2) \cdot \text{precision} \cdot \text{recall}}{(\beta ^2 \cdot \text{precision}) + \text{recall}} $ Does a $\beta$ of 0.5 for example ...
0
votes
0answers
40 views

Comparing AUC, logloss and accuracy scores between models

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem: ...
0
votes
0answers
35 views

Considerations of surrogate end point measures

Data from various studies indicate that men who exercise are less likely to sustain a bone fracture compared to men who don’t exercise. For a number of reasons, it is difficult to use bone fracture ...
0
votes
0answers
8 views

How to plot the area under curve and the calibration curve for cox regression model applied to a test sample?

How to plot the area under curve and the calibration curve for cox regression model applied to a test sample using rms package? please
1
vote
1answer
32 views

Area under the curve in risk prediction model

The area under the curve for my data set is $0.63$. However, when I divided my data randomly into two parts, development (67%) and validation (33%), the value of the area under the curve became $0.58$...
0
votes
0answers
12 views

The next logical/legitimate step if a model performs poorly on testing set?

I have a fairly basic question but am sure that this is something every machine learning practitioner has encountered in real life. That is, what is a legitimate next step when the best model selected ...
0
votes
0answers
30 views

Average precision biased in object detection when validation set has easy examples

I am computing average precision (AP) for object detection as in Pascal VOC dataset. Sometimes my results were too good to be true and I suspected that I might overlook something. Then I realised that ...
0
votes
0answers
13 views

Explaining Profitability in Banks Using Panel Data Dynamic/Fixed Effects Model

I am currently looking to study the profitability of banks. My 4 independent variables, which are taken from banks balance sheet, are just financial ratios. My time is 13 Years, and I have 21 banks. ...
0
votes
1answer
24 views

Machine Learning - Training/Validation Sets

I have a very general question that I can't seem to get a straight answer on. Machine Learning - I understand how it works - you have your dataset for which you want to answer either a prediction or ...
1
vote
0answers
31 views

Can test results be considered reliable if it is not possible to have true out of sample validation data set?

We are working with an anonymized dataset (similar to commercially available consumer data) to create a binary classification based model. We have data for entire US population that is commercially ...
0
votes
0answers
18 views

How can underfitting and overfitting Classifiers of the same Type perform equally worse?

An rbf support vector classifier is instantiated with different values for C and gamma. Then learning curves are plotted for each of the classifiers. Some classifiers consistently achieve very high ...
0
votes
0answers
16 views

Choosing the correct moving average model

I am working with the in built Air Passenger data set in R to learn forecasting. After splitting the data in 120:24 data points, I am trying to extract trend ...
1
vote
2answers
107 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
0
votes
0answers
32 views

Why the Logistic regression model trained with tensorflow performed so poor

I trained a logistic regression model with tensorflow but the accuracy of the model was so poor (accuracy = 0.68). The model was trained using simulated dataset and the result should be very good. is ...
0
votes
1answer
14 views

Validate Z-score Formula

I need need help validating a formula for a z-score. I have a book that I enjoy but in my re-read of the materials, I have found that the author has made some fundamental errors in their writing. So, ...
0
votes
0answers
13 views

Why my binary classification neural network performance oscillates a lot through epochs? [duplicate]

I am training a CNN with Keras, vgg16-like model and i don't understand the results. For example, in epoch 15 i have good results but in 14 and 16 it's horrible (you can see it in the loss). What ...
0
votes
1answer
43 views

How to check if a Machine Learning model is applicable for newly input data?

Suppose we have a good Machine Learning model, with good cross-validation and test score. How can we estimate whether a newly input data instance belongs to the domain of data where model predictions ...
0
votes
0answers
19 views

Calculating Accuracy in Binary classification Using SVM

I am a beginner; studying SVM with OpenCv and Python I am bit confuse about SVM and SVC; when I searched and find out " SVC is the support vector machine algorithm fot the multiclass problem and ...