Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
19 views

What does this evaluation function mean?

I am solving a prediction problem where I am supposed to predict the number of clicks, given some historical Data. I am told that my predictions will be evaluated by a normalized weighted mean square ...
0
votes
0answers
21 views

What test can I use for Model Validation?

I ran the BTYD package in R which predicts the number of transactions that a customer is expected to make in the future.These expected values that I get are not integers but are in the form 0.14, ...
0
votes
0answers
17 views

Inproper model validation

Let us say that I have a data set D that contians N objects from K different species, words for example, where most of them occur one time. Now my advisor want's me to draw random samples with size ...
0
votes
0answers
49 views

What are the reasons why a classifier could produce bad results?

I know of four possible reasons: overfitting underfitting input data doesn't represent the problem (which I guess is underfitting) classifier isn't suitable (e.g. problem is not linear) Are there ...
4
votes
1answer
129 views

ML / train-test-validate: What is allowed when?

As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to ...
0
votes
1answer
69 views

How to pick the best model with cross validation?

Based on my understanding the leave one out cross validation is to hold a sample out as the test set and fit a model with remaining data and then calculate the error of prediction of the test sample ...
1
vote
1answer
48 views

What test shall I use to validate the use of a certain score to predict my outcome in a survival analysis?

I validate usage of a clinical cardiovascular score to predict the risk of dementia using data from a longitudinal study. Therefore, my outcome is binary (dementia yes or not) and the independent ...
0
votes
0answers
39 views

Which statistical test is preferred when comparing two predictive models?

Say, I have two predictive models A and B and recorded AUC (Area under curve) by running both the models on 20 different data sets. Now, if I want to find the best model out of these two for these ...
0
votes
0answers
24 views

Large condition number, but no effect on validation

My question is in regards to the relation between the condition number (or other multicollinearity diagnostics) and validation of a linear regression (lm in R). Specifically, I have extremely high ...
0
votes
0answers
6 views

How to model the behavior of Webservice and doing validation?

The goal is to find a smart way to validate the behavior of webservice such as CMS (wordpress, joomla). To give more detail, we want to be able based on observing host metrics ( cpu utilization,... ) ...
1
vote
0answers
22 views

Bootstrap within bootstrap?

I want to use .632 bootstrapping to validate an algorithm. The problem is that the algorithm itself also has bootstrapping procedures. Will that cause bias? Since bootstrapping with replacement will ...
2
votes
0answers
49 views

What should I do when logistic regression model doesn't perform well on test sample?

For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training ...
0
votes
1answer
54 views

Measures for Clustering Validations [duplicate]

I have unlabeled dataset and I'm using the hierarchical clustering to generate a groups from this data. I had a look to the lit and I found that there are two approaches to evaluate the clustering ...
1
vote
1answer
25 views

which paper(s) to cite when stating that out-of-sample validation is better than in-sample val

Validating a model (meaning, computing metrics on the prediction) using the training samples is, well, rubbish. Out-of sample validation should be preferred. This is well known by statisticians and ...
1
vote
0answers
15 views

The burden for verification/validation data

I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that ...
0
votes
0answers
57 views

Validation Questionnaire Sample or format

I want to know on how to make a validation questionnaire for my research paper but I have no idea on how to create one. I searched about validation questionnaire but there's no format or example - ...
2
votes
2answers
78 views

The role of validation in estimation and hypothesis testing

Validation, with or without statistical/machine learning procedures, is used often, if not universally, in prediction. In estimation or hypothesis testing that does not seem to be the case, yet I ...
0
votes
0answers
56 views

Calibration curve confidence intervals interpretation. Is to reduce a high Bias Corrected possible in Bootstrapping?

I have been working on a mixed generalized model, glmer, and now I am trying to validate it by a calibrate curve, and also would like generate its CIs. For this, I thought in Bootstraping and found ...
0
votes
0answers
21 views

Using one model to validate another? (Validation with no ground truth?)

My question is on model validation when there is no ground truth. Suppose I have a data set of attributes: $x = \{x_1,\ldots, x_n\}$ These attributes are different attributes that describe animals ...
2
votes
2answers
74 views

Double (nested,wrapper) CrossValidation - final trained model

I'm performing a study where I'm selecting kernel type and hyperparameters in an inner CV loop and an outer loop doing 10-fold CV (using SVR). The output is 10 trained models and performance measures. ...
0
votes
0answers
33 views

Avoiding over/under fitting & validation set size

I'm using a PLS model associated to FT-NIR spectroscopy to predict some properties of biodiesel. At the moment, I'm trying to optimize the number of factors I should use to do the calculations. ...
0
votes
0answers
100 views

SOM Validation method

I made a project using Self Organizing Map (SOM) for clustering multi dimensions data in R. But I have difficulty to find what method that I can use to: validate the clustering result based on data ...
0
votes
0answers
16 views

How to account for drift in system identification output?

I am trying to identify the lateral dynamics of a vehicle. I use steering angle as input, and lateral displacement, lateral velocity and yaw rate as output. I use idss method of MATLAB. The issue ...
1
vote
1answer
53 views

What are the assumptions to test on unseen data to use a predictive model?

I am trying to understand what are the assumptions I have to test, to use a predictive model on unseen data. After the validation phase of my predictive model and the estimation of its accuracy on a ...
0
votes
0answers
8 views

Correcting for multiple data points

I am validating a new measure of therapist competency in psychology. I am getting 4 clients who have taken part in 40 sessions (10 each) to answer a questionnaires about their therapy sessions ...
0
votes
1answer
74 views

Calibration and validation of an accelerated failure time model on new data

I have an existing AFT (accelerated failure time) model which I'm using on a new dataset, with the obvious intent of testing whether the model predicts the new data well. A first step is to look at ...
0
votes
0answers
23 views

validation set with ranking variables

i'm working on an approach of feature selection with SVM model and i have some questions about validation , training and test sets. the idea is to rank variables in decreasing order of relevance ...
0
votes
0answers
20 views

Why use clustering + external measure for validation, instead of supervised learning?

Lets say I have set of genes in individuals and I will use clustering to find groups of individuals with similar genes and then I will check how these groups express itself in form of some disease. I ...
0
votes
0answers
17 views

How do I conduct model validation with glm models that have proportional response variable?

I would like to determine which environmental factors influence turtle nest survival. I ranked glm models with AIC using a step-wise procedure. The response variable is the proportion of eggs that ...
0
votes
0answers
22 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
0
votes
0answers
41 views

Validating formative model - statistical analysis

Based upon literature I have developed a model which I want to validate. Data gathering will be done with the use of surveys. My knowledge of statistics, or more the lack thereof, has raised some ...
2
votes
2answers
160 views

Performing k-means clustering on a set of lines

I have a set of lines (y = numbers between 1 and 100, x= discrete) that I am trying to cluster to group similarly-shaped profiles. I have found that the profiles seem to cluster the cleanest when ...
4
votes
2answers
384 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, ...
1
vote
2answers
216 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
2
votes
0answers
88 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
0
votes
0answers
129 views

K-fold cross validation for a glmer model with nested data

I'm working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). I'm using a generalized linear mixed effects modeling procedure (lme4 ...
2
votes
1answer
697 views

How do you use test data set after Cross-validation

In some lectures and tutorials they suggest to split your data in three parts: training, validation and test. But it is not clear how test data set should be used and how this approach is better than ...
0
votes
0answers
43 views

Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$ d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}} $$ Here, $C(xy)$ is the concatenation ...
0
votes
0answers
16 views

Evaluate and report fit of a model on validation cohort(s)

I trained a random forest regression model M on a training set. I am interested in how well the model predicts the responses in 3 different validation sets. I am also interested in the characteristics ...
7
votes
2answers
184 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
2
votes
1answer
56 views

Why do clustering internal validation indices are decreasing with the number of clusters?

I get the same pattern for 3 different indices: Silhouettes, Dunn and Connectivity- as the number of clusters increases, the score decreases. I am using several clustering methods and several distance ...
1
vote
1answer
38 views

Missing variable in the external validation cohort

I have developed a model on my training cohort and I intent to use an external dataset to validate the model. However, one of the predictors used in the model was not captured in the external ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
2
votes
2answers
408 views

GLMM validation: weird qq & fit vs residual plots

I'm encountering problems with the results of a glmer model (lme4-package). Im trying to answer the question, whether a beaver ...
1
vote
1answer
65 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
1
vote
0answers
123 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
0
votes
1answer
49 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
89 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
6
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
2
votes
1answer
182 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...