All Questions
Tagged with train-test-split classification
17 questions
0
votes
0
answers
24
views
Model Stacking - Out of Fold Procedures
I am attempting to use a model stacking procedure where I am using a time-series split on a set of data I have (around 5000 entries). The goal is binary classification.
After obtaining hyper ...
- 47
2
votes
1
answer
36
views
Model Stacking Train Test Split Methdods
I am trying to validate my processes in terms of how I am engaging in model stacking for binary classification. Say I have two models as my base models, models A and B both with different classifiers ...
- 47
3
votes
2
answers
684
views
test & train for very very small data
I have just 25 observations. I'm not sure would it possible to test & train the data. For example 15 observations for train and 10 observations for test set. 15 observations is so small for ...
- 179
0
votes
0
answers
15
views
Is GroupKFold needed if some samples have some of their feature values equal?
I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
- 621
1
vote
0
answers
36
views
Choosing a suitable sample size for a Random Forest Model
I know that this isn't always a straight forward question to answer, but I am working on a provincial wide wetland classification model that has $7$ classes and $32$ or so explanatory variables. In my ...
- 11
1
vote
1
answer
48
views
CV score vastly different from Train-Test score
I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
- 11
0
votes
0
answers
39
views
Is it normal to have a sharp increase in validation error when using 10% of the data instead of something like 20-30%?
Scenario:
I'm training a relatively simple neural network to classify pairs of tabular datapoints (~150k), lets say drugs and diseases, whether they are related (positive) or not (negative). As I only ...
- 185
1
vote
0
answers
19
views
Which one to choose: subject-wise or record-wise
I want to do some classifications on open access "Activity Recognition Using Wearable Physiological Measurements" data set. In this dataset, there are 40 subjects whose Electrocardiogram (...
- 111
1
vote
3
answers
138
views
How to design cross-validation and testing scheme when N is small?
I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make ...
- 25
0
votes
0
answers
84
views
Statistical comparison of two probabilistic classifiers
TL;DR There are well-known tests to compare classifiers. How can we generalize to classifiers with random training steps?
I am comparing two classification algorithms (A and B). I can see that ...
1
vote
0
answers
41
views
How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?
My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
- 23
4
votes
1
answer
642
views
Rule based label - random split vs time-based split
We have a dataset of 977 records (77:23 class ratio) where we try to predict a binary outcome using random forests and neural networks. whether supplier met the target or not. However, we didn't have ...
- 3,342
2
votes
1
answer
3k
views
random split vs time based split of train and test data
I have been working on binary classification problem using algorithms such as Random Forest, Boosting methods, neural networks and logistic regression. I have data from Jan 2017 to Jan 2022. We wish ...
- 3,342
3
votes
1
answer
2k
views
Why does error rate of kNN increase when k approaches size of training set?
I've been experimenting with the effect that different values of k have on the generalisation error of kNN classifier, and I've gotten some unexpected results towards the end when k approaches the ...
- 63
1
vote
1
answer
52
views
Train/dev/test split with limited and skewed positive labels
(Because of the sensitive nature of the actual project, I am using an analogy here. I hope it's clear, if not, please let me know!)
My goal is to classify images as cats or dogs (binary classification)...
- 113
1
vote
0
answers
100
views
Splitting Data Into Training and Test set [closed]
I am trying to split a data set into training and test set with these codes
...
- 11
1
vote
0
answers
48
views
Classifier score is higher on test set than on training set. Is this an error?
I am currently getting used to scikit-learn and I trained a simple logistic regression model on the iris dataset. One interesting thing I noticed was when I looked at the classifier.score, namely:
<...
- 11