Questions tagged [data-preprocessing]
A step of cleaning data in data mining for analysis purposes
516
questions
0
votes
0
answers
45
views
+50
Data mining for categorical variables that correspond to modes of a continuous variable
Is there a good method or test to find categorical variables associated with modes of a continuous variable?
Take the velocity of a car as an example. You would expect there to be at least two modes ...
- 38
0
votes
1
answer
87
views
Outlier Detection and Removal
I am reading a paper on wind power forecasting and the authors present a plot of the data before outliers are removed and a plot after. However, they don't actually say what method was employed to ...
- 1
0
votes
0
answers
11
views
Cleaning Panel Data with Merging Firms
Im cleaning some panel data for a regression. Subjects are firms and a substantial amount of firms have been involved in mergers. Should I just drop all firms that have ever been part of a merger? ...
- 1
0
votes
0
answers
8
views
Using Unordered Text Features in Machine Learning Model
I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
- 103
2
votes
2
answers
87
views
Non-constant treatment periods for A/B test samples
I have 4 A/B testing samples with 4 treatment groups and their 4 respective control groups (n=10000~), with known (but not distinct) treatment periods. The issue is that the treatment periods overlap ...
0
votes
0
answers
24
views
How do i prioritize which features to use in my machine learning model before the feature engineering stage?
I am encountering a probably fairly common problem where I have too many features, lets say 500 possible features.
I only want to pick the top 10-50 features that would be the most predictive of y, or ...
- 831
0
votes
0
answers
18
views
Using pycaret's preprocessing on unseen dataset
I'm thoroughly enjoying pycaret to handle much of the legwork in my analysis. I'm making heavy use of the setup() method in preprocessing to handle normalization, ...
- 113
2
votes
1
answer
99
views
Geometric Intuition Behind Whitening for ICA
I know there are a couple posts asking about why we whiten the data for ICA. I understand why we whiten to fix scaling invariants between the sources and to increase the computationally efficiency.
...
- 21
2
votes
0
answers
30
views
Data leakage: Train test split before or after data preprocessing? [duplicate]
A while ago I came across the word "data leakage" for the first time, and after some research, I found that it is a common mistake among data science/machine learning practitioners. But the ...
- 21
0
votes
0
answers
9
views
organizing analytical file for persons with multiple records
My stakeholders have manually recorded data for patients enrolled in an intervention, which is causing data issues that I need to resolve in order to move to determine what is the appropriate ...
0
votes
0
answers
10
views
How can I get knowledge graph embedding by using TransE? [duplicate]
I saw nobody answered Ssong, however now I'm struggling with the same question. Can anyone help? Thanks.
2
votes
2
answers
47
views
Univariate vs. Multivariate Standardization
There are several common methods for scaling input features to machine learning models prior to training the model. The most popular methods seem to be standardization (centering by the mean and ...
- 38
0
votes
1
answer
21
views
How to organize an overlapping ordinal variables?
I am analyzing some data with attributes about how long it takes to commute, and it is an overlapping ordinal variable like the following:
Time
Number of People
1-5 minutes
10
6-10 minutes
24
less ...
0
votes
0
answers
11
views
Should the data be scaled before normalisation to enable the use of the model as a pre-trained model?
I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data.
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
- 101
1
vote
0
answers
50
views
Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?
After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
- 375
1
vote
1
answer
21
views
Changing scale and rounding off Target
I am training a regression machine learning model to predict an airplane's maximum take-off weight based on some pre-project features. The airplane weights in the dataset I'm working with are ...
0
votes
0
answers
47
views
Is the correlation method used within your dataset problem dependent?
Question
Say I have a dataset $D$ with $N$ features that are trying to predict a target $y$. I would like to build a model from $D$ and part of that process is removing correlated columns to reduce ...
- 451
0
votes
0
answers
18
views
Cleaning Data Before SVM
I want to classify diabetic retinopathy grades using SVM. I have 32 extracted features, and those features won't all be used in classification stage. Before entering feature selection, I want to clean ...
- 21
0
votes
0
answers
38
views
How to find (and interpret) outliers in set of time series data?
I face with a problem of doing a time series forecasting on multivariate data in the form where different entities have their own 100-day (daily) series of 10 variables, and I'm expected to predict 10-...
- 725
2
votes
1
answer
120
views
Do we need preprocessing before applying a sophisticated undersampling method?
My question is around applying undersampling methods to an imbalanced, and highly dimensional dataset, with mixed data. Lets say as an example, I have 150 features, also a highly imbalanced binary ...
- 21
1
vote
0
answers
11
views
Training computationally cheap models for feature selection and then extending to more flexible models
To what extent can I apply lessons learnt about the model performance on a simple "cheap" model (e.g. Naive Bayes, LDA) to a more flexible model (i.e. ensemble of trees, neural networks)?
...
- 53
0
votes
1
answer
30
views
Deep learning classification with multiple temporal data
I'm working on a project to predict the category of music segments in an audio file (represented in pianoroll format with an additional column for the corresponding class). Each row represents the ...
- 1
0
votes
0
answers
14
views
How to apply dimensionality reduction to a data set with outliers?
I try to apply dimensionality reduction to a multidimensional data set (with numerical features) with significant outliers. I have managed to identify outliers with Isolation Forest but now I'm in a ...
- 725
1
vote
0
answers
27
views
Clustering on thousands of product feature clicks and pages viewed
I want to classify 120k customers into 5-6 clusters basis the product usage, say, hundreds of product features clicked and hundreds of product pages viewed. The data will be like a customer_id has ...
- 31
1
vote
0
answers
19
views
Stardardization for Random Forest, SVM and Logistic Regression
I have a classification project and I want to compare three models: Random Forest, SVM and Logistic Regression.
Random Forest are tree based algorithms wheras, SVM is a distance based model and LR is ...
- 11
1
vote
0
answers
37
views
data preprocessing with zeros dominating
I'm working on a machine learning classification project and i faced some difficulties: all of my features distributed like this:
I'm not sure what should i do, should i use any scalers/other ...
- 11
2
votes
1
answer
316
views
Do we lose information when we normalize an image? [closed]
Before training a machine learning algorithms, it is advisable to perform feature scaling. Suppose we have a "toy" dataset where each image is composed of two pixels $x_0$ and $x_1$. Lets ...
- 452
1
vote
1
answer
469
views
change point detection of time series
Currently, I am working on a research project that involves forecasting electricity consumption using data mining. In my analysis, I have detected the change point for time series using the ...
- 31
1
vote
0
answers
18
views
When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?
I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing.
It's given that ...
- 1,155
0
votes
0
answers
23
views
Estimating lost precision in time series data
I have some time series floating point distance measurements $x_n$ with corresponding integer time rounded to the nearest seconds $t_n$. If I calculate velocity (eg. forward difference), there is ...
- 1
0
votes
1
answer
88
views
Order of pre-processing the dataset
suppose I have categorical dataset, I'm doing data pre-processing.
what is the correct order of applying these 3 techniques
Train Test split
SMOTEN to over sampler the minority class
Categorical ...
0
votes
0
answers
21
views
How does centering the data reduce the risk of numerical problems when doing PCA?
In Mathematics for Machine Learning (page 336), the authors state that centering the data (subtracting from the data its the empirical mean) reduces the risk of numerical problems.
Which numerical ...
- 1,665
0
votes
0
answers
24
views
What does it mean when you scale categorical features and it increases the F1 score on validation data?
I am working on a dataset with some categorical features. Those categorical features are encoded as numbers.
For example: I have a feature with the name ...
- 1,360
2
votes
3
answers
153
views
What is "information leak from test to train" ? Is stratification by target a leak?
It's common practice to do procedures such as standardization and even missing value imputation (commonly based on some means) after train/test split - otherwise it is treated as information leak from ...
- 31
0
votes
1
answer
66
views
Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?
I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
- 11
1
vote
0
answers
39
views
Different preprocessing steps for time series data vs. regular cross sectional data?
What are the different preprocessing steps for time series data vs. regular cross sectional data?
Eg 1. When doing train/test or cross validation, you cannot randomly split the data. The data must be ...
- 831
0
votes
0
answers
37
views
How do I prepare data for a multivariate LSTM model that includes multiple patients
I want to predict the blood glucose levels using time series data with multiple features such as time, glucose levels, carbohydrates, fat, and protein. I have a dataset with hundreds of patients but ...
- 1
0
votes
1
answer
82
views
Preprocessing on training set only or both training & test set? Seems like there would be errors for both answers
Let's say I have a dataset that hasn't been split into train/test yet.
Upon loading it, I discover that there are columns where there are nulls that need to be filled in, some quadratic relationships ...
- 831
1
vote
0
answers
117
views
Better way to create a pandas dataframe with variable feature length for regression?
I am doing a project trying to see whether we can infer the power factor (cos(phi) = P/S) from voltage (U) and current (I) data. What this means exactly is not very important for my question. My ...
- 11
0
votes
0
answers
67
views
How to handle a column with both float and categorical values
This sounds like a question that should have come up before but I couldn't find it on CV.
I am trying to use a column called limit_price as in the limit price of an order for a machine learning ...
- 143
2
votes
1
answer
50
views
Should I remove 0 values from my dataset if they seem to be from instrument error?
I have conducted an experiment looking at the decay rate of a DNA target in flowing water over time, with 4 replicates of each treatment. The data is collected via dPCR Quiacuity instrument that ...
0
votes
0
answers
43
views
Not balancing the validation and test set gives me a very bad dev set result
Here is the resulting heatmap for train, val, test set. I also apply PCA to train,val,and test set. Because from the train set, there are a lot of features that has high correlation with each other.
...
1
vote
2
answers
732
views
how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?
I am trying to pre-process data following a statement in a paper.
They said
for the normalization, each dataset is normalized on a per channel basis with a sample range from -1 to 1 and a mean value ...
- 13
1
vote
0
answers
35
views
Input Normalization for Preprocessing Time Series Data
I have data traces for separate days that I am reading in from separate files. I'm splitting the data into smaller sequences to train a state-prediction model on, but all the data in a sequence must ...
- 11
2
votes
1
answer
46
views
How do I normalize this feature, I've tried almost everything
I'm trying to normalize this skewed data as part of data preprocessing, but it doesn't normalize no matter which transformation I use to the point it's making me crazy :') .
The methods which I've ...
- 21
1
vote
1
answer
58
views
How to handle machinery shut-down in Time Series for Anomaly Detection
I have this data coming from sensors installed in a industrial machinery and my ultimate goal is implementing an anomaly detection method on it.
Now, the data is quite noisy and with lots of missing ...
- 23
0
votes
0
answers
30
views
Why do we scale features in PCA? Wouldn't that mean the variance in all dimensions is just $1$? [duplicate]
According to https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html,
Feature scaling through standardization (or Z-score normalization) can be an important ...
- 255
1
vote
1
answer
282
views
Oversampling for Continuous Values
I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
- 11
0
votes
1
answer
18
views
Training data from real hardware
I am working on machine learning model. The target is to learn the behavior of a black box which has a couple of inputs and one output signal. I have generated some training data by applying different ...
1
vote
0
answers
89
views
How should I use TimeSeriesSplit for validation of Classifier models?
My problem is whether I should use TimeSeriesSplit for cross validation of my classifier models (SVC, Random Forest etc).
The input data has a bit of temporal sense. The X features are rolling(1yr, ...
