Questions tagged [data-preprocessing]
A step of cleaning data in data mining for analysis purposes
503
questions
1
vote
0
answers
9
views
When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?
I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing.
It's given that ...
- 1,129
0
votes
0
answers
9
views
How to simplify a dataframe to signal if a level does or does not have a value in another column [migrated]
I would like to reduce my dataframe so that I can determine if a checklist does or does not have a presence value (i.e., X) instead of a number. Each row in the current dataframe corresponds to an ...
- 81
0
votes
0
answers
19
views
Estimating lost precision in time series data
I have some time series floating point distance measurements $x_n$ with corresponding integer time rounded to the nearest seconds $t_n$. If I calculate velocity (eg. forward difference), there is ...
- 1
0
votes
1
answer
23
views
Order of pre-processing the dataset
suppose I have categorical dataset, I'm doing data pre-processing.
what is the correct order of applying these 3 techniques
Train Test split
SMOTEN to over sampler the minority class
Categorical ...
0
votes
0
answers
19
views
How does centering the data reduce the risk of numerical problems when doing PCA?
In Mathematics for Machine Learning (page 336), the authors state that centering the data (subtracting from the data its the empirical mean) reduces the risk of numerical problems.
Which numerical ...
- 1,655
0
votes
0
answers
10
views
What does it mean when you scale categorical features and it increases the F1 score on validation data?
I am working on a dataset with some categorical features. Those categorical features are encoded as numbers.
For example: I have a feature with the name ...
- 1,330
2
votes
3
answers
83
views
What is "information leak from test to train" ? Is stratification by target a leak?
It's common practice to do procedures such as standardization and even missing value imputation (commonly based on some means) after train/test split - otherwise it is treated as information leak from ...
- 21
0
votes
0
answers
10
views
Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?
I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
- 11
1
vote
0
answers
35
views
Different preprocessing steps for time series data vs. regular cross sectional data?
What are the different preprocessing steps for time series data vs. regular cross sectional data?
Eg 1. When doing train/test or cross validation, you cannot randomly split the data. The data must be ...
- 513
0
votes
0
answers
23
views
How do I prepare data for a multivariate LSTM model that includes multiple patients
I want to predict the blood glucose levels using time series data with multiple features such as time, glucose levels, carbohydrates, fat, and protein. I have a dataset with hundreds of patients but ...
- 1
0
votes
1
answer
37
views
Preprocessing on training set only or both training & test set? Seems like there would be errors for both answers
Let's say I have a dataset that hasn't been split into train/test yet.
Upon loading it, I discover that there are columns where there are nulls that need to be filled in, some quadratic relationships ...
- 513
1
vote
0
answers
59
views
Better way to create a pandas dataframe with variable feature length for regression?
I am doing a project trying to see whether we can infer the power factor (cos(phi) = P/S) from voltage (U) and current (I) data. What this means exactly is not very important for my question. My ...
- 11
0
votes
0
answers
48
views
How to handle a column with both float and categorical values
This sounds like a question that should have come up before but I couldn't find it on CV.
I am trying to use a column called limit_price as in the limit price of an order for a machine learning ...
- 133
2
votes
1
answer
44
views
Should I remove 0 values from my dataset if they seem to be from instrument error?
I have conducted an experiment looking at the decay rate of a DNA target in flowing water over time, with 4 replicates of each treatment. The data is collected via dPCR Quiacuity instrument that ...
0
votes
0
answers
35
views
Not balancing the validation and test set gives me a very bad dev set result
Here is the resulting heatmap for train, val, test set. I also apply PCA to train,val,and test set. Because from the train set, there are a lot of features that has high correlation with each other.
...
1
vote
2
answers
329
views
how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?
I am trying to pre-process data following a statement in a paper.
They said
for the normalization, each dataset is normalized on a per channel basis with a sample range from -1 to 1 and a mean value ...
- 13
0
votes
0
answers
33
views
Should I remove/cap "Extreme Values" from Variables and at which percentile should capping be done?
In my dataset, I have a variable corresponding to the monthly average playtime hours of the players. The dataset has 200K records. I estimated various percentiles of the variable - from 0 to 1 for ...
- 143
0
votes
0
answers
8
views
Sequence of pre processing methods of machine learning
I have a big dataset. It is a categorical data set. I used label encoder to change the categorical values to the integer values. I would like to find out the co relation between class attribute and ...
- 155
1
vote
0
answers
28
views
Input Normalization for Preprocessing Time Series Data
I have data traces for separate days that I am reading in from separate files. I'm splitting the data into smaller sequences to train a state-prediction model on, but all the data in a sequence must ...
- 11
0
votes
0
answers
45
views
How to compare variables from different time periods?
I am about to compare weather's influence on green-up day across US states and different years (2001 to 2010). I have green-up day of the year as dependent variable and weather data as independent ...
- 418
2
votes
1
answer
43
views
How do I normalize this feature, I've tried almost everything
I'm trying to normalize this skewed data as part of data preprocessing, but it doesn't normalize no matter which transformation I use to the point it's making me crazy :') .
The methods which I've ...
- 21
1
vote
1
answer
47
views
How to handle machinery shut-down in Time Series for Anomaly Detection
I have this data coming from sensors installed in a industrial machinery and my ultimate goal is implementing an anomaly detection method on it.
Now, the data is quite noisy and with lots of missing ...
- 23
0
votes
0
answers
19
views
Why is data whitening (decorrelation) not tried with a post multiplication of a whitening matrix?
The general procedure of whitening(decorrelating) a data $X$ with dimensions $(D, n)$ (n being the number of samples) rests upon finding a matrix $W$ with dimensions $(D, D)$ such that the transformed ...
- 2,119
0
votes
0
answers
29
views
Why do we scale features in PCA? Wouldn't that mean the variance in all dimensions is just $1$? [duplicate]
According to https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html,
Feature scaling through standardization (or Z-score normalization) can be an important ...
- 205
0
votes
0
answers
43
views
Is there an accepted standard for storing factor order information? (And if not ...)
I use a lot of survey data, and often get a csv of the responses -- however, this almost always requires an enormous amount of extremely tedious transformation, before it can be analyzed. In ...
- 1,605
1
vote
1
answer
75
views
Oversampling for Continuous Values
I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
- 11
0
votes
1
answer
16
views
Training data from real hardware
I am working on machine learning model. The target is to learn the behavior of a black box which has a couple of inputs and one output signal. I have generated some training data by applying different ...
1
vote
0
answers
34
views
How should I use TimeSeriesSplit for validation of Classifier models?
My problem is whether I should use TimeSeriesSplit for cross validation of my classifier models (SVC, Random Forest etc).
The input data has a bit of temporal sense. The X features are rolling(1yr, ...
0
votes
0
answers
40
views
How to properly use isolation forest for data cleaning in a machine learning project?
I am trying to clean my data set (X) from outliers with isolation forest. Then I am going to use the cleaned data set for supervised learning. Should I first split ...
- 1
4
votes
0
answers
51
views
Examples of Leakages in the Training Data
I was wondering about Data Leakage in the data preparation phase during the training of a model. By definition, data leakage happens when information is revealed to the model giving it an unrealistic ...
0
votes
0
answers
52
views
How to interpret a missing data combination-frequency plot (e.g. VIM::aggr)
I am exploring some missing data, and would like to know what principles to apply in interpreting the following type of plot. This example is from VIM::aggr() in R, although this question is not about ...
- 155
2
votes
2
answers
75
views
Detect quoted text in emails
I have small dataset (<10k) of emails (plaintext) that need to be classified. Currently I'm doing research on topic of email preprocessing and I can't find any suitable solution for quoted text ...
- 23
1
vote
0
answers
14
views
Standardization on production dataset
Say I trained a Logistic regression model on a training dataset (sample = 2000), using Standardization. I than test the model on a test dataset (sample = 400) using the Standardization parameter ...
- 11
1
vote
0
answers
27
views
Constant boolean variables in multivariate time series observations
I am observing multivariate time series with some of its variables being boolean.
One example of such observation has its var1=1 during all time series, and another observation has var2=0 for example ...
1
vote
1
answer
113
views
Data Preparation in R: normalization, log - order [closed]
I have a little problem with my data (GDP per capita, some control variables with negative minimums (e.g. FDI) and explanatory variables without negative values but also different ranges.
Originally, ...
- 11
0
votes
0
answers
27
views
What is the procedure for data preprocessing for time-dependent LSTM classifier?
I attempt a beginner level LSTM classification task with a time-series numerical data, but my task is finding changes in features over time (in which those changes describe the outcome or the classes),...
0
votes
0
answers
22
views
Handling new customers in customer propensity model
I'm using last four years' data to predict whether they will buy or not buy in the next quarter.
One problem I'm facing is customers who are not four years old. Is it right to keep them in the data ...
- 203
2
votes
2
answers
24
views
Transformation of mix normal and skew features [duplicate]
I have a weather dataset containing four features that are continuous values. Temperature is almost normal, but precipitation is highly negatively skewed. In addition, wind speed and humidity are ...
- 51
1
vote
1
answer
30
views
Removing categorical predictors with mode frequency above certain threshold
Is there a rule of thumb for dropping certain categorical predictors if the most frequent value in the column is above a certain threshold? For example, should I drop a column/predictor if more than ...
- 131
1
vote
0
answers
37
views
Unsupervised learning (clustering) before supervised learning [closed]
Is it a common practice to do clustering before supervised learning to eliminate "noisy data"? Obviously, depending on the type of task. It seems like it makes sense in my case and my neural ...
- 11
1
vote
1
answer
107
views
How to transform prediction std of gaussian process back to origin
I am looking for a way of rescaling the predictions of my Gaussian Process Model back to the original scale.
The data is scaled for training using a ...
- 13
0
votes
2
answers
94
views
How to handle highly correlated observations (rows)
What is the best practice to handle highly similar/ autocorrelated observations (rows) in a data set. These highly similar rows could come from recording (some of the) observations at too close ...
- 431
1
vote
1
answer
145
views
How to distinguish numerical categorical (Ex: White = 1, Latino = 2 etc) from numerical continuous or discrete variables on a dataset? [duplicate]
I am currently working on a project that involves a data processing pipeline, and in it I might come across all sorts of data. I would like to know if there are any references on an automatic way to ...
- 11
1
vote
1
answer
19
views
Optimal way to create a feature set?
I have a time series data (say weather data for each day for one week) that changes at each time step. Along with this, I have some data that is fixed (eg - the latitude and longitude of this place ...
- 115
1
vote
1
answer
30
views
My max target value is seen quite frequently since the data source has a threshold on what they can measure. Should I remove these data?
I'm working on a regression problem involving nutrient concentrations. The lab I'm getting my data from can measure up to 9000ppm of a particular nutrient. Beyond that, everything is reported as ...
- 11
2
votes
1
answer
62
views
Which data preprocessing steps do I need to perform on which data subset?
I want to make predictions using several supervised Machine Learning algorithms and apply 10-fold-cross validation. For doing so, I randomly divided my dataset into in-sample and out-of-sample sets. ...
- 21
1
vote
0
answers
51
views
Model which achieves good results on the Adult dataset, especially on the minority class
I'm looking for a model which achieves an high prediction and recall on the Adult dataset. I've tried with random forests, but couldn't get a model with good results on the minority class. Here are my ...
- 111
1
vote
0
answers
52
views
Machine Learning for Stock Price Prediction Issue
In financial application, someone might make use of machine learning techniques in stock price prediction, e.g. LSTM. In general, before training the model, in light of the model robustness, some ...
- 11
0
votes
0
answers
51
views
Necessary Data Preprocessing for SVM
All the lecture notes that I read about support vector machine assumes that $w$ is orthogonal to the hyperplane. I am trying to prove that the vector $w$ is orthogonal to the hyperplane $L=\{w^Tx + b =...
- 303
5
votes
1
answer
230
views
Do Transformers pad input sentences?
Assume we have a Transformer (Attention is all you need paper) and we give to it an input sequence S of length $n_{words}$.
If no padding is applied, the output of the encoder model would be a matrix ...
- 71