Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Large amount of missing values in as input features for LSTM time series

I am using an LSTM to predict a time series chart from multiple other time series charts as input features. The problem is that some of these input charts have much ...
0
votes
0answers
15 views

sklearn ColumnTransformer based preprocessor outputs different columns on Train and Test dataset [migrated]

I was trying to learn how to use Pipelines and ColumnTransformer to effectively preprocess data before Regression. Here's my attempt: ...
0
votes
0answers
18 views

Find Correlation between Grouped and Ungrouped Data

I have 2 datasets. First is a Quarter-yearly data that has PTR (Pay-through-rate) value associated with an Agent for every Quarter. The second dataset has the detailed data for Sales related to those ...
0
votes
0answers
7 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
2
votes
1answer
16 views

Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]

First post! If you don't want to read the background you can skip to the Problem heading below. Background Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
0
votes
1answer
13 views

Handle various shapes of distribution in data preprocessing

Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
9 views

Freedmans rule to find number of bins

I have a data set that is 30162 rows long. I am trying to split age into bins but I am struggling to understand the concept of the rule with my data set. As a quick examples this is what I did: <...
0
votes
0answers
21 views

How to preprocess performance counter input data for anomaly detection using autoencoders

I am working with more than 250 input features that include system performance counters and SQL Server database counters to predict anomalies / system outages. I am looking to use an autoencoder ...
1
vote
1answer
21 views

Wilks.test usage in R

I want to calculate the Wilks' lambda for a given data set. The "rrcov" library in R appears to have a function for this purpose. The examples provided in the documentation run without a problem; ...
0
votes
0answers
10 views

Feature Engineering: How to deal with imbalanced numerical/categorical features

I'm analyzing a data set and solving a classification problem and find that values concentrate on one number in many features. For example, a categorical feature 'loan' indicating a person having loan ...
3
votes
1answer
73 views

Classification of data tables (each table is an item)

I have to work on a binary classification task where single items to be classified are not single rows of a data matrix, but groups of rows. In other words, I have $N$ data tables of varying size $n_i ...
0
votes
0answers
13 views

Transformation vs Scaling

I'm not sure I understand the different uses between the 2 methods: Scaling - scale features to same scale (Normalization or Standardization) Transformation - makes the data normally distributed ...
0
votes
0answers
19 views

What is the best way to handle 2 (or more) missing values in same row?

one of the methods to handle missing data is using predictive model. If we have a row with 2 or more missing values, is it right (and accurate) to use ...
0
votes
0answers
18 views

Is it right to use different feature scaling techniques to different features?

I read this post about feature scaling: all-about-feature-scaling The two main feature scaling techniques are: min-max scaler -...
1
vote
1answer
18 views

Feature Engineering with Focus on KNN

I have seen a number of helpful posts such as this one on feature engineering, but I am specifically looking for something that may be helpful when using KNN. In my experience, some features work best ...
1
vote
0answers
14 views

Questions about pre-processing/transformation of data

For an assignment for a ML Online course I have to find the best classifier for a given data set using 4 different methods: Logistic regression, Decision tree, Support Vectors and K Nearest Neighbours....
0
votes
1answer
28 views

Data pre-processing on test files

I am working on a classification problem. I have a training file with a label and a separate test file without a label field. I needed to remove some rows that contained missing values from the ...
1
vote
0answers
29 views

Highly correlated variable, when to remove

I have a dataset that includes both course number and course name. For some models such as regression, a Pearson Correlation matrix is obtained and one of any pair of features that are highly ...
0
votes
0answers
13 views

Building an analytical base table from a relational database

I need to train a (supervised) machine learning model. However, up to now I have only made model for data that was given to me as 1 single data frame. Now I need to create a model for data that is ...
1
vote
1answer
17 views

What is a good way to compare two data pre-processing methods e.g better predictions and/or narrower HPDs?

Given one dataset and two different data pre-processing pipelines, does it make sense to say that one of the processing pipelines is better if, given a regression model, it subsequently leads to a ...
0
votes
1answer
49 views

When should you use different scaling methods?

There are already some good answers on when feature scaling is desired and when to center vs. scale. However, they don't explain which scaling method to use in which situation. For context, assume ...
0
votes
0answers
14 views

Do non-linearly correlated variables affect model interpretability? If so, how can they be detected and removed?

My understanding of multicollinearity is that it's the linear correlation of 2+ predictor variables, and it impedes model interpretability because it makes it difficult to isolate the effects of a ...
1
vote
1answer
25 views

Is it advisable to impute missing values and scale features before computing the Variance Inflation Factor (VIF)?

As far as scaling, Wikipedia says: Finally, note that the VIF is invariant to the scaling of the variables (that is, we could scale each variable Xj by a constant cj without changing the VIF). ...
0
votes
0answers
24 views

Median Absolute Deviation for new outlier

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Assume, that based on Median Absolute Deviation (MAD) $$ MAD = \frac{\sum_{i=1}^{n}|x_{i} - m|}{n}, $$ where $m = median(x_{1}, \dots, x_{n})...
0
votes
0answers
12 views

FCNN with stacking/sequencing data rows vs RNN

I'm training a binary classifier on a time-series data set. I want to know whats the differences between these two scenarios and could the second one perform better in any cases? 1 - using RNN or ...
0
votes
1answer
20 views

Creation of a Target Variable

So i'm new to machine learning and data science, i've been looking tutorials and i'm working on self made project at the moment and i'm having an analysis paralysis with the data preprocessing portion....
1
vote
1answer
33 views

Low memory time series input for deep learning

Background I have some data that looks like this: ...
0
votes
0answers
12 views

Time series object with multiple station data with different time periods?

How can make a time series object from a set of data. I have data set from a number of regional meteorological stations. And I don't know how to create a time series object that includes time series ...
0
votes
1answer
14 views

Do non-linear Support Vector Machines need data to be scaled with zero mean and unit std?

I am trying to understand the difference between standardisation, normalisation and scaling for features. My data roughly follows a bell shaped/Gaussian distribution but the magnitude of their ...
1
vote
0answers
34 views

Time Series Preprocessing

I am just here for brainstorming as time series prediction can be storming to the brain. so basically working with time series prediction doesn't usually follow the same rules as working with other ...
0
votes
0answers
12 views

Spatial Sign Transformation not plotting circular shape as expected

I am attempting to execute Spatial Sign transformation using Phyton. However, I also found that there are not many libraries to use this concept, thus I had to create a function from scratch based on ...
3
votes
2answers
37 views

Binary representation: -1 vs 0

I commonly see {0,1} used in representing pass/fail data, but I recently found a case where I was given a data set with {-1,1} to perform some pre-processing. So, I'm wondering whether I should keep {-...
0
votes
0answers
38 views

What kind of model would be best for customized similarity measures between textual documents?

So one of my projects is to build a bot that brings forth relevant pieces of a document based on an input doc. For example, if my reference document is the bible: ...
1
vote
0answers
22 views

Data Quality Checks---Taking the standard deviation on the number of rows in a dataset?

I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks. My checks are as plain as checking for ...
0
votes
0answers
8 views

Entropy of log normalized data

Is it ok to calculate the entropy of log(2) normalized data? Or should I make to transformation to a linear scale Some more background - I’m dealing with SingleCell data which is by definition is very ...
1
vote
0answers
24 views

rolling removing outliers: include or not include

In the paper "Realized kernels in practice: trades and quotes" by O. E.Bandorff-Nielsen etc. cf. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1368-423X.2008.00275.x in the section dedicated to ...
1
vote
0answers
23 views

Mean Absolute Deviation and data preprocessing

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Next, based on Mean Absolute Deviation (MAD) we aim to decide if the last point $x_{n+1}$ is outlier or not. First, let us compute the MAD: ...
0
votes
0answers
20 views

Graph node binary classification: Training set labels contain false negatives

I'm working on a graph node classification problem. Given a graph G = (V, E) where each vertex v has a set of features x, and each edge e may also have a set of features, I want to predict which (...
0
votes
1answer
29 views

Correct preprocessing strategy in machine learning

Question In order to avoid data leakage, when preprocessing data, we need to Step1: split the dataset into train, validation, and test set. Step2: fit a transformer on train set and transform train,...
2
votes
1answer
51 views

Should I interpolate missing data, or is the last-known value good enough?

I want to study the yearly weight change of a population of about 100 people. I have about 100k measure taken at random intervals over a 10 year period. Each samples point is represented as a 3uple: $...
3
votes
1answer
88 views

Why would we want to normalize (L1) by row?

I've read this question which seems similar but answers didn't address when we would want to normalize by sample (row): Should I normalize featurewise or samplewise I ask specifically about ...
1
vote
1answer
19 views

Organizing repeated measures data with a grouping variable for MLM

I just started reading about multilevel modeling so I can apply the method to my research data. However, I have yet to come across an illustrative example or even discussion how to organize and set up ...
0
votes
0answers
32 views

How should we preprocess our data before logistic regression? [duplicate]

Is it necessary to normalize continuous variables before logistic regression ? Or rather use min-max transformation ?
0
votes
1answer
127 views

is this way of applying data augmentation correct [closed]

I'm training a CNN and want to apply some data augmentation to my input images. I combined some code from tensorflow tutorials and have the following workflow: I have a dataset containing all ...
0
votes
0answers
45 views

For outliers treatment, clipping, winsorizing or removing?

I came across three different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
0
votes
1answer
16 views

how to treat binary variables when normalizing for DL model?

So I am doing preprocessing for deep learning model. I have mix of binary and continuous variables. I will normalize continuous variables to z-scores (to standard normal distribution). Should I "...
1
vote
0answers
78 views

How can I preprocess my data better? Help finding issues with the Dataset to improve accurracy

The 'DATA' and my Jupyter notebooks can be found here Now my issue is that for my data-set I get an mean accuracy from grid search of 0.64 (I get different for other models, like for svm 0.75) but ...
1
vote
1answer
45 views

Machine Learning - Data Pre-Processing [closed]

I'm currently working on a machine learning project. I have data from over 50 sensors but the time at which data was recorded are not the same, the data is not synchronized and there are missing data. ...
3
votes
0answers
21 views

Are there any models that do worse on standardized datasets?

Background I am currently an undergraduate student beginning to explore the field of data science. Recently, our professor introduced us to the concept of standardizing a dataset. My professor ...
1
vote
1answer
58 views

Fix wrong data coming from a sensor

I have data coming from a sensor that I store in a time serie. When I graph them, I obtain: These data are supposed to be "continuous", like temperatures, not going up and down so fast. After ...

1
2 3 4 5 6