Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Which metrics should be used in preprocessing in continual learning?

So my idea is to train an LSTM - autoencoder for anomaly detection by continual learning, i.e., I want to update the model after each 10 time steps. Firstly I will train it on source data, then re-...
1
vote
0answers
21 views

Padding - which values for standardized time series data?

I have different time series sequences with varying lengths and want to use them as input for an LSTM. My approach is to use padding to fill shorter sequences. Is there any best practice about which ...
0
votes
0answers
6 views

Should a model be ready to handle a missing observation in the test set for any given feature?

Say imputation is sufficient for handling missing values in our problem and that our test set is locked in a vault. Which of the following approaches is recommended? Fit imputation for every feature, ...
1
vote
1answer
27 views

Preprocessing of target data set in Transfer learning approach

So the idea of transfer learning approach is to pre-train a model on source data set and then re-train (or fine-tune) the model on the target data set. But what about preprocessing? If I choose to ...
5
votes
1answer
144 views

When should I do train test split?

I'm new to Machine Learning. I'm basically confused about when to perform train test split. Is the order given below correct? Split entire data into training and test set Extract Features from ...
0
votes
0answers
11 views

What can go wrong if we don't scale and center data in a predictive model? [duplicate]

I have started using R recently, along with the caret package. I note that there is an option to preprocess data so that it is scaled and centered. I am interested in multiple linear regression. What ...
1
vote
0answers
38 views

How to normalize data in R

This is my data: ...
0
votes
0answers
17 views

Can regression coefficients be guaranteed to be normalized?

When solving a multivariate linear regression problem of the form $A\vec{x}=\vec{y}$ where $A$ and $\vec{y}$ are known. Is there any from of preprocessing or scaling that can be done $A$ and or $\vec{...
0
votes
0answers
11 views

Tools to store & visualize larger quantities of data with many permutations?

I have a lot of data coming in with high number of permutations, like: duration, date, country, other configuration data that's specific to use case I want to create counts by permutation and ...
1
vote
0answers
25 views

how to estimate means and SD from clustered means in a study

for conduction of a meta-analysis I am collecting data from various studies. Unfortunately, not every author provides means for all groups, instead, means are clustered. Say, I have independent ...
0
votes
1answer
37 views

When to preprocess data for neural network

In https://cs231n.github.io/neural-networks-2/, the authors say that preprocessing should only be done on the training set, and then the mean, variance, etc. of the training set should be used on the ...
1
vote
0answers
15 views

LabelBinarizer gives too many features on test

Let's say I have a Dataset with a coulum called countries. Lots of the values are usa and there is a small amount of values wich are either ...
2
votes
0answers
41 views

Preprocessing data for the learning step

I am currently reading "Human level control through deep reinforcement learning" and I came across the algorithm in the paper. I am confused because the algorithm uses a different notation ...
2
votes
1answer
34 views

Is randomly split dataset to train and test in lstm model reasonable?(human activity recognition)

I have build an lstm model to predict human activity recognition with dataset OPPORTUNITY. I did two experiment with different oder of processing as below, normalized the dataset with minmax scaler, ...
0
votes
0answers
41 views

Preprocessing on ImageNet for MobileNetV2

I have saved an ImageNet model in ONNX for MobileNetV2 by doing the following: ...
11
votes
3answers
1k views

Why can't scikit-learn SVM solve two concentric circles?

Consider the following dataset (code for generating it is at the bottom of the post): Running the following code: ...
0
votes
0answers
9 views

How to construct a regression with regressors such as mean, std, max,

I want to construct a logistic regression for binary classification. My data are data about computer network traffic (specifically CIC-IDS 2017 publicly available data set). These data consist of ...
0
votes
0answers
14 views

What are some ways to improve the accuracy in this SVM?

I am an absolute beginner in the field of Machine Learning. I am trying to teach a binary classifier using the following dataset https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients ...
0
votes
1answer
20 views

Encoding Approach for Paired value

I'm having a problem with preprocessing my input data for later downstream analysis. Example data: ...
2
votes
2answers
215 views

Does the feature selection matter for learning algorithm with regularization?

Let's assume we have infinite computing power. When we consider two algorithms, learning algorithm + regularization and feature selection + (learning algorithm + regularization), Which one would ...
0
votes
0answers
13 views

Does Outliers in Categorical Feature varibales exist?

So, I was working on an Exploratory Data analysis project, after dealing with all the preprocessing of numerical features of the dataset when I started analyzing categorical data(nominal) there I ...
1
vote
1answer
13 views

Simple preprocessing large columns set

I have a huge dataset about the effect of the drug on cancerous cell lines with a 17k column. I need to prepare a simple regression, but I don't know, how to pick the most important columns. I ...
1
vote
1answer
14 views

The correct way of performing OneHotEncoding for neural network models

I have a dataset of multiclass (0,1,2) labels. Since I am using Keras, I need to first perform OneHotEncoding on the class column in the dataset using ...
0
votes
1answer
18 views

Is it possible not to perform transform() on test data

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
0answers
9 views

Large amount of missing values in as input features for LSTM time series

I am using an LSTM to predict a time series chart from multiple other time series charts as input features. The problem is that some of these input charts have much ...
0
votes
0answers
23 views

Find Correlation between Grouped and Ungrouped Data

I have 2 datasets. First is a Quarter-yearly data that has PTR (Pay-through-rate) value associated with an Agent for every Quarter. The second dataset has the detailed data for Sales related to those ...
0
votes
0answers
10 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
2
votes
1answer
24 views

Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]

First post! If you don't want to read the background you can skip to the Problem heading below. Background Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
0
votes
1answer
15 views

Handle various shapes of distribution in data preprocessing

Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
10 views

Freedmans rule to find number of bins

I have a data set that is 30162 rows long. I am trying to split age into bins but I am struggling to understand the concept of the rule with my data set. As a quick examples this is what I did: <...
0
votes
0answers
31 views

How to preprocess performance counter input data for anomaly detection using autoencoders

I am working with more than 250 input features that include system performance counters and SQL Server database counters to predict anomalies / system outages. I am looking to use an autoencoder ...
1
vote
1answer
23 views

Wilks.test usage in R

I want to calculate the Wilks' lambda for a given data set. The "rrcov" library in R appears to have a function for this purpose. The examples provided in the documentation run without a problem; ...
0
votes
0answers
11 views

Feature Engineering: How to deal with imbalanced numerical/categorical features

I'm analyzing a data set and solving a classification problem and find that values concentrate on one number in many features. For example, a categorical feature 'loan' indicating a person having loan ...
4
votes
1answer
79 views

Classification of data tables (each table is an item)

I have to work on a binary classification task where single items to be classified are not single rows of a data matrix, but groups of rows. In other words, I have $N$ data tables of varying size $n_i ...
0
votes
0answers
18 views

Transformation vs Scaling

I'm not sure I understand the different uses between the 2 methods: Scaling - scale features to same scale (Normalization or Standardization) Transformation - makes the data normally distributed ...
0
votes
0answers
23 views

What is the best way to handle 2 (or more) missing values in same row?

one of the methods to handle missing data is using predictive model. If we have a row with 2 or more missing values, is it right (and accurate) to use ...
0
votes
0answers
21 views

Is it right to use different feature scaling techniques to different features?

I read this post about feature scaling: all-about-feature-scaling The two main feature scaling techniques are: min-max scaler -...
1
vote
1answer
20 views

Feature Engineering with Focus on KNN

I have seen a number of helpful posts such as this one on feature engineering, but I am specifically looking for something that may be helpful when using KNN. In my experience, some features work best ...
1
vote
0answers
14 views

Questions about pre-processing/transformation of data

For an assignment for a ML Online course I have to find the best classifier for a given data set using 4 different methods: Logistic regression, Decision tree, Support Vectors and K Nearest Neighbours....
0
votes
1answer
31 views

Data pre-processing on test files

I am working on a classification problem. I have a training file with a label and a separate test file without a label field. I needed to remove some rows that contained missing values from the ...
1
vote
0answers
60 views

Highly correlated variable, when to remove

I have a dataset that includes both course number and course name. For some models such as regression, a Pearson Correlation matrix is obtained and one of any pair of features that are highly ...
0
votes
0answers
22 views

Building an analytical base table from a relational database

I need to train a (supervised) machine learning model. However, up to now I have only made model for data that was given to me as 1 single data frame. Now I need to create a model for data that is ...
1
vote
1answer
17 views

What is a good way to compare two data pre-processing methods e.g better predictions and/or narrower HPDs?

Given one dataset and two different data pre-processing pipelines, does it make sense to say that one of the processing pipelines is better if, given a regression model, it subsequently leads to a ...
0
votes
1answer
65 views

When should you use different scaling methods?

There are already some good answers on when feature scaling is desired and when to center vs. scale. However, they don't explain which scaling method to use in which situation. For context, assume ...
0
votes
0answers
15 views

Do non-linearly correlated variables affect model interpretability? If so, how can they be detected and removed?

My understanding of multicollinearity is that it's the linear correlation of 2+ predictor variables, and it impedes model interpretability because it makes it difficult to isolate the effects of a ...
1
vote
1answer
61 views

Is it advisable to impute missing values and scale features before computing the Variance Inflation Factor (VIF)?

As far as scaling, Wikipedia says: Finally, note that the VIF is invariant to the scaling of the variables (that is, we could scale each variable Xj by a constant cj without changing the VIF). ...
0
votes
0answers
26 views

Median Absolute Deviation for new outlier

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Assume, that based on Median Absolute Deviation (MAD) $$ MAD = \frac{\sum_{i=1}^{n}|x_{i} - m|}{n}, $$ where $m = median(x_{1}, \dots, x_{n})...
0
votes
0answers
12 views

FCNN with stacking/sequencing data rows vs RNN

I'm training a binary classifier on a time-series data set. I want to know whats the differences between these two scenarios and could the second one perform better in any cases? 1 - using RNN or ...
0
votes
1answer
31 views

Creation of a Target Variable

So i'm new to machine learning and data science, i've been looking tutorials and i'm working on self made project at the moment and i'm having an analysis paralysis with the data preprocessing portion....
1
vote
1answer
35 views

Low memory time series input for deep learning

Background I have some data that looks like this: ...

1
2 3 4 5 6