Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

What ANN architectures can be used for variable length feature vectors (audio and video)? [closed]

How to input variable length feature vectors (audio and video) to the deep model, instead of fixed length input (e.g. x frames per second), for character recognition based on speech recognition based ...
1
vote
1answer
30 views

How to combine two or more columns which are giving same information?

I am trying to predict the closing price of a cryptocurrency. As its price depends on lots of factors, so I have lots of columns. But a bunch of columns in my dataset are kind of adding the same set ...
1
vote
0answers
14 views

Predicting the lifetime of events using ML, based on observation window

I am trying to predict the lifetime of events based on the features X of the event. There are some caveats to the samples that make training the NN non-trivial: Samples were gathered in a small ...
0
votes
0answers
16 views

Is it okay to use raw body temperature data (data that have a normal range in between abnormal values) as input to a neural network?

While I was doing some research applying machine learning/deep learning on medical data, I came across the question: Is it okay to use raw body temperature data (data that have a normal range in ...
0
votes
1answer
28 views

Is there a right way to aggregate timeseries data for anomaly detection and forecasting?

Hello Cross Validated Users, I have had a question on my mind for a long time whenever I work with timeseries but I have never quite found the right answer. Here is the thing: when creating an ...
0
votes
0answers
13 views

Why whitening is not used with CNN

In Stanford CS231N course notes, it says that whitening is not used with CNN. Is there any reason not to use it? I am also wondering if there are some cases whitening is necessary for preprocessing ...
0
votes
1answer
24 views

How to do feature engineering with scikit learn pipelines? [closed]

When doing preprocessing I've always used pandas to impute, encode, or scale my data. In other words, I've done all of the steps "manually". However, this takes a long time, generates a lot ...
1
vote
0answers
30 views

Categorical variable that is both nominal and ordinal [closed]

In a dataset with house prices I encountered the variable BsmtCond that describes the general condition of the basement. It takes 6 values: ...
0
votes
1answer
36 views

How do I prepare clinical data for multivariate time series analysis using LSTM or RNNs?

I am trying to predict the progression of disease using certain clinical data (time series data) and covariates (such as age, sex, race etc.). I am aware of the existence of mainstream machine ...
2
votes
0answers
31 views

How to preprocess binary labeled sequences of ordered in time data points to fit into RNN?

I am observing some repeated events which are guaranteed to give a binary outcome on termination and I am doing some data preparation so I can fit it into a model. Each event has its own properties ...
1
vote
0answers
22 views

Correct way of mean-centering and scaling time series data used as input to an LSTM?

I am training an LSTM network in tensorflow/Keras which takes eight different time series/features as input. The input matrix given to the network has the form ( nSamples x nTimesteps x nFeatures ). ...
0
votes
0answers
14 views

Image normalization with scaled pixel values

I was going through my code and I realized that I first scale values of the image to <0;1> before calculating the channel-wise mean and std. Later down the line, I normalize the images channel-...
1
vote
1answer
170 views

When should you remove Outliers - Entire Dataset or Train Dataset?

I have been trying to understand the concepts of data leakage and outlier analysis as I am new to data analysis and machine learning. I have googled these topics and understand data leakage but it is ...
1
vote
1answer
13 views

What data sets should be used when calculating scalers?

When doing maching learning tasks, it is common to divide the whold data set into three unoverlapping subsets, namely training set, validation set and test set. I understand that the test set should ...
1
vote
0answers
32 views

How to handle missing data in this case?

I'm practicing data cleaning on this very messy dataset on graduation outcome of a school, I attached some snapshots here: There are a lot of cohorts that don't have the statistics, they seem to be ...
0
votes
0answers
15 views

General strategies to flag distribution shift in live text systems

Does someone know some basic strategies for flagging distributional shift in a live system, particularly for text systems? Many of the solutions online revolve around correcting for this; however, I ...
1
vote
0answers
15 views

What is the influence of the number of channels on the input of a CNN? [closed]

Let's say I want to pass an RGB image through a CNN. It has three channels, each containing a 8 bit integer per pixel, ranging from 0 to 255. Let's say I know encode this three channels in only one ...
0
votes
0answers
8 views

Ways to handle features that are not applicable for all records [duplicate]

I have a data set that contains records of distinct groups identified by a 'type' variable. Depending on value of this 'type' variable certain other variables are either applicable or not. Effectively ...
2
votes
1answer
25 views

Should I scale targets when building regression model with multiple objects?

I'm using TensorFlow 2 to build a regression neural network with four numeric output objects. Each object has a distribution that is close to the normal ...
1
vote
0answers
27 views

Is it reasonable to do Feature Engineering before Data Preprocessing? [closed]

I want to do data preprocessing with scikit learn and create a pipeline, among other things to avoid data leakage and streamline the process. The problem though is that after fitting the data to the ...
0
votes
0answers
22 views

How to prepare MoCap data for training on LSTM model?

I am working with Motion Capture (MoCap) Data, which's considered multivariate time series data as it contains multiple time instants that each describes the XZY rotation specified joints of a ...
0
votes
0answers
12 views

Data whitening removes linear correlations, but how about non linear one?

Data whitening makes covariance matrix to identity matrix, so linear relationship will be removed through data whitening. This makes learning faster. But how can we remove non linear relationship? Is ...
0
votes
1answer
12 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
-1
votes
1answer
131 views

should stemming and lemmatization both be used together or not? what is Best practice in NLP preprocessing?

Since stemming chops of the word and gives back the stem and lemmatization focuses on root form of the word, individually both of them serves the purpose of reducing the length of the word tokens with ...
0
votes
0answers
15 views

Preprocessing deterministic data with sklearn

I am trying to create a set of ML models that will serve as a replacement for a complex deterministic simulation. The simulation requires 4 inputs (x1, x2, x3 and x4) to determine 4 different outputs (...
0
votes
1answer
28 views

Data preparation in cross-validation scenario

In a k-fold cross-validation scenario for validating a regression model, how/when should I do preprocessing of the data? I'd like to know especially when to remove outliers, empty columns etc, i.e. if ...
1
vote
1answer
88 views

How does the kNN imputer actually work?

I've understood that the kNN imputer, being a multivariate imputer, is "better" than univariate approaches like SimpleImputer in the sense that it takes multiple variables into account, ...
1
vote
0answers
44 views

How to identify careless responders when items negatively correlate with the total scale

I'm calculating the Cronbach's alpha on a dataset with the scores of the Coping Orientations to the Problems Experienced (COPE-NVI-25) with R and the psych package. ...
0
votes
1answer
34 views

Which should go first, test/train splitting or standardization/normalization?

I am used to normalizing everything before splitting them up until recently. I ran into a Kaggle notebook where the author split the dataset first without anything modified. That makes me think about ...
1
vote
1answer
56 views

Avoiding data leakage in preprocessing

I'm a data science newbie and a bit confused with the following: I usually do the preprocessing on all predictors of a dataset, meaning I create X by concatenating <...
1
vote
0answers
154 views

Fine-tuning VGG-Face for Facial Expression Recognition on FER2013 - Grayscale vs RGB Images

I am experimenting with Facial Expression Recognition and want to use a pretrained CNN model and a multi-stage fine tuning strategy to deal with scarce data. I came across the work of Knyazev et al. (...
1
vote
0answers
36 views

Winsorizing and ratios [closed]

Say I have a ratio c = a/b. Should I winsorize both a and b and then ...
2
votes
1answer
67 views

Data cleaning on entire data set vs train and test

I'm taking a class now and they have contradicted themselves, so I'm looking for some clarification. Can you standardize your values on the ENTIRE data set or does it have to be done only on train (...
0
votes
0answers
7 views

Is it always recommended to scale features to be predicted the same way as the training data was scaled?

I don't know if this has been asked before... And the question might seem somewhat silly, so let me start briefly with the standard approach: The general advice I have always seen is that, if you ...
2
votes
2answers
30 views

Does the mean normalization step of k-means affect its performance?

A common step when clustering using k-means is to first standardize the dataset so that each feature has zero mean and unit variance. I understand why forcing unit variance helps k-means generate ...
0
votes
0answers
8 views

What uni-variate feature selection method to apply to high-dimensional data?

For context, my task is imbalanced binary classification. I am trying to reduce the 60+ features in my data set with 260 000+ rows. I read that information gain method is not a very reliable solution ...
1
vote
0answers
18 views

EDA and Data cleaning can be done before selecting data samples from population?

I have a huge dataset from which I need to select a sample for machine learning. The data set has high NAN and noise. Hence is it good approach to do EDA, cleaning before taking samples?
1
vote
1answer
36 views

One hot coding in Train Validation and Test set (Production data) [closed]

For example I have below train set. name values 0 Tony 100 1 Smith 110 2 Sam 120 3 Shane 130 4 Sam 140 5 Ram 160 After ...
0
votes
1answer
52 views

How to choose the correct dataset transformation

I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction ...
1
vote
0answers
11 views

Replacing NA values in race column? How should this issue be best approached?

I am working with some data for my university to predict an individual's likelihood to accept an admission letter from the school. One predictor column is race. There are ~12,000 rows, and only ~450 ...
1
vote
1answer
21 views

Data normalization of test data in machine learning

I have 117 samples which I used to select and train a model. What I did: 1) pre-processed the 117 samples (normalization, statistics, etc); 2) created 4 folds (random split); 3) performed a nested-...
0
votes
0answers
40 views

Adding variable and value labels in SPSS across large number of variables

I have a longitudinal dataset with a large number of variables that require variable and value labels. All of the variables have a prefix (e.g., time1., time2., time3.), a suffix (e.g., .age,.sex, ....
0
votes
0answers
10 views

How to take the keywords from the given dataset to train GPT-2 based chatbot?

I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
0
votes
0answers
13 views

Preprocessing on a "branch-like" dataset of varying density

I'm trying to classify a dataset with unsupervised learning. Based on a limited amount of hand-labelled data, I was able to identify two larger sets of points that probably belong to the same class, ...
1
vote
0answers
16 views

Find control chart limits based on existing data

I'd like to produce a control chart that tells me if a given process will be within bounds in the future or not. Currently, the process gives me simple timeseries data, per minute ...
1
vote
1answer
30 views

How to decide whether to keep the duplicate rows or remove them. I have two duplicate records but they refer to two different persons

I am trying to build an NLP model on this data set where I have data from some accidents where I need to predict the Accident Level. There are a total of 13 duplicate rows. But on looking into them I ...
1
vote
1answer
47 views

Should I use different types of normalization on the same dataset when preprocessing for machine learning

I am working to preprocess a dataset where half of it is already normalized between 0 and 1. I was planning on using z-score to normalize the rest of the dataset but I was wondering if that was a bad ...
0
votes
0answers
33 views

How do I handle separate standardization for test and training when doing Cross-Validation

I understand that if I am going to standardize numeric columns in preparation for a machine learning algorithm, I should do this scaling separately for training and testing data, which is fine and ...
0
votes
1answer
28 views

Detecting anomalies question

This would be a data cleaning question, but I guess there are many related phrases and for sure one of them may be anomaly detection. If I have a single feature say height of humans. Question: If I ...

1
2 3 4 5
9