Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Is it always recommended to scale features to be predicted the same way as the training data was scaled?

I don't know if this has been asked before... And the question might seem somewhat silly, so let me start briefly with the standard approach: The general advice I have always seen is that, if you ...
2
votes
2answers
24 views

Does the mean normalization step of k-means affect its performance?

A common step when clustering using k-means is to first standardize the dataset so that each feature has zero mean and unit variance. I understand why forcing unit variance helps k-means generate ...
0
votes
0answers
7 views

What uni-variate feature selection method to apply to high-dimensional data?

For context, my task is imbalanced binary classification. I am trying to reduce the 60+ features in my data set with 260 000+ rows. I read that information gain method is not a very reliable solution ...
1
vote
0answers
9 views

EDA and Data cleaning can be done before selecting data samples from population?

I have a huge dataset from which I need to select a sample for machine learning. The data set has high NAN and noise. Hence is it good approach to do EDA, cleaning before taking samples?
1
vote
1answer
20 views

One hot coding in Train Validation and Test set (Production data) [closed]

For example I have below train set. name values 0 Tony 100 1 Smith 110 2 Sam 120 3 Shane 130 4 Sam 140 5 Ram 160 After ...
0
votes
1answer
34 views

How to choose the correct dataset transformation

I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction ...
1
vote
0answers
9 views

Replacing NA values in race column? How should this issue be best approached?

I am working with some data for my university to predict an individual's likelihood to accept an admission letter from the school. One predictor column is race. There are ~12,000 rows, and only ~450 ...
1
vote
1answer
16 views

Data normalization of test data in machine learning

I have 117 samples which I used to select and train a model. What I did: 1) pre-processed the 117 samples (normalization, statistics, etc); 2) created 4 folds (random split); 3) performed a nested-...
0
votes
0answers
8 views

Adding variable and value labels in SPSS across large number of variables

I have a longitudinal dataset with a large number of variables that require variable and value labels. All of the variables have a prefix (e.g., time1., time2., time3.), a suffix (e.g., .age,.sex, ....
0
votes
0answers
8 views

How to take the keywords from the given dataset to train GPT-2 based chatbot?

I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
0
votes
0answers
13 views

Preprocessing on a “branch-like” dataset of varying density

I'm trying to classify a dataset with unsupervised learning. Based on a limited amount of hand-labelled data, I was able to identify two larger sets of points that probably belong to the same class, ...
1
vote
0answers
16 views

Find control chart limits based on existing data

I'd like to produce a control chart that tells me if a given process will be within bounds in the future or not. Currently, the process gives me simple timeseries data, per minute ...
1
vote
1answer
19 views

How to decide whether to keep the duplicate rows or remove them. I have two duplicate records but they refer to two different persons

I am trying to build an NLP model on this data set where I have data from some accidents where I need to predict the Accident Level. There are a total of 13 duplicate rows. But on looking into them I ...
1
vote
1answer
22 views

Should I use different types of normalization on the same dataset when preprocessing for machine learning

I am working to preprocess a dataset where half of it is already normalized between 0 and 1. I was planning on using z-score to normalize the rest of the dataset but I was wondering if that was a bad ...
0
votes
0answers
32 views

How do I handle separate standardization for test and training when doing Cross-Validation

I understand that if I am going to standardize numeric columns in preparation for a machine learning algorithm, I should do this scaling separately for training and testing data, which is fine and ...
0
votes
1answer
23 views

Detecting anomalies question

This would be a data cleaning question, but I guess there are many related phrases and for sure one of them may be anomaly detection. If I have a single feature say height of humans. Question: If I ...
0
votes
0answers
12 views

Is it valid to use different scaling techniques for different features in a dataset?

I am currently working with a dataset that has a few different features. Ultimately, I would like to train a binary classification model on this dataset. I would like to scale my data, as I plan on ...
0
votes
0answers
23 views

Is it a bad idea to always standardize all features by default? [duplicate]

Is there a reason not to standardize all features by default? I realize it may not be necessary for e.g., decision trees but for certain algorithms such as KNN, SVM and K-Means. Would there be any ...
0
votes
0answers
19 views

Effect of data scaling on model training

Standardization of a dataset is a common requirement for many machine learning estimators. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective ...
0
votes
1answer
10 views

Joining train and test dataset when preprocessing data with encoders

So I've been looking at several beginner Kaggle kernels for the 'Titanic - Machine Learning from Disaster' competition. In these kernels, I noticed that sometimes they combine train and test data, ...
0
votes
0answers
21 views

Tensorflow unit scale preprocessing layer

I would like to have a keras model self-contained to reduce the training/serving skew. It would mean here having a preprocessing layer that is doing essentially what MinMaxScaler from scikit learn is ...
0
votes
0answers
15 views

Proving Yeo-Johnson transform

I am reading “A new family of power transformations to improve normality or symmetry” by Yeo and Johnson, but cannot go through the equation (3.1) where authors declare $l_n(\theta|x)=-\frac{n}{2}log(...
0
votes
0answers
20 views

Dealing with skewed data but cannot use log-transformation

It seems that the popular solution to dealing with skewed data is to apply log-transformation. But in my case, the data is a rating score (range form 0-5). The distribution of the data looks like ...
0
votes
0answers
12 views

When conducting a reliability test using Cronbach's alpha, is it better to conduct the reliability test on the raw data or the cleaned data?

I'm working on a longitudinal study, and I had to take out some participants who didn't meet the required number of days completed. When conducting a reliability test using Cronbach's alpha, is it ...
0
votes
1answer
17 views

Handling imbalanced data for regression based tasks

I have an imbalanced google analytics dataset. I'm interested in predicting total.tranactionRevenue but, of the 70,000 data points only 700 have transactions. The value of these transactions varies ...
0
votes
0answers
7 views

Data lineage/traceability in pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
0
votes
0answers
13 views

Is Data Normalization absolutely required for training a neural network? [duplicate]

I am relatively new to Deep Learning and wanted to implement a Variational Autoencoder for images. For data preprocessing I only rescaled the pixel values from the range of [0..255] to [0...1]. The ...
0
votes
0answers
39 views

How to preprocess data that is not mean reverting?

One of the most common ways of preprocessing data before feeding it to a machine learning model is to subtract the mean value of the data and dividing it by its standard deviation. Such that the data ...
0
votes
0answers
9 views

Subset dataframe by conditioning on two observations within a column for each subject (group)

I am looking to compare the results of a survey (not shown) before and after teaching. You can see below that the last column column specifies whether this observation contains the information from &...
0
votes
0answers
20 views

How should I approach clustering with features with different lengths for different data?

I'm trying to cluster different audio files and will be using features that vary in length with the length of the audio file. For example, one of the features is the pitch over time of each audio ...
0
votes
0answers
30 views

How to add attributes to nodelist from calculations on an edgelist?

I have a bimodal network of people who write letters soliciting money from various entities made from an edgelist and nodelist (MRE code below). How do I add columns to my nodelist based on ...
1
vote
0answers
29 views

Database for Macroeconomic Time-series [closed]

I have decided to improve my well-being and in case being successful write a note about this and share it with my peers for free and try to help them improve there well-being as well. But on this road,...
0
votes
0answers
6 views

How to process multidimensional feature in python?

Hi there so my dataset looks as follow: Patient ID Medicine Death 1 A,B,C,D,E 1 2 B,D 0 3 A,D,E 1 So my dependent feature is death and my independent feature is medicine. I am trying to predict ...
1
vote
1answer
32 views

how do I Clean the Data when iqr is qual to 0?

I want to clean my data but i faced a variable with more than 86 percent zero and consequently iqr=0 when i want to clean outlier, that feauture are eliminated because all of nonzeros were defined as ...
0
votes
0answers
15 views

Different behavior of MinMaxScaler() on similar range of series

I am trying to understand Sklearn's MinMaxScaler's different behavior for the similar series. I've 2 sets of series with 2 of each in it call it Normal and New. In each of the set, I want to reduce ...
0
votes
0answers
17 views

How much does upcasing benefit NLP?

I have been wokring on an NLP project at work where the available training data has all been preprocessed (upcased, some characters removed etc.). However, I have just been informed that the data I ...
2
votes
2answers
43 views

Numeric variable with outliers as a categories

I'm working with a dataset that has a few variables that I'm having difficulty trying to preprocess. So one of them is called MENTHLTH where it is a numeric variable. The point of the variable is to ...
0
votes
0answers
19 views

How to clean up outliers in regression which cannot be visualized?

Recently I meet a problem in an interview. Given a dataset $\{(X_i, y_i) \}$ for regression problem, how to detect and clean up outliers before starting using any regression algorithm. The following ...
0
votes
0answers
8 views

How can I regularize data pre-processing parameter estimates (center, scale, rotation matrix)?

Suppose I have an $N×1$ vector $Y_{in}$ of response values an $N×P$ matrix of predictors $X_{in}$ whose individual columns exhibit significant correlation. Let's further suppose that this matrix $X$ ...
0
votes
0answers
18 views

Can you use a single missingness indicator for several variables?

I have a data set of property sales where information on previous sales are included as predictors. If a property has not been previously sold, the value for predictors such as "Previous sale ...
0
votes
2answers
50 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
0answers
23 views

Can different types of feature scaling result in different prediction performance and how to choose one type?

Being new to machine learning and currently making use of a MLP-Classifier from scikit learn to solve a multi-class multi-label classification problem, I was wondering how to decide on a type of ...
0
votes
0answers
30 views

Does standardization require the existence of normally distributed features?

In some resources, websites, and blogs, I saw an assertion related to standardization operation in machine learning, and I could not make sure whether it is really true or not. The assertion is given ...
2
votes
0answers
63 views

Predicting an annual event – modeling on an annual or monthly basis?

Suppose I'm interested in predicting which of my current customers are likely to renew their insurance at some point in the year. The renewal can happen at any time in the year. I want to proactively ...
0
votes
0answers
55 views

How to approach variable engineering for panel data with much lower number of time steps than subjects?

I have an unbalanced panel data with binary label for each subject at each time step. The maximal number of time steps is 113 and minimal is 3. I have 97 variables and over 100.000 subjects which are ...
0
votes
0answers
27 views

Image intensity normalization in preprocessing

Suppose having two images on a given scale, for example it could be the classic [0-255], representing the same thing but with different value intensities, i.e. the first could have a maximum pixel ...
3
votes
1answer
39 views

Padding a time series for neural networks during cross validation

I am trying to train a neural network on some time series data and decided to implement cross validation for my model. The cross validation method I'm trying to implement is the Day Forward-Chaining ...
0
votes
2answers
218 views

Different input size for training and prediction in CNN for image segmentation?

I’m relatively unexperienced when it comes to deep learning and I’m trying to reimplement a CNN architecture for segmentation of medical images based on a paper. In the paper they state that they use ...
0
votes
0answers
20 views

power transformation with negative values?

Is there a name for this type of transformation: \begin{equation} sgn(x) * |x|^p \end{equation} where $p$ is an arbitrary number (e.g., 1/2 and 1/3 for square and cube roots, respectively) and $sgn$ ...
1
vote
1answer
34 views

Hierarchical Categories as Input Features

I have a regression problem. Two input features describe a category and subcategory. For illustrative explanation, let's consider we speak about city and district. Some more details about the ...

1
2 3 4 5
9