Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0 votes
0 answers
2 views

Leaving duplicated entries in a dataset at pretraining stage

I'm adopting a fine-tuning approach after having pretrained a deep learning model (transformer) on a source dataset (let's call it dataset A) and then fine-tuning it on a target dataset (B). Dataset A ...
James Arten's user avatar
1 vote
0 answers
47 views

Valid approach: Winsorizing data for main analysis and then doing sensitivity analysis without winsorizing?

I've got a variable with psychological data (N=75) which is distributed pretty symmetrical, but has very few cases with very extreme values, more extreme to the left tail. But nevertheless this data ...
Malea Dondé's user avatar
0 votes
0 answers
20 views

Scaling data to a sample that is neither training nor validation. Is this data leakage?

TL;DR: Data being scaled to a sample that is neither training nor validation. Is this data leakage? Hi, I have a data set from samples that are distributed on a plate. Precisely, there are 96 wells ...
Luiz Gustavo's user avatar
0 votes
0 answers
20 views

How to adjust for double counting data to account for multiple perspectives in logistic regression

I am trying to measure if referees are biased in favor of players from the same geographic region (as themselves) in a sport. I want to see if players from the referee's region are more likely to win. ...
ral's user avatar
  • 101
2 votes
1 answer
105 views

How to normalise outputs of neural networks with different distribution?

I have a NN model that predicts 8 different variables. I use a multi-task learning approach, where I compute the loss between predictions and targets for each of ...
bird's user avatar
  • 95
2 votes
2 answers
239 views

Is it okay to use different data preparation procedures for different models?

I'm currently learning some basics of building models, and I was wondering whether it's okay to use different data preparation procedures for different models. Specifically, I was considering an ...
mrepic1123's user avatar
0 votes
0 answers
29 views

Help interpreting normalized HMM (or otherwise) results

I have run a hidden markov model with five variables on very different scales. Because of this I normalized the input data beforehand using Carets preprocessing: ...
Jcarroll's user avatar
3 votes
1 answer
27 views

How to find correlations of the features in a dataframe? The features are of mixed types (nominal, ordinal, discrete, and continuous)

I am working with python using pandas, and seaborn libraries. I have a dataframe, that I am using for some machine learning. My dataframe has a target variable, along with several other features. ...
letdatado's user avatar
  • 325
1 vote
1 answer
236 views

How to interpret pairplots()

I have been working on a Classification problem. And I want to see how many features associate with the target variables. Let me share an example. I got this pairplot using ...
letdatado's user avatar
  • 325
0 votes
0 answers
19 views

How to prepare sequential word data for many-to-many LSTM?

I want to build a many-to-many LSTM and I need some guidance on how to prepare the data that I have for such a model. My data frame consists of subject_ID (dance couple), Time (in s, time step in a ...
Rūta Slivkaitė's user avatar
0 votes
0 answers
77 views

Data mining for categorical variables that correspond to modes of a continuous variable

Is there a good method or test to find categorical variables associated with modes of a continuous variable? Take the velocity of a car as an example. You would expect there to be at least two modes ...
noNameTed's user avatar
  • 113
0 votes
1 answer
95 views

Outlier Detection and Removal

I am reading a paper on wind power forecasting and the authors present a plot of the data before outliers are removed and a plot after. However, they don't actually say what method was employed to ...
x0929's user avatar
  • 1
0 votes
0 answers
15 views

Cleaning Panel Data with Merging Firms

Im cleaning some panel data for a regression. Subjects are firms and a substantial amount of firms have been involved in mergers. Should I just drop all firms that have ever been part of a merger? ...
Liam's user avatar
  • 21
0 votes
0 answers
9 views

Using Unordered Text Features in Machine Learning Model

I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
Omnitragedy's user avatar
2 votes
2 answers
94 views

Non-constant treatment periods for A/B test samples

I have 4 A/B testing samples with 4 treatment groups and their 4 respective control groups (n=10000~), with known (but not distinct) treatment periods. The issue is that the treatment periods overlap ...
scooptywoopty's user avatar
0 votes
0 answers
30 views

How do i prioritize which features to use in my machine learning model before the feature engineering stage?

I am encountering a probably fairly common problem where I have too many features, lets say 500 possible features. I only want to pick the top 10-50 features that would be the most predictive of y, or ...
Katsu's user avatar
  • 911
0 votes
0 answers
53 views

Using pycaret's preprocessing on unseen dataset

I'm thoroughly enjoying pycaret to handle much of the legwork in my analysis. I'm making heavy use of the setup() method in preprocessing to handle normalization, ...
neanderslob's user avatar
2 votes
1 answer
131 views

Geometric Intuition Behind Whitening for ICA

I know there are a couple posts asking about why we whiten the data for ICA. I understand why we whiten to fix scaling invariants between the sources and to increase the computationally efficiency. ...
user19402204's user avatar
2 votes
0 answers
32 views

Data leakage: Train test split before or after data preprocessing? [duplicate]

A while ago I came across the word "data leakage" for the first time, and after some research, I found that it is a common mistake among data science/machine learning practitioners. But the ...
jairiidriss's user avatar
0 votes
0 answers
9 views

organizing analytical file for persons with multiple records

My stakeholders have manually recorded data for patients enrolled in an intervention, which is causing data issues that I need to resolve in order to move to determine what is the appropriate ...
Ronald Sanchez's user avatar
0 votes
0 answers
10 views

How can I get knowledge graph embedding by using TransE? [duplicate]

I saw nobody answered Ssong, however now I'm struggling with the same question. Can anyone help? Thanks.
Yolanda Zhu's user avatar
2 votes
2 answers
73 views

Univariate vs. Multivariate Standardization

There are several common methods for scaling input features to machine learning models prior to training the model. The most popular methods seem to be standardization (centering by the mean and ...
noNameTed's user avatar
  • 113
0 votes
1 answer
24 views

How to organize an overlapping ordinal variables?

I am analyzing some data with attributes about how long it takes to commute, and it is an overlapping ordinal variable like the following: Time Number of People 1-5 minutes 10 6-10 minutes 24 less ...
Po Sang Yu's user avatar
0 votes
0 answers
13 views

Should the data be scaled before normalisation to enable the use of the model as a pre-trained model?

I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data. Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
Janikas's user avatar
  • 101
1 vote
0 answers
86 views

Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?

After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
Museful's user avatar
  • 375
1 vote
1 answer
24 views

Changing scale and rounding off Target

I am training a regression machine learning model to predict an airplane's maximum take-off weight based on some pre-project features. The airplane weights in the dataset I'm working with are ...
Heitor Nunes Rosa's user avatar
0 votes
0 answers
48 views

Is the correlation method used within your dataset problem dependent?

Question Say I have a dataset $D$ with $N$ features that are trying to predict a target $y$. I would like to build a model from $D$ and part of that process is removing correlated columns to reduce ...
Connor's user avatar
  • 625
0 votes
0 answers
21 views

Cleaning Data Before SVM

I want to classify diabetic retinopathy grades using SVM. I have 32 extracted features, and those features won't all be used in classification stage. Before entering feature selection, I want to clean ...
anastasia's user avatar
0 votes
0 answers
47 views

How to find (and interpret) outliers in set of time series data?

I face with a problem of doing a time series forecasting on multivariate data in the form where different entities have their own 100-day (daily) series of 10 variables, and I'm expected to predict 10-...
Fredrik's user avatar
  • 725
2 votes
1 answer
165 views

Do we need preprocessing before applying a sophisticated undersampling method?

My question is around applying undersampling methods to an imbalanced, and highly dimensional dataset, with mixed data. Lets say as an example, I have 150 features, also a highly imbalanced binary ...
Ali Kılınç's user avatar
1 vote
0 answers
11 views

Training computationally cheap models for feature selection and then extending to more flexible models

To what extent can I apply lessons learnt about the model performance on a simple "cheap" model (e.g. Naive Bayes, LDA) to a more flexible model (i.e. ensemble of trees, neural networks)? ...
Douw Marx's user avatar
0 votes
1 answer
34 views

Deep learning classification with multiple temporal data

I'm working on a project to predict the category of music segments in an audio file (represented in pianoroll format with an additional column for the corresponding class). Each row represents the ...
Tria Ufo's user avatar
0 votes
0 answers
16 views

How to apply dimensionality reduction to a data set with outliers?

I try to apply dimensionality reduction to a multidimensional data set (with numerical features) with significant outliers. I have managed to identify outliers with Isolation Forest but now I'm in a ...
Fredrik's user avatar
  • 725
1 vote
0 answers
27 views

Clustering on thousands of product feature clicks and pages viewed

I want to classify 120k customers into 5-6 clusters basis the product usage, say, hundreds of product features clicked and hundreds of product pages viewed. The data will be like a customer_id has ...
Nebulum's user avatar
  • 31
1 vote
0 answers
22 views

Stardardization for Random Forest, SVM and Logistic Regression

I have a classification project and I want to compare three models: Random Forest, SVM and Logistic Regression. Random Forest are tree based algorithms wheras, SVM is a distance based model and LR is ...
OneManArmy's user avatar
1 vote
0 answers
42 views

data preprocessing with zeros dominating

I'm working on a machine learning classification project and i faced some difficulties: all of my features distributed like this: I'm not sure what should i do, should i use any scalers/other ...
alffff's user avatar
  • 11
2 votes
1 answer
474 views

Do we lose information when we normalize an image? [closed]

Before training a machine learning algorithms, it is advisable to perform feature scaling. Suppose we have a "toy" dataset where each image is composed of two pixels $x_0$ and $x_1$. Lets ...
ado sar's user avatar
  • 471
3 votes
2 answers
703 views

change point detection of time series

Currently, I am working on a research project that involves forecasting electricity consumption using data mining. In my analysis, I have detected the change point for time series using the ...
rashmi's user avatar
  • 51
1 vote
0 answers
19 views

When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?

I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing. It's given that ...
Arpit Sisodia's user avatar
0 votes
1 answer
184 views

Order of pre-processing the dataset

suppose I have categorical dataset, I'm doing data pre-processing. what is the correct order of applying these 3 techniques Train Test split SMOTEN to over sampler the minority class Categorical ...
Mohamed Ahmed's user avatar
2 votes
3 answers
205 views

What is "information leak from test to train" ? Is stratification by target a leak?

It's common practice to do procedures such as standardization and even missing value imputation (commonly based on some means) after train/test split - otherwise it is treated as information leak from ...
Ars ML's user avatar
  • 31
0 votes
1 answer
125 views

Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?

I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
iris's user avatar
  • 11
1 vote
0 answers
39 views

Different preprocessing steps for time series data vs. regular cross sectional data?

What are the different preprocessing steps for time series data vs. regular cross sectional data? Eg 1. When doing train/test or cross validation, you cannot randomly split the data. The data must be ...
Katsu's user avatar
  • 911
0 votes
1 answer
147 views

Preprocessing on training set only or both training & test set? Seems like there would be errors for both answers

Let's say I have a dataset that hasn't been split into train/test yet. Upon loading it, I discover that there are columns where there are nulls that need to be filled in, some quadratic relationships ...
Katsu's user avatar
  • 911
1 vote
0 answers
147 views

Better way to create a pandas dataframe with variable feature length for regression?

I am doing a project trying to see whether we can infer the power factor (cos(phi) = P/S) from voltage (U) and current (I) data. What this means exactly is not very important for my question. My ...
D T's user avatar
  • 11
0 votes
0 answers
74 views

How to handle a column with both float and categorical values

This sounds like a question that should have come up before but I couldn't find it on CV. I am trying to use a column called limit_price as in the limit price of an order for a machine learning ...
wizz's user avatar
  • 143
2 votes
1 answer
51 views

Should I remove 0 values from my dataset if they seem to be from instrument error?

I have conducted an experiment looking at the decay rate of a DNA target in flowing water over time, with 4 replicates of each treatment. The data is collected via dPCR Quiacuity instrument that ...
Mitchell Liddick's user avatar
0 votes
0 answers
50 views

Not balancing the validation and test set gives me a very bad dev set result

Here is the resulting heatmap for train, val, test set. I also apply PCA to train,val,and test set. Because from the train set, there are a lot of features that has high correlation with each other. ...
UrDailyCS's user avatar
1 vote
2 answers
880 views

how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?

I am trying to pre-process data following a statement in a paper. They said for the normalization, each dataset is normalized on a per channel basis with a sample range from -1 to 1 and a mean value ...
Margie Shi's user avatar
1 vote
0 answers
35 views

Input Normalization for Preprocessing Time Series Data

I have data traces for separate days that I am reading in from separate files. I'm splitting the data into smaller sequences to train a state-prediction model on, but all the data in a sequence must ...
big_vibes's user avatar

1
2 3 4 5
11