Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

How to eliminate subset and eliminate duplicate rows most efficiently (python & pandas) [closed]

I have a dataset that has many columns: among them AMS card number, registration date, and first purchase date. The data has duplicates for a large number of AMS card numbers. The final dataset ...
0 votes
0 answers
11 views

What is the proper order of preprocessing steps in a dataframe containing categorical and numerical variables? [closed]

I have a dataframe where X is comprised of categorical (nominal and ordinal) and numerical variables, and y is numerical (continuous). Sort out X's nominal, ordinal and numerical variables. Ordinal ...
0 votes
0 answers
6 views

Why is data whitening (decorrelation) not tried with a post multiplication of a whitening matrix?

The general procedure of whitening(decorrelating) a data $X$ with dimensions $(D, n)$ (n being the number of samples) rests upon finding a matrix $W$ with dimensions $(D, D)$ such that the transformed ...
0 votes
0 answers
29 views

Why do we scale features in PCA? Wouldn't that mean the variance in all dimensions is just $1$? [duplicate]

According to https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html, Feature scaling through standardization (or Z-score normalization) can be an important ...
  • 133
0 votes
0 answers
33 views

Is there an accepted standard for storing factor order information? (And if not ...)

I use a lot of survey data, and often get a csv of the responses -- however, this almost always requires an enormous amount of extremely tedious transformation, before it can be analyzed. In ...
1 vote
1 answer
16 views

Oversampling for Continuous Values

I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
0 votes
1 answer
12 views

Training data from real hardware

I am working on machine learning model. The target is to learn the behavior of a black box which has a couple of inputs and one output signal. I have generated some training data by applying different ...
1 vote
0 answers
12 views

How should I use TimeSeriesSplit for validation of Classifier models?

My problem is whether I should use TimeSeriesSplit for cross validation of my classifier models (SVC, Random Forest etc). The input data has a bit of temporal sense. The X features are rolling(1yr, ...
0 votes
0 answers
9 views

How to properly use isolation forest for data cleaning in a machine learning project?

I am trying to clean my data set (X) from outliers with isolation forest. Then I am going to use the cleaned data set for supervised learning. Should I first split ...
  • 1
3 votes
0 answers
14 views

Examples of Leakages in the Training Data

I was wondering about Data Leakage in the data preparation phase during the training of a model. By definition, data leakage happens when information is revealed to the model giving it an unrealistic ...
0 votes
0 answers
25 views

How to interpret a missing data combination-frequency plot (e.g. VIM::aggr)

I am exploring some missing data, and would like to know what principles to apply in interpreting the following type of plot. This example is from VIM::aggr() in R, although this question is not about ...
2 votes
2 answers
61 views

Detect quoted text in emails

I have small dataset (<10k) of emails (plaintext) that need to be classified. Currently I'm doing research on topic of email preprocessing and I can't find any suitable solution for quoted text ...
  • 23
1 vote
0 answers
8 views

Standardization on production dataset

Say I trained a Logistic regression model on a training dataset (sample = 2000), using Standardization. I than test the model on a test dataset (sample = 400) using the Standardization parameter ...
  • 11
1 vote
0 answers
14 views

Constant boolean variables in multivariate time series observations

I am observing multivariate time series with some of its variables being boolean. One example of such observation has its var1=1 during all time series, and another observation has var2=0 for example ...
1 vote
1 answer
73 views

Data Preparation in R: normalization, log - order [closed]

I have a little problem with my data (GDP per capita, some control variables with negative minimums (e.g. FDI) and explanatory variables without negative values but also different ranges. Originally, ...
0 votes
0 answers
11 views

What is the procedure for data preprocessing for time-dependent LSTM classifier?

I attempt a beginner level LSTM classification task with a time-series numerical data, but my task is finding changes in features over time (in which those changes describe the outcome or the classes),...
0 votes
0 answers
16 views

Handling new customers in customer propensity model

I'm using last four years' data to predict whether they will buy or not buy in the next quarter. One problem I'm facing is customers who are not four years old. Is it right to keep them in the data ...
  • 203
2 votes
2 answers
21 views

Transformation of mix normal and skew features [duplicate]

I have a weather dataset containing four features that are continuous values. Temperature is almost normal, but precipitation is highly negatively skewed. In addition, wind speed and humidity are ...
  • 51
1 vote
1 answer
19 views

Removing categorical predictors with mode frequency above certain threshold

Is there a rule of thumb for dropping certain categorical predictors if the most frequent value in the column is above a certain threshold? For example, should I drop a column/predictor if more than ...
  • 131
1 vote
0 answers
35 views

Unsupervised learning (clustering) before supervised learning [closed]

Is it a common practice to do clustering before supervised learning to eliminate "noisy data"? Obviously, depending on the type of task. It seems like it makes sense in my case and my neural ...
  • 11
1 vote
1 answer
35 views

How to transform prediction std of gaussian process back to origin

I am looking for a way of rescaling the predictions of my Gaussian Process Model back to the original scale. The data is scaled for training using a ...
0 votes
2 answers
70 views

How to handle highly correlated observations (rows)

What is the best practice to handle highly similar/ autocorrelated observations (rows) in a data set. These highly similar rows could come from recording (some of the) observations at too close ...
  • 197
1 vote
1 answer
52 views

How to distinguish numerical categorical (Ex: White = 1, Latino = 2 etc) from numerical continuous or discrete variables on a dataset? [duplicate]

I am currently working on a project that involves a data processing pipeline, and in it I might come across all sorts of data. I would like to know if there are any references on an automatic way to ...
1 vote
1 answer
18 views

Optimal way to create a feature set?

I have a time series data (say weather data for each day for one week) that changes at each time step. Along with this, I have some data that is fixed (eg - the latitude and longitude of this place ...
1 vote
1 answer
28 views

My max target value is seen quite frequently since the data source has a threshold on what they can measure. Should I remove these data?

I'm working on a regression problem involving nutrient concentrations. The lab I'm getting my data from can measure up to 9000ppm of a particular nutrient. Beyond that, everything is reported as ...
2 votes
1 answer
42 views

Which data preprocessing steps do I need to perform on which data subset?

I want to make predictions using several supervised Machine Learning algorithms and apply 10-fold-cross validation. For doing so, I randomly divided my dataset into in-sample and out-of-sample sets. ...
1 vote
0 answers
21 views

Model which achieves good results on the Adult dataset, especially on the minority class

I'm looking for a model which achieves an high prediction and recall on the Adult dataset. I've tried with random forests, but couldn't get a model with good results on the minority class. Here are my ...
0 votes
0 answers
41 views

Machine Learning for Stock Price Prediction Issue

In financial application, someone might make use of machine learning techniques in stock price prediction, e.g. LSTM. In general, before training the model, in light of the model robustness, some ...
0 votes
0 answers
29 views

Necessary Data Preprocessing for SVM

All the lecture notes that I read about support vector machine assumes that $w$ is orthogonal to the hyperplane. I am trying to prove that the vector $w$ is orthogonal to the hyperplane $L=\{w^Tx + b =...
  • 261
2 votes
0 answers
38 views

Do Transformers pad input sentences?

Assume we have a Transformer (Attention is all you need paper) and we give to it an input sequence S of length $n_{words}$. If no padding is applied, the output of the encoder model would be a matrix ...
  • 31
-1 votes
0 answers
16 views

data cleaning procedures [duplicate]

If you were provided with a sample data set as follows: Date Open High Low Close Adj Close Signal 202106-01 627.80 633.80 620.55 620.55 623.90 85.11 2021-06-02 620.13 623.36 599.14 620.13 605.12 76....
  • 1
0 votes
0 answers
41 views

Linearly dependent columns in dataset

I need to do the 'cleaning' of the dataset, ie preprocessing. I noticed that I have two columns in the dataset that are totally linearly dependent. Is it okay to delete one, both or am I not allowed ...
0 votes
1 answer
49 views

what if range of normalized of data in machine learning goes beyond?

Normalization in machine learning is the process of translating data into the range 0 -> 1 or -1 -> 1. What if the values goes above 1 or below -1. What does that mean? Is it again an outlier? ...
  • 19
0 votes
1 answer
199 views

Calculating Shannon Information of Data Augmentation Strategies [closed]

I recently caught Andrew Ng's 2021 talk on MLOps (MLOps: From Model-centric to Data-centric AI). At 26:40, he talks about calculating the effectiveness of cleaning your data (training examples) vs. ...
1 vote
0 answers
25 views

Seasonal and trend adjustment for irregularly spaced time series

I know of different methods that exist to remove seasonality and trend in the data to make it stationary. However, that exists only for regular time series; that is, a series that follows a fixed ...
0 votes
0 answers
30 views

Feature preprocessing (standardize and normalize) and variable independence

I can't find clarity on this question so here goes: Suppose I have 3 features, x, y, z. I know x and ...
  • 143
1 vote
1 answer
47 views

Should I trim/winsorize raw data or computed metric used in models?

Question: Should I rather winsorise (or trim, where relevant) my raw data, or the intermediary metric I use in my models? Context: My analysis consists in 3 steps: Collect raw data, Compute ...
  • 129
0 votes
0 answers
33 views

Using machine learning model trained on standardized data for real world low volume data

I have developed a machine learning model which has been trained on a preprocessed data by scaling and centering using h2o package of R. I am able to use this model ...
  • 197
0 votes
0 answers
24 views

Should you clean your data after or before selecting a sample?

Assuming a 500k dataset. For statistical modeling purposes (selecting up to 10% of 500k as a sample). Should I clean the 500k dataset first before selecting a sample or select the sample first and ...
2 votes
0 answers
48 views

Avoiding data leakage in preprocessing and handling unseen values in test data

I've been reading up on avoiding data leakage in the preprocessing step of a machine-learning/data-science pipeline, specifically that it is wrong to apply preprocessing to both training and test data ...
  • 121
4 votes
1 answer
121 views

Does watermark/text on images at the same position influence the classification of images using CNN?

I'm working on a multi-class classification problem using CNN. Most of the images of each class have a text/watermark at a specific position on the image. I have a couple of questions. Does the ...
  • 53
0 votes
1 answer
43 views

how to deal with data leakage in historical data

I have a dataset containing matches from 2000 TO 2018 and I am asked to predict match outcomes for the year 2017 to avoid data leakage I am going to just train my model from 2000 to 2016. in the ...
2 votes
1 answer
50 views

Derivatives of output w.r.t input on a neural network trained with standardized data

I'm using a neural network to model an unknown function for which I would also like to know the derivatives. The nn has four inputs and four outputs, and the training data is preprocessed using scikit-...
  • 25
0 votes
0 answers
39 views

Find dips in the time series data

I need help to remove dips(trough) from the signals. The red circle in the image indicates the dip and yellow circles indicates contributing points. Means there’re multiple points in the dip. I tried ...
  • 101
0 votes
0 answers
11 views

Compositional data analysis with rounded values: validity of log ratio methods

I want to use histopathology compositional data with percentage rounded in 10% increments (for example 4 classes percentages: 0% - 50% - 30% - 20%). I am interested in data exploration (e.g. PCA, CCA) ...
0 votes
1 answer
373 views

Upweight minority class vs. downsample+upweight majority class?

I've been getting some conflicting advice from various ML podcasts/videos/articles lately for how to deal with imbalanced datasets. Let's say my independent variable for a classification problem has a ...
  • 11
1 vote
1 answer
63 views

How to model seasonal variance in a high frequency time series?

I have a time series that exhibits a clear seasonal pattern regarding its variance. Basically at night and morning the variance is low and at midday it is high. The time granularity of the data is 5 ...
  • 11
0 votes
0 answers
14 views

feature engineer 2 numeric features with odd values

I am not yet sure what type of regression model I will eventually use but I wonder what method(s) exist to leave 2 numeric features as numeric, when I know from some explorative data analysis that the ...
  • 1,346
0 votes
0 answers
24 views

Factor Analysis or PCA on part of or grouped variables

I am working on industrial time series data. (Such as sensor and controller signals, etc.) I have 150 features. Some of my features/indepentdent variables are highly correlated with each other. (75-80-...
  • 21
0 votes
0 answers
15 views

When applying transfer learning for semantic segmentation, do you need to apply the same preprocessing to the new images?

I want to use transfer learning. It is not a popular model for semantic segmentation(Unet, etc). So, do I have to apply the same preprocessing to the new dataset? Does this include the masks? The ...
  • 1,139

1
2 3 4 5
10