Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Feature preprocessing (standardize and normalize) and variable independence

I can't find clarity on this question so here goes: Suppose I have 3 features, x, y, z. I know x and ...
user avatar
  • 143
1 vote
1 answer
13 views

Should I trim/winsorize raw data or computed metric used in models?

Question: Should I rather winsorise (or trim, where relevant) my raw data, or the intermediary metric I use in my models? Context: My analysis consists in 3 steps: Collect raw data, Compute ...
user avatar
  • 129
0 votes
0 answers
28 views

Using machine learning model trained on standardized data for real world low volume data

I have developed a machine learning model which has been trained on a preprocessed data by scaling and centering using h2o package of R. I am able to use this model ...
user avatar
  • 197
0 votes
0 answers
22 views

Should you clean your data after or before selecting a sample?

Assuming a 500k dataset. For statistical modeling purposes (selecting up to 10% of 500k as a sample). Should I clean the 500k dataset first before selecting a sample or select the sample first and ...
user avatar
1 vote
0 answers
16 views

Avoiding data leakage in preprocessing and handling unseen values in test data

I've been reading up on avoiding data leakage in the preprocessing step of a machine-learning/data-science pipeline, specifically that it is wrong to apply preprocessing to both training and test data ...
user avatar
  • 111
4 votes
1 answer
39 views

Does watermark/text on images at the same position influence the classification of images using CNN?

I'm working on a multi-class classification problem using CNN. Most of the images of each class have a text/watermark at a specific position on the image. I have a couple of questions. Does the ...
user avatar
  • 53
0 votes
1 answer
24 views

how to deal with data leakage in historical data

I have a dataset containing matches from 2000 TO 2018 and I am asked to predict match outcomes for the year 2017 to avoid data leakage I am going to just train my model from 2000 to 2016. in the ...
user avatar
2 votes
1 answer
24 views

Derivatives of output w.r.t input on a neural network trained with standardized data

I'm using a neural network to model an unknown function for which I would also like to know the derivatives. The nn has four inputs and four outputs, and the training data is preprocessed using scikit-...
user avatar
  • 25
0 votes
0 answers
14 views

Find dips in the time series data

I need help to remove dips(trough) from the signals. The red circle in the image indicates the dip and yellow circles indicates contributing points. Means there’re multiple points in the dip. I tried ...
user avatar
  • 101
0 votes
0 answers
7 views

Compositional data analysis with rounded values: validity of log ratio methods

I want to use histopathology compositional data with percentage rounded in 10% increments (for example 4 classes percentages: 0% - 50% - 30% - 20%). I am interested in data exploration (e.g. PCA, CCA) ...
user avatar
0 votes
1 answer
90 views

Upweight minority class vs. downsample+upweight majority class?

I've been getting some conflicting advice from various ML podcasts/videos/articles lately for how to deal with imbalanced datasets. Let's say my independent variable for a classification problem has a ...
user avatar
  • 11
1 vote
1 answer
49 views

How to model seasonal variance in a high frequency time series?

I have a time series that exhibits a clear seasonal pattern regarding its variance. Basically at night and morning the variance is low and at midday it is high. The time granularity of the data is 5 ...
user avatar
  • 11
0 votes
0 answers
11 views

feature engineer 2 numeric features with odd values

I am not yet sure what type of regression model I will eventually use but I wonder what method(s) exist to leave 2 numeric features as numeric, when I know from some explorative data analysis that the ...
user avatar
  • 1,306
0 votes
0 answers
17 views

Factor Analysis or PCA on part of or grouped variables

I am working on industrial time series data. (Such as sensor and controller signals, etc.) I have 150 features. Some of my features/indepentdent variables are highly correlated with each other. (75-80-...
user avatar
  • 11
0 votes
0 answers
7 views

When applying transfer learning for semantic segmentation, do you need to apply the same preprocessing to the new images?

I want to use transfer learning. It is not a popular model for semantic segmentation(Unet, etc). So, do I have to apply the same preprocessing to the new dataset? Does this include the masks? The ...
user avatar
  • 1,019
0 votes
0 answers
16 views

How to deal with a predictor that is 25% unknown and the rest discrete? [duplicate]

For a sales prediction task, I have 2 datasets, one pertaining to the sales with each store occurring multiple times and the other that has extra information on each of the 1115 distinct stores. The ...
user avatar
1 vote
0 answers
34 views

Combining multiple columns in R to get frequency count? [closed]

I have incident data and for each incident, there are multiple individuals each with sex/age. The data looks something like this: incident person1 personl1sex person1age person2 person2sex person2age ...
user avatar
0 votes
0 answers
40 views

Training a machine learning model on data that has several rows for each user

I have a dataset consisting of log files from a smartphone application. Currently, it creates a row each time a user clicks on something, i.e. a user clicks on the homepage, and a new row is created ...
user avatar
1 vote
0 answers
15 views

Train a model using a merged dataset consisted of values obtained from two different methods

I have 100 samples and I performed a certain analysis which resulted in approx. 60 features with measures ranging between 0 and 1 (after a preprocessing round). For the same 100 samples, I was given ...
user avatar
-1 votes
1 answer
28 views

How to Interpret this lagged model? [duplicate]

Just a Quick Overview there are 2 lagged variables, Unemploy.L1 - lagged for 1 month, Unemploy.L12 lagged for 12 months. Overview of variables - umemployment (unemploy, in thousands) (date, month of ...
user avatar
0 votes
1 answer
126 views

How to get glmnet to work for proportions as response variable

I am trying to run a penalized logistic regression in R. My response variables are proportions (they are winning percentages for a sports team), and I have the number of total games played by each ...
user avatar
0 votes
1 answer
20 views

No change in algorithm performance following removal of important variable

performed a classification task with XGBoost where I aim to predict cardiovascular disease (CVD) with a dataset of 12 vars and 70 0000 data points and got an f1 score of 0.73. After obtaining a ...
user avatar
  • 1
1 vote
1 answer
43 views

prepare data for autoencoder

I have for example 500 engines and for each engine I have 100 features and each feature is measured 1000 times. It means the data table is: 500,000 rows and 100 columns. (for each engine I have only a ...
user avatar
  • 113
1 vote
0 answers
44 views

How to account for weights of a skewed dataset in a machine learning problem?

I, a novice, have a dataset which I would like to use for multiclass classification. I know that the data is skewed, but luckily, my dataset contains an observation weights column. The observation ...
user avatar
0 votes
0 answers
42 views

How to clean dataset in order to fit to a curve? [duplicate]

I'm trying to fit a dataset to a curve for while, but I'm not managing. The goal is to obtain a curve with equation that fits the data so I can get the parameter x to any value of y. The blue dataset ...
user avatar
0 votes
0 answers
21 views

Is it necessary to preprocess the data before doing sentiment analysis using Veder, TextBlob and Corenlp?

I want to perform unsupervised sentiment analysis using Veder, TextBlob and CoreNlp. Is it ...
user avatar
1 vote
0 answers
26 views

Bibliography on how to data preprocessing

Very often we're given a data set which contains duplicate rows, missing values, outliers, etc. Most of the information on how to deal with these situations (from a statistical perspective) is ...
1 vote
0 answers
42 views

How to deal with missing values in a 300 GB file (more than 1.5 billion data points)? Statistical Insight required [closed]

Scenario I'm working on a binary classification problem involving a 300GB dataset. The dataset's interpretability is low due to privacy concerns. All I have is 6 independent columns (features) and a ...
user avatar
  • 21
1 vote
1 answer
128 views

How to interpret the output of caret::findCorrelation function?

The output I received after applying findCorrelation function from the caret package is: ...
user avatar
1 vote
0 answers
12 views

What activation function or pre-processing to use for features describing when a certain event occurred in the past?

I have a series of features that describe how long ago a certain event happened and whether it happened at all. Of course we could break down this features into two, whether it did happen or no, and ...
user avatar
  • 151
2 votes
0 answers
36 views

How to deal with output transformation at inference/prediction time?

Suppose A machine learning model (e.g. RandomForest) which uses $x$ as input and produces $y$. Now as part of preprocessing and feature engineering, I applied some ...
user avatar
0 votes
0 answers
19 views

Identifying outliers in data collection with binary variables

I have a bunch of data collected on a set of engines. Most of my data is a flag if a particular feature works/doesn't or Issue_X exists/doesn't exist and then a final engine rating is given. I want to ...
user avatar
  • 450
1 vote
2 answers
84 views

Classification with imbalanced data

I have a dataset that is highly imbalanced. I did some research on Internet, however I did not find what I was looking for. What is the correct sequence for dealing with imbalanced data? Should we ...
user avatar
0 votes
0 answers
26 views

What "rules of thumb" preprocessing techniques to use for neural nets?

Usually when we have an algorithm, we have some rules regarding what kind of data preprocessing techniques that algorithm needs. For example LinearRegression has the following rules: There must be a ...
user avatar
  • 255
1 vote
1 answer
32 views

Suggestions to deal with a noisy testing dataset

I'm actually training a model on a healthy training dataset to perform a regression task. To validate the model, I'm running it with different testing datasets. However, some testing datasets may ...
user avatar
  • 19
2 votes
0 answers
51 views

Feature preprocessing for neural nets

Suppose that you have a feature where for some instances the value is exactly 0, while for others it is a continuous value. An example could be money earned doing some specific type of work, or ...
user avatar
  • 521
0 votes
0 answers
33 views

Advice for how to approach encoding non-numerical data

I have a dataset where the task is to classify whether someone looking for a new job will leave their current job, based on a number of factors. (dataset: https://www.kaggle.com/arashnic/hr-analytics-...
user avatar
0 votes
0 answers
69 views

Number of bins for discretization

How do I decide on the right number of bins to discretize my continuous data? Are there are tests/techniques to do the same? Could someone give me some idea into existing approaches?
user avatar
1 vote
0 answers
33 views

Long versus short format in fitting a binomial GLM

This question arises after reading the question Input format for response in binomial glm in R and its answer. Are the long and short format to input the data in a binomial GLM truly equivalent? While ...
user avatar
0 votes
0 answers
15 views

An efficient way to encode & embed tabular data of a video into a transformer?

So a little bit of a background: I have a folder which contains video files of lets say humans doing a certain action (i.e. walking) where each .2 seconds is documented in a ...
user avatar
  • 111
1 vote
0 answers
17 views

Preprocessing for the final model to be deployed [duplicate]

Typically for a ML workflow, we import the data (X and y), split the X and ...
user avatar
  • 255
0 votes
0 answers
13 views

Handling time series data with same start and end datetime but different vector lengths

I am dealing with time series data with 15 different features from a machine. The thing is that the different feature datasets that have been collected have been done at different frequecies. That is ...
user avatar
0 votes
0 answers
9 views

When did the data change significantly to recalculate neural network?

In my classification model I have one neural network that is trained with normalized data to the interval [0,1], normalization is done using this formula: $$ I = I_{...
user avatar
  • 239
0 votes
0 answers
49 views

How to fix label errors in text corpus?

I have a huge text corpus (around 60k+ documents with 40 classes. But the corpus suffers from class imbalance problem. Also, the ...
user avatar
  • 185
0 votes
0 answers
73 views

Data preprocessing before lasso regression

I am doing lasso for variable reduction from a bunch of 100 odd variables. Some numeric variable have extreme values. for e.g **count of rooms in house ** have values like 1,2,3,4,5,6,7,100. 100 is ...
user avatar
  • 203
0 votes
1 answer
27 views

How do I roll data within groups in a dataframe? [closed]

The original dataframe is like df1. df1 = pd.DataFrame({'group': [1,1,1,1,1,2,2,2,2,2], 'in': list(range(1,11)), 'out': [0,0,0,1,1,0,0,0,1,1]}) df1 The output I ...
user avatar
5 votes
2 answers
560 views

Can duplicate examples create multi-collinearity?

We know if any linear dependency exists in training data, therefore, the feature matrix becomes singular and hence, we cannot solve it. But apart from the features (columns), a matrix can still can be ...
user avatar
  • 185
0 votes
0 answers
25 views

How to make multivariate windowed time series data work for LSTM?

My dataset comprises 4-timestep sliding windows for multiple features and is structured as shown in the first image below. It is important to note that rows can be either (i) the next sliding window ...
user avatar
3 votes
1 answer
51 views

Need data preparation cheatsheet / guidelines / first principles to train team members! [closed]

Really ran out of ideas and hence such a basic question to the community - Despite the repeated emphasis on ensuring data accuracy/validity, team members just do not spend enough time on it because ...
0 votes
0 answers
75 views

How to combine data from different time periods for time-series prediction?

Consider a multivariate time series forecasting task where I have two datasets A and B. A goes from 1960 to 2020 and B goes from 2010 to 2020. There is a feature f ...
user avatar
  • 25

1
2 3 4 5
10