Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Data Quality for Multivariate Time Series

Background Assume $x(t)$ and $y(t)$ are two dependent time series. The time series $y(t)$ takes discrete value of either $0$ or $1$, such that $x(t)$ has different characteristic depending on the ...
0
votes
0answers
7 views

Inconsistent Entries in Panel Data

I am working with a longitudinal dataset, and one of the users [unit of observation] has inconsistent gender entries for different readings. Something like: Is there a general rule on how to ...
0
votes
0answers
6 views

Should a model be ready to handle a missing observation in the test set for any given feature?

Say imputation is sufficient for handling missing values in our problem and that our test set is locked in a vault. Which of the following approaches is recommended? Fit imputation for every feature, ...
0
votes
0answers
3 views

Suggestions for Data Wrangling with Large Dataset of String Types

I have a fairly large dataset (1934X150000) which I got from this Kaggle challenge. https://www.kaggle.com/c/springleaf-marketing-response/data I have no problem reading the file into my SparkSession, ...
3
votes
2answers
77 views

How do you spot errors in data?

I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
0
votes
1answer
30 views

Dealing with Continuous Variable in Logistic Regression?

I Am trying to create a Logistic model and I have two columns which are important features but it has continuous values ... I have 7044 rows so creating dummies might not be optimal which will burst ...
0
votes
0answers
17 views

Best practices and procedures for the imputation of missing data in a repeated measures design

I have a dataset from an experiment I conducted that examined the accuracy and precision of eye movements. It was a repeated measures, 3 x 4 x 10 design with no nesting of subjects and no between-...
1
vote
1answer
20 views

List values in tidy dataset

Can a table that has a list of values in a field, say comma-separated, be tidy? For example, the CSV: id,name,tags 1,fork,"utensil,cutlery" 2,plate,crockery My ...
0
votes
0answers
16 views

replacing nas when 50% or more of the data is missing

I have to perform a linear regression on a dataset. However, I am having trouble figuring out what type of imputation I should do on the data because in some cases the majority of the the data is ...
0
votes
0answers
26 views

Median Absolute Deviation for new outlier

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Assume, that based on Median Absolute Deviation (MAD) $$ MAD = \frac{\sum_{i=1}^{n}|x_{i} - m|}{n}, $$ where $m = median(x_{1}, \dots, x_{n})...
0
votes
1answer
26 views

Combining data sets using an ID with multiple possible names

I have two data sets, both with dog breed names. I'd like to combine the data sets somehow, but a lot of dogs have multiple names, so for instance African Hairless Dogs are also called African ...
0
votes
0answers
10 views

Identifying Gap Regions in Time Series Data [duplicate]

I have order millions of time series and order tens of thousands of them have sudden drops to near-zero and then return to legitimacy at random times. A couple of examples: First, my challenge is ...
1
vote
0answers
23 views

How are the values of the Grubb's table calculated for use in the Grubbs filter?

I am looking at Grubb's test for outliers. The approach seems simple enough but for completeness I have two questions. The Grubbs test is defined as $$G_{\rm{test}} = \frac{\left| x_{i} - \bar{x} \...
0
votes
0answers
27 views

How to do image classification in a dataset with mislabel and totally irrelevant data?

I meet a problem when doing image classification on a dataset. There are two problems in this dataset: mislabel, that is, an image should be labeled as 1 but in ...
0
votes
0answers
8 views

Different normalization techniques (mean and min-max) on different columns in a data frame

Is it possible to have different normalization techniques (mean and min-max) applied on different columns in a data frame?
0
votes
0answers
37 views

Right skewed distribution of a continuous variable with outliers: replace outliers with mode or median of that column?

When I replace my outliers with the median value of that column/feature, my mode for that column/feature also changes. Is that correct?
0
votes
0answers
10 views

Are data outliers are data cleaning error for machine learning process?

I found a lot of ways and examples for data cleaning errors, but should we remove dataset outliers from our dataset in machine learning pre-process like data cleaning? Because sometimes in linear ...
0
votes
0answers
63 views

Do you clean the data before calculating MASE (Mean Absolute Scaled Error)

The denominator in the MASE calculation for seasonal data is the MAE of the seasonal naive forecast calculated in-sample. Is it common to do imputation before calculating the seasonal naive MAE or ...
1
vote
0answers
31 views

rolling removing outliers: include or not include

In the paper "Realized kernels in practice: trades and quotes" by O. E.Bandorff-Nielsen etc. cf. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1368-423X.2008.00275.x in the section dedicated to ...
1
vote
0answers
24 views

Mean Absolute Deviation and data preprocessing

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Next, based on Mean Absolute Deviation (MAD) we aim to decide if the last point $x_{n+1}$ is outlier or not. First, let us compute the MAD: ...
0
votes
0answers
35 views

Do I impute missing values with the response?

I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
1
vote
1answer
61 views

Fix wrong data coming from a sensor

I have data coming from a sensor that I store in a time serie. When I graph them, I obtain: These data are supposed to be "continuous", like temperatures, not going up and down so fast. After ...
0
votes
0answers
19 views

Training with unclean data

I am working on a supervised machine learning problem (text classification). My data consists of labels that were assigned by an automated system and, in theory cleaned by humans when needed. After ...
1
vote
0answers
23 views

Impact of empty data in ML models

Working on a project for my computational intelligence class at college doing machine learning models for crime hotspot predictions. Essentially what I am doing is laying a grid over the specific city ...
0
votes
0answers
32 views

How do I perform data cleaning for threshold detection using method of limits?

I have a dataset collected using the Method of Limits (https://en.wikipedia.org/wiki/Psychophysics#Method_of_limits) and am looking for any approaches to outlier detection that are accepted practice. ...
2
votes
1answer
40 views

What is a sensible way to truncate data to a region that fits a model?

I want to use an exponential decay model in python to relate the flow rate in a device to the mass left inside it, in particular $flow=a−b×e^{−c×mass}$ where a, b and c are the parameters of the model....
1
vote
1answer
20 views

Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
2
votes
1answer
47 views

Is data that has been entered incorrectly treated the same as missing data?

I am doing an online study and have just started looking at the data. I noticed two of my participants have listed ages that they couldn't possibly be (e.g 450 and 220). I'm wondering what the ...
1
vote
1answer
526 views

Standardizing data produces negative values

I am working on a basic house price prediction problem with traditional ML algorithms, not NN since the size of data is small comparing to the number of features. The issue I am having is that many ...
2
votes
1answer
226 views

what should be done first, handling missing data or finding correlation between features and drop irrelevant features?

In data science, Which process should come first, handling missing data or finding correlation between features. I am asking this question because I have problem in following cases: 1) Handling ...
4
votes
2answers
208 views

what should be done first, handling missing data or dealing with data types?

In data science, Which process should come first, handling missing data or handling data types. I am asking this question because I have problem in following cases: 1) Handling Missing data first, ...
1
vote
1answer
38 views

Is there a point where you wouldn't use dummy variables? I.e., if getting dummy vars would lead to hundreds of variables? [duplicate]

I built a web scraper that drew in a bunch of data and I have more qualitative variables than I expected. Originally there were just a few quantitative variables that I had intended to consider but, ...
-1
votes
1answer
90 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
-1
votes
1answer
36 views

group_by() and summarize() is not resulting in one row in the output for each group [closed]

I am learning basics of {dplyr} package in R and working with summarize() function. When I create groups using group_by() ...
1
vote
1answer
149 views

Normalising data with only min and max values [closed]

I'm working with salary datasets and wants to normalize the data as much as possible. I have data in the form of: years of experience, salary range 0-1, 28-34 2-5, 32-44 ... Each dataset uses ...
0
votes
0answers
66 views

How to build a machine learning model from sparse features with class imbalance?

I have around 10 numerical features and 1 class/target (e.g., visitors count of a website). All of them are sparse. Sparsity is around %70-80. The median of the class/target is zero. Is there a good ...
-1
votes
3answers
350 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
0
votes
1answer
295 views

Pre-Processing audio data for whale sound classification using CNN

Previous researchers have used techniques like Denoising using Spectral Subtraction method and calculating Short Time Fourier Transform (STFT) by dividing the audio data into fixed size chunks and ...
0
votes
0answers
21 views

Clearing out errors from a data set

Sorry for the vagueness of the title, I am having a hard time even coming up with sort of problem I am facing (if there is a specific name for it....) In a nutshell, I have a time series of points, ...
2
votes
2answers
411 views

Is it 40% or 0.4%? [closed]

A variable, which should contain percents, also contains some "ratio" values, for example: 0.61 41 54 .4 .39 20 52 0.7 12 70 82 The real distribution parameters ...
2
votes
0answers
113 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
1
vote
0answers
24 views

Underperforming participants (<50% accuracy) in LMM

I have to perform a linear mixed model analysis on behavioural conflict paradigm data (ie analysis of congruency effects) and I'm struggling to find reliable sources on what to do with respect to ...
4
votes
2answers
187 views

Regression algorithm on [0,1] with lots of mislabeled data

I have a training set mapping some Likert-scale variables (integers between 1 and 7, rescaled to real numbers between 0 and 1) to predict a continuous variable between 0 and 1. The data set is ...
1
vote
1answer
943 views

How long does it take to clean data? [closed]

I am trying to plan out how long it will take me to clean my survey data. I have about 200 responses. The survey takes about 15 minutes, about 40-60 questions (depending on the logic). I have very few ...
1
vote
0answers
35 views

Two predictor variables are partially related. How do I handle them before performing logistic regression

I have a predictor variable which has three levels (success, failure, nonexistent). For all nonexistent values another predictor variable value is 0 but success and failure could fall within a range ...
0
votes
2answers
35 views

What is the rule for deciding when to normalize Variables In pre-processing?

Some techniques, Like boosting For classification, Do not require The Variables to be normalized.For other techniques, Normalization seems very important How Do I know When I need to normalize My ...
0
votes
0answers
127 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
1
vote
0answers
21 views

is there any tools or fast-track techniques through which millions of review data can be clean automatically

My research domain pertains to Deep Learning, and with reviews. To make better feature selection and extraction by deep learning algorithm I have a collection of close to a million review data. But, ...
0
votes
1answer
42 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
-1
votes
1answer
20 views

Concatenating distinct values contained within a string [closed]

I have a string variable (drugs) that contains a list of drugs prescribed at each line of treatment. I'm looking to create a new string that contains only the unique drugs from each line. In the ...