Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

2
votes
0answers
15 views

Split dataset before or after preprocessing in training, validation, test

Suppose you have a super large table $T$ with data, which is pretty messy, and it's columns aren't useful features. So you clean it, remove/impute missing values, define new columns, which are more ...
-1
votes
3answers
58 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
0
votes
1answer
48 views

Pre-Processing audio data for whale sound classification using CNN

Previous researchers have used techniques like Denoising using Spectral Subtraction method and calculating Short Time Fourier Transform (STFT) by dividing the audio data into fixed size chunks and ...
0
votes
0answers
30 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
0
votes
0answers
18 views

Clearing out errors from a data set

Sorry for the vagueness of the title, I am having a hard time even coming up with sort of problem I am facing (if there is a specific name for it....) In a nutshell, I have a time series of points, ...
2
votes
2answers
397 views

Is it 40% or 0.4%? [closed]

A variable, which should contain percents, also contains some "ratio" values, for example: 0.61 41 54 .4 .39 20 52 0.7 12 70 82 The real distribution parameters ...
2
votes
0answers
47 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
0
votes
0answers
12 views

How to verify structural errors in a data set with huge number of variables

I have a data-set with 80 predictors and 1 outcome. As part of data cleaning i want to see the structure of each data, how they are distributed and whether there are any culprits(like NULL instead of ...
1
vote
0answers
21 views

Underperforming participants (<50% accuracy) in LMM

I have to perform a linear mixed model analysis on behavioural conflict paradigm data (ie analysis of congruency effects) and I'm struggling to find reliable sources on what to do with respect to ...
3
votes
2answers
144 views

Regression algorithm on [0,1] with lots of mislabeled data

I have a training set mapping some Likert-scale variables (integers between 1 and 7, rescaled to real numbers between 0 and 1) to predict a continuous variable between 0 and 1. The data set is ...
1
vote
1answer
82 views

How long does it take to clean data? [closed]

I am trying to plan out how long it will take me to clean my survey data. I have about 200 responses. The survey takes about 15 minutes, about 40-60 questions (depending on the logic). I have very few ...
1
vote
0answers
28 views

Two predictor variables are partially related. How do I handle them before performing logistic regression

I have a predictor variable which has three levels (success, failure, nonexistent). For all nonexistent values another predictor variable value is 0 but success and failure could fall within a range ...
0
votes
0answers
70 views

Distribution Comparison on Multiple Dimensions in Python

I work as an analyst at a company that is in the midst of transitioning to more cloud-friendly tools. That means we've gone from using SAS/TD SQL (from my actual analyst days) to working with Python ...
0
votes
2answers
31 views

What is the rule for deciding when to normalize Variables In pre-processing?

Some techniques, Like boosting For classification, Do not require The Variables to be normalized.For other techniques, Normalization seems very important How Do I know When I need to normalize My ...
0
votes
0answers
92 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
1
vote
0answers
20 views

is there any tools or fast-track techniques through which millions of review data can be clean automatically

My research domain pertains to Deep Learning, and with reviews. To make better feature selection and extraction by deep learning algorithm I have a collection of close to a million review data. But, ...
0
votes
1answer
34 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
-1
votes
1answer
16 views

Concatenating distinct values contained within a string [closed]

I have a string variable (drugs) that contains a list of drugs prescribed at each line of treatment. I'm looking to create a new string that contains only the unique drugs from each line. In the ...
0
votes
0answers
12 views

Is there any model that can detect and eliminate the fake boosting part of the data(data cleaning model?)

As far as I know, there usually were some fake part in some data For example, In many company's sale, there are some fake part in the data, it actually sale is 100000 dollars, but for some reason ...
-1
votes
1answer
201 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished in a race in 45 seconds,35 ...
2
votes
1answer
145 views

Is it possible that GridSearchCV make the test data leakage?

I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and ...
0
votes
0answers
27 views

Testing quality of survey count data

I have a dataset. We asked the charities we fund what results they have achieved with the money we gave them. For example, the number of policies they impacted or the amount of news they generated. ...
7
votes
3answers
129 views

Using ML to assist human labelling in dataset with highly unbalanced classes

Are there scientific issues with using ML to assist human annotation? I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. The labels arn't ...
1
vote
2answers
44 views

PCA variances pattern changed greatly after data cleaning

I have data, when I normalize it and then performed PCA, I calculated the variance of PC components, I found that, the first component is 72% and seconed component is 8% (total 72+8=80%) and so on. ...
0
votes
0answers
64 views

Cleaning a time-series before ARIMA

My target is to fit an ARIMA-GARCH model with non-normal innovations to a dataset. Before I run the model should I clean the series, for example via the tsclean() function of R? Should I remove the ...
0
votes
0answers
578 views

MCAR test for large number variables and small sample size

I have a dataset with 101 observations and 402 columns (those columns comprise several multiple-item questionnaires). Among those 402 columns, 10 of them are categorical and the remaining are ...
0
votes
1answer
27 views

How to handle a column having the number of children with some values being represented with some dummy value?

I have a dataset which has a column representing the number of children. The values vary from 0 to 6. But the issue is that there are many entries which have values like 84, 94, 97, 98 and 99, which ...
1
vote
1answer
421 views

Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?

From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
1
vote
0answers
37 views

making n-grams, best practices

if I encode my data in 2-grams, thats (26+26+10)^2~3800 possible pairs if I use 3-grams that's ~200,000 possible triplets. We can reduce this number by using only lower case, but basic combinatorics ...
0
votes
1answer
37 views

Data cleaning: Derived variables

I am cleaning data that I will use with machine learning prediction algorithms. Several of my variables in my data set are sums of other variables. eg) given variables x1, x2, x3, x3=x2+x1 or even x4=...
0
votes
1answer
35 views

Should I be cleaning data that needs to be classified?

I have a Naive-Bayes model that does sentiment analysis. For the training of the model, the training data was cleaned, i.e.: stop words were removed, certain punctuation, etc... When I want to ...
0
votes
0answers
43 views

Handle data with high variation

I'm analyzing som product data that shows sells over the past day, week and month. In some cases I have a very high number but in most cases this number is quite low, and this is affecting quite a lot ...
1
vote
0answers
199 views

Multivariate imputation in R

I am working with a dataset from the world bank, it's a relatively simple dataset with 11 variables for 211 countries from 1990-2015: ...
2
votes
2answers
1k views

Outlier removal before Boosted Decision Tree Regression

I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe ...
0
votes
1answer
3k views

how to handle sparse data problem in unsupervised learning .i'm going to use k means on data set

how to handle sparse data problem in unsupervised learning .i'm going to use k-means on the dataset. I have 200 variables, nearly in each column have 70% zeros. how can I handle without discarding any ...
4
votes
3answers
5k views

Difference between preprocessing train and test set before and after splitting

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?...
1
vote
2answers
68 views

Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
0
votes
0answers
875 views

Improving RMSE of my model

I'm trying to build a model based on some training set. The training set contains 1460 observations, with 79 variables each (features). I'm using linear regression to build a model and after that ...
1
vote
1answer
319 views

What criteria to use when having with 'unengaged' , ‘straightlined’ or ‘patterned’ responses in questionnaires?

Please consider the following situation, which happens a lot in real life applied research in social sciences. Some percentage of the respondents give ‘straightlined’ or ‘patterned’ responses to a ...
0
votes
1answer
39 views

Should I remove data with known computational error before doing linear regression?

I have a dataset having 252 observations. There are two variables having a defined and known linear relationship. By computation, I found 4 observations do not follow the linear equation, and I ...
5
votes
0answers
78 views

Does tidiness of data differ by application?

After reading a recent paper by Hadley (link), I got to thinking about whether what we'd refer to as tidy data changes by application. For example, consider a sample dataset: ...
3
votes
1answer
702 views

What is the best way to perform sentence segmentation for textual analysis?

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:...
1
vote
1answer
770 views

Fitting linear model through noisy data

I'm currently working on a predictive modeling project. I have to predict $Y$ given variables $X_1,X_2,X_3$ and $X_4$ that are not necessarily independent. Our first idea was to propose a linear ...
2
votes
0answers
1k views

How to perform data quality check on large number of features using Spark?

I am used to work with manageable number of features. I usually print some descriptive statistics and visualise the histograms of each feature using Python and Pandas or R. I check for outliers and if ...
0
votes
0answers
76 views

Which classifier should I use for sparse boolean features?

I have training data that is classified into 2 categories: X and Not X Each piece of training or experimental data has a variable number of boolean features. Each piece of data may have ~100 features,...
1
vote
0answers
38 views

How to identify outlying raters

I have collected data from 500 anonymous online respondents who rated the acceptability of two variations of a large number of sentences (3-level categorical data: excellent, acceptable, impossible). ...
1
vote
0answers
212 views

Would it make a difference to clean the data before using R findCorrelation() method?

I'm new in data analysis area and didn't have very strong statistical background... Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to ...
1
vote
1answer
63 views

How to organize lots of data and extract information

I am working in a project where I need to extract data from lots of files provided to me in different formats (.exe, .doc, .txt, .dbf, and even .pdf). The data I am interested in are water quality and ...
1
vote
3answers
5k views

Data Imputation in R with NAs in only one variable (categorical)

I have data frame with 44,353 entries with 17 variables (4 categorical + 13 continuous). Out of all variables only 1 categorical variable (with 52 factors) has NAs No of factors in the categorical ...
3
votes
0answers
17 views

How to deal with implausible answers to filter questions? [duplicate]

I work with a questionnaire that contains a variety of filter questions. In the following I will give an example how one block of this successive questions look like. Have you ever skipped ...