Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

0
votes
0answers
27 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
0
votes
0answers
18 views

Clearing out errors from a data set

Sorry for the vagueness of the title, I am having a hard time even coming up with sort of problem I am facing (if there is a specific name for it....) In a nutshell, I have a time series of points, ...
2
votes
2answers
388 views

Is it 40% or 0.4%? [closed]

A variable, which should contain percents, also contains some "ratio" values, for example: 0.61 41 54 .4 .39 20 52 0.7 12 70 82 The real distribution parameters ...
2
votes
0answers
33 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
0
votes
0answers
7 views

How to verify structural errors in a data set with huge number of variables

I have a data-set with 80 predictors and 1 outcome. As part of data cleaning i want to see the structure of each data, how they are distributed and whether there are any culprits(like NULL instead of ...
1
vote
0answers
21 views

Underperforming participants (<50% accuracy) in LMM

I have to perform a linear mixed model analysis on behavioural conflict paradigm data (ie analysis of congruency effects) and I'm struggling to find reliable sources on what to do with respect to ...
3
votes
2answers
140 views

Regression algorithm on [0,1] with lots of mislabeled data

I have a training set mapping some Likert-scale variables (integers between 1 and 7, rescaled to real numbers between 0 and 1) to predict a continuous variable between 0 and 1. The data set is ...
1
vote
1answer
46 views

How long does it take to clean data? [closed]

I am trying to plan out how long it will take me to clean my survey data. I have about 200 responses. The survey takes about 15 minutes, about 40-60 questions (depending on the logic). I have very few ...
1
vote
0answers
26 views

Two predictor variables are partially related. How do I handle them before performing logistic regression

I have a predictor variable which has three levels (success, failure, nonexistent). For all nonexistent values another predictor variable value is 0 but success and failure could fall within a range ...
0
votes
0answers
30 views

Distribution Comparison on Multiple Dimensions in Python

I work as an analyst at a company that is in the midst of transitioning to more cloud-friendly tools. That means we've gone from using SAS/TD SQL (from my actual analyst days) to working with Python ...
0
votes
2answers
31 views

What is the rule for deciding when to normalize Variables In pre-processing?

Some techniques, Like boosting For classification, Do not require The Variables to be normalized.For other techniques, Normalization seems very important How Do I know When I need to normalize My ...
0
votes
0answers
65 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
1
vote
0answers
20 views

is there any tools or fast-track techniques through which millions of review data can be clean automatically

My research domain pertains to Deep Learning, and with reviews. To make better feature selection and extraction by deep learning algorithm I have a collection of close to a million review data. But, ...
0
votes
1answer
33 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
-1
votes
1answer
16 views

Concatenating distinct values contained within a string [closed]

I have a string variable (drugs) that contains a list of drugs prescribed at each line of treatment. I'm looking to create a new string that contains only the unique drugs from each line. In the ...
0
votes
0answers
11 views

Is there any model that can detect and eliminate the fake boosting part of the data(data cleaning model?)

As far as I know, there usually were some fake part in some data For example, In many company's sale, there are some fake part in the data, it actually sale is 100000 dollars, but for some reason ...
-1
votes
1answer
145 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished in a race in 45 seconds,35 ...
2
votes
1answer
122 views

Is it possible that GridSearchCV make the test data leakage?

I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and ...
0
votes
0answers
26 views

Testing quality of survey count data

I have a dataset. We asked the charities we fund what results they have achieved with the money we gave them. For example, the number of policies they impacted or the amount of news they generated. ...
7
votes
3answers
126 views

Using ML to assist human labelling in dataset with highly unbalanced classes

Are there scientific issues with using ML to assist human annotation? I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. The labels arn't ...
1
vote
2answers
43 views

PCA variances pattern changed greatly after data cleaning

I have data, when I normalize it and then performed PCA, I calculated the variance of PC components, I found that, the first component is 72% and seconed component is 8% (total 72+8=80%) and so on. ...
0
votes
0answers
45 views

Cleaning a time-series before ARIMA

My target is to fit an ARIMA-GARCH model with non-normal innovations to a dataset. Before I run the model should I clean the series, for example via the tsclean() function of R? Should I remove the ...
0
votes
0answers
479 views

MCAR test for large number variables and small sample size

I have a dataset with 101 observations and 402 columns (those columns comprise several multiple-item questionnaires). Among those 402 columns, 10 of them are categorical and the remaining are ...
0
votes
1answer
27 views

How to handle a column having the number of children with some values being represented with some dummy value?

I have a dataset which has a column representing the number of children. The values vary from 0 to 6. But the issue is that there are many entries which have values like 84, 94, 97, 98 and 99, which ...
1
vote
1answer
338 views

Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?

From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
1
vote
0answers
35 views

making n-grams, best practices

if I encode my data in 2-grams, thats (26+26+10)^2~3800 possible pairs if I use 3-grams that's ~200,000 possible triplets. We can reduce this number by using only lower case, but basic combinatorics ...
0
votes
1answer
35 views

Data cleaning: Derived variables

I am cleaning data that I will use with machine learning prediction algorithms. Several of my variables in my data set are sums of other variables. eg) given variables x1, x2, x3, x3=x2+x1 or even x4=...
0
votes
1answer
35 views

Should I be cleaning data that needs to be classified?

I have a Naive-Bayes model that does sentiment analysis. For the training of the model, the training data was cleaned, i.e.: stop words were removed, certain punctuation, etc... When I want to ...
0
votes
0answers
39 views

Handle data with high variation

I'm analyzing som product data that shows sells over the past day, week and month. In some cases I have a very high number but in most cases this number is quite low, and this is affecting quite a lot ...
1
vote
0answers
171 views

Multivariate imputation in R

I am working with a dataset from the world bank, it's a relatively simple dataset with 11 variables for 211 countries from 1990-2015: ...
2
votes
2answers
1k views

Outlier removal before Boosted Decision Tree Regression

I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe ...
0
votes
1answer
2k views

how to handle sparse data problem in unsupervised learning .i'm going to use k means on data set

how to handle sparse data problem in unsupervised learning .i'm going to use k-means on the dataset. I have 200 variables, nearly in each column have 70% zeros. how can I handle without discarding any ...
4
votes
3answers
5k views

Difference between preprocessing train and test set before and after splitting

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?...
1
vote
2answers
66 views

Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
0
votes
0answers
777 views

Improving RMSE of my model

I'm trying to build a model based on some training set. The training set contains 1460 observations, with 79 variables each (features). I'm using linear regression to build a model and after that ...
1
vote
1answer
289 views

What criteria to use when having with 'unengaged' , ‘straightlined’ or ‘patterned’ responses in questionnaires?

Please consider the following situation, which happens a lot in real life applied research in social sciences. Some percentage of the respondents give ‘straightlined’ or ‘patterned’ responses to a ...
0
votes
1answer
37 views

Should I remove data with known computational error before doing linear regression?

I have a dataset having 252 observations. There are two variables having a defined and known linear relationship. By computation, I found 4 observations do not follow the linear equation, and I ...
5
votes
0answers
77 views

Does tidiness of data differ by application?

After reading a recent paper by Hadley (link), I got to thinking about whether what we'd refer to as tidy data changes by application. For example, consider a sample dataset: ...
3
votes
1answer
627 views

What is the best way to perform sentence segmentation for textual analysis?

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:...
1
vote
1answer
709 views

Fitting linear model through noisy data

I'm currently working on a predictive modeling project. I have to predict $Y$ given variables $X_1,X_2,X_3$ and $X_4$ that are not necessarily independent. Our first idea was to propose a linear ...
2
votes
0answers
1k views

How to perform data quality check on large number of features using Spark?

I am used to work with manageable number of features. I usually print some descriptive statistics and visualise the histograms of each feature using Python and Pandas or R. I check for outliers and if ...
0
votes
0answers
72 views

Which classifier should I use for sparse boolean features?

I have training data that is classified into 2 categories: X and Not X Each piece of training or experimental data has a variable number of boolean features. Each piece of data may have ~100 features,...
1
vote
0answers
36 views

How to identify outlying raters

I have collected data from 500 anonymous online respondents who rated the acceptability of two variations of a large number of sentences (3-level categorical data: excellent, acceptable, impossible). ...
1
vote
0answers
207 views

Would it make a difference to clean the data before using R findCorrelation() method?

I'm new in data analysis area and didn't have very strong statistical background... Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to ...
1
vote
1answer
60 views

How to organize lots of data and extract information

I am working in a project where I need to extract data from lots of files provided to me in different formats (.exe, .doc, .txt, .dbf, and even .pdf). The data I am interested in are water quality and ...
1
vote
3answers
5k views

Data Imputation in R with NAs in only one variable (categorical)

I have data frame with 44,353 entries with 17 variables (4 categorical + 13 continuous). Out of all variables only 1 categorical variable (with 52 factors) has NAs No of factors in the categorical ...
0
votes
0answers
143 views

outlier detection : use trimmed data to calculate STD?

I can use box-plot to examine data and detect outliers based on box-plot. Most often Box-plot uses a q-range (q3-q1) to define the data boundary and then display "outliers". Sometimes I see people ...
3
votes
0answers
17 views

How to deal with implausible answers to filter questions? [duplicate]

I work with a questionnaire that contains a variety of filter questions. In the following I will give an example how one block of this successive questions look like. Have you ever skipped ...
2
votes
1answer
116 views

Cleaning a text: R vs Python [closed]

I have a "dirty" text with statements as L'intervento ÃẀ mirato all'eliminazione nonché alla... where I have ÃẀ instead of è, é instead of é, etc. Which is the best language for manipulating ...
3
votes
0answers
466 views

Sensor data cleaning

Underneath is a picture of a sensor measuring the fill rate of a container on an hourly basis. It goes up to 100% and is then emptied. There is some natural deviation of the sensor due to temperature ...