Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

0
votes
0answers
10 views

Two predictor variables are partially related. How do I handle them before performing logistic regression [on hold]

I have a predictor variable called poutcome variable which has three levels (success, failure, nonexistent). For all nonexistent values another variable pdays value is 0 but success and failure could ...
0
votes
0answers
12 views

Distribution Comparison on Multiple Dimensions in Python

I work as an analyst at a company that is in the midst of transitioning to more cloud-friendly tools. That means we've gone from using SAS/TD SQL (from my actual analyst days) to working with Python ...
0
votes
2answers
31 views

What is the rule for deciding when to normalize Variables In pre-processing?

Some techniques, Like boosting For classification, Do not require The Variables to be normalized.For other techniques, Normalization seems very important How Do I know When I need to normalize My ...
0
votes
0answers
25 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
1
vote
0answers
20 views

is there any tools or fast-track techniques through which millions of review data can be clean automatically

My research domain pertains to Deep Learning, and with reviews. To make better feature selection and extraction by deep learning algorithm I have a collection of close to a million review data. But, ...
0
votes
0answers
22 views

how imputation with extension works?

i was going through kaggle learning (https://www.kaggle.com/dansbecker/handling-missing-values), in handling the missing data section ,i found use of imputation with extension (i.e) adding extra a ...
0
votes
1answer
32 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
-1
votes
1answer
15 views

Concatenating distinct values contained within a string [closed]

I have a string variable (drugs) that contains a list of drugs prescribed at each line of treatment. I'm looking to create a new string that contains only the unique drugs from each line. In the ...
0
votes
0answers
10 views

Is there any model that can detect and eliminate the fake boosting part of the data(data cleaning model?)

As far as I know, there usually were some fake part in some data For example, In many company's sale, there are some fake part in the data, it actually sale is 100000 dollars, but for some reason ...
-1
votes
1answer
84 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished in a race in 45 seconds,35 ...
2
votes
1answer
67 views

Is it possible that GridSearchCV make the test data leakage?

I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and ...
0
votes
0answers
26 views

Testing quality of survey count data

I have a dataset. We asked the charities we fund what results they have achieved with the money we gave them. For example, the number of policies they impacted or the amount of news they generated. ...
7
votes
3answers
123 views

Using ML to assist human labelling in dataset with highly unbalanced classes

Are there scientific issues with using ML to assist human annotation? I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. The labels arn't ...
1
vote
2answers
37 views

PCA variances pattern changed greatly after data cleaning

I have data, when I normalize it and then performed PCA, I calculated the variance of PC components, I found that, the first component is 72% and seconed component is 8% (total 72+8=80%) and so on. ...
0
votes
0answers
37 views

Cleaning a time-series before ARIMA

My target is to fit an ARIMA-GARCH model with non-normal innovations to a dataset. Before I run the model should I clean the series, for example via the tsclean() function of R? Should I remove the ...
0
votes
0answers
395 views

MCAR test for large number variables and small sample size

I have a dataset with 101 observations and 402 columns (those columns comprise several multiple-item questionnaires). Among those 402 columns, 10 of them are categorical and the remaining are ...
0
votes
1answer
27 views

How to handle a column having the number of children with some values being represented with some dummy value?

I have a dataset which has a column representing the number of children. The values vary from 0 to 6. But the issue is that there are many entries which have values like 84, 94, 97, 98 and 99, which ...
0
votes
1answer
241 views

Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?

From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
0
votes
0answers
19 views

Data cleaning/clustering/grouping query? (novice)

Working on a project with numerous time-related onset values from a number of subjects and am looking for a way to group them by temporal centricity. Another way to look at it: I am trying to find the ...
1
vote
0answers
26 views

making n-grams, best practices

if I encode my data in 2-grams, thats (26+26+10)^2~3800 possible pairs if I use 3-grams that's ~200,000 possible triplets. We can reduce this number by using only lower case, but basic combinatorics ...
0
votes
1answer
31 views

Data cleaning: Derived variables

I am cleaning data that I will use with machine learning prediction algorithms. Several of my variables in my data set are sums of other variables. eg) given variables x1, x2, x3, x3=x2+x1 or even x4=...
0
votes
1answer
33 views

Should I be cleaning data that needs to be classified?

I have a Naive-Bayes model that does sentiment analysis. For the training of the model, the training data was cleaned, i.e.: stop words were removed, certain punctuation, etc... When I want to ...
0
votes
0answers
38 views

Handle data with high variation

I'm analyzing som product data that shows sells over the past day, week and month. In some cases I have a very high number but in most cases this number is quite low, and this is affecting quite a lot ...
1
vote
0answers
150 views

Multivariate imputation in R

I am working with a dataset from the world bank, it's a relatively simple dataset with 11 variables for 211 countries from 1990-2015: ...
1
vote
2answers
829 views

Outlier removal before Boosted Decision Tree Regression

I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe ...
0
votes
1answer
2k views

how to handle sparse data problem in unsupervised learning .i'm going to use k means on data set

how to handle sparse data problem in unsupervised learning .i'm going to use k-means on the dataset. I have 200 variables, nearly in each column have 70% zeros. how can I handle without discarding any ...
4
votes
3answers
4k views

Difference between preprocessing train and test set before and after splitting

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?...
1
vote
2answers
64 views

Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
0
votes
0answers
695 views

Improving RMSE of my model

I'm trying to build a model based on some training set. The training set contains 1460 observations, with 79 variables each (features). I'm using linear regression to build a model and after that ...
1
vote
1answer
255 views

What criteria to use when having with 'unengaged' , ‘straightlined’ or ‘patterned’ responses in questionnaires?

Please consider the following situation, which happens a lot in real life applied research in social sciences. Some percentage of the respondents give ‘straightlined’ or ‘patterned’ responses to a ...
0
votes
1answer
37 views

Should I remove data with known computational error before doing linear regression?

I have a dataset having 252 observations. There are two variables having a defined and known linear relationship. By computation, I found 4 observations do not follow the linear equation, and I ...
5
votes
0answers
75 views

Does tidiness of data differ by application?

After reading a recent paper by Hadley (link), I got to thinking about whether what we'd refer to as tidy data changes by application. For example, consider a sample dataset: ...
3
votes
1answer
535 views

What is the best way to perform sentence segmentation for textual analysis?

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:...
1
vote
1answer
644 views

Fitting linear model through noisy data

I'm currently working on a predictive modeling project. I have to predict $Y$ given variables $X_1,X_2,X_3$ and $X_4$ that are not necessarily independent. Our first idea was to propose a linear ...
2
votes
0answers
1k views

How to perform data quality check on large number of features using Spark?

I am used to work with manageable number of features. I usually print some descriptive statistics and visualise the histograms of each feature using Python and Pandas or R. I check for outliers and if ...
0
votes
0answers
68 views

Which classifier should I use for sparse boolean features?

I have training data that is classified into 2 categories: X and Not X Each piece of training or experimental data has a variable number of boolean features. Each piece of data may have ~100 features,...
1
vote
0answers
34 views

How to identify outlying raters

I have collected data from 500 anonymous online respondents who rated the acceptability of two variations of a large number of sentences (3-level categorical data: excellent, acceptable, impossible). ...
1
vote
0answers
205 views

Would it make a difference to clean the data before using R findCorrelation() method?

I'm new in data analysis area and didn't have very strong statistical background... Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to ...
1
vote
1answer
59 views

How to organize lots of data and extract information

I am working in a project where I need to extract data from lots of files provided to me in different formats (.exe, .doc, .txt, .dbf, and even .pdf). The data I am interested in are water quality and ...
1
vote
3answers
5k views

Data Imputation in R with NAs in only one variable (categorical)

I have data frame with 44,353 entries with 17 variables (4 categorical + 13 continuous). Out of all variables only 1 categorical variable (with 52 factors) has NAs No of factors in the categorical ...
0
votes
0answers
130 views

outlier detection : use trimmed data to calculate STD?

I can use box-plot to examine data and detect outliers based on box-plot. Most often Box-plot uses a q-range (q3-q1) to define the data boundary and then display "outliers". Sometimes I see people ...
0
votes
0answers
172 views

Recoding Ordinal Variables to Binary: Justifiable?

Suppose I have a model $Y = \alpha + \beta X$ where I am interested in the point estimate and hypothesis testing for $\beta$. Furthermore, $Y$ is an ordinal variable taking on a small number of value ...
3
votes
0answers
17 views

How to deal with implausible answers to filter questions? [duplicate]

I work with a questionnaire that contains a variety of filter questions. In the following I will give an example how one block of this successive questions look like. Have you ever skipped ...
2
votes
1answer
114 views

Cleaning a text: R vs Python [closed]

I have a "dirty" text with statements as L'intervento ÃẀ mirato all'eliminazione nonché alla... where I have ÃẀ instead of è, é instead of é, etc. Which is the best language for manipulating ...
3
votes
0answers
432 views

Sensor data cleaning

Underneath is a picture of a sensor measuring the fill rate of a container on an hourly basis. It goes up to 100% and is then emptied. There is some natural deviation of the sensor due to temperature ...
1
vote
1answer
27 views

Data Manipulation

I currently have $n$ datapoints along $k$ dimensions ($n$ observations and $k$ variables). I need to drop a lot of these observations by some criteria. To be more specific, I have to perform a few ...
-1
votes
1answer
276 views

dealing with meaningless inconsistent data

Am trying to develop a predictive model, in the data set instances have an attribute subject with 40 possible values such as ...
3
votes
3answers
7k views

Best Practices with Data Wrangling before running Random Forest Predictions

When doing predictions with Random Forests, we very often (or always) need to perform some pre-processing. Since I have a background of Computing and pretty much all I know from statistics comes from ...
0
votes
1answer
41 views

List of Combination

Hallo I have this problem about some data that i wish to reconcile. I do not really know the specific statistical software that could allow me to generate the list of combinations that add up to a ...
1
vote
1answer
1k views

Sample time series to equal interval

I have data with timestamp and associated values. time interval between two consecutive data is not constant. How to standardize the the time series and associated value ? eg- Input data is Timestamp ...