Structured data files in any format, collected together with the documentation that explains their production or use.

learn more… | top users | synonyms (1)

0
votes
0answers
8 views

Identifying duplicated rows [migrated]

I have a larger data frame (~50K rows and 50 to 75 columns) that has a small number of row that are duplicated in, say, 7 of the 75 columns. Although it's simple enough to locate rows that duplicate ...
1
vote
0answers
10 views

Regression of irregular time series

I am beginning to look at high frequency data sets. I have only ever dealt with regularly spaced data up until now so wondering what best practice is. If I have two time series X and Y with Y being ...
0
votes
1answer
25 views

Simulate data by using existing dataset

I have a complete dataset with input variables and response variables. I would like to perform a simulation where I give the input variables and generate randomly the response variables. Is there a ...
0
votes
0answers
5 views

Getting a list of zip codes with its adjacent zip codes [migrated]

I need a list in the form of adjacency list that contains all zip codes in the United States that each has a list of adjacent zip codes. I tried searching for it on the internet. Many of them only ...
11
votes
1answer
102 views
+200

Good data example needed with covariate affected by treatments

I have looked at a lot of R datasets, postings in DASL, and elsewhere, and am not finding very many good examples of interesting datasets illustrating analysis of covariance for experimental data. ...
0
votes
0answers
22 views

Help on small data set with correlated features

My data set is quite small and feature rich. It has 30 observations and 30 features/predictors. In addition to that, many features are correlated (often not linear). I have tried several things- ...
2
votes
1answer
22 views

What do NORB and CIFAR stand for?

The MNIST dataset is a standard benchmark data set of digit images. MNIST stands for 'Mixed National Institute of Standards and Technology'. The NORB dataset is a commonly used dataset of binocular ...
0
votes
0answers
11 views

Invalid measurement data removing

I have a lot of data (temperatures and similar things) collected using automatic devices. Lots of these data is let's call it smooth: even when the temperature increases/decreases fast there are few ...
2
votes
0answers
30 views

Averaging z scores when doing a “group by”

I have a dataset where each row is an hourly measurement of certain fields (columns). For each column I then add another column that is its respective z score relative to the entire population. If I ...
1
vote
0answers
10 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
1answer
19 views

Plotting Scatterplot Matrix or Correlation matrix or both?

I have a problem where I want to use a classifier for it. So I defined a set of features and created a dataset. Now I want to generate some plots to understand the features. I came across the ...
0
votes
0answers
17 views

Data Mining from excel [migrated]

new to the site as well as microsoft excel I need help developing a macro to make a manual process into an automated one regarding 2 workbooks. If data in column A and column C of Workbook1, named ...
2
votes
0answers
16 views

Anonymizing survey data, plus an ethics question

This is a fairly simple question about raw data in Excel. I have a survey asking respondents to rate 17 questions on a rating scale from 1 to 5. The goal is for individuals who answer the survey to be ...
4
votes
2answers
73 views

Should types of data (nominal/ordinal/interval/ratio) really be considered types of variables?

So for instance here are the definitions that I get from standard text books Variable - characteristic of population or sample. ex. Price of a stock or grade on a test Data - actual ...
3
votes
2answers
57 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
2
votes
3answers
62 views

Gaps in time series and time series validity

After doing some reading on CrossValidated, I understood that we can use "imputation" techniques to fill in the gaps (if they are random). But I am not clear on following questions: How many ...
2
votes
2answers
50 views

How to get rid of bias in data?

I have been trying to classify a set of data into one of four classes. The data has already been generated and I have set aside 10,000 for training and 2,000 for testing. I have also generated the ...
0
votes
0answers
8 views

LETOR data sets, which contain the queries and documents?

While researching information retrieval evaluation, I found LETOR. This is a test collection used for evaluation of ranking algorithms. Since the data contains relevance judgements for queries, it can ...
0
votes
0answers
34 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
0
votes
1answer
57 views

Data Mining, generating a model from a database

I have a database with different variables which contains information such as age, date of vaccination, number of doses as well as the number of antibodies (which is my target variable). If the number ...
0
votes
0answers
52 views

Method for detecting anomalies in timeseries?

I am exploring creating alerts for anomalies in a timeseries. Say that I have a graph like this: There are two anomalies here that I would wanted to be alerted on: First, on April 7th with there ...
1
vote
1answer
42 views

Predictive analysis of data using R

I am trying to analyze data to predict the effect of 8 predictors on a response variable. I ant to do it using R. There are 8 predictor variables (x1, x2, ..., x8) and one response variable (y). I ...
2
votes
1answer
54 views

Interaction effects in big data sets

I'm looking for a method to identify a shortlist of potentially good 2-way interaction terms rather than trying all possible interactions. This question is similarly asked before here but in a more ...
0
votes
1answer
32 views

Data preparation and machine algo for ad click prediction

I am an ml noob. I have a task at hand of predicting click probability given user information like city, state, os version, os family, device, browser family browser version, city, etc. I have been ...
0
votes
0answers
11 views

Data Mining technique for preferred vendor problem?

Currently I use open auction method for selecting a transporter and want to switch to preferred transporter where I would select a transporter based on my preferred transporter list. The project ...
0
votes
0answers
10 views

Dataset for classifying image data, containing spatial info

I am looking for an object classification data set that meets some criteria: It should already have high-level local features extracted from the images (ideally taken from a grid) (less than 100 ...
0
votes
1answer
43 views

Linear regression on several trials of experiment

Sorry about asking such a basic question. Assuming I have data like this ...
1
vote
2answers
114 views

How to deal with unbalanced data

I'm doing data analysis with a dataset of 11795 data points (with 88 features). 85% (9973 points) of these data points correspond to data points belonging to class 1, 5% (589 points) belong to class 2 ...
4
votes
3answers
270 views

The idea of making the data have a zero-mean

I often see people making a dimension/feature of a dataset be of a zero-mean by removing the mean from all the elements. But I never I understood why to do so? What is the effect of doing that as a ...
0
votes
1answer
31 views

Best practice to understand a dataset

My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct. My questions: What are ...
2
votes
2answers
55 views

Are there any good publicly accessible wisdom of crowd data sets?

I am looking for publicly accessible data sets to explore some wisdom of the crowd ideas. An ideal data set would contain predictions from a number of individuals (who were not collaborating with ...
3
votes
0answers
58 views

How to create appropriate number of data points that would be accurate enough for creating regression equations?

I found a similar question in this forum. As a rule of thumb, since there are 4 independent variables in my case, I need 4*10=40 data points. However, my question differs slightly, since I want to ask ...
1
vote
1answer
35 views

'Panel' data without unit identifier

I need help trying to figure out what data-structure I'm dealing with, and what estimation & inference models I can therefore use. There are multiple units and periods. No unit identifiers, so ...
0
votes
0answers
9 views

Econometrics: Panel data without unit identifier [duplicate]

I need help trying to figure out what data-structure I'm dealing with, and what estimation & inference models I can therefore use. There are multiple units and periods. No unit identifiers, so ...
0
votes
0answers
10 views

Grouped and Ungrouped Data- What's the difference?

Would data in two columns such as Shoe Size (variate x) and Number of Pairs (frequency f) be considered grouped or ungrouped data? Does grouped data necessarily imply the existence or classification ...
0
votes
0answers
8 views

augmenting test set

In training a neural network for an image classification task, is it ok to augment the test set by affine transforms when the dataset is small (~1000 images), rather than doing k-fold cross ...
0
votes
0answers
8 views

Quantify possible model improvement given samples X in Y dimensions and Z% accuracy in a random forest model

I've been doing some Machine Learning as a hobby and am playing around with Random Forest models and basically trying to apply that towards predicting things in my own life. So I have a series of ...
4
votes
3answers
256 views

How to organise the variable names in R without messing up?

When I work in R, every time I create a new variable I try to name it simple so that it iss easy to type like x1, x2, ... or ...
2
votes
0answers
31 views

Under what conditions should Likert scales be used as nominal data?

Similar to this question, Under what conditions should Likert scales be used as ordinal or interval data?, I am curious under what conditions should/could Likert scales be used as nominal data?
2
votes
1answer
41 views

Dataset for Least Angle Regression

I have read that least angle regression is good for high dimensional data. I didn't actually understand the meaning of high dimensional data, so does this mean $p>>n$ case? And does anyone know ...
1
vote
0answers
25 views

How can i perform Cosinor analysis in Matlab?

I have monthly and daily data from births in Greece and i want to to do a cosinor analysis to find the rhythm, (circadian,circannual,half yearly etc etc ) How can i use Matlab to perform the cosinor ...
1
vote
1answer
63 views

Original source for the “play tennis” dataset

A famous toy example in machine learning, especially with learning decision trees, is the well known "play tennis" dataset. Is there an official source for the dataset which could be quoted in ...
0
votes
1answer
20 views

Statistical analysis pre + post testing- with control and intervention 35 participants

What statistical analysis would I use when testing 35 participants. pre & post test and checking for changes after 1 intervention and a control group with no intervention? I have one mediating ...
0
votes
0answers
19 views

Where can I download sample data sets for mobile app downloads?

For statistical testing, I am in need for android and apple mobile app download data. I have checked on AWS (Amazon) and Infochimps. No luck so far. Can anyone suggest other resources or is there a ...
0
votes
0answers
15 views

How to handle imbalanced data using SMOTE-N (nominal data) to generate multiple synthetic data?

I have a classification problem with two classes working on nominal data. I want to apply SMOTE-N to deal with imbalanced data. However, it is not clear to me how to use SMOTE-N for generating N ...
0
votes
1answer
30 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
0answers
18 views

Examples of derived data quality features?

I have a predictive classification task and the end users want to get initial understanding of: the quality (not specified further) of each data point in a data set whether there are systematic ...
0
votes
0answers
37 views

Fixed effects analysis in Stata

I have a large data set on which I need to run a fixed effect regression. The two variables that I want to control for: year (2001–2008) and industries (20 types). But I apparently need to set the ...
1
vote
1answer
12 views

Compare and quantify quality of data

We get vendor quotes (their prices for different financial securities) every month. I want to compare which of these vendors 'agree' (i.e come close to) with our internal prices [the gold standard]. ...