Structured data files in any format, collected together with the documentation that explains their production or use.

learn more… | top users | synonyms (1)

0
votes
0answers
10 views

Invalid measurement data removing

I have a lot of data (temperatures and similar things) collected using automatic devices. Lots of these data is let's call it smooth: even when the temperature increases/decreases fast there are few ...
2
votes
0answers
29 views

Averaging z scores when doing a “group by”

I have a dataset where each row is an hourly measurement of certain fields (columns). For each column I then add another column that is its respective z score relative to the entire population. If I ...
1
vote
0answers
9 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
1answer
18 views

Plotting Scatterplot Matrix or Correlation matrix or both?

I have a problem where I want to use a classifier for it. So I defined a set of features and created a dataset. Now I want to generate some plots to understand the features. I came across the ...
0
votes
0answers
17 views

Data Mining from excel [migrated]

new to the site as well as microsoft excel I need help developing a macro to make a manual process into an automated one regarding 2 workbooks. If data in column A and column C of Workbook1, named ...
2
votes
0answers
15 views

Anonymizing survey data, plus an ethics question

This is a fairly simple question about raw data in Excel. I have a survey asking respondents to rate 17 questions on a rating scale from 1 to 5. The goal is for individuals who answer the survey to be ...
4
votes
2answers
70 views

Should types of data (nominal/ordinal/interval/ratio) really be considered types of variables?

So for instance here are the definitions that I get from standard text books Variable - characteristic of population or sample. ex. Price of a stock or grade on a test Data - actual ...
3
votes
2answers
56 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
2
votes
3answers
62 views

Gaps in time series and time series validity

After doing some reading on CrossValidated, I understood that we can use "imputation" techniques to fill in the gaps (if they are random). But I am not clear on following questions: How many ...
2
votes
2answers
48 views

How to get rid of bias in data?

I have been trying to classify a set of data into one of four classes. The data has already been generated and I have set aside 10,000 for training and 2,000 for testing. I have also generated the ...
0
votes
0answers
8 views

LETOR data sets, which contain the queries and documents?

While researching information retrieval evaluation, I found LETOR. This is a test collection used for evaluation of ranking algorithms. Since the data contains relevance judgements for queries, it can ...
-2
votes
0answers
31 views

Imputing missing responses in test exam [on hold]

I am working with a database that consist in the answers of 100 students to about 300 questions (1: right answer, 0: wrong ...
0
votes
0answers
29 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
0
votes
1answer
53 views

Data Mining, generating a model from a database

I have a database with different variables which contains information such as age, date of vaccination, number of doses as well as the number of antibodies (which is my target variable). If the number ...
0
votes
0answers
51 views

Method for detecting anomalies in timeseries?

I am exploring creating alerts for anomalies in a timeseries. Say that I have a graph like this: There are two anomalies here that I would wanted to be alerted on: First, on April 7th with there ...
1
vote
1answer
36 views

Predictive analysis of data using R

I am trying to analyze data to predict the effect of 8 predictors on a response variable. I ant to do it using R. There are 8 predictor variables (x1, x2, ..., x8) and one response variable (y). I ...
2
votes
1answer
53 views

Interaction effects in big data sets

I'm looking for a method to identify a shortlist of potentially good 2-way interaction terms rather than trying all possible interactions. This question is similarly asked before here but in a more ...
0
votes
1answer
28 views

Data preparation and machine algo for ad click prediction

I am an ml noob. I have a task at hand of predicting click probability given user information like city, state, os version, os family, device, browser family browser version, city, etc. I have been ...
0
votes
0answers
11 views

Data Mining technique for preferred vendor problem?

Currently I use open auction method for selecting a transporter and want to switch to preferred transporter where I would select a transporter based on my preferred transporter list. The project ...
0
votes
0answers
10 views

Dataset for classifying image data, containing spatial info

I am looking for an object classification data set that meets some criteria: It should already have high-level local features extracted from the images (ideally taken from a grid) (less than 100 ...
0
votes
1answer
41 views

Linear regression on several trials of experiment

Sorry about asking such a basic question. Assuming I have data like this ...
1
vote
2answers
111 views

How to deal with unbalanced data

I'm doing data analysis with a dataset of 11795 data points (with 88 features). 85% (9973 points) of these data points correspond to data points belonging to class 1, 5% (589 points) belong to class 2 ...
4
votes
3answers
262 views

The idea of making the data have a zero-mean

I often see people making a dimension/feature of a dataset be of a zero-mean by removing the mean from all the elements. But I never I understood why to do so? What is the effect of doing that as a ...
0
votes
1answer
30 views

Best practice to understand a dataset

My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct. My questions: What are ...
2
votes
2answers
51 views

Are there any good publicly accessible wisdom of crowd data sets?

I am looking for publicly accessible data sets to explore some wisdom of the crowd ideas. An ideal data set would contain predictions from a number of individuals (who were not collaborating with ...
3
votes
0answers
56 views

How to create appropriate number of data points that would be accurate enough for creating regression equations?

I found a similar question in this forum. As a rule of thumb, since there are 4 independent variables in my case, I need 4*10=40 data points. However, my question differs slightly, since I want to ask ...
1
vote
1answer
35 views

'Panel' data without unit identifier

I need help trying to figure out what data-structure I'm dealing with, and what estimation & inference models I can therefore use. There are multiple units and periods. No unit identifiers, so ...
0
votes
0answers
9 views

Econometrics: Panel data without unit identifier [duplicate]

I need help trying to figure out what data-structure I'm dealing with, and what estimation & inference models I can therefore use. There are multiple units and periods. No unit identifiers, so ...
0
votes
0answers
9 views

Grouped and Ungrouped Data- What's the difference?

Would data in two columns such as Shoe Size (variate x) and Number of Pairs (frequency f) be considered grouped or ungrouped data? Does grouped data necessarily imply the existence or classification ...
0
votes
0answers
8 views

augmenting test set

In training a neural network for an image classification task, is it ok to augment the test set by affine transforms when the dataset is small (~1000 images), rather than doing k-fold cross ...
0
votes
0answers
8 views

Quantify possible model improvement given samples X in Y dimensions and Z% accuracy in a random forest model

I've been doing some Machine Learning as a hobby and am playing around with Random Forest models and basically trying to apply that towards predicting things in my own life. So I have a series of ...
4
votes
3answers
251 views

How to organise the variable names in R without messing up?

When I work in R, every time I create a new variable I try to name it simple so that it iss easy to type like x1, x2, ... or ...
2
votes
0answers
30 views

Under what conditions should Likert scales be used as nominal data?

Similar to this question, Under what conditions should Likert scales be used as ordinal or interval data?, I am curious under what conditions should/could Likert scales be used as nominal data?
2
votes
1answer
39 views

Dataset for Least Angle Regression

I have read that least angle regression is good for high dimensional data. I didn't actually understand the meaning of high dimensional data, so does this mean $p>>n$ case? And does anyone know ...
1
vote
0answers
23 views

How can i perform Cosinor analysis in Matlab?

I have monthly and daily data from births in Greece and i want to to do a cosinor analysis to find the rhythm, (circadian,circannual,half yearly etc etc ) How can i use Matlab to perform the cosinor ...
1
vote
1answer
60 views

Original source for the “play tennis” dataset

A famous toy example in machine learning, especially with learning decision trees, is the well known "play tennis" dataset. Is there an official source for the dataset which could be quoted in ...
0
votes
1answer
19 views

Statistical analysis pre + post testing- with control and intervention 35 participants

What statistical analysis would I use when testing 35 participants. pre & post test and checking for changes after 1 intervention and a control group with no intervention? I have one mediating ...
0
votes
0answers
16 views

Where can I download sample data sets for mobile app downloads?

For statistical testing, I am in need for android and apple mobile app download data. I have checked on AWS (Amazon) and Infochimps. No luck so far. Can anyone suggest other resources or is there a ...
0
votes
0answers
15 views

How to handle imbalanced data using SMOTE-N (nominal data) to generate multiple synthetic data?

I have a classification problem with two classes working on nominal data. I want to apply SMOTE-N to deal with imbalanced data. However, it is not clear to me how to use SMOTE-N for generating N ...
0
votes
1answer
30 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
0answers
17 views

Examples of derived data quality features?

I have a predictive classification task and the end users want to get initial understanding of: the quality (not specified further) of each data point in a data set whether there are systematic ...
0
votes
0answers
36 views

Fixed effects analysis in Stata

I have a large data set on which I need to run a fixed effect regression. The two variables that I want to control for: year (2001–2008) and industries (20 types). But I apparently need to set the ...
1
vote
1answer
12 views

Compare and quantify quality of data

We get vendor quotes (their prices for different financial securities) every month. I want to compare which of these vendors 'agree' (i.e come close to) with our internal prices [the gold standard]. ...
0
votes
0answers
31 views

Delete observations for regression modelling

I am implementing robust regression on some data and when i tested the accuracy on a hold set of data, the accuracy was very bad. I have about 500 000 observations for the regression model and i ...
0
votes
1answer
26 views

use of validation set on lasso cross validation

When training a model a train, a validation and test set are used. I was wondering if there is any paper or example that proves that the use of an independent validation set increase the performance ...
0
votes
0answers
10 views

correlate/compare two scales measuring different constructs

I need to compare a multidimensional pain scale item response >3 pain is present and a scale measuring emotional suffering with classified levels of suffering (High, intermediate and low). Will ICC ...
0
votes
1answer
38 views

Organize data with multiple levels of a categorical variable per entry, for easy R analysis

I need some help here. I have some data in which every entry can take one or more levels of a categorical variable. for example, I have a category with 3 levels: ...
2
votes
2answers
66 views

Building training set from unlabeled data

I want to build classifier using naive bayes. I know that naive bayes is supervised learning that requires labeled data for training set. However, I only have data with no label on it. Is there any ...
0
votes
0answers
20 views

I have data that includes a record of each event and its time

I want to see if this other data I have predicts that this first event will occur. In other words, I have a data field with a record of farts, with columns for day, month, year, and then I have ...