Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

Filter by
Sorted by
Tagged with
135
votes
16answers
37k views

Are large data sets inappropriate for hypothesis testing?

In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
62
votes
8answers
99k views

How to simulate data that satisfy specific constraints such as having specific mean and standard deviation?

This question is motivated by my question on meta-analysis. But I imagine that it would also be useful in teaching contexts where you want to create a dataset that exactly mirrors an existing ...
99
votes
25answers
41k views

Locating freely available data samples

I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics. While the method works ...
32
votes
3answers
3k views

Datasets constructed for a purpose similar to that of Anscombe's quartet

I've just come across Anscombe's quartet (four datasets that have almost indistinguishable descriptive statistics but look very different when plotted) and I am curious if there are other more or less ...
5
votes
3answers
6k views

Tool for generating correlated data sets

Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)? I am working on ...
28
votes
2answers
4k views

What aspects of the “Iris” data set make it so successful as an example/teaching/test data set

The "Iris" dataset is probably familiar to most people here - it's one of the canonical test data sets and a go-to example dataset for everything from data visualization to machine learning. For ...
10
votes
2answers
2k views

Should types of data (nominal/ordinal/interval/ratio) really be considered types of variables?

So for instance here are the definitions that I get from standard text books Variable - characteristic of population or sample. ex. Price of a stock or grade on a test Data - actual ...
93
votes
6answers
7k views

Essential data checking tests

In my job role I often work with other people's datasets, non-experts bring me clinical data and I help them to summarise it and perform statistical tests. The problem I am having is that the ...
40
votes
2answers
76k views

How to normalize data between -1 and 1?

I have seen the min-max normalization formula but that normalizes values between 0 and 1. How would I normalize my data between -1 and 1? I have both negative and positive values in my data matrix.
19
votes
2answers
13k views

Testing Classification on Oversampled Imbalance Data

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are: SMOTE: Synthetic ...
17
votes
2answers
81k views

Calculating the 95th percentile: Comparing normal distribution, R Quantile, and Excel approaches

I was trying to compute the 95th percentile on the following dataset. I came across a few online references of doing it. Approach 1: Based on sample data The first one tells me to obtain the ...
13
votes
4answers
3k views

Separating two populations from the sample

I'm trying to separate two groups of values from a single data set. I can assume that one of the populations is normally distributed and is at least half the size of the sample. The values of the ...
4
votes
1answer
674 views

Coding of semi-numerical variables

I've asked this question before, in the context of logistic-regression, but I'd like to pose it in a broader context here. There are a lot of variables with interval or ratio-type values for some ...
2
votes
1answer
4k views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...
40
votes
2answers
6k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
26
votes
9answers
6k views

What do statisticians do that can't be automated?

Will software eventually make statisticians obsolete? What is done that can't be programmed into a computer?
11
votes
3answers
934 views

Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?

The i.i.d. assumption states: We are given a data set, $\{(x_i,y_i)\}_{i = 1, \ldots, n}$, each data $(x_i,y_i)$ is generated in an independent and identically distributed fashion. To me, ...
15
votes
3answers
7k views

How to do data augmentation and train-validate split?

I am doing image classification using machine learning. Suppose I have some training data (images) and will split the data into training and validation sets. And I also want to augment the data (...
5
votes
1answer
333 views

Difference between NA and None in biological dataset

I have collected environmental data on animals, and am seeking help on how to deal with certain variables where a non-numeric value is informative, but also problematic. I have three variables that ...
18
votes
10answers
13k views

Social network datasets

I am looking for social network datasets (twitter, friendfeed, facebook, lastfm, etc.) for classification tasks, preferably in arff format. My searches via UCI and Google weren't successful so far......
18
votes
2answers
11k views

Quality assurance and quality control (QA/QC) guidelines for a database

Background I am overseeing the input of data from primary literature into a database. The data entry process is error prone, particularly because users must interpret experimental design, extract ...
22
votes
2answers
19k views

Data augmentation techniques for general datasets?

In many machine learning applications, the so called data augmentation methods have allowed building better models. For example, assume a training set of $100$ images of cats and dogs. By rotating, ...
11
votes
2answers
11k views

Optimal Binning with respect to a given response variable

I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter. example:...
12
votes
4answers
2k views

Examples of costly consequences from improper use of statistical tools

I suspect that most users of statistical tools are ancillary users (folks who have had little to no formal training in statistics). It’s very tempting for researchers and other professionals to apply ...
16
votes
6answers
25k views

Where to find a large text corpus? [closed]

I am looking for large (>1000) text corpus to download. Preferably with world news or some kind of reports. I have only found one with patents. Any suggestions?
8
votes
2answers
950 views

Likelihood function of truncated data

I am having a little trouble understanding the concept and derivation of the likelihood of truncated data. For example, if I want to find the likelihood function based on a sample from a distribution,...
7
votes
1answer
1k views

Construct artificial slightly overlapping data for PCA plot

I am trying to construct artificial data which show two distinct groups in a PCA plot. However, the two groups should still slightly overlap. The following approaches came the closest but I am still ...
3
votes
3answers
4k views

Learning multiple output

Can you suggest me an algorithm and probably a real code for multiple output learning, where input of the model is vector of around 10 000 values and output is, for each input vector, an output vector ...
9
votes
1answer
6k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
7
votes
3answers
23k views

Handling unbalanced data using SMOTE - no big difference?

I have a classification problem with 2 classes. I have nearly 5000 samples, each of which is represented as vector with 570 features. The positive class samples are nearly 600. Meaning, I have a 1:8 ...
6
votes
3answers
6k views

Identify probability distributions

Given a sample data set of floating point numbers, how do we determine its probability distribution and prove it? Also generate random numbers of the same distributions thereafter.
7
votes
2answers
401 views

Generating a high-dimensional dataset where nearest neighbor becomes meaningless

In the paper "When Is 'Nearest Neighbor' Meaningful?" we read that, We show that under certain broad conditions (in terms of data and query distributions, or workload), as dimensionality increases,...
5
votes
3answers
2k views

A data set with missing values in multiple variables

I'm trying to analyze a set of data related to the health area but I'm not sure how to proceed with the missing values. Objective: To adjust a model with a discrete response, to study the influence ...
5
votes
1answer
3k views

How do I clean up inconsistent survey data?

I have some survey response data that are inconsistent and I'm not sure what is the best way to handle them. This is the nature of the inconsistency: Have you ever had a food poisoning episode that ...
4
votes
3answers
2k views

How should we convert sports results data to perform a valid logistical regression?

Say we want to perform a logistical regression analysis (although my question pertains to regressions in general) on sports results to determine the effects of various factors on who wins and who ...
2
votes
1answer
7k views

How to organize the data for this repeated-measure within-subject setup?

This question is a follow up to the question I asked yesterday. Here is the current situation again. I have data for a within subject measurement, where each variable is measured repeatedly. The ...
1
vote
0answers
6k views

Data Preparation for Cluster Analysis

Updated answer to "Data Preparation for Cluster Analysis": Based on the discussions, data normalization and removing correlation among data are often recommended. References posts: 1) Are mean ...
1
vote
1answer
83 views

How can I create meaningful / interesing data following multiple linear models?

I would like to create many(!) examples for exercices, where I would like to control various parameters such as the coefficients, their uncertainty, t-values and p-values of the linear model. I know, ...
3
votes
1answer
190 views

Does retraining a model on all available data necessarily yield a better model?

A (simplified) typical workflow in machine learning might be: Train $m$ models on a training set. Validate the $m$ models on a validation set to yield the best model with parameters $\theta$. Retrain ...
2
votes
3answers
167 views

Are the variable types here considered correct?

If we want to determine the variable types, will it be as follows for the below variables? Age ---> quantitative, discrete (we ...
2
votes
1answer
3k views

Standard formula for quick calculation of scores

I have three datasets as follows: Dataset A can have a maximum score of 30. (Individual scores can be from 1 to 30) Dataset B can have a maximum score of 45. (Individual scores can be from 1 to 45) ...
2
votes
1answer
990 views

Should I use multi-label classification?

I have a classification problem with 2 classes (positive and negative). Usually, in such classification problems, all the samples will be labelled either 'positive' or 'negative'. In my dataset, some ...
2
votes
2answers
2k views

What is the recommended way to Normalize data to a NN (or CNN)?

I've seen several ways to normalize a data (features or even images) before use as input in a NN or CNN. The most common I saw are: [0, 1]: (data - min(data)) / (max(data) - min(data)) z-score: (...
1
vote
2answers
116 views

What is the statistical method that I should use for the following problem?

I have been given a problem and need to find the best statistical method that would help. The problem is as follows: A business has approx. 100,000 active customers who spend between £500 per annum ...
0
votes
2answers
237 views

If the model fits well, nothing can be done?

I am playing a data without any background information. First, I try multiple linear regression. The model fits well, since the $r^2$ is larger than 90%. I deleted several variables by AIC. The fits ...
0
votes
3answers
132 views

How to deal with a data-set having all the samples with every feature-point almost similar?

I have a very large training set where each of the feature's datapoints are very similar (float values in the range 95.0 to 97.0)...
53
votes
3answers
20k views

Data APIs/feeds available as packages in R

EDIT: The Web Technologies and Services CRAN task view contains a much more comprehensive list of data sources and APIs available in R. You can submit a pull request on github if you wish to add a ...
35
votes
5answers
15k views

Free data set for very high dimensional classification [closed]

What are the freely available data set for classification with more than 1000 features (or sample points if it contains curves)? There is already a community wiki about free data sets: Locating ...
43
votes
9answers
32k views

Tiny (real) datasets for giving examples in class?

When teaching an introductory level class, the teachers I know tend to invent some numbers and a story in order to exemplify the method they are teaching. What I would prefer is to tell a real story ...
10
votes
3answers
18k views

Should feature selection be performed only on training data (or all data)?

Should be feature selection performed only on training data (or all data)? I went through some discussions and papers such as Guyon (2003) and Singhi and Liu (2006), but still not sure about right ...