Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

550 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
1answer
50 views

Visualising activity

I work for an organisation that provides free activities that benefit people's health. We have a well maintained database that records participation and I am using it in order to research the active ...
5
votes
0answers
789 views

Best practices for documenting a data-science pipeline

Even though I try to keep it as simple as possible, the pipelines for some of my data science projects get rather complex. At some point it becomes necessary to document this pipeline so that someone ...
5
votes
1answer
208 views

Share repurchase

I am doing a project for share repurchase in the US. I am trying to estimate the operating performance before and after the repurchase announcement. And I am trying to adjust the performance to the ...
5
votes
1answer
865 views

Assessing the representativeness of population sampling

I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing. In this dataset I am looking at the relationship between two variables (e.g., X and Y)...
4
votes
0answers
35 views

How to combine noisy and noise-free datasets to train a model

Overview Suppose I have two datasets, both of which consist of rows of features and their matching labels. One of these datasets is noise-free and its labels correspond to the ground truth, but the ...
4
votes
0answers
356 views

Simulate/Generate Data for Multinomial Logistic regression

How to simulate data for Multinomial Logistic regression? For Example i want to generate a high dimensional data set with 90 subjects and 500 independent predictors. The ratio of Classes should given ...
4
votes
0answers
224 views

Correcting Sample Selection Bias given actual Distribution

I have two datasets, both from the same population: The samples from the first survey are quite representative of the underlying truth. However, the second survey comes with a change in distribution ...
4
votes
0answers
65 views

How does one design a data set from polynomial target function such that logistic regression separates the data perfectly?

I want to design a target function for a classification task of the form: $$ f_{target}(x) = \mathbb{1}_{>0}[\sum^{D^*}_{i=0} w^*_i x^i] = \mathbb{1}_{>0}[ \langle w^*, \Phi(x)\rangle]$$ and ...
4
votes
2answers
93 views

Evaluating TAs grading

Let me introduce the context: I need to evaluate how my TAs tend to grade students. Ideally I would like to pinpoint whether they show some sort of internal consistency (e.g., a TA usually grades ...
4
votes
0answers
610 views

How to measure similarity between features in different datasets, if possible?

Is it possible to measure the similarity between two different features in different datasets? It could sound like a non-sense question, but it has sense in the context of merging two almost ...
4
votes
0answers
16k views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
4
votes
0answers
876 views

Why would SVD be 'unstable' if you don't standardize your data first?

I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
3
votes
1answer
77 views

why has author divided by 1.5 in hands on machine learning with scikit learn

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
3
votes
0answers
197 views

Benchmark data sets for anomaly detection algorithms in multivariate time series

as per title, which datasets are commonly used to benchmark novel methods to detect anomalies in multivariate time series? I'm particularly interested in moderately high-dimensional (10-40 components),...
3
votes
2answers
105 views

Handling missing data for participants who have not completed any standardised measures and have only provided demographic answers

When managing missing data, how many questions should participant have completed, at a minimum, before imputing the remainder of their missing data? For example, a number of my participants only ...
3
votes
0answers
121 views

What information should be released to characterize a dataset for text classification?

I am releasing a dataset for text classification. ​What information would a researcher in natural language processing or machine learning may want to have about this dataset? Here some some ...
3
votes
0answers
61 views

How do I compare frequency data of more than two groups?

I am comparing the frequencies of a disease presenting in particular season. The expected proportion would be 0.25 and the observed frequencies for spring, summer, autumn, and winter were 7, 9, 11, ...
3
votes
0answers
1k views

Google Trends: Stitching 90 day periods of daily data together

Google Trends lets you see the amount of researches made on for a term on google during a set period of time, normalized between 0 and 100 (depending on the highest value in that period). I would like ...
3
votes
1answer
60 views

Parameter Inference when Model is a bad fit to the data.

I am working with gamma-ray data from the Fermi Satellite. The data has been binned into energy dependent maps of the sky -- e.g. three dimensions (energy, latitude, longitude) and is extremely high ...
3
votes
0answers
71 views

differences between fake computer-generated data & real data

generally, is there a/some formula(s) or algorithm(s) to prove that a data set is a fake one generated using computer and not by field work? more specific, recently I got a research questionnaire data ...
3
votes
0answers
8k views

What is robust data?

This is the rather confusing go-to internet definition for robust data: Robust data is data that is constructed to survive and function in multiple settings. It's reusable. It can be updated. Anyone ...
3
votes
0answers
4k views

Averaging z scores when doing a “group by”

I have a dataset where each row is an hourly measurement of certain fields (columns). For each column I then add another column that is its respective z score relative to the entire population. If I ...
3
votes
0answers
146 views

Anonymizing survey data, plus an ethics question

This is a fairly simple question about raw data in Excel. I have a survey asking respondents to rate 17 questions on a rating scale from 1 to 5. The goal is for individuals who answer the survey to be ...
3
votes
0answers
228 views

How to create appropriate number of data points that would be accurate enough for creating regression equations?

I found a similar question in this forum. As a rule of thumb, since there are 4 independent variables in my case, I need 4*10=40 data points. However, my question differs slightly, since I want to ask ...
3
votes
0answers
91 views

Fuzzy Record Linkage of Spatial Datasets

I have two datasets describing real-estate properties Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
3
votes
0answers
2k views

What is the best way to enter data to analyze answers from a Likert scale?

I need to analyze data regarding an item based on a Likert scale. Field data was arranged in Excel using tables that show the frequency of answers among respondents. What is the best approach to ...
3
votes
0answers
158 views

Choosing the best model to predict second hand widget prices

I am looking for some thoughts on how best to approach a dataset and pick the right model for predicting the second hand price of widgets. Some background on the properties of these widgets: The ...
3
votes
1answer
149 views

Handling NLP probability results

I'm a programmer without a statistics background. I've been working with NLP lately to classify documents, and I'm pretty up to speed with NLP concepts. I've gotten to the point where the NLP ...
2
votes
0answers
12 views

Target class shape & type in train_test_split

In train_test_split function of sklearn, what should be the shape & type of the target class? Does it make a difference? ...
2
votes
0answers
30 views

Is there a non-arbitrary criterion to choose which points to use for interpolation?

Look at the image. I have to interpolate this experimental points with a voigt function to find the position of the center of the peak, I have to choose how many points and which points to use in the ...
2
votes
0answers
18 views

How to use weights from our data set properly? Should we use them at all?

The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when ...
2
votes
0answers
13 views

what are some classic data sets, and they stories behind them that made them famous?

I'm familiar with some famous datasets and the stories behind them, but I'd love to know more. for instance: The Iris dataset - a botanician said that it is possible to differentiate between species ...
2
votes
0answers
27 views

How can I use spatial heterogeneity within regions as predictors about those regions?

Please forgive if I am misusing the term "spatial heterogeneity". I am trying to predict county level crop productivity using satellite acquired vegetative health (NDVI). Since this is pixel data, I ...
2
votes
0answers
46 views

Sum of exponentials and variance of the exponents

Let's assume we have two sets of non-negative real-valued numbers $\{a_i\}_{i=1}^{N}$ and $\{b_i\}_{i=1}^{N}$ ($N$ can be large) such that $\frac{1}{N}\sum_{i=1}^{N}a_i=\frac{1}{N}\sum_{i=1}^{N}b_i=...
2
votes
0answers
47 views

Data obfuscation strategies that keep statistical properties intact

I am getting data from a provider that contains identifiable information from a number of different organizations. The data aren't particularly sensitive, but to be safe, the provider wants to perform ...
2
votes
0answers
32 views

Ordinal or Interval Level of Measurement?

If we have a score of number of students in a kindergarden, which goes "one, two, three, four, five, six and more", is this variable interval or ordinal? The differences between them are exact, yet ...
2
votes
0answers
196 views

Similarity metric for sets of sets of nominal data

I felt it important to include the full motivation for my question, since part of my trouble I think is simply not knowing the terms to search for. Similarity of set of sets and Similarity metric ...
2
votes
0answers
78 views

Comparing three datasets where one is a combination of the other two

I have two experimental datasets $D_1$, $D_2$. An algorithm $A$ is applied to both resulting in errors $E_1$, $E_2$. From $D_1$, $D_2$I can create another dataset by randomly sampling half of the data ...
2
votes
0answers
1k views

how to handle the sparse data in machine learning for novelty detection using SVM in r?

I have a data frame wherein 89000 records 60% of the values are missing I am replacing the missing values by ...
2
votes
0answers
15 views

Identify clusters of components that vary together

I have a number $M$ of populations. From each population, I sample $R$ vectors, where each vector has $N$ real components. The entire dataset consists of $M\times R\times N$ real numbers. In this ...
2
votes
0answers
40 views

What is a statistical test I can use to show that, between Model A and Model B, Model A can be better distinguished from some Dataset X?

I have a dataset (Dataset X) with error bars. I have two different models (Model A and Model B) which each try to fit the dataset. On visual inspection, it is very obvious that Model A fits Dataset X ...
2
votes
0answers
152 views

Data Set Representation of Survival Data that Includes Time-Varying Covariates

I want to perform a survival analysis which includes time-varying covariates, using the aalen() function from an R package called ...
2
votes
0answers
425 views

problems with HSD.test

I would like to do a HSD.test. I have found an example of this and this example is working fine. This is the example: ...
2
votes
0answers
42 views

Does using decision trees to predict risky alcohol consumption make sense?

Context: I'm currently writing my bachelor thesis and have decided to focus on what factors that contribute to students that have risky alcohol habits at my university. I am planing on doing a big ...
2
votes
0answers
148 views

Input geolocation Data in Neural Network

I am working on a Neural Network that should predict certain transport-tariffs based on start and end region of the transport. The region can take different forms, from a whole country to a range of ...
2
votes
0answers
118 views

Regression analysis for a big set of data

Well, it's my first post but I have been struggling hard with this problem so I had to look after help. The problem: I have a high set of data like this one (black dot are the data): I have to find ...
2
votes
0answers
36 views

Using Data on Different Spatial Scales for Econometrics?

Background: Using a Production Function in form of Autoregressive Regression Model for Regional Perfomance Hi, This question is more of a theoretical nature. I have dataset that comprises all data ...
2
votes
0answers
80 views

Is more data better for ML even if it causes imbalance between classes?

The question stems from attempting to predict if a loan in a peer-to-peer lending portal will be repaid or not. I have a historical data set of about 32000 loans, but not all of these loans are ...
2
votes
0answers
520 views

Missing data MAR not MAR - multiple imputation

I am preparing my data for CFA and multiple regression. I want to see how different types of parental involvement influence children's motivation and academic results. I have a sample size of 5000 ...
2
votes
0answers
66 views

Measuring Difference Between Two Sets of Likert Values

I asked users to complete a likert survey (1-5) at the start of an activity and at the conclusion of the activity. What is the best manner to show the rate of change/difference between these two ...

1
2 3 4 5
11