Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

0
votes
1answer
20 views

Sort datasets by how much it's increasing

I'm not familiar with statistic so please bear with me :) I'm a programmer and have a task to sort datasets by how much they have increased in values. So for example, we have 3 datasets : ...
0
votes
0answers
9 views

Shorter audio files in the input of a network

I am working with a given network. The dataset contains 60-sec audio files which, after pre-processing, results in a vector $M\times T$ map as an input to the network. I tried cutting this into 20-...
0
votes
0answers
8 views

unbalanced panel data: is it possible to use?

I am trying to analyze unbalanced panel data. Honestly, it is first time to do longitudinal study, so I have many difficulties. I really need your help. I consider to use GEE or RE (in STATA) ...
1
vote
0answers
93 views

Choosing the Right Data Mining Classification Technique

From the different Data Mining tasks, I want to train the Classification. For that, I: Took this dataset (can be used as an example for the answer). Got to know the data (data objects, attribute ...
0
votes
0answers
29 views

Why should we demean data?

In my case, I'm performing a vibration analysis using FFT and all that, and in this algorithm that I've been using, the author demeans the amplitude data before applying the FFT to it. Is it a thing ...
0
votes
0answers
12 views

Formatting Panel Data [closed]

I wish to format a panel dataset that I will use to run a regression down the line... I have both individual (people) level data from survey data, as well as country level economic variables such as ...
0
votes
0answers
11 views

How precisely do we ensemble output as in this Omni-Supervised Data Distillation paper?

After reading the paper Data Distillation: Towards Omni-Supervised Learning, I saw that after getting the results, I ensemble them before using them in order to train the student model with respect to ...
0
votes
0answers
14 views

Are there any data formats for storing text worth looking into, besides CSV? [closed]

I have noticed Pandas has several storage options, pickle, feather, parquet, sql, hdf5, etc. Are any of these worth looking into for simple text data? If it makes a difference, I am mostly looking ...
0
votes
0answers
29 views

Data augmentation techniques for numeric datasets?

I'm writing a paper about Data Augmentation and I'm looking for some way of increasing the size of a dataset. I'm already aware of the techniques used for images (transformations, PCA, blurs, etc.) ...
0
votes
0answers
15 views

Training a deep neural network using a changing (increasing) dataset

I am trying to train a deep autoencoder (applies also for other architectures) in the following way: Step 1) I start with a fix dataset of e.g. 10k samples. Step 2) A training "loop" consists of ...
0
votes
0answers
20 views

what is data mapping? [on hold]

I need some help with my internship project. The company in which I'm doing my internship is changing the tool that manages all the data. As a result of that, they want me to do a data mappin. After ...
1
vote
2answers
31 views

Why Permuted MNIST is good for evaluating continual learning model?

while I was reading papers about continual learning, I found that many researchers use permutated MNIST to evaluate their approach. I understand what it is but it ...
1
vote
0answers
24 views

Polynomial regression in a circular data set

I'm working on a data set where I have a set of events per time of the day (in a 1h time slot). I made the sum of the events and plotted a bar as show below. In order to define an equation that could ...
2
votes
1answer
41 views

Standard Deviation in qualitative sample

Can there be a meaningful standard deviation for qualitative data?
0
votes
1answer
100 views

Machine learning when there are two “answers”

I have a problem that I am trying to use machine learning on. At a very high level, I am looking to do a transformation where in the training data I have x and it ...
0
votes
1answer
12 views

ATGC sequence of gene expression data

I am not a pro in genetics so please excuse my non-technical language. I need dataset which contains the gene expression as well as the associated ATGC sequence with each gene expression value. For ...
0
votes
1answer
30 views

What is the best way to reperesent this data (R)?

I am trying to show the correlations between life expectancy age and state pension age for men and women, per country. I have the following data here (working example): ...
1
vote
3answers
45 views

Commonly used example data sets

Is there a place where one can get standard example data for various statistics tools to try on? For example, if one is learning about ARIMA models, where would one get data that can be modelled well ...
1
vote
0answers
24 views

Cross validation decreases precision?

I've been working on a simple logistic regression model and I'm trying to improve its precision by cross validation. This is the code I've done so far (without including the imports): ...
0
votes
1answer
18 views

Randomize dataset for Restricted Boltmann Machines

Suppose I want to train a RBM (or even a DBN architecture) and then fine-tune the parameter training a Feedforward NN. In my case the dataset is composed of time series, so in principle there is a ...
0
votes
0answers
17 views

Ratios and Probability

So I had this question in one of my classes and I wrote a super simple version of it and what I did. The only reason I'm asking is because either this question was super easy and I did it right, or I ...
0
votes
0answers
18 views

Should I use a panel/longitudinal data model or a time series model?

Suppose I want to assess the impact of school attendance rates on test scores in a particular Canadian province, Alberta. I have data on both variables for Alberta, but also for the other Canadian ...
0
votes
0answers
13 views

Modeling both linear and non-linear relationship

Apology for being verbose and all the typos or mistakes. This problem has been bothering me for a while and I really hope you can help me with it. Let's say I want to model quarterly sales for a ...
0
votes
0answers
25 views

How to treat missing data involving multiple entries

I'm developing a model used to predict the probability of a client changing telephone companies based on their daily usage. My dataset has information from two weeks (14 days). My datasets include in ...
0
votes
1answer
38 views

What are general practises used to divide the data into training / dev and test set?

Example: I have am building a dog vs cat classifier and I have collected data from 15 countries. Europe: 1. UK 2. France 3. Germany 4. Italy 5. Finland Asia: 1. India 2. China 3. Japan 4. Russia 5. ...
2
votes
0answers
15 views

Split dataset before or after preprocessing in training, validation, test

Suppose you have a super large table $T$ with data, which is pretty messy, and it's columns aren't useful features. So you clean it, remove/impute missing values, define new columns, which are more ...
1
vote
1answer
30 views

Different common meanings of training set, validation set, test set [closed]

In this previous question of mine it was pointed out in an insightful comment by ReneBT that the usage of the 3 terms training data set, validation set and test set is not uniform across the cross ...
0
votes
0answers
5 views

Generate scenarios of multiple related parameters

Assume I have three industry datasets: interest rates, inflation and unemployment. Data contains information of last ten years and it's monthly. Now, I would like to create N possible scenarios of ...
0
votes
0answers
11 views

Questions related to an upcoming databse project

Starting this summer, I will work on a project where I am supposed to create a database in Excel. Without going into much detail, it would be nice to know if anyone has done something similiar to see ...
0
votes
0answers
7 views

Common statistical Information about the multiple datasets

What is the common statistical information that can be extracted from the collection of multiple datasets that can help us to choose the appropriate machine learning models?
0
votes
0answers
14 views

When is categorical data ordinal?

I've obtained some datasets on which I should perform the analysis. Since I have not made the questionnaires I have some doubts about the questions with categorical outcomes. I'm unsure whether I ...
0
votes
0answers
12 views

Can someone explain the answer this question about Apriori?

Consider the following set of frequent 3-itemsets: {1,2,3},{1,2,4},{1,2,5},{1,3,4},{1,3,5},{2,3,4},{2,3,5},{3,4,5}. Assume that there are only five items in the data set. List all candidate 4-...
4
votes
0answers
21 views

How to combine noisy and noise-free datasets to train a model

Overview Suppose I have two datasets, both of which consist of rows of features and their matching labels. One of these datasets is noise-free and its labels correspond to the ground truth, but the ...
0
votes
0answers
28 views

Methods to classify observed data into states

Given a time series (that sometimes unpredictably alternates between random and non-random and is not stationary). What mathematical methods are there to classify the observed set of data into ...
0
votes
1answer
18 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
1answer
36 views

Given the following table of class probabilities, find the TPR and FPR at thresholds 0.78 and 0.53

Below is a table of tuples that contain class probabilities for a ROC curve. How can I compute the TPR (True Positive Rate) and FPR (False Positive Rate) at the thresholds 0.78 and 0.53? Note: the ...
0
votes
0answers
25 views

Survival analysis: Individuals with event time = 0, exclude or not?

Data set: 50000 participants Assessments of various risk factors at baseline Dates when participants were included in the study Dates when participants died or were censored (after 2 years) ...
0
votes
1answer
41 views

Difference between Cross validation,GridSearchCV and does cross validation refer the train test split?

what is difference between Cross Validation and GridSearchCV and i know how train test split works but i heard many times cross validation, does it refer the same (train test split)? Can someone ...
0
votes
0answers
6 views

How to find windows of non-standard data in graph with a certain minimum width?

I'm doing a project involving EEG brain data; I have a relatively noisy graph representing a user's calculated concentration level over time, and would like to find time intervals at which their ...
0
votes
1answer
21 views

What does it mean by the generation of the dataset by probability distribution?

Let us consider the following dataset $ D = \{d_1, d_2, \cdots, d_k\} $ where $ d_i \in \mathbb{R}^n$ for all $1 \leqslant i \leqslant n$ and $n, k, i \in \mathbb{N}$. A probability distribution $...
0
votes
0answers
8 views

Evaluating a rule based detection tool for test smells

Measurement of efficeny in smell detection tool Hi guys! I know that this is not strictly speaking statistics but I find the evaluation part related. Maybe this belongs at Stack? It's more about TP,...
0
votes
0answers
29 views

how to statisticaly test data collection methods

I have 2 dataset collected with different data collection methods (very large datasets). I use the same classification algorithm on both dataset and get a greatly improved result for one compare to ...
1
vote
1answer
33 views

can you help me to do a difference of CDF?

I have 2 CDF's with equal number of points that I want to compare. These are from: Temperature of 1 month from 2012 Mean temperature across months What can I do to obtain this difference, this is ...
1
vote
1answer
46 views

Feature engineering with cross validation, then testing on a holdout data set?

We have 3000 samples for two classes, roughly 2000:1000. Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a ...
1
vote
1answer
28 views

Should I use 5KFold or 10 KFold CV for data about 300000 sample?

I use logistics regression for binary classification. and my data is about 300 000 samples. I have used both kfold = 5 and kfold =10 cross-validation ...
0
votes
1answer
17 views

Conditional expectation in Basic Linear Unobserved Effects Panel Data Model

I want to see that $ E[u_t|X_t,c] \Rightarrow E[X_t' u_t]$ in a Panel Data Models
0
votes
0answers
210 views

Research question: Which stain is most effective for improving tissue contrast in CT scans of the larynx?

I'm doing research into which stain out of I2KI, PMA and PTA is most effective for improving tissue contrast in CT scans of the larynx. I have 9 specimens in I2KI (CT scanned after 0, and then either ...
0
votes
0answers
11 views

Confidence vs. Count in association rule mining: which one is better?

I am writing a program that mines association rules from a large data set. I have an array of association rules, and I have to decide which ones are more representative of the patterns I am studying. ...
0
votes
0answers
14 views

how to check the distribution of the training set and testing set are similar

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
0
votes
0answers
12 views

Converting Annual Data into Biannual Frequency

I need to convert annual data into biannual, since most of my data have biannual frequency. However, some macroeconomic variables - like business environment quality or Herfindahl-Hirschman Index are ...