Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Looking for a data source (for Ancel Keys 7 country study) [closed]

As some of you now, there is controversy associated with Ancel Keys and the 7 country study (which was one of the first major studies suggesting a link between heart disease and diet). I am a ...
2
votes
0answers
16 views

How to use weights from our data set properly? Should we use them at all?

The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when ...
0
votes
0answers
7 views

Should I include top level categories in a training set? [closed]

If I'm creating a training set for multilabel category classification, using a category tree, should I include the top level category when labeling? Maybe easier with an example: This: ...
1
vote
0answers
13 views

What are the limitations of non-negative matrix factorisation when reducing the dimensions of a data set?

From what I understand NFM (non-negative matrix factorisation) is constrained by the factor that it only supports data sets with non-negative values when reducing the dimensions of a data set. ...
0
votes
0answers
13 views

What is the use of x_train.astype('float32') in CNN? [closed]

I an working with the mnist dataset through some tutorial using CNN.After loading the data, x_train is converted into float32.I cant see the significance of it as if i divide it as x_train/255, the ...
0
votes
1answer
28 views

Can I apply PCA on continuous data and reduce the dimensions and keep categorical data as it is?

I have a dataset which contains 95 highly correlated continuous variables and other 3 categorical variables. I want to reduce the dimension of the data and by that I can deal with correlation as well. ...
1
vote
1answer
37 views

Dataset with 10% missing without either using a multiple imputation technique or removing the samples. How would I address the missingness?

I have a dataset comprising of 10% missing fraction where missingness can be predicted from the data which I believe is Missing Not At Random. How would I address the missingness of a dataset without ...
0
votes
0answers
15 views

Combining datasets when working with PCA

I'm largely a statistics novice, and I've gotten into a discussion with my lab mate about the application of PCA. Previously I've applied PCA only to a uniform dataset (results from cells all ...
1
vote
0answers
18 views

Data scale for regression analysis

I am working on my PhD thesis.i have dependent variable as daily stock return for many firms and independent variable are inflation, exchange rate etc. My problem is 1-my dependent variable stock ...
3
votes
2answers
37 views

Dataset having about 1% missing fraction, missing completely at random. How would I address the missingness of the dataset?

I have a dataset comprising of 1% missing fraction where data is missing completely at random. How would I address the missingness of the dataset with a multiple imputation technique? For example, is ...
1
vote
0answers
10 views

Appropriate way to report cox proportional hazards model

I have a dataset of 150 patients and 10 variables. I wanted to run a survival analysis, so I did the following: Univariate test (log-rank for my categorical variables, univariate Cox proportional ...
0
votes
0answers
14 views

Is GAN effective enough to replace data augmentation and manual annotation?

We all know that GAN can be used to augment and expand our dataset Can a GAN be used for data augmentation?. But my question is, is it effective and fast enough? For example I have done experiment ...
1
vote
0answers
15 views

How to simulate 1000 datasets, with each data set consisting of 100 random draws from the normal distribution? [closed]

I'm really new to R, and can't seem to figure out how to create a dataset consisting of 100 random draws. I can create 1 data set of 100 random draws using rnorm = (100,mean,sd) but I don't know how ...
1
vote
1answer
29 views

Need some eyes on my dataset: can we argue here that there is a random spread around a constant?

I have this huge data set where I am looking to find trends. However, this one quantity (Figure) seems to randomly spread around an almost constant line (with the exception between 1600-2300). I did ...
1
vote
1answer
27 views

5-point likert scale into low, medium, high categories?

I am designing a study where the objective is to collect the % for low, medium and high to populate probability tables. For example, my objective is to know what is the probability in % to have a {low,...
0
votes
0answers
18 views

Is there a significant relationship between two data sets?

I am interested in image processing in python. I worked on detecting and tracking particles in an image. I obtained velocity data of particles in different regions. I believe that the difference ...
1
vote
0answers
13 views

Why do people produce large data sets with >90% of features redundant, when analysed computationally?

Why do people produce large data sets with >90% of features redundant, when analysed computationally? I've seen more than one example of this kind of data set. The gathering of the data is perhaps ...
0
votes
0answers
8 views

How much is the Class Imbalance Problem rates?

I'm working on a data set and wanted to know is there a standard rate about Class Imbalance problem or not? I have 47 samples in Class A and 150 Sample in class B , should I use Class Imbalance ...
0
votes
0answers
13 views

Avoiding bias when labelling to construct a test/train data set

I have a multiclass classification problem, where I have a massive database of text documents and the objective is to predict what class a document belongs to. There are many classes (>10) but the ...
1
vote
0answers
8 views

Aggregation of daily futures prices to quarterly prices

We are working to incorporate the EIA future prices into our quarterly data set of financial balance sheet data. There are four different contracts, which are available at the EIA website. The idea is ...
1
vote
1answer
31 views

What is the difference between Intrinsic dimension and Embedding dimension of the data

I have read in this paper that there are Intrinsic dimension and the Embedding dimension of the data. I am not familiar with these two concepts. The only I know is that the number of variables in a ...
0
votes
1answer
21 views

What Chi Square test should I use?

I am attempting to assess the ecological validity of an animal testing procedure as follows: I have notated x number of events that occur in the wild. During each event, the speed of the animal is ...
0
votes
3answers
39 views

How to increase data samples to prevent over fitting?

As you know increase a data samples on training is a way to preventing of over fitting. I'm working on an UCI data set with 198 samples and 34 feature this is my data set's dimension and I wanted to ...
1
vote
1answer
25 views

What statistical test should be performed in this setting?

I am analyzing a dataset. There I have 3 different "Lab test report findings" and 1 "clinical findings" which is obtained by the clinical examination of the patient (most of the time this clinical ...
2
votes
0answers
13 views

what are some classic data sets, and they stories behind them that made them famous?

I'm familiar with some famous datasets and the stories behind them, but I'd love to know more. for instance: The Iris dataset - a botanician said that it is possible to differentiate between species ...
0
votes
0answers
4 views

Constructing dataset split considering biased subgroups

I'm working on a classificator that is supposed to classify images taken from within a driving vehicle with the camera pointing forwards. I have multiple datasets, consisting of pictures taken at a ...
1
vote
1answer
63 views

How can I create meaningful / interesing data following multiple linear models?

I would like to create many(!) examples for exercices, where I would like to control various parameters such as the coefficients, their uncertainty, t-values and p-values of the linear model. I know, ...
0
votes
0answers
6 views

Interpretation of PCSE results

Hi I would like some help in the interpretation of results with regards to a regression analysis being conducted: One with a simple linear model: enter image description here The other with PCSE ...
1
vote
0answers
13 views

How to determine statistical significance of physics dataset?

I am working on a physics paper, which looks at the dependence of resistivity on physical temperature. I obtained data showing a linear trend of the form $$\rho(T)=\rho_0[1+\alpha(T-T_0)]$$ Based on ...
0
votes
2answers
18 views

Clustering temporal data

I have a large collection of bus arrival times (a specific bus number at a specific stop). I am trying to determine the best way to cluster the data so I can see what time the bus usually shows up at -...
1
vote
1answer
50 views

How to choose an appropriate method to fit this data?

I want to fit a model curve to the plot below. Each fine curve corresponds to a vehicle driving a certain distance and measureing some quantities of interest in certain timesteps (discrete ...
1
vote
0answers
9 views

Impact of empty data in ML models

Working on a project for my computational intelligence class at college doing machine learning models for crime hotspot predictions. Essentially what I am doing is laying a grid over the specific city ...
1
vote
1answer
19 views

Using lagged independent variable in place of its current value

I am estimating the impact of migration on a binary outcome (employment) using Diff-and-Diff with multiple periods in an unbalanced panel data (with non-migrants as a control group). To this end, one ...
0
votes
1answer
16 views

How can we include hourly traffic series data in the rows of train data set for training?

I have a classification problem where I am planning to use hourly traffic data for a day. Is there any way to compress it? instead of creating 24 predictors which account for hourly traffic?
0
votes
0answers
5 views

What kind of test to run on regeneration data over several days?

So I have data of worms that are regenerating under different conditions. I measure their length every 3 days as well as a control. These measurements are then converted into a percentage of growth ...
0
votes
0answers
7 views

Is there a way to use HDBSCAN to fit on batches?

I would prefer to use a DBSCAN like algorithm that supports batching data. I can use minibatch k-means but that would require a lot of additional work for optimal performance that HDBSCAN takes care ...
1
vote
1answer
28 views

What is the name for n dimensional data sampled at regular intervals in n-1 dimensions?

I'm looking for a word to describe n-dimensional data sampled at regular intervals in n-1 dimensions. For example, a 3d dataset would have data sampled at regular intervals on a 2d grid. A 4d dataset ...
1
vote
1answer
30 views

Normalizing input to DNN with structured data

I have a structured dataset of around 100 gigs, and I am using DNN for classification. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in ...
0
votes
0answers
10 views

How to choose a threshold for error measurment in a given dataset

I am working with a given dataset of measurements, each one with an associated error. I want to establish a threshold for this error, to maximize the data quality. Is there an statistical approach on ...
0
votes
0answers
19 views

Principles for choosing standard deviation of gaussian convolution filter

I have collected data from a chemical analysis, and I need to find the maximum of its first derivative; however it is too noisy to find this by simply taking the maximum. The noise can be smoothed out ...
0
votes
1answer
25 views

Within-subject model training based on multiple subjects

I have the following situation: - ~300 participants, for each of them I have ~30 participant-specific data (from questionnaire) - For each participant I have ~200 points of data consisted of 1 ...
1
vote
1answer
47 views

Splitting dataset with respect to categorical variable

Assume that we have a data set with some features and a goal is to perform a classification. Let's assume that a dataset is moderately large compare to the total number of features. Next, assume that ...
3
votes
1answer
24 views

What statistical test should I use? Visitors distribution in fitness center

I conducted a field study where I gathered data about the distribution of visitors over the day in a fitness center. The goal of the study was to implement a treatment that would shift visitors from ...
0
votes
0answers
7 views

Is there a dynamic dataset definition? how do I deal with dynamic data?

I wish to convolve audio file (from urbansounds8k dataset) with an RIR function to represent the room characteristics before a neural network. My plan was to this on the fly, that is, randomize the ...
0
votes
0answers
3 views

Calculating DALY for global population

I have been tasked with calculating the DALY. From what I understand, the first part for YLL requires N*L where N is the number of deaths from the decease, and L is the life expectancy at the age of ...
0
votes
1answer
59 views

Where can I find data for multiple linear regression? [closed]

I'm doing a paper about multiple linear regression where I need to have 1 dependent variable and 3 independent variables. Anyone have some good data that I can use? I already tried Google Database, ...
0
votes
0answers
11 views

Should I omit certain data points in the set

I am writing a script in R to estimate Pi using the Buffon Needle Experiment. If you are unfamiliar with the experiment it is where you drop a needle of length l onto a floor of wooden slats where ...
0
votes
1answer
22 views

How can I determine the original dataset of an observation after rbind? [closed]

I have 113 different datasets and use the R-command "rbind" to put them into one single dataset. Now my question: How can I determine from which original dataset a specific observation stems? Is there ...
0
votes
1answer
57 views

likelihood functions

Assuming that a linear regression model is fitted to 5 distinct datasets and that in all cases the regression line is y = ax+b+e . Will The likelihood functions are the same for the two different ...
0
votes
0answers
10 views

Require suggestions on data handling

I have below tables in my database with record count as shown below I would like to join all these tables and make one final/main table. I tried the below approaches but nothing helped 1) Currently ...