Skip to main content

Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

Filter by
Sorted by
Tagged with
2 votes
0 answers
22 views

Using whole training set for choosing model

I am working on a classification problem with what I understand as a big dataset. I have first of all splitted it in my "train" dataset and the "test" one. (Actually I am convinced ...
Videgain's user avatar
0 votes
0 answers
9 views

Research Paper for Introductory Data Analytics [closed]

I am taking an introductory data analytics course that requires us to write a small research paper using statistical methods. We are meant to find past research that uses fairly simple techniques ...
Trevor Medeiros's user avatar
0 votes
0 answers
8 views

Advice on ideal data structure for storing Long Jsons within SQLite/RDBMS [closed]

I am developing a project that displays a live rendering of bus, when the user clicks on a bus, i would like a linestring representing the bus route to appear/highlight. I am currently attempting to ...
Bispar426's user avatar
2 votes
1 answer
26 views

Object detection for finite dataset

Consider the following scenario If I want to train a model to detect and count these squares: These squares will never be different. They will always look exactly the same, and be of exactly the ...
Fresh Prince Of Nigeria's user avatar
2 votes
2 answers
35 views

Imputation Missing data

I have a longitudinal data set with 2 dependent variables (couple) - a husband and a wife. There were 2 waves for the husbands and 3 waves for the wives. Since there is a lot of missing data, I ...
eagersquirrel's user avatar
1 vote
1 answer
80 views

How can I augment a 1D tablar dataset using an additional 2D dataset?

I have the following two types of datasets: The dataset-1 is a tabular data that describe 7000 proteins. dataset-1 is only one file. dataset-2 consists of 7000 individual data files that are 2D ...
user366312's user avatar
  • 2,201
2 votes
1 answer
23 views

Analyzing lists and variables of multiple answers

My current issue lies within EMR extracted data for medications. There are multiple variables named: Medication_1, Medication_2, Medication_3, etc... This data may overlap and analyzing each column ...
Abdallah Al-Ani's user avatar
2 votes
0 answers
14 views

What is the best metric to use to discard annotators with low IAA (inter-annotator agreement) with all others?

This question is specific to ordinal data collected on the likert scale What is the best metric to discard annotators with low inter-annotator agreement (IAA) with others? from e.g., Cohen’s Kappa, ...
user2160809's user avatar
0 votes
0 answers
22 views

How can i create a linear regression model not having the exact same dataset

i want to create a data regression between two financial indexes, but they don't have a perfect correspondence in the data of observation (for example one has the relevations for 17/6 18/6 19/6 but ...
ConfusedConsultant's user avatar
1 vote
1 answer
42 views

Poisson regression given multiple predictors on a repeating ID variable

I was wondering how a poisson regression would work given my dataset which describes a series of zip codes stratified by age groups, gender and death counts. The regression would use death counts as ...
Seyong Chang's user avatar
0 votes
0 answers
31 views

Is there any technique that I can use to achieve the required computation without reading all 10 million rows?

I have a data file from a Monte Carlo simulation of fifteen protein chains. The file contains 10 million r_end_to_end 3D vectors as rows and 3 x 15 = 45 columns. My ...
user366312's user avatar
  • 2,201
0 votes
0 answers
48 views

What is dependency on sequential data?

We know that Whenever the points in the dataset are dependent on the other points in the dataset the data is said to be Sequential data. A common example of this is a Timeseries such as a weather data....
D. S.'s user avatar
  • 69
1 vote
0 answers
54 views

Sales data trend

I have my historical sales data and I want to check for the trend (increase, decrease or no change). When I do my annual line graph, the slop of my linear equation is positive (indicating increase) ...
monique's user avatar
  • 31
1 vote
0 answers
12 views

Problems using custom dataset using Minirocket classification [closed]

I'm working on a bigger school project, trying to classify timeseries measurements with Minirocket/Rocket. My trainingdata consists of a 1D matrix containing the measurements, and a seperate 1D matrix ...
Michael's user avatar
  • 11
1 vote
0 answers
43 views

How to deal with outliers in panel data? [closed]

When we have cross-sectional data, we can easily detect and remove outliers. But how should one approach outliers when we are dealing with panel data? Since we have $i$ entities and $t$ times periods, ...
TFT's user avatar
  • 345
2 votes
0 answers
27 views

Testing time series data stationarity

I am working with time series and want test different forecasting methods but first I need to test if my time series (sales) data is stationary or not. So I have been learning about KPSS and Dickey-...
monique's user avatar
  • 31
0 votes
0 answers
10 views

Imbalanced dataset with multiple classes [duplicate]

I have an imbalanced dataset with multiple classes where some have less than 100 some are more than 10k,where i want to apply random forest(the dataset is confidential so i cant share),i used all ...
Deepak kumar's user avatar
2 votes
1 answer
23 views

How to train Logistic regression model with multiple inputs for 1 target value?

My data looks like similar to this: (the picture below is not mine, but describes perfectly my situation) where the IDs are not unique but for each ID value I have a unique target value The following ...
Moez Daly's user avatar
1 vote
0 answers
50 views

Improving prediction accuracy for Gaussian process with derivative information

I am fitting two different GPs with derivative observations (one with 9 dimensional input and one 12 dimensional input), however for some reason I am getting much worse results for the 12 dimensional ...
m-julian's user avatar
1 vote
0 answers
34 views

Best method to identify layered clusters

The Problem Hello everyone. I'm working with a dataset that has 15300 samples with 49 features each, equally distributed amongst three classes. I used TSNE to reduce the dimensions of the feature ...
Amyr14's user avatar
  • 11
4 votes
1 answer
41 views

Generating synthetic data with multiple records per ID

I would like to generate a synthetic dataset where there are multiple records per ID, and self-consistency is maintained among records of each ID. For example, imagine a dataset where the ID is a ...
user12138762's user avatar
0 votes
0 answers
19 views

"How can I address the lack of correlation and a low R-squared value in my univariate linear regression when the data is scattered?"

** "I'm trying to find a correlation between the confirmed cases and deaths rates against HUMIDEX values. As you can see, the data is very scattered, so I understand that polynomial and ...
Carlos Leonel Guerrero Rodrigu's user avatar
0 votes
1 answer
44 views

Data analysis for school project

I have to create a hypothetical study focusing on the relationship between sBCMA (soluble B-cell maturation antigen in blood) and the expression of BCMA on bone marrow cells in patients with multiple ...
youknow 321's user avatar
2 votes
0 answers
48 views

Problem with mathematical formulas in gap statistic

I'm studying the article "Estimating the number of clusters in a data set via the gap statistic" by R. Tibshirani, G. Walther and T.Hastie: https://academic.oup.com/jrsssb/article/63/2/411/...
user2702's user avatar
0 votes
0 answers
13 views

Given variable A and B containing data of lemma sentiments, what is the correct term for the variable containing average of var A and var B?

I have a data visualization, showing the sentiment of two lemmas "гей" (var a) and "трансгендер" (var b) in a news corpus throughout the year. Here is the dataframe sample of my ...
pindakazen's user avatar
1 vote
0 answers
13 views

Consistent way of doing paired-trial validation (and leave-one-dataset-out validation)

In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. ...
Roger V.'s user avatar
  • 4,429
4 votes
2 answers
74 views

Autocorrelation of discontinuous time series data [closed]

I am attempting to perform an autocorrelation study using python on a discontinuous time series dataset. To share a bit about how my data looks like, it is a single column of values, which spans over ...
Sam's user avatar
  • 83
0 votes
1 answer
8 views

Should the testing data be uniquely distinct and come from different source/dataset than the training data?

I am building an audio classification system using CNN. My dataset consists of different audio I have recorded and spliced to equal time lengths. Like with any other common ML or DL tasks, I am to ...
Flash's user avatar
  • 1
0 votes
0 answers
20 views

How to calculate reliability of difference scores?

I am trying to calculate the reliability of a difference score. Specifically, the data have, for each participant, scores for 10 items in Condition X (1s and 0s), as well as 10 different items in ...
Altair555's user avatar
0 votes
0 answers
24 views

How do I calculate the weighted average of a 2D data set for a 3D structure?

I have a 2D data array indicating a chemical percentage content (PC) in a chemical droplet. I am trying to calculate the average PC in the droplet. The image of one of these arrays is shown below (the ...
user7077252's user avatar
0 votes
0 answers
12 views

How can I represent highly fluctuating dataset of multiple signals without lowering actual signal?

I am looking for some good suggestions. I have highly fluctuating data from multiple samples (three samples are shown in light blue). When I calculate the mean (black line) of multiple samples, it ...
Debajyoti Kabiraj's user avatar
0 votes
0 answers
12 views

Statistical Analysis Query: Assessing Odour Intensity Ratings with Concentration Levels in Odour A and B Groups

I'm currently working on analyzing data from an odour rating task, and I have some questions regarding the statistics approach. In the experiment, participants were exposed to 6 different odour ...
Steve's user avatar
  • 1
3 votes
1 answer
183 views

Can I change values in data from yes and no to binary

I have a dataset that I want to perform a regression on. However, some of the columns are not in numerical form. For example, the extra classes column. What I ...
Charlotte's user avatar
1 vote
1 answer
71 views

What's the justification for comparing two separate models built on subsets of data versus using one model that uses the whole dataset?

I've noticed that there are some data analysis being done in some scientific field where the authors would split out an entire dataset into subsets based on a particular property. One classic example ...
Syuma's user avatar
  • 125
0 votes
1 answer
26 views

Difference-in-difference age groups

I would appreciate your help with a question I have. I'm creating a Difference-in-Difference study to examine how a conditional cash transfer to individuals 18 years of age to be spent in sport ...
Retir's user avatar
  • 1
1 vote
0 answers
80 views

Datasets with multiple maximum likelihood estimators

There is a sizeable body of literature on the issue of multiple maximizers in maximum likelihood estimation, such as https://projecteuclid.org/journals/statistical-science/volume-15/issue-4/...
Tom Solberg's user avatar
0 votes
0 answers
21 views

Method for pairwise ordering two datasets

Given two rather small but unordered multidimensional vectors/datasets (e.g sets of a handful of 3D coordinates), what is a simple method for pairwise alignment/ordering? I've though about using ...
joaocandre's user avatar
0 votes
0 answers
10 views

Comparing two datasets of communication flows

I know the question of how to compare two datasets has been asked multiple times and I have read a lot about the different tests available but I'm a little lost on what exactly the tests show and what ...
Milan D's user avatar
  • 101
3 votes
0 answers
183 views

Recreating data variance from the posterior distribution

Recreating data variance from the posterior distribution Take a set of data points $(x, y)$ with (Gaussian) uncertainties $\sigma_y$ on the $y$ coordinate; they are modeled as $y \sim f(x; \alpha) + \...
Jacopo Tissino's user avatar
0 votes
0 answers
24 views

Reference datasets for conditional density estimation

[In case you feel inclined to close this question because I'm asking for a dataset - I'm looking for solutions in the spirit of point 2 (on-topic) in the accepted answer to this question about asking ...
Scriddie's user avatar
  • 2,439
0 votes
0 answers
20 views

Should I indicate "success" of an experimental run at the beginning of the data?

Because I'm that guy, I wanted to run some statistical analysis on the results of a number of experiments; specifically, I'm wanting to track my progress on different runs of the turn-based strategy ...
John Doe's user avatar
1 vote
0 answers
32 views

What is the difference between construct validity and reliability? [closed]

I want to design a questionnaire and examine a new construct (variable) in my research with a five point scale from 1 to 5. How can I test whether the questionnaire satisfies the requirements of ...
Dr. Subhash Chander's user avatar
1 vote
0 answers
14 views

Correlation of event occurrence in multiple sectors

I have the following problem to analyze: I divided an area into several sectors (i.e.: S1,S2,S3,…,Sn) and there is an event that can happen in one or more sectors at the same time. I considered a ...
Rodrigo's user avatar
  • 111
0 votes
1 answer
33 views

Make Predictions with an RNN Using a Multi-dimensional Training Set

I have a 2D matrix TD of training data that is a collection of N non-linear signals that are functions of time (hence the ...
Jonathan Frutschy's user avatar
1 vote
0 answers
57 views

Plotting 3 points per data set in lollipop-type chart [closed]

I am wanting to plot a graph where I have multiple data points per category of data. For some context, I have done some analysis on different samples and now have up to 3 3 data points for each sample ...
Charllotte's user avatar
2 votes
1 answer
30 views

Missing data for Cox regression and HR

I'm conducting a research in which patients went through a surgery, for some the surgery was successful (outcome = 1) and for some it wasn't (outcome = 0). The risk factors were calculated using a Cox ...
AREEEL's user avatar
  • 21
0 votes
0 answers
19 views

Deriving log-likelihood contribution involving fixed effects (panel model)

I would appreciate some help with the following problem: We have the following panel model: $y_{it} = h(x_{it}\beta + c_i) + \epsilon_{it}, \quad t=1,\ldots,T, \quad i=1,\ldots,N $ where \begin{align*}...
TheSnailSurgeon's user avatar
1 vote
1 answer
61 views

How can I best visualize & compare this data? Should I create weighted composite variables for a scatter plot?

I have data from a survey which was asked people 1) how often they used a particular tool (daily, weekly, monthly, annually, etc) and 2) many hours they usually spent using it (0 - 4 hrs, 5 - 9, 10 - ...
Arctic's user avatar
  • 71
2 votes
1 answer
50 views

Analyzing microbiome and clinical data for event-prediction

I am analyzing clinical data and complex microbiome data in a longitudinal study. I already compared different groups at baseline and between baseline and "events" using linear mixed models (...
BHO_1990's user avatar
6 votes
1 answer
374 views

Name of academic field studying geometric structure of data sets [closed]

I have questions about the geometric structure of data sets, esp. as it relates to the relationships between predictors. Is there a name for this field?
Chris Science's user avatar

1
2 3 4 5
39