Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

Filter by
Sorted by
Tagged with
0
votes
0answers
31 views

Is there a way to differentiate between linear and non-linear datasets in simulations studies?

I'm trying to find a a measure of "linearity" among different simulated data sets and how I could control for such differences. This idea came from studying Hastie's and Tibshirani's work around ...
3
votes
0answers
28 views

Check whether data forms a curve or not [on hold]

I have a set of lists and need to determine if the data of every list forms a curve. The picture below is an example of how the data looks plotted. Humanly it is easy to identify which ones are the ...
0
votes
0answers
9 views

What is a good example of a hybrid data-set which has over 5000 entries? [closed]

I'm looking for a dataset which has the following properties: Has more than 5000 entries Has both numerical and categorical data Is publicly available, free of cost I want to test an algorithm, and ...
0
votes
1answer
35 views

What is the method should I use to calculate similarity in a data set with outliers that must be included?

Information: So I have a data set with 18 vectors with 167 components, each of with has a value with a range of $[-2, 2]$. I am trying to calculate the similarity between one arbitrary vector in ...
0
votes
0answers
17 views

how to create features from datasets with date columns [closed]

I'm working on a project to score my leads and customers based on the meeting our sales guys do with them. These are the fields- customer_id date (date when visit is completed) schedule_status (date ...
0
votes
2answers
40 views

how to generate random data based on simple statistical meassures

0 I currently have a test data set that has 500k data points. I have an algorithm that process that data and returns some information. In order to establish the statistical significance of the ...
1
vote
1answer
54 views

How to identify if my data set is skewed or not?

I think my assumptions are a bit naive regarding this matter. I have two metrics about my data set: the number of items and the cardinality of the items. Low cardinality means a lot of repeated items ...
-1
votes
0answers
14 views

Comparing two subsets of data for same observations

I have a data set with 50 test subject and 500 features. The features can be divided into two subsets: profile data of the test subject (pass/fail, age, education, IQ,...) and scoring on the test(...
0
votes
2answers
53 views

How to manipulate open-source datasets for coursework

I have an open-source dataset that I would like to use in a deep learning course, but the downside of using open-source datasets is that there are open-source solutions. I want to manipulate the data ...
0
votes
1answer
37 views

How can I apply a Logistic Regression model in real life? [duplicate]

I just run a LR multivariate analysis to class probability check. From this analysis I could see that the combination of 4-variables would increase my test accuracy. Here is the output of this ...
2
votes
0answers
34 views

Sum of exponentials and variance of the exponents

Let's assume we have two sets of non-negative real-valued numbers $\{a_i\}_{i=1}^{N}$ and $\{b_i\}_{i=1}^{N}$ ($N$ can be large) such that $\frac{1}{N}\sum_{i=1}^{N}a_i=\frac{1}{N}\sum_{i=1}^{N}b_i=...
-1
votes
1answer
41 views

Combine multiple rows into one row in R [closed]

I am creating a sample of 3 observations with 3 values. I would like to then combine them into one observation and determine the frequency of duplicate values but would like to know what values are ...
0
votes
0answers
16 views

Machine translation dataset is actually not parallel

I'm currently working on implementing the Transformer model for machine translation. I'm taking a look at the data that was used in the actual paper and also used by many other implementations ...
1
vote
0answers
24 views

What is the difference between validation and cross validation?

I hear the dataset is divided into three parts 1-training 2-validation 3-test in the 2- validation Do you mean cross validation or what difference does it make?
1
vote
1answer
20 views

When to validate your data? [closed]

I have simulated a scenario under ideal conditions. This simulation was done using the Friis equation to determine the power received by a RFID tag for a particular frequency, distance, power of the ...
0
votes
0answers
6 views

CRSP/Compustat - Issue with gaps in data

I would like to test empirically CAPM for my dissertation. I have downloaded monthly data for period 01-1990 - 12-2018 for NYSE from CRSP/Compustat database using condition "exchangecode" = 1 however, ...
1
vote
0answers
20 views

Sparsify meaningful data following a Gaussian distribution

This is following a post I made a couple of days ago on the Data Science Stack Exchange. Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the ...
1
vote
3answers
84 views

What kind of statistical model I should use?

Here is the project that I'm currently working on it and would be more than happy if I can get your opinion about it. The project is about dispatching the EMS helicopters to the incident locations to ...
0
votes
0answers
10 views

Bivariate Data Variables

When comparing two different variables to determine whether or not one determines the other, do both data sets of the variables need to be from the same year? Such as life expectancy from 2004 and HDI ...
0
votes
1answer
33 views

What is the cumulants of a whole data in terms of the cumulants of its parts?

I have around 8 billion data points, and I need to calculate the distribution and the cumulants of this distribution. However, due to technical restrictions, and time constraints, I can only ...
0
votes
1answer
44 views

Dealing with imbalanced dataset without using Undersampling or Oversampling

Let's say I have a dataset with 100,000 class A training observations and 400 class B training observations. I want to use Support vector machine for this binary classification problem. Instead of ...
1
vote
0answers
24 views

What types of analysis will work on a dataset of medians?

The dataset I currently have is wages for different occupations across different geographical regions, however each observation is a median. For example, a doctor in region A, the wage reported would ...
0
votes
0answers
18 views

How can I get a greatful order considering two different variables using z-score?

I have the data of different stores which have different values ​​of number of sales and different price ranges. I want to order them in such a way that the first ones are the most expensive stores ...
0
votes
0answers
35 views

Least squares with correlated noise

Assume I have the system \begin{equation} x(k+1) = \begin{bmatrix}A & B\end{bmatrix}\begin{bmatrix}x(k) \\ u(k)\end{bmatrix}+ w(k) \end{equation} where $x(0)\sim\mathcal{N}(0,\sigma_x^2)$ and $...
0
votes
1answer
27 views

What options do I have for preprocessing data before entering it into a convolutional neural network?

So I am using about 1300 lines of statistical data from the 2017-2018 NHL season and I want to run it through a neural network, but I want to know what kind of pre-processing options I have. I have ...
1
vote
2answers
50 views

What statistical test would I use to test that both data sources are matching [closed]

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
1
vote
0answers
30 views

Generating new(fake) probability distributions based on many sample probabilty distributions

I wish to be able to generate new probability distributions that incorporate the characteristics of many sample distributions available to me. I currently have data for the individual daily return ...
3
votes
1answer
37 views

How to calculate standard error from a 95% confidence interval

I am conducting a meta analysis and one of studies doesn't have the standard deviation so i've been stuggling to work out the standard error. CI's are of 95% I was wondering if I could calculate the ...
1
vote
2answers
37 views

Are total number of products purchased always correlated w/ total sales ($)?

Sorry if this qualifies as a dumb question, but would it be possible for this relationship to be uncorrelated if a lot of customers buys a lot of inexpensive product that don't contribute to sales ...
0
votes
0answers
11 views

what are “adversarial distributions”

I ran into a paper that talked about some k selection algorithms working better or worse with "adversarial distributions", full excerpt here: 1.3. Organization In this paper we present three ...
0
votes
0answers
24 views

Features after PCA are not normalized

You are supposed to normalise data before doing a PCA analysis. But when one has a PCA transformed dataset which contains 10 continuous features/variables, but the scale of feature one go from -6 to +...
1
vote
1answer
27 views

Running SMOTE on unbalanced data

I've read a few answers to similar questions that advise the completion of SMOTE after splitting the data set into Train and Test sets however, the documentation and other examples I've seen run SMOTE ...
1
vote
0answers
37 views

Why do I get worse result after normalizing my input data of LSTM?

The first pic shows my data before normalization: And the second pic show my data after normalization using sklearn.preprocessing.normalize : I get better result ...
0
votes
1answer
34 views

Proper way to add noise into a dataset

I feel this question is trivial but I also couldn't find the answer (hope I am not bad at searching online). Put simply, I generate data from a normal distribution with ...
3
votes
1answer
33 views

analysis of composite scores

I've created a task comprising 24 task items. Each item asks a 'right/wrong' type binary answer. For data analysis, I calculated the percent of the right answers (e.g. 20 out of 24) and treated the ...
1
vote
0answers
14 views

What can i conclude about SalePrice (dependent variable) between independent variables in correlation matrix python?

I'm a beginner with Kaggle House Price dataset to predict Sale Price, just doing data exploration to find out the correlations between the attributes and Sale Price (target variable). I came up with ...
0
votes
0answers
5 views

Score model for valdiation

I hope I am on the right part of stackexchange, because my question is a bit more data related. I post here for some conceptual help. I hope you can help. My problem is the following: I work on a ...
0
votes
0answers
2 views

Feature-construction for regression on samples consisting of varying numbers of sub-samples

I recently came across the following regression problem: The goal is to predict some value (e.g. the duration) of an event. One such event consists of multiple sub-events. The amount of sub-events ...
2
votes
1answer
105 views

Clear explanation of dummy variable trap [duplicate]

I have a confusion in multiple regression about dummy variable trap, so far I had seen tutorials explaining about dummy variable trap and multicollinearity but I'm unable to understand it fully.
0
votes
0answers
8 views

What is surrogate training data?

I've seen a lot of talks that use the term "surrogate training data" when talking about unsupervised training. I'm not able to quite wrap my mind around this and the resources I've found online have ...
0
votes
0answers
9 views

Vector representation of subsets of data in python

I have 141575 data points that can be divided into 153 blocks. There are around 70 features in my dataset. I want to measure similarity among the blocks and rule out outliers during training my model. ...
0
votes
0answers
12 views

How to get a threshold value?

I have a dataset of genes and their frequency in the dataset as: Target_Gene:A,B,C,D,E,F,G,H....,Z Count:260,150,120,90,50,40,36,27,.....,1 Now I take a Sample from this: Target_Gene:A,C,F,J,K.......
0
votes
1answer
42 views

What statistical technique could be used to extrapolate CO2 data

I am new to statistics but scraped data from NOAA's web site to plot CO2 level vs time. Is there a good statistical technique to extrapolate past limit of data X axis to near future like 2050? One ...
2
votes
2answers
30 views

How should I perform cross-validation of 10-fold?

Suppose that the circles in red are negative samples and the circles in blue are positive samples and that the green boxes are the validation set and the blank boxes are the training set. In addition, ...
0
votes
1answer
20 views

When to use Normalized Root-Mean-Squared Error vs Spearman Correlation?

I am doing some Machine Learning experiments with Azure and the graphs that it gives me are measured in Spearman Correlation vs Iteration Number (part of the machine learning) However I was just in ...
0
votes
0answers
3 views

Opensource data set- workload(request rate/second) for Amazon(e-commerce) or Facebook type of applications

Is there any opensource data available for Facebook or Amazon (e-commerce) applications? The dataset I'm looking for should include the number of requests coming to the application on per second basis....
0
votes
0answers
10 views

unbalanced panel data: is it possible to use?

I am trying to analyze unbalanced panel data. Honestly, it is first time to do longitudinal study, so I have many difficulties. I really need your help. I consider to use GEE or RE (in STATA) ...
1
vote
1answer
119 views

Choosing the Right Data Mining Classification Technique

From the different Data Mining tasks, I want to train the Classification. For that, I: Took this dataset (can be used as an example for the answer). Got to know the data (data objects, attribute ...
0
votes
0answers
33 views

Why should we demean data?

In my case, I'm performing a vibration analysis using FFT and all that, and in this algorithm that I've been using, the author demeans the amplitude data before applying the FFT to it. Is it a thing ...
0
votes
0answers
11 views

How precisely do we ensemble output as in this Omni-Supervised Data Distillation paper?

After reading the paper Data Distillation: Towards Omni-Supervised Learning, I saw that after getting the results, I ensemble them before using them in order to train the student model with respect to ...