Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

0
votes
1answer
17 views

can you help me to do a difference of CDF?

I have 2 CDF's with equal number of points that I want to compare. These are from: Temperature of 1 month from 2012 Mean temperature across months What can I do to obtain this difference, this is ...
0
votes
1answer
15 views

Feature engineering with cross validation, then testing on a holdout data set?

We have 3000 samples for two classes, roughly 2000:1000. Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a ...
0
votes
0answers
13 views

Big Data Analysis_Logistic Regresson or Support Vector Machine [on hold]

Can anyone share some resources or links in which I could get access to some large data sets(>1GB) suitable for Support Vector Machine or Logistic Regression Analysis?
1
vote
1answer
21 views

Should I use 5KFold or 10 KFold CV for data about 300000 sample?

I use logistics regression for binary classification. and my data is about 300 000 samples. I have used both kfold = 5 and kfold =10 cross-validation ...
0
votes
1answer
16 views

Conditional expectation in Basic Linear Unobserved Effects Panel Data Model

I want to see that $ E[u_t|X_t,c] \Rightarrow E[X_t' u_t]$ in a Panel Data Models
0
votes
0answers
31 views

Research question: Which stain is most effective for improving tissue contrast in CT scans of the larynx?

I'm doing research into which stain out of I2KI, PMA and PTA is most effective for improving tissue contrast in CT scans of the larynx. I have 9 specimens in I2KI (CT scanned after 0, and then either ...
0
votes
0answers
11 views

Confidence vs. Count in association rule mining: which one is better?

I am writing a program that mines association rules from a large data set. I have an array of association rules, and I have to decide which ones are more representative of the patterns I am studying. ...
0
votes
0answers
8 views

Import ARFF dataset using RWeka in RStudio [on hold]

I am currently using R for Windows verison 3.5.3 and RStudio version 1.2.1335. My goal is to import an ARFF dataset using the RWeka package in order to do some Association analysis, more specifically,...
0
votes
0answers
7 views

how to check the distribution of the training set and testing set are similar

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
0
votes
0answers
10 views

Converting Annual Data into Biannual Frequency

I need to convert annual data into biannual, since most of my data have biannual frequency. However, some macroeconomic variables - like business environment quality or Herfindahl-Hirschman Index are ...
0
votes
0answers
12 views

Optimization to search connected components with a cloud of points (2 Dimensions) and a distance D [on hold]

I'm searching a way to optimize this problem: In entry : a list of point (Python List) , an integer D. I have to search all the connected components knowing that a point is theorically linked to ...
0
votes
0answers
23 views

Method of comparing the evenness of data sets [closed]

I need to compare and rank multiple sets of data for their evenness, flatness or in other words average deviation from their relative mean. I'm intending for each set to have a value which would be ...
0
votes
0answers
17 views

how I can make the code for censoring proportions gamma distr [closed]

Can someone helps me with R studio code? I know that the Weibull censoring proportions can be expressed as P: res=function(lambda,alpha,beta,p,N){ Z=rexp(N,rate=1) cens=((1/N)sum(exp(-((...
0
votes
0answers
7 views

Model evaluation based on WoE: test and train set

I am training Logistic Regression model. First, I prepare the variables. I am confused with the following: Should I transform variables to the ones with WoE (Weight of Evidence) values based on the ...
0
votes
0answers
6 views

Use time series analysis to analyze game players behaviours and interactions (in seconds & miliseconds)

I am working on a mobile game in which the player needs to respond to a quiz.. And i am storing the player interactions and behaviours during the game like : response time ,clicks , number of answer ...
0
votes
0answers
18 views

Prediction Challenge AI. Machine learning and data analysis.(Inconsistency data)

I am working on a personal rento in which based on some input data I have to predict some output data. The challenge is to predict the expenses in transactions, receipts and cards that users will have ...
0
votes
0answers
7 views

Why some missing values are recorded as -1? [duplicate]

I have a lot of missing values in a dataset in a velocity column. Some of the missing values are just blank cells, some are recorded as NaN, but for some column (velocity), the missing value was ...
0
votes
0answers
20 views

What type of data is this? Nominal, ordinal, interval, ratio

If you were to do a survey and ask, “how many people have lived in a town for less than 2 years, 2-9 years or greater than 9 years?” Would this be considered nominal, ordinal, interval or ratio?
0
votes
1answer
16 views

Classification with different types of data

I'm wondering if there is any methodology or work that illustrates learning on different types of data, but with the same goal/prediction task. As an example, if I have two different data sources: ...
0
votes
0answers
38 views

Multidimensional Principal Component Analysis

I found a lot of question about principal component analysis as well as functional PCA but only in the sense of time series and not in spatial case and multidimensional principal component analysis. ...
-1
votes
3answers
34 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
0
votes
0answers
13 views

Similar plots of 2 closely related data sets

Suppose my data set contains 1000 observations and 10 variables. I consider 2 data sets: the original one and the one which contains only the first 5 variables (the number of observations is still the ...
0
votes
0answers
13 views

Using Concordance correlation in r, for data sets with slightly different number of points

Just wondering if there is a way in r to define a data point based on 3 values, for instance, I have 4 columns. Im type, Feature class, Feature and Value I can't use the data point number as the ...
0
votes
1answer
23 views

What kind of analysis to use for continues value data set [closed]

I'm new in the data mining field and I'm using RStudio to do the analysis. I have a dataset and I don't know what could be the best method to use. I've tried using correlation and I don't know if ...
1
vote
1answer
35 views

Can I use Wilcoxon Signed-Rank paired Test to compare binary data?

I have two series of binary outcomes: Series1 Series2 Test1 0 1 Test2 1 1 Test3 1 1 Test4 0 1 Test5 0 0 ... I want to know ...
-1
votes
0answers
29 views

Ordinal Logistic Regression In R [closed]

I am applying Ordinal Logistic Regression to a dataset. And I have total of 18 variables. Class is my ordinal variable. It gives this error when I execute the code: Error in polr(Class ~ ., data = ...
0
votes
1answer
24 views

prepare a data set to be fed into a machine learning model

so I am pretty new to the whole Machine Learning concept and I'm coming from a complete beginners standpoint. I am trying to detect whether there is steganography present within jpg files using a ...
0
votes
1answer
14 views

Is this variable suitable for a categorical regression (multinomial logistic regression)?

I have created a dataset starting from a series of multiple choice (3 choices) questions. ...
0
votes
0answers
21 views

Imbalanced Dataset - Poor Evaluation

My dataset has about ~75,000 records with 39 features. Most of the features are categorical, so I have one-hot encoded them. About 14% are minority with label 0 and the rest 86% with label 1. I have ...
0
votes
1answer
24 views

Unsupervised methods for Detecting Spam

What are some common unsupervised methods used to detect whether something is spam or not? For example, we may be given a large corpus of emails and need to determine whether any one of them is spam. ...
2
votes
1answer
32 views

Do error bars give information about the dispersion of the raw data from the study?

Do error bars give information about the dispersion of the raw data from the study? If data in a graph has standard error bars will this provide information about the dispersion of the raw data ...
1
vote
0answers
12 views

How to Convert Quarterly Data into Annual Data

I'm doing a project where I'm trying to compare the median wage of workers that had federal job training to the minimum wage indexed to productivity. The Department of Labor only offers the median ...
0
votes
0answers
28 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
2
votes
1answer
56 views

Dealing with imbalanced data-set and cross-validation

I have a data set of brain tumours, 700 malignant, and 225 benign. And I want to build a classification model using SVM, to classify the tumours types based on the data I have. My first question, is ...
0
votes
0answers
15 views

May negative dataset cause CNN model over/under-fitting?

To put you into context, let me explain a bit what I'm trying to achieve. I'm using YOLOv3 (doesn't really matter now) convolutional neural network to detect traffic signs in full images. I'm training ...
0
votes
0answers
12 views

Ways to anonymize the data for logistic regression

We are testing a Proof of Concept with an external vendor to whom we need to provide the data in an anonymized way. Apart from removing the client details/identifiers, what are the best ways to ...
0
votes
0answers
12 views

Reorganized Panel data in R

I am currently working on a fixed effect model, and have a dataset (csv file) in the form of the following (represents two city over three years): year feature1 feature2 2000 10 14 2001 12 13 2002 13 ...
0
votes
2answers
59 views

What is the meaning of these principal components?

I have a matrix of data. I computed the principal components of my matrix using SVD (code shown below): subtract mean...then $$[U,S,V] = SVD({\rm matrix})$$ for $V$, which is the principal ...
0
votes
2answers
27 views

Does it make sense to apply PCA or Z-Score to any dataset?

Suppose we have a given dataset whose variables represent different things. For instance, one of them could represent the time a user spends on the phone while another one can represent the continent ...
0
votes
1answer
23 views

What defines the qualitative and quantitative characteristics of a dataset?

Can somebody tell the differences between quantitative and qualitative characteristics of a dataset? After reading multiple articles on the subject, with different explanations, I am a bit confused.
0
votes
1answer
16 views

Dataset partitioning for k-fold cv

In general, is the size of the validation set for k-fold CV given by n/k and that of the training set given by n(k-1)/k and if so, why? (this is based on the ISLR book) Alternatively, what are some ...
1
vote
0answers
15 views

Use historical data to build surface with limited number of points [closed]

Lets say I have historical surfaces of data which vary slightly during time, but keeps similiar dynamics. I would like to use these historical data sets to estimate todays surface (given a limited ...
0
votes
0answers
19 views

Real 1 dimensional non-binary data where Cox Proportional hazards model fails

I am learning about the Cox Proportional Hazards model and understood that it is very flexible, even when the assumption of proportional hazards is not met, e.g. for additive hazards. But say we have ...
0
votes
1answer
51 views

Deciding on population vs. sample

I have been researching when to classify a dataset as population vs. sample for many hours now for a statistics project I am doing. I have a dataset of NFL combine data and one thing I need to clarify ...
3
votes
1answer
41 views

Does retraining a model on all available data necessarily yield a better model?

A (simplified) typical workflow in machine learning might be: Train $m$ models on a training set. Validate the $m$ models on a validation set to yield the best model with parameters $\theta$. Retrain ...
0
votes
0answers
8 views

What type of analysis to perform?

I am working on in-vitro evaluation of efficacy of multiple antibiotics. I've selected 8 antibiotics with 3 concentration of each antibiotic applied with 3 replications each concentration and ...
1
vote
0answers
28 views

Modeling daily purchase events / website traffic, time series or independent events?

I've been thinking about how best to model sequential events like purchases on a website, or website traffic. I'm curious if people think this is best modeled as a poisson process (or something like ...
0
votes
1answer
21 views

Evaluate Sample Representativeness of Dataset

I'm trying to check / evaluate how representative a sample is from a dataset. I'm interested in two things: Which sample best represents the original dataset Is the sample a "good enough" ...
0
votes
0answers
30 views

Problem doing Machine Learning in this dataset

I have a problem to obtain good performances with a dataset. I have to predict the flow of visitors in my city given the distance of origin of tourists and the number of inhabitants of their city of ...