Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Transforming data on a per-instance basis

I am currently facing the following situation (in my feature engineering process): I am inspecting a step function for a set of instances (step function per instance), i.e., in each iteration "i&...
0
votes
0answers
14 views

How to analyze expanding area of data [closed]

I've got a large set of data that when graphed as a scatter plot looks like the image I've created below (minus a few outliers). I'm completely stuck trying to get useful info out of this chart as I ...
0
votes
0answers
17 views

What size Dataset do I need before I make a decision? [closed]

I am trying to figure out how big the data set needs to be to decide that an analysis result is statistically significant (in my use-case - how many clicks before I can make a decision). What should I ...
1
vote
1answer
22 views

How to use labeled and unlabled data together in machine learning

Say I have a dataset of labeled elements and an unlabeled dataset that I would like to apply my machine learning model to after training. How would I go about reporting accuracy/ROC AUC/etc. in this ...
0
votes
0answers
4 views

ProPublica's COMPAS Data | Feature Descriptions

ProPublica kindly provided the COMPAS Recidivism data in the github: https://github.com/propublica/compas-analysis/blob/master/compas-scores-two-years-violent.csv Most of the features are clear from ...
1
vote
0answers
42 views

Neural Networks: cross validation necessary?

I am training some networks to detect certain characteristics in x-ray images. The images are divided into three classes: A, B and C. I have currently used the neural network committee building ...
0
votes
0answers
11 views

Anova test number of files in system types, extremely small p-value

To get more practice with Anova tests, I am attempting to use it on a dataset that I have, but the result that I have received has an incredibly small P-value (e^-62). Below is the result as well as ...
1
vote
1answer
30 views

Correlation uses observed values, Mutual information uses probability values

I have a paired dataset whose observed values are: ...
3
votes
2answers
72 views

How do you spot errors in data?

I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
1
vote
1answer
26 views

Correct term to describe data that has had values removed/trimmed?

What terms can I use to describe data that does not include +- 3 z-values? More importantly which are flat out wrong to use? a) sample b) sample group c) dataset d) population e) other
2
votes
0answers
50 views

Confidence Intervals - What's going on?

I'm looking for help with this figure. Plotted are percentages and confidence intervals for each year (I didn't collect the data). I am wondering, under what circumstances would the confidence ...
0
votes
1answer
38 views

Taking the mean of several data entries on same date to make one

I am analyzing the number of likes of several influencers over time and I want to plot it to see if there is a trend. Some influencers post several times on the same date, and then overall several ...
0
votes
0answers
9 views

recode Na and NaN values for specific rows based on column value [migrated]

I want to recode NA and NaN values for rows where a column value is X. Here is my data structure. ...
4
votes
1answer
80 views

How to Make Meaningful Conclusions?

I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question. He asked the same question to a friend ...
0
votes
0answers
14 views

Possible to make predictions with overlapning data

I've the following dataset. The task is whether "UNEMP_Q1" is correlated with "UNEMP_AVG_Q47" and whether it is possible to use "UNEMP_Q1" (or/and UNEMP_Q2 and UNEMP_Q3 ...
3
votes
2answers
17 views

How to present ML model performance before deployment?

I have machine learning model performance metrics on training set with cross validation and metrics from when I ran the model on the test set. The model performs similar on both sets, so I want to ...
4
votes
1answer
55 views

Standard deviation/variance for the sum, product and quotient of two Poisson distributions

What would be the standard deviation for $A+B$, $AB$ and $\frac{A}{B}$ for $A$ and $B$ Poisson distributed?
0
votes
1answer
29 views

Averaging quarterly level data to the annual level

From the Quarterly Workforce Indicators, I have obtained a measure of data measured at the quarterly level for earnings and employment. so in each quarter, they take the total earnings, and divide by ...
0
votes
0answers
8 views

how to work with dataset having single target for a multiple rows

I am a beginner in ML/DL. I have majorly worked on a single data file with each row having a target value Currently, I trying to build a CNN model for a given dataset. However, I am struggling with ...
4
votes
0answers
33 views

Which financial time series have a PDF and/or a CDF?

Consider the following types of financial time series for a single publicly-listed stock: Price data Log returns Cumulative returns Each is computed from the item listed before it: log returns are ...
0
votes
0answers
6 views

Can I average a Volume-Weighted Average for the same result?

I'm currently consuming metering data for a large utility company. I want to calculate the volume weighted average return temperature of cooling systems, but I have a lot of data points. To reduce the ...
1
vote
1answer
27 views

How to deal with zero inflated columns in dataset?

I have a dataset on which I am trying to fit a Linear Regression model. It has 4 independent variables. I am trying to predict my dependent variable using these four columns. However, 2 out of these 4 ...
0
votes
0answers
8 views

how to separate 2 numbers within a column in R [migrated]

In R, I want to separate numbers that are in the same column. My data appear like this: id time 1 12 2 34 I want it to appear like this: 1 1 1 2 2 3 2 4 Does anyone have advice to ...
0
votes
0answers
38 views

Generating Multivariate Outliers in R

I have generated a data-set following the Gaussian distribution with 5 dimensions. The data-set is 1Million record and I am seeking planting 20% outliers, is there a more neat way in R to generate ...
3
votes
1answer
98 views

How are artificially balanced datasets corrected for?

I came across the following in Pattern Recognition and Machine Learning by Christopher Bishop - A balanced data set in which we have selected equal numbers of examples from each of the classes would ...
0
votes
0answers
13 views

Can a new independent variable be produced from the dependent variable in variable engineering?

I have a question. I have a data set, my goal here is to estimate player salaries. My variables are salary, number, year, error etc. I thought of creating a variable such as average salary by making ...
0
votes
0answers
13 views

Combining two datasets for Random Forest binary classification

I was tasked to perform a brief meta-analysis between three datasets, all with identical features, that studied different sources of neurodegeneration. As a meta-analysis these datasets were drawn ...
0
votes
1answer
34 views

is there a way to train the models in batches? i m not talking about deep learning

I have 800,000 records. The training with all the records at once is not possible. Is there a way i can train a model on 50k records then continue training on next 50k records and so on... basically i ...
0
votes
0answers
22 views

Logistic regression: dependent observations

I have a time series dataset. One series contains ~200 data points. Each series describes one sports match. I have converted each series into ~200 samples such that each sample contains somehow ...
1
vote
1answer
21 views

Plotting categorical data along with numerical variable

I have data like as shown below I would like to represent the above tabular data in a visual form. However, the below graph may not work because my real data as 50K unique drug names. Doing so, will ...
0
votes
1answer
42 views

What is the actual infection rate (per 100,000) of Covid-19 in my area (I'm in the UK)?

I think I'm misunderstanding some basic data given by the UK Government. https://www.bbc.co.uk/news/uk-51768274 On the above page, about three fifths of the way down, there is a table entitled, "...
0
votes
0answers
6 views

Advantage of DEA VS. simple KPI benchmarking?

im wondering what the actual advantage of data envelopment analysis over a simple KPI benchmarking is? Image you have let's say 5 inputs and 1 output. Then I can just create 5 KPIs for each input, by ...
0
votes
1answer
8 views

Reducing training dataset for architecture testing

I want to quickly test many CNN architectures on the MNIST dataset, but I don't want to train each on the whole dataset. Is it possible to reduce the training dataset size and still be able to ...
10
votes
3answers
594 views

Is it in general helpful to add “external” datasets to the training dataset? [closed]

Several people have already asked "is more data helpful?": What impact does increasing the training data have on the overall system accuracy? Can increasing the amount of training data make ...
0
votes
1answer
18 views

Is data observed from the internet counted as primary data?

I need to find primary data (2 variable) to conduct a research about correlation. Are data that I observed from the internet (example: from youtube.com/amazon.com) counted as primary data?
0
votes
1answer
19 views

Should we try imputation on cases with slightly problematic datasets or prefer ommiting observations

I would like to ask a general question which makes me worry when I try to impute NA values. We know that most of the imputation methods are based on the rest non-NA values. However, if we know that ...
1
vote
0answers
9 views

Crossing categorical features that are stored as integers

Newbie here. I'm experimenting with the following dataset: https://archive.ics.uci.edu/ml/datasets/Teaching+Assistant+Evaluation Data Set Information: The data consist of evaluations of teaching ...
1
vote
2answers
54 views

Crime Statistics Analysis: how to properly compare apples to apples? [closed]

first this is non-partisan. Now, I have heard a lot of people in the news using certain simple statistics such as the number of uses of lethal forces by cops particularly where the victim is a ...
0
votes
0answers
25 views

How to transform data in order to visualize it with my own drawing software?

I want to build my own data visualization toolset using Blender3D and Python. Let's think of a racing bar chart. As a dataset, I have 'google search frequencies of 10 keywords over a period of 10 ...
0
votes
1answer
21 views

Do GLMM account for interdependence of data points?

I never used Generalised linear mixed models but from what I understand they can be used when the observations are correlated. I'm interested in knowing whether I could potentially use GLMM to analyse ...
0
votes
0answers
17 views

Similarity metric for one-dimensional time-series data-sets

I have a setup consisting of measured (one-dimensional) sensor data from n subjects. For each subject, measurements were performed from different angles. The amounts of angles and acquired ...
0
votes
0answers
8 views

Why does my lagged target variable have a high p value in multifactor regression?

I'm running a multifactor regression which uses the following data: The first column is my target variable, the following 5 columns are my factors. The final column is my target variable with a lag of ...
0
votes
0answers
19 views

Splitting a single large data set that contains two populations into two data sets

I have a single data set that contains two populations(This is a fact.) The means are different. It's obvious when charting the data, but not obvious to me how to split them. Is there a method or ...
0
votes
0answers
7 views

What's a good way to select a bound that's close to zero?

I have a bunch of position data that I transformed into speed data. I'm assuming that I have some noise in my data and that the noise got worse after transforming to speed. I used a Kalman filter to ...
0
votes
0answers
15 views

Analyzing cost of treatment in a cohort (varying follow up/available data) at multiple time points

I would like to look at total cost of treatment from an index date (treatment initiation) after 1 month, 3 month, and 6 months. However, my data has has issues where patients drop out at varying time ...
0
votes
0answers
25 views

Should I feed my neural network with “not-approved” profiles, even though I want to predict the approvals?

I have a categorical classification problem to solve in which I have a profile of a consumer and a loan order (which has a unique partner) and an end state of that order. There are 10% approvals. I ...
0
votes
0answers
6 views

How can I transform evidence scores of single-video machine analysis output to confidence scores?

For my research purpose, I need to convert the evidence scores of machine vision output to confidence scores. I found below codes for R in the website: https://rdrr.io/github/bigpas/face_exp_analysis/...
0
votes
0answers
3 views

Issue in course classing and fine classing for A score development

Im a bit stuck in something, for which i need assistance Can u help I have a set of data for which i am developing an Application score card In this data, the applicant's income is only rank ordering ...
2
votes
2answers
46 views

Missing data in a pattern - How to proceed?

I am working with regression on the following dataset: https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime Which relates the crime rate per capita (last column) with 120 different metrics, ...
0
votes
0answers
12 views

SPSS modeler error Type node read values output typeless

Whenever I try to read the values of my categorical fields in my dataset, the measurement is changed to typeless and the values are empty. According to IBM Streams created through Modeler Advantage ...

1
2 3 4 5
32