Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

0
votes
0answers
20 views

Fitting 2D data

I have a 2D data-set. If we plot, it looks like the image below (to some extent): This distribution mostly follow a Gaussian shape with some mismatch. For the sake of simplicity let's assume it a ...
0
votes
0answers
2 views

Fitting of 2D data

I have a 2D data-set. If we plot, it looks like above. Now if I want to know the central value of the distribution $\theta$ and width of the distribution $W$. How do I do that?
0
votes
0answers
7 views

data normalization for one sided specification [on hold]

how to Normalize data when we have only only side specification Ex:upper specification parameter is given & the lower specification value is blank or vice versa & the observed value of the ...
-1
votes
0answers
12 views

How can I add subscript to Y axis without using the expression() and paste() command? [duplicate]

I was able to add subscript and superscript for the title but I am unable to do that for the Y-axis! I want to write µgml-1NO3--N. Can any one help me on this? Thank you! ...
1
vote
0answers
10 views

mlbench synthetic datasets for python [on hold]

Is there any implementation of the synthetic datasets from mlbench in python? I am looking for these implementations in particular. I've found sklearn.datasets for synthetic examples, but they do not ...
0
votes
0answers
11 views

R: Ordering data in a cell of data frame [on hold]

I have huge data frame of multiple columns, one of the column contain data like that ...
0
votes
0answers
5 views

How to reduce false positives when novel negative images are visually similar to training images?

I am noticing that my ResNet model is showing some false positives in cases where novel negative example images are somewhat visually similar to positive examples. In these cases, it's not simply ...
0
votes
1answer
55 views

How to statistically analyze the relationship of right skewed data

I struggle to analyze these continuous data: The last four plots show the diagnostic plots on my model (model <- lm(data 1 ~ data 2). My aim is to investigate the relationship between data 1 and ...
0
votes
1answer
27 views

How to deal with a poor quality data set? [closed]

I have come across a data set in which almost half of the data points are unusable (as in when further processed, they give negative results when the results should all be above 0). This is largely ...
3
votes
1answer
64 views

why has author divided by 1.5 in hands on machine learning with scikit learn

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
0
votes
0answers
10 views

Transforming data before wavelet analysis [closed]

I would like opinions on log-transforming the data before performing a wavelet analysis. Is it advisable? Is it right?
0
votes
0answers
10 views

Assessing performance of an agent based on commission rate, market share and revenue

I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following: Number ...
5
votes
1answer
50 views

Which statistical methods are best suited for distribution with two peaks?

My data shows this distribution: I am looking for a statistical distribution which my data follows. Thought about poisson distribution, but goodness of fit test shows p < 0.05
0
votes
1answer
22 views

Train classifiers on a subset and validate on a full dataset

Let's say I want to train a classifier and use it on 20% of "known" data; the rest of the "known" data is reserved for training (60%) and testing (20%). In the end I also want to apply it on "unknown" ...
0
votes
0answers
24 views

How is dev set used to tune hyperparameters?

I'm new to the deep learning domain and still did not understand clearly enough the idea behind the dev set. I read that dev set is usually used to tune hyperparameters and to compare the performance ...
1
vote
0answers
24 views

Supervised classification with different row length dataset

I need to make some predictions based on some situations: the problem is how to enconde situations themself. A row in the dataset will represent a single situation but here it comes the problem. A ...
3
votes
2answers
39 views

Example of dataset where the data collected at time-points $g(t_1), g(t_2), \ldots$

What would be some practical scenarios where we collect data at time-points $g(t_1), \ldots, g(t_n)$, where $g$ is an increasing function? For example, $g(t) = \exp(t)$ or $\ln t$. To be more clear, ...
0
votes
1answer
23 views

Can y of datasets be increasing/decreasing ratio when train regression model?

I found many cases in kaggle to predict the quantity or trends. They all set the real quantity as y. But I have question is that does anyone set the changing ratio as y? Like: ...
0
votes
0answers
10 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
34
votes
5answers
2k views

What if my linear regression data contains several co-mingled linear relationships?

Let's say I am studying how daffodils respond to various soil conditions. I have collected data on the pH of the soil versus the mature height of the daffodil. I'm expecting a linear relationship, ...
1
vote
0answers
14 views

Different conditions on data measurement for ml

Is it possible to train a prediction model (on my case classifier with four classes) between data taken on different conditions? To be more specific I have two data sets and for my task I am allowed ...
3
votes
1answer
63 views

How to solve a classification problem when the independent variables/covariates/feature vectors form a time series?

Say we've a time indexed sequence of feature vectors/covariates/independent variables $x_t$ at time $t$. Say also we've a corresponding time indexed sequence of variates/dependent variables $y_t$. Now ...
0
votes
0answers
10 views

What are some techniques to augment tabular data?

As we know we can perform data augmentation to "image dataset". We can apply random rotation, shifts, shear and flips over images. Are there techniques to augment tabular small dataset? I know the ...
0
votes
0answers
9 views

Forecast of sales data (daily or weekly) with random missing values (No sales)

I have a daily sales data of a medicinal brand by hospital accounts. I want to forecast the sales of the brand by accounts however the hospital orders are not regular. Their is a randomness in order ...
1
vote
2answers
65 views

How can we estimate (conditional) probabilities from a dataset?

Given three random variables $X$, $Y$ and $Z$, how can we estimate $P(X)$, $P(X\mid Y)$ and determine whether $P(X \mid Y, Z) = P(X \mid Z)$ from a dataset (of e.g. $N$ observations) which contains ...
1
vote
0answers
28 views

Test to determine whether the empirical distribution for a given day is an outlier compared with other days

Say you have multiple data samples from different days (or some other unit of time) and you want to answer the question: is the distribution for a given day an outlier (compared to other days)? Is ...
1
vote
0answers
24 views

Two predictor variables are partially related. How do I handle them before performing logistic regression

I have a predictor variable which has three levels (success, failure, nonexistent). For all nonexistent values another predictor variable value is 0 but success and failure could fall within a range ...
0
votes
0answers
9 views

How to handle varying sample size in a panel study?

Our panel study has an initial sample size of 17 and increased to 25 in the second data collection. How can be avoid bias in the data analysis? Method: We used a survey form with 23 likert-scale ...
2
votes
0answers
35 views

Data obfuscation strategies that keep statistical properties intact

I am getting data from a provider that contains identifiable information from a number of different organizations. The data aren't particularly sensitive, but to be safe, the provider wants to perform ...
1
vote
1answer
27 views

What does each cluster represent?

I have a dataset from a questionnaire with over 10000 rows and 30 variables. I am trying to have an insight of the data so I tried to cluster similar items. I first made a dimension reduction ...
5
votes
4answers
154 views

Does sampling from a large dataset lead to correct inferences?

Say we have some population, and we obtain a "representative" random sample of that population, $(y_i, x_i)_{i = 1}^n$, where $n$ is very large (millions) and $x_i = (x_{i1}, x_{i2}, ... x_{ip})'$ is ...
0
votes
0answers
12 views

How do I get intervals classes (groups) from a histogram of data?

in my problem setting, I have data for 1 variable. The data is numerical, and in my problem setting, I just generate a frequency count of the data, for example: If data is: I create a frecuency ...
0
votes
0answers
17 views

All items missing for various questionnaires

this is actually the first time I'm working on a big dataset and I really hope someone can give me some advice on how to handle missing data. I tried to find information regarding my problem but can't ...
1
vote
1answer
33 views

How much data is needed to train CNN from scratch?

Any rule of thumb, on how many input images would be needed to have a reasonable chance not to overfit the data when training a CNN from scratch? In other words, what is a reasonable amount of data (...
1
vote
1answer
43 views

“Thinning” a sample until its distribution matches another distribution

I am an astronomer studying galaxies. I have observations of about 500 galaxies. For each galaxy, I can measure two quantities (call them $X$ and $Y$). I want to compare my observations to some ...
0
votes
1answer
23 views

Check to see if sample data could be uniformly distributed

I have data and I want to see if it is plausible that it comes from some uniform distribution. Is it uniformly distributed?
3
votes
1answer
191 views

Should data be split into test / training prior to descriptive statistics being carried out on it?

I have a data set that I have added to and plan to carry out some modelling with. I'm wondering whether I should split the data into test / training prior to carrying out the modelling, or if I should ...
0
votes
1answer
22 views

How can I predict movie rating based on actors/directors in it?

Okay so my goal is to create a program to predict average rating of a movie, based on release date, director and actors playing in it. Some movies have one/two directors, some movies have one/two/...
0
votes
0answers
16 views

Can SMOTE Be Used for Pure Data Augmentation and not Just Imblanaced Classes

I have learned that SMOTE can be used to deal with imbalanced class datasets. Could it also be used to create a larger dataset, preserving the original structure/distribution and thus also the ...
0
votes
2answers
41 views

Use LSTM to predict large number instead of poboabilities

I want to predict datasets like this I'm using Keras' LSTM: Not like a classification problem which returns probabilities, this model needs to output the exact value depending on input giving to it. ...
0
votes
0answers
18 views

How can i interpret my residual deviance and degrees of freedom?

Ive made a binomial GLM in r and my resid. deviance and df are rather high; Null deviance: 4396.2 on 3207 degrees of freedom Residual deviance: 3679.4 on 3205 degrees of freedom (4346 ...
0
votes
1answer
14 views

retrieving element from a list based on an element from another list [closed]

I have some data manipulation problem that I couldn't solve. I have a list that contains BIC value from GLM model, basically I have 8 models, and I put each model into a list: ...
0
votes
0answers
24 views

Prepare data for stateful/stateless/return_sequences RNN

I'm trying this Tensorflow example that using GRU to predict and generate text. Suppose there is the text "Hello World yes", and the sequence length is 5. Then we prepare the training data to be (...
0
votes
0answers
23 views

Decrease in False Positive Rate when True Positive Rate increases (ROC curve analysis)

I am trying to plot a ROC curve for classifier performance varied with size of input data set (effect of data augmentation on testing accuracy). With an increase of data size, the classifier is ...
0
votes
1answer
15 views

CMUDict normalized for word frequency

I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
0
votes
1answer
35 views

Data analysis mess! Huge data set, 2 factors with 2 levels each, but the levels are the same

To explain the hypothess a little bit: The factors would be emotional words and neutral words; the levels of both factors are left and right visual hemifields. The data is the participants response ...
0
votes
0answers
40 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
0
votes
0answers
29 views

Multiple regression data analysis

https://docs.google.com/spreadsheets/d/1czrXmJuxvwalDwNDLHIvt64QqNb8cSTahEEK8DIiQPM/edit?usp=sharing the study: "right and left visual hemifields and duration of the presented stimulus were ...
0
votes
1answer
29 views

Statistical analysis, repeated measures t test or anova? Trouble with input when running any test

I've tried using google sheets/excel, but I can't figure how to run a 2 factor without repetition ANOVA (if that's the appropriate test for this data set). I think it's the way I set the spreadsheet ...
2
votes
3answers
913 views

How to normalize data between 0 and 1?

I have seen the min-max normalization formula in several answers (e.g. [1], [2], [3]), where data is normalized into the interval $\left[0,1 \right]$. However, is there a method to normalize data ...