New answers tagged

0 votes
Accepted

Data with a lot of zeroes

The keyword you are after is zero-inflated models. You can find many questions concerning them under the zero-inflation tag. If your data is counts, a possible candidate would be zero-inflated Poisson ...
  • 123k
2 votes

How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy?

I can do this with zero data points. My estimate is 42. Say the true distribution is around 0 with a deviation of 10 (but it can be different as well). Given that, if you define the accuracy as 90%, ...
17 votes

How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy?

We can't tell you. It depends on your situation and how easy prediction is in your situation. How many coin tosses do you need to observe before you can predict the next one with 90% accuracy? Related:...
0 votes

What is the correct method for training NLP models with augmented data?

With only 50 samples, I think there is very little one can do. I would suggest training BERT on the "open data" and treating the 50 samples as our hold-out set. I would advise against ...
  • 37.6k
3 votes

How to find the strongest correlation with big data in R?

I have some experience with this question, but it may not be the best. Suppose the correlation coefficient is stored in corr_df(a data frame [9000,1]) Try this code: ...
  • 31
0 votes

Conditions to Select Pairwise Deletion

First and foremost, you need to determine the mechanism through which your data is missing. It appears from your post that you are assuming that your data is MCAR - which would mean that listwise/...
0 votes

Generating synthetic time series data with limited data

Generating extra data can sometimes help you train a neural net. (Adding noise layers is equivalent to generating extra data, since it fuzzes the existing data.) But you still need to have enough data ...
  • 1,335
1 vote

Generating synthetic time series data with limited data

Data sparsity should result in uncertainty about the model - it's functional form, presence of outliers, periodicity, etc. 500 rows of data can reliably estimate some stable series; and functions can ...
  • 55.1k
1 vote

calculate an equation based on conditional existing data

I'm not sure what is doing your R code (I'm not very familiar with the tidiverse philosophy), but I think you are after this <...
  • 3,734
0 votes
Accepted

how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?

The following code in R was my attempt of to solve the problem of finding an algorithm to convert a vector of values to a new vector of values with mean of 0 and range of [-1, 1]. After one pass, it ...
1 vote
Accepted

Prudent to reduce data size for the sake of model performance?

This sounds like a standard case of model drift, in other words, the relationship between your predictors and your outcome change over time so that a model trained on old data might not make good ...
  • 7,044
0 votes

When is an unbalanced dataset large enough for calculating a decision threshold?

1% of $10^6$ is 10,000, which is a pretty good sample size. You could be having problems because 10,000 samples isn't enough to fill the feature space you're dealing with. If I had to guess, though, I'...
  • 171
1 vote

When is an unbalanced dataset large enough for calculating a decision threshold?

This is fun and not a trivial problem, I would attack as follows: 1. Find formulas where the variance of the confidence intervals of the statistic of interest is a function of $n$, the sample size ...
  • 37.6k
1 vote

how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?

I believe the intended application of this preprocessing technique is to first standardize the data with mean zero then change its range to $[-1,1]$ for a neural network input. To quote this referece :...
  • 123
1 vote

Machine Learning Models for predicting probability

When you are faced with a problem like this, an early step is to try to access the original counts that generated the probabilities. An easy solution arises if you have the number of attempts by which ...
  • 38.9k
0 votes

How are artificially balanced datasets corrected for?

I know this is a late reply and you probably do not need this answer, however I believe that I can add valuable information for future Pattern Recognition and Machine Learning by Christopher Bishop ...
0 votes

Salary of a group of people is continuous or discrete

Go back to the definition of what is meant by a discrete and what is meant by a continuous random variable. It also helps to know that the real line is dense and asking whether it is possible that ...
1 vote
Accepted

Should I use long-format or recode the condition column?

It's hard to tell a priori if a long format is better than a wide format since it really depends on what are your aims. Anyway, for modelling purposes, it may be more useful to have it in a long ...
  • 3,734
1 vote

What happens if I fit my model on the same training dataset multiple times?

TECHNICALLY, IT IS POSSIBLE, BUT SUCH A DESCRIPTION IS A STRETCH. It depends on the type of model and, critically, how it is trained. For a neural network, you could start again from where the network ...
  • 38.9k
1 vote
Accepted

Using SVM for subsets

The answer depends on exactly what you want to do,i will just give you some pointers. If you want to combine predictions from multiple models, than that is called ensemble learning, in your case it ...
  • 6,943

Top 50 recent answers are included