New answers tagged

0 votes

Lasso regression prediction on test set is predicting towards the mean of the train set?

As an initial guess, overfitting the test data set probably isn't your problem. For linear models, Statistical Learning with Sparsity (SLS) notes on page 18: Somewhat miraculously, one can show that ...
user avatar
  • 62.3k
0 votes

Is it possible to use linear regression to score a personality quiz?

This was the code that I ended up using: ans = [[], [], [], [], []] print('Through this quiz you will find out which new girl character you relate the most to.') q1 = input('\nHow would your friends ...
user avatar
  • 1
2 votes

In elastic net regularisation, will dividing the OLS term the number of observations cause misleading results when cross-validating?

It's correct that when the sample size is fixed, there is not a difference between the two statements of the optimization problem. Your demonstration in the revised question makes it clear that the ...
user avatar
  • 78.2k
0 votes

finding sparse regions in time series data

Essentially, you are trying to estimate the number of events per unit time, given the times of the events. The usual way to approach a problem like this is an algorithm called kernel density ...
user avatar
  • 1,207
0 votes

finding sparse regions in time series data

Interesting question! and interesting data. You can focus on the waiting times between baptism, so take the successive differences. Then you could make a local estimation of the mean waiting times (...
user avatar
1 vote

Wilcoxon.test in R will not calculate exact distribution due to ties (scipy.stats.wilcoxon will)

I don't have enough reputation points to add this as a comment, so although not an answer, hope it does help. I've adjusted your code a bit for convenience: ...
user avatar
  • 11
3 votes

Finding the position of the global optimum with Pytorch

The description of the particular network is not specific enough to understand what the model is, or how it works. Additionally, the terminology seems confused because datasets don't have parameters, ...
user avatar
  • 78.2k
2 votes
Accepted

Interpretation of the linear predictor of a OLS model on binomial data

The difference is that OLS is 1) not using a link function and 2) assumes a different distribution of the data. Fitting a straight line model OLS works well when you have a linear model for the mean ...
user avatar
0 votes

Analyze a time series to predict a value

There are a few approaches you can use for this type of regression model using time series data. I'll mention a couple that are commonly used in earth/environmental science applications. The first is ...
user avatar
  • 454
1 vote
Accepted

Correcting repeated measures data to display error bars that show within-subjects differences

Although I did not check the program, your logic is indeed correct and probably the best approach. This is the approach used in superb (article found here) for the R implementation. The whole process ...
user avatar
5 votes
Accepted

Multinomial logistic regression R vs Python

In case you are not sure whether a variable is being treated as categorical, you can manually one-hot-encode (=dummy coding) the categories to make sure you are using the variable as categorical. Then,...
user avatar
  • 22.3k
1 vote

Getting different AIC / BIC values for AR(2) estimation via AutoReg(2) vs ARIMA(2,0,0) through python statsmodels

When AutoReg was first included in Statsmodels in e.g. v0.12, it used the AIC definition from Lutkepohl's book New Introduction to Time Series Analysis, which ...
user avatar
  • 1,203
0 votes

Better Visualization for Correlation Plots

I always find horizontal barcharts to be very informative when looking at how all variables are correlated to a single variable
user avatar
0 votes

Inter annotator agreement when the number of annotators is not consistent across the samples

You just need to use a formulation of Fleiss' kappa (or another chance-adjusted index of categorical agreement) that allows for missing data. If you want to use Python, see the irrCAC library.
user avatar
1 vote
Accepted

Bayesian Regression Credible Intervals/Standard Deviation extremely large

The bug is not in your implementation of Bayesian linear regression but in how you sample the errors in Y. Aside: You don't cite Pattern Recognition and Machine Learning by Bishop properly. In ...
user avatar
  • 2,156
0 votes
Accepted

How to compare multiple survival scores regarding its accuracy?

Frank Harrell discusses this in a blog post. He recommends against his c-index for comparing models because it is a low-power procedure. This is because the c-index is a rank measure ... that does ...
user avatar
  • 62.3k
1 vote
Accepted

How to simulate non-gaussian stochastic paths

Posting as an answer as too long for a comment: The reason you're seeing the central limit theorem crop up here is because your returns at each time point are independent. I think what you want to do ...
user avatar
  • 126
0 votes

How to compute a 'pair confusion matrix'?

Per sklearn documentation I understand it this way. Let say there are 2 label clusterings: true and pred ...
user avatar
  • 161
2 votes
Accepted

using logsumexp in softmax

The idea of working in log-space to avoid underflow requires that the intermediate objects you use to track progress are themselves on the log-scale --- you only convert back to regular scale at the ...
user avatar
  • 94.4k
1 vote

using logsumexp in softmax

matrix is on the log scale. If matrix were not on the log scale, then you would only want to do ...
user avatar
  • 78.2k
0 votes
Accepted

How to get the True Negative Rate from this code?

The True Negative Ratio is the fraction of the correctly classified negative samples from all negative samples. You can compute it using numpy: ...
user avatar
  • 222
0 votes

evaluating scoring metrics during hyperparameter tuning

This isn't the best way to go about validating models. If you're doing hyperparameter selection, you need to validate the process of selecting the hyperparameters vis a vis a nested cross validation ...
user avatar
2 votes

transformation of a kernel density estimate to uniform distribution

Background As stated in this as well in his prior question the OP wants to perform Bayes quadrature of an expensive function against a density, which is a Gaussian mixture as the result of applying a ...
user avatar
  • 1,663
0 votes

Log-normal mean and standard deviation change after sampling

I did this in R and got very similar results to you, even when I tried sampling from the corresponding normal then exponentiating. The issue here is the scale of the values you're trying to sample. ...
user avatar
4 votes

transformation of a kernel density estimate to uniform distribution

The multivariate $d$ dimensional extension of the inverse cdf generation is incorrect, both because $F^{−1}(\cdot)$ does not exist and because $F(X)$ is not Uniform (0,1). (For instance, in the ...
user avatar
  • 91.8k
0 votes

Goodness of Fit for (presumably) poisson distributed data

So I think the Chi-square approach works OK for low mean Poisson data, since setting the bins at integer values is the logical choice. With higher means though, it becomes more tricky -- you will get ...
user avatar
  • 15.4k
1 vote

Online clustering approach

Yes, there are. An online variant of $k$-means clustering is pretty trivial and can be implemented by hand. For ready implementations of many different online algorithms, you can check Python's River ...
user avatar
  • 113k

Top 50 recent answers are included