All Questions

3
votes
2answers
2k views

Negative BIC in k-means

Probably a simple question but I'm trying to interpret BIC for k-means. I have some k-means clustering and calculating BIC gives me a negative value, with a plot something like this: ...
0
votes
0answers
5 views

Coefficient Interpretation when dependent and independent variables are percentages

I have built a linear mixed regression model with fund returns (measured in percentage ie. 0.01 denotes one percent) as the dependent variable. For the independent variables I have percentage level of ...
2
votes
0answers
5 views

How to compute confidence intervals from *weighted* samples?

Imagine we have a webserver, which serves a total of N static URLS. There are users visiting the URLs every day. At the end of each day, we have data like this: ...
0
votes
0answers
4 views

Mixing average and the Renewal Reward Theorem

I tried to solve the following problem: Buses arrive to an archeological site according to the discrete renewal process with i.i.d inter-arrival times T1, T2, T3, ... which are distributed Geo(p). ...
0
votes
0answers
6 views

Computing slope of data with correlated errors using Monte Carlo method

I have $N$ pairs of $(x,y)$ measurements. I have the (estimated) uncertainty of each $y$ value as well as the correlations between each pair of $y$ values. I want to know whether there is a ...
0
votes
0answers
2 views

How to interpret Estimated Marginal Means results

I have used the emmeans() package to calculated the difference between the difference of estimated marginal means. I did this by first calculating the EMMs of location|treatment, and then the ...
11
votes
1answer
95 views
+100

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
3
votes
3answers
4k views

Function with multiple local minima

I am trying to understand gradient descent algorithm by plotting the error vs value of parameters in the function. What would be an example of a simple function of the form y = f(x) with just just one ...
4
votes
2answers
607 views

Dealing with zeros in a poisson regression

Our code goes through multiple stages of review. I wish to use the number of defects at an earlier stage of review as a "defect density" estimate for later stages. It sometimes happens that code has ...
2
votes
2answers
352 views

Computational shortcuts/approximations for beta-binomial and beta-negative binomial CDFs

There are several simplifications that can be done so that computing cumulative distribution functions of beta-binomial and beta-negative binomial distributions, but still computing CDF as $F(x) = \...
0
votes
1answer
17 views

Basic question on ANOVA vs. T-test

I'm conducting an exercise intervention with office workers. The intervention will include Yoga classes and one to one coaching. I will be looking at the effects of the intervention on two dependent ...
0
votes
0answers
5 views

How to compare and select the better GARCH model

I am reading this article which is talking about a GARCH trading strategy. I follow the steps and tried different parameters, window size (stock historical data). From the back test, I can see that ...
0
votes
1answer
12 views

Coefficient of determination relationship?

$R^2 = \frac{SSREG}{SSTOT}$ or $R^2 = 1-\frac{SSRES}{SSTOT}$ If $X$ is the predictor random variable for science SAT and $Y$ is the predictor random variable science GPA given by equation $$\hat Y =...
1
vote
1answer
6 views

Demeaning with two (n) fixed effects in panel regressions

I would like to ask for your help concerning the following issue. In a nutshell, I would like to know, how demeaning works in a panel regression with two (separate) fixed effects. Let us assume I ...
1
vote
0answers
33 views

XGBoost and AdaBoostClassifier feature importances

I try to compare XGBoost and AdaBoostClassifier (from sklearn.ensemble) feature importances charts. From this answer: https://stats.stackexchange.com/a/324418/239354 I get know that ...
0
votes
0answers
6 views

What is the difference between a discrete and continuous rolling window?

What I have read so far suggests that a discrete rolling window is say a yearly window, calculating somethig every year, which is from 01-01/31-12 and a continuous rolling window is a window that ...
0
votes
1answer
15 views

Machine Learning problem - label over multiple lines

Currently I'm trying to work out a project where I would like to recognize movements from videos using machine learning and python. What I've done so far is extracting the x and y values of body ...
0
votes
0answers
6 views

Monthly mortality rate from annual rate formula?

My problem is I know the death rate (default rate) on an annual basis, and I would like to be able to neatly calculate the monthly rate from that. I know how to do it in Excel via Solver, but I'd ...
9
votes
2answers
3k views

Why does logistic regression produce well-calibrated models?

I understand that one of the reason logistic regression is frequently used for predicting click-through-rates on the web is that it produces well-calibrated models. Is there a good mathematical ...
1
vote
1answer
343 views

ordered probit regression with only categorical variables

I would like to know if it makes sense to run an ordered probit regression (dependent variable is ordinal with three outcomes) with only categorical explanatory variables (some are dichotomous e.g. ...
0
votes
0answers
2 views

Modified Mann-Kendall trend test on hourly data?

Hi have a ten year dataset of hourly readings of water quality parameters (pH, temperature, dissolved oxygen, turbidity and specific conductivity). Since the data is on an hourly basis, it probably ...
1
vote
0answers
9 views

What statistical test compares changes over time between two groups?

Sorry if this question doesn't make sense, but my knowledge of SPSS and statistics is very limited. However, I am trying to make sense of some data I have. I have data from two groups - let's call ...
0
votes
0answers
14 views

Inverse distribution of gaussian mixture

In one of the papers I've encountered, the authors propose a copula function $$ c(u_1, \ldots, u_d; \Theta) = \frac{\psi(y_1, \ldots, y_d; \Theta)}{\prod_{j=1}^{d}\psi_j (y_j)}$$ where $\psi(y_1, \...
0
votes
0answers
3 views

converting tanh activation output to a probability

I am trying to implement PPO for continuous action spaces so need probability of taking actions from a neural network with a tanh activation in the output layer since the action space ranges from [-2, ...
0
votes
0answers
4 views

OPTICS Reachability Curve: Scikit-learn VS PyClustering implementations

I am comparing the Reachability curves generated by the OPTICS algorithm using the Scikit-learn (new implementation) VS the PyClustering one. The hyper parameters are not the same, but it seems that ...
0
votes
1answer
11 views

Difference between “distractors” and “foils”

What is the difference between "distractors" and "foils" in experimental design in psychology?
0
votes
0answers
4 views

How to calculate accuracy of Mamdani fuzzy inference system?

I have make a Mamdani fuzzy system and know I want to test my 25% test data in that in MatLab, Firstly, I made c4.5 decision tree based on 75% of data then I extract the rules from them, and I import ...
0
votes
1answer
12 views

Correct C-Vine Configuration to emulate a predefined correlation matrix

I am trying to simulate a vine copula with a predefined dependence structure, so I can generate samples for my desired dependence structure. However, I am having trouble defining the Vine structure ...
0
votes
0answers
6 views

What is the encoding in VAEs?

The question is actually less broad than it sounds. I generally do understand how variational autoencoders work. From the encoding step we get four parts: mean $\mu$ standard deviation $\sigma$ ...
0
votes
0answers
4 views

Stochastic gradient variance reduced methods

I'm doing stochastic gradient descent on a non-convex optimization problem. Gradient corresponds to an intractable expectation which I approximate via Monte Carlo averaging. I'm trying to infer the ...
0
votes
0answers
9 views

Nonlinear Regression with Correlated Error

Assume I have two data sets: Experimentally measured data y for known values $x$. Assume the nature of the error in $y$ is random only and $\sigma_y$ is known/estimated. Simulated (deterministic) ...
1
vote
0answers
18 views

What derivative to use in Gradient boosting decision tree for a semi-supervised model

I am trying to build a semi-supervised prediction model with a Gradient Boosting decision trees. The learning phase is done using the following input: $X \in \mathbb{R}^{n\times p} $ $O(X) \in \...
0
votes
0answers
4 views

Nested cross-validation with resampling

I am aware that sckit-learn has a provision to perform nested cross-validation. I wanted to perform the same with resampling. My confusion stems from whether their example listed here performs ...
1
vote
1answer
364 views

Finding the most important factor driving the target in a regression problem

I am working on an assignment in which I am given a set of features and a (continuous) target and I need to find what is the most important factor driving the target. I thought about several ...
2
votes
0answers
7 views

Probability of having an increasing trend in normal variates

Let $x\sim N(\mu,\sigma)$ and $x_i$ is ordered instances of random variate of $x$ for $i=1...n$. What is the probability that the series is in increasing (or decreasing) order? The problem is finding ...
0
votes
0answers
3 views

Clinical Method Comparison (Bland-Altman, Deming) with large measurement range

I am trying to compare two methods using a combination of Bland-Altman plots and Deming regression. The measurement range of the assay is large (0.15 to 200) so we have a panel of samples to reflect ...
0
votes
0answers
2 views

QR decomposition used for estimating condition numbers

I have read that the QR decomposition is often used to estimate the condition number of a matrix but I don't understand... what is the benefit of using the QR decomposition for this? Is it purely ...
0
votes
1answer
22 views

How to generate a VAR(1) model?

I already written the code but something went wrong and I don't know why... here is the code. ...
0
votes
0answers
4 views

Statistics for one group pretest-posttest design and a confounding variable

I am writing a research proposal where I have one group pre test post test design and there is a variable that I hypothesize might also affect the dependent variable. What can be the best statistic to ...
0
votes
0answers
17 views

Calculation of a sample size

During the financial year, 6700 payments to suppliers were recorded. Payment to suppliers in that undertaking is subject to verification of the delivery of the goods in storage in accordance with ...
1
vote
1answer
18 views

Hierachical Random Mixed effect sizes

When using a mixed effect model the rule of thumb seems to be that you need at least 5 levels to use a random factor . Is this still True when you have a hieachical model. i.e A - 4 level factor B - ...
1
vote
1answer
331 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
0
votes
0answers
6 views

Create TensorFlow Dataset with multiple data sources

I am trying to create a TensorFlow Dataset by joining data from 2 different sources. I am trying to replicate the dataset creation from the following paper: https://arxiv.org/abs/1809.01984 ...
0
votes
0answers
8 views

Clustering Algorithms- Sample Size

I have a small sample (290 observations) of categorical data and I am trying to decide on what clustering method to use. I am considering k-medoids, hierarchical clustering and Latent Class Clustering....
0
votes
0answers
6 views

understanding time-homogeneous markov chain

Could anyone make me understand the definition here 1 on page 7 definition 2.25, I quite do not understand the notation $(P(a))(A)$, what does this mean? Also, is $P(a, A)$ a probability measure from ...
0
votes
0answers
5 views

Descriptive statistics for complementary random variables

I have two datasets, each one contains 157 images of microscopic field processed with a different preparation tecnique. My task is to show if there any differences like cells density, distribution and ...
1
vote
0answers
12 views

Why using RMSE as loss function in logistic regression takes non convex form but doesn't in linear regression?

I am taking this deep learning course from Andrew NG. In the 3rd lecture of 2nd week of the first course, he mentions that we can use RMSE for logistic regression as well but it will take a nonconvex ...
0
votes
0answers
9 views

Statistical power for AB test

I am planning to run an AB test in Jul-Aug 2019 with 20% size of B group. Before launching, I want to estimate the power in order to make sure I set the correct sample size and make sure the test is ...
0
votes
0answers
11 views

Predicting a seasonal signal

Consider a noisy signal $X$ and a time variable $T$ as seen in Fig. 1. This is a simulated generalization of data I am currently facing with the goal of making predictions for future time points $t_f$....
1
vote
0answers
23 views

missForest Data imputation vs. MICE using RF as imputation method?

Is the missForest package a special case of MICE using Random Forest as imputation (for just a single imputation)? The missForest algorithm is described here: https://academic.oup.com/bioinformatics/...

15 30 50 per page