Recent Questions - Cross Validated most recent 30 from stats.stackexchange.com 2022-05-24T03:36:37Z https://stats.stackexchange.com/feeds https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/576389 0 What method can be used to reformulate this kind of Chance Constraint CZzzzzzzzz https://stats.stackexchange.com/users/358936 2022-05-24T03:24:27Z 2022-05-24T03:24:27Z <p><a href="https://i.stack.imgur.com/BGryP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BGryP.png" alt="Chance Constraint" /></a></p> <p>In the above chance constraint, x and y are binary decision variables, <a href="https://i.stack.imgur.com/36X8L.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/36X8L.png" alt="enter image description here" /></a> is the integer random variable with the range of [4,20], and the right side of the constraint is the probability level.</p> <p>And I want to know if there is any approach to reformulate this chance constraint? Thanks.</p> https://stats.stackexchange.com/q/576386 0 Non-Dirichlet Prior for $Cat(\theta)$ parameter that can tractably be integrated out (for Latent Dirichlet Analysis)? cataclysmic https://stats.stackexchange.com/users/96507 2022-05-24T02:09:59Z 2022-05-24T02:34:57Z <p>In LDA Topic Models, it is standard to 'integrate out' the <span class="math-container">$\theta$</span> parameter, which contains a document's Categorical probabilities of drawing each topic. Because the prior for <span class="math-container">$\theta$</span> is Dirichlet, integrating out' <span class="math-container">$\theta$</span> leaves just a simple term <span class="math-container">$const.*\Gamma(x+\alpha)$</span> that is a function of the Dirichlet hyperparameter <span class="math-container">$\alpha$</span> and the count of words <span class="math-container">$x$</span> assigned to a particular topic (the derivation is in eqs. 20-25 of this <a href="https://lingpipe.files.wordpress.com/2010/07/lda3.pdf" rel="nofollow noreferrer">paper</a>).</p> <p><strong>QUESTION</strong> In my case, I would prefer to replace the Dirichlet prior on <span class="math-container">$\theta$</span> with something simpler---either an improper uniform distribution over all <span class="math-container">$\theta$</span>, or perhaps a prior that just penalizes the count of nonzero entries in the <span class="math-container">$\theta$</span> vector of probabilities.I <strong>don't</strong> need to obtain a normalized marginal posterior after integrating out <span class="math-container">$\theta$</span>---I just want a clean expression for what it is proportional to. <em>Are there choices of priors (even trivial ones) other than Dirichlet for which integrating out' a categorical parameter <span class="math-container">$\theta$</span> is tractable?</em></p> <p>Apologies if the answer is obvious---I'm in a CS department but Bayesian stats is (clearly) not by subfield.</p> <p><strong>Background/for reference</strong> LDA generates a set of <span class="math-container">$M$</span> documents that each contain <span class="math-container">$N_m$</span> words <span class="math-container">$w_{m,n}$</span>. Each document has a document-specific Categorical distribution <span class="math-container">$Cat(\theta_m)$</span> over <span class="math-container">$K$</span> topics, with prior <span class="math-container">$\theta_m \sim Dirichlet(\alpha)$</span>. We generate each document's words by repeatedly drawing a topic <span class="math-container">$t_{m,n}$</span> from <span class="math-container">$Cat(\theta_m)$</span>, then drawing a word <span class="math-container">$w_{m,n}$</span> from this topic <span class="math-container">$Cat(\phi_{t_{m,n}})$</span>. The joint probability is <span class="math-container">\begin{align} P(\mathbf{w}, \mathbf{t}, \mathbf{\theta}, \mathbf{\phi} \ | \alpha, \beta) = \prod^M_{m=1} \prod^{N_m}_{n=1} Cat (t_{m, n_m} | \theta_m) \times \prod^M_{m=1} Dir(\theta_d|\alpha) \times \prod^M_{m=1} \prod^{N_m}_{n=1} Cat (w_{n,m} | t_{n,m}, \phi_t) \times \prod^{K}_{k=1} Dir(\phi_k|\beta) \\ \end{align}</span> We want other priors for <span class="math-container">$\theta$</span> that let us derive: <span class="math-container">\begin{align} P(\mathbf{t}, \mathbf{\phi} \ | \mathbf{w}, \alpha, \beta) \underbrace{\propto} \int \underbrace{ \prod^M_{m=1} \prod^{N_m}_{n=1} Cat (z_{m, n_m} | \theta_m) \times \prod^M_{m=1} \mathbf{Pr}(\theta_m|\alpha) d\theta} \times \prod^M_{m=1} \prod^{N_m}_{n=1} Cat (w_{n,m} | t_{n,m}, \phi_t) \times \prod^{K}_{k=1} Dir(\phi_k|\beta) \end{align}</span></p> https://stats.stackexchange.com/q/576385 0 Why do we need to Define "Valid" State Transitions in a Multi-State Model? stats_noob https://stats.stackexchange.com/users/77179 2022-05-24T02:01:25Z 2022-05-24T02:01:25Z <p>I was watching this video (<a href="https://www.youtube.com/watch?v=Wy-WmY6x4tg" rel="nofollow noreferrer">https://www.youtube.com/watch?v=Wy-WmY6x4tg</a>) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of &quot;States&quot; within the model and which &quot;Transitions&quot; between these States are &quot;Valid&quot;:</p> <p><a href="https://i.stack.imgur.com/28XRJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/28XRJ.png" alt="enter image description here" /></a></p> <p>Based on this diagram, I can understand that for the Multi-State Model to work, (obviously) the user must specify how many States are in the model, <strong>but I am not sure why it is necessary for the user to specify which Transitions are &quot;Valid&quot;</strong>. In the above example, there are 3 States - The First State can transition to the Second State or the Third State, the Second State can only transition to the Third State, and the Third State is an &quot;absorbing&quot; State (i.e. no transitions are allowed).</p> <p>This transition structure has been codified into the generator matrix &quot;Q&quot; - but I still do not understand why this transition structure is necessary. For example, suppose we assume a model in which &quot;any state can be reached from any state&quot; - but based on the observed data, there is only evidence of certain transitions happening (e.g. like the above diagram): <strong>When it came time to estimate the entries of the Q Matrix, wouldn't the entries corresponding to transitions between states that was never observed in the data be estimated as 0 (regardless if they were codified as &quot;valid transitions&quot; into the transition structure) ?</strong></p> <p>Thus, if the Q matrix is specified as in the diagram - or if the Q matrix is specified in such a way, such that there are no 0's : For the same observed data, wouldn't the estimated values of the entries in both matrix specifications be identical?</p> <p>Thanks!</p> https://stats.stackexchange.com/q/576384 0 sample size calculation based on survival rate at 2-year alittleboy https://stats.stackexchange.com/users/14156 2022-05-24T01:56:41Z 2022-05-24T01:56:41Z <p>I encountered a paper which shows the calculation of n=1240 patients:</p> <p><a href="https://i.stack.imgur.com/7MUmu.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7MUmu.jpg" alt="enter image description here" /></a></p> <p>Because the 2-year PFS is very high for the treatment (81%) and control (73%), which means that the median PFS is not reached, the sample size is probably calculated based on this 2-year PFS difference... I think I need to calculate the hazard rate from proportion surviving until a given time T0 (here, 2 years): The proportion surviving is transformed to a hazard rate using the relationship h = –ln(S(T0)) / T0. With T0=24 months, how to calculate the hazard &quot;h&quot; in R? Thanks!</p> https://stats.stackexchange.com/q/576383 0 Defining an independent variable? James Zouave https://stats.stackexchange.com/users/358935 2022-05-24T01:51:17Z 2022-05-24T01:51:17Z <p>I am conducting a small study where one class is taught similar (but not the same) concepts over two separate lessons. The first lesson, the teacher will deliver the lesson to the class. The second lesson, the students will watch instructional videos without teacher intervention. A post test is conducted after each lesson to see if the way the content was delivered showed an increase in student learning. So from the reading I have done I would think:</p> <p>I have a control group: explicit instruction from the teacher</p> <p>I have an intervention group: content delivered through instructional videos</p> <p>Student learning is my dependant variable</p> <p>My independent variable consists of two levels, the control and the intervention ie: the way the lesson is delivered.</p> <p>Does this also mean my independent variable is actually the two groups - the control group and the intervention group?</p> <p>Just trying to get my terminology right so I don’t look like a total twit when proposing this :)</p> https://stats.stackexchange.com/q/576382 0 Decision tree feature importances on a test set ajanss https://stats.stackexchange.com/users/358933 2022-05-24T01:27:49Z 2022-05-24T01:27:49Z <p>Many tree based models come with a built in feature importance method usually based on impurity decrease (ex. sklearn RandomForest). Would it be possible to calculate feature importances on a test set (out-of-sample data), in other words the per-feature contribution to impurity decrease with respect to an external dataset? If so does anyone know of any implementations? It seems fairly straightforward and more robust than the usual approach, I just haven't seen this done before.</p> https://stats.stackexchange.com/q/576380 0 Can I tell the significance of a treatment using linear regression? Blossom https://stats.stackexchange.com/users/189159 2022-05-24T01:24:34Z 2022-05-24T01:24:34Z <p>I have 10 stores. 5 of the stores were given a new promotion that was meant to increase the number of units purchased in an order (for each order we look at the number of units).</p> <p>If being give the promo was completely randomly assigned to each order, I could do a t-test and see if the promo had a positive effect with statistical significance. Since it is at the store level, I can not do that.</p> <p>I want to create a linear regression at the order level where the dependent variable is number on units, one of the independent variables is whether or not the store had a promo and all the other independent variables were confounding factors (day of week, traffic to store) etc. If I do this can I look at the p-value of the independent variable of whether or not the store had a promo and use that as my &quot;significance&quot; (instead of a t-test)?</p> https://stats.stackexchange.com/q/576378 0 Moment Generating Function of a Product of Two Random Variables Connor Cleaver https://stats.stackexchange.com/users/358932 2022-05-24T01:16:59Z 2022-05-24T01:16:59Z <p><a href="https://i.stack.imgur.com/rwFvU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rwFvU.png" alt="enter image description here" /></a> I know this question requires condition on U and then using condtional expection to solve for the moment generating function but cannot simplify the final MGF into a clean distribution. I did solve for W - Gamma(1-p, lamba) but not sure if this is correct. Any help would be greatly appreciated.</p> https://stats.stackexchange.com/q/576375 0 Can reparameterization always yield tight CRBs? Christian Chapman https://stats.stackexchange.com/users/10461 2022-05-24T00:51:56Z 2022-05-24T01:10:24Z <p>In general, given a family of distributions parametrized by <span class="math-container">$\theta$</span>, the Cramer-Rao bounds on variance for a (biased or unbiased) estimator of <span class="math-container">$\theta$</span> are unattainable. In the proof for the 1D case, the looseness is introduced by a Cauchy-Schwarz inequality.</p> <p>Is there a reparameterization of the family of distributions which yields tight CRLBs, or is the looseness from a different reason?</p> https://stats.stackexchange.com/q/576373 0 Z-scores and comparisons of means MQ99 https://stats.stackexchange.com/users/347370 2022-05-24T00:42:01Z 2022-05-24T00:42:01Z <p>I am interested in comparing mean protein levels (dependent variable) across four groups (independent variable), while controlling for covariates/ confounders. I will perform this (essentially ANCOVA) analysis for several proteins, however the unit of measurement is arbitrary. It cannot be used to compare the levels of different proteins. Thus, I will z-score transform each protein separately and perform the ANCOVA to compare group differences for that specific protein.</p> <p>In the end, I will plot the results on a single graph. Can z-scores be used for comparing mean differences? Does it also make sense to plot the mean z score for each group, or differences in z-score means?</p> <p>Thank you in advance</p> https://stats.stackexchange.com/q/576372 0 Is multilevel modeling necessary: only 1 member of each group at individual level with group-level moderator BLS22 https://stats.stackexchange.com/users/355092 2022-05-24T00:32:04Z 2022-05-24T03:33:31Z <p>I have data where I am examining the moderating effect of a group-level variable (performance) on the relationship between two individual-level variables, X (2 experimental conditions) and Y (attitude change score calculated from a within-person repeated measure); i.e., my research question is: Does group performance moderate the effect an experimental manipulation has on group leaders' attitude change? (Attitude change = attitude at time 2 - attitude at time 1)</p> <p>Thus, the sample I'm examining at the individual level are group leaders (one leader per group), so they should not share variance with one another on the group-level variable (an objective group performance measure) and I'm not including any other group members in the analysis. Am I able to conduct a simple moderation analysis (as though I had all individual-level variables), or is there a multilevel approach more appropriate for this situation?</p> https://stats.stackexchange.com/q/576371 0 How would I separate a data set into a large number of pairs? James McL. https://stats.stackexchange.com/users/358930 2022-05-24T00:27:17Z 2022-05-24T02:43:11Z <p>I have <span class="math-container">$n$</span> observations in <span class="math-container">$\mathbb{R}^2$</span> (where <span class="math-container">$n$</span> is large and even), representing points in physical space. What I am trying to do is to produce a set of <span class="math-container">$\frac{n}{2}$</span> pairs, such that the distances between paired points are minimised.</p> <p>My first thought was that this seems to be a clustering problem. I first considered k-means clustering; indeed I even found a module for Python which performs <em>constrained</em> k-means clustering, which allows the cluster sizes to be fixed. However, while I found that module to be efficient (and produces good results) if the clusters are relatively large, when I attempted to use the algorithm to split the data into <span class="math-container">$\frac{n}{2}$</span> clusters with two members in each cluster, it was incredibly slow. It also produced results which seemed far from optimal, even after initialising the algorithm with many different random starting points and selecting the best result. Therefore, I'm wondering whether a different approach may be required - and whether this is even a clustering problem at all. I'd appreciate any advice on a potential alternative way to solve this problem.</p> https://stats.stackexchange.com/q/576364 0 Interpreting the Explanatory Effect of Dropping Non-Significant Variables bshelt141 https://stats.stackexchange.com/users/121027 2022-05-23T22:54:02Z 2022-05-24T01:23:41Z <p>I have a data set comprised of an overdispersed Poisson response variable, two standardized continuous features, and a categorical feature. I am using deviation coding on my categorical feature and I am manually applying the deviation coding, because I want to visualize how the removal of non-significant categories from effect the coefficients of all the other model inputs.</p> <p>As you can see in the reproducible example below, removing the non-significant categories from the model makes the <code>1 WHITE</code> category go from significant to not significant, using <code>p-value &lt; .05</code> as the standard for variable significance.</p> <p><strong>Should I consider the <code>1 WHITE</code> category as significant or not, since it is significant when including non-significant variables, but becomes non-significant when the other non-significant variables are removed.</strong></p> <p><strong>My goal at this point is explanation, not prediction.</strong></p> <p><a href="https://i.stack.imgur.com/Io7dC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Io7dC.png" alt="enter image description here" /></a></p> <pre><code>library(dplyr) library(MASS) library(ggplot2) # Download data dat_loc &lt;- &quot;https://github.com/b-shelton/stack_questions/blob/main/explanatory_effect_change_example.csv?raw=true&quot; dat &lt;- read.csv(dat_loc) # Manually create deviation coding features races &lt;- array(sort(unique(dat<span class="math-container">$race))) contrasts(dat$</span>race) &lt;- contr.sum(length(races)) race_contrasts &lt;- contrasts(dat$race) for (i in c(1:(length(races)-1))) { focal_race &lt;- races[i] contrast_df &lt;- data.frame(&quot;race&quot; = row.names(race_contrasts)) contrast_df[, focal_race] &lt;- race_contrasts[, i] dat &lt;- dplyr::inner_join(dat, contrast_df, by = &quot;race&quot;) print(paste0(&quot;Created contrast-encoded variable: &quot;, focal_race)) } # Negative Binomial model using all variables mod1 &lt;- glm.nb(response ~ conditions + doses_per_month +1 WHITE +2 HISPANIC +3 BLACK +4 ASIAN +5 NATIVE HAWAIIAN +6 AMERICAN INDIAN +7 OTHER +8 DECLINED , data = dat) summary(mod1) #Call: #glm.nb(formula = response ~ conditions + doses_per_month + 1 WHITE + # 2 HISPANIC + 3 BLACK + 4 ASIAN + 5 NATIVE HAWAIIAN + # 6 AMERICAN INDIAN + 7 OTHER + 8 DECLINED, data = dat, # init.theta = 0.3100583261, link = log) # #Deviance Residuals: # Min 1Q Median 3Q Max #-1.9391 -0.6595 -0.5773 -0.4190 5.4887 # #Coefficients: # Estimate Std. Error z value Pr(&gt;|z|) #(Intercept) -1.50614 0.16753 -8.990 &lt; 2e-16 *** #conditions 0.29339 0.02585 11.351 &lt; 2e-16 *** #doses_per_month 0.30618 0.02554 11.987 &lt; 2e-16 *** #1 WHITE 0.34367 0.17429 1.972 0.048627 * #2 HISPANIC 0.31243 0.17062 1.831 0.067076 . #3 BLACK 0.70055 0.17746 3.948 7.89e-05 *** #4 ASIAN -0.74274 0.19497 -3.810 0.000139 *** #5 NATIVE HAWAIIAN 0.24217 0.63144 0.384 0.701334 #6 AMERICAN INDIAN -1.38039 0.99884 -1.382 0.166976 #7 OTHER 0.75020 0.18799 3.991 6.59e-05 *** #8 DECLINED -0.34630 0.63741 -0.543 0.586926 #--- #Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 # #(Dispersion parameter for Negative Binomial(0.3101) family taken to be 1) # # Null deviance: 6117.7 on 9999 degrees of freedom #Residual deviance: 5322.6 on 9989 degrees of freedom #AIC: 14154 # #Number of Fisher Scoring iterations: 1 # # # Theta: 0.3101 # Std. Err.: 0.0138 # # 2 x log-likelihood: -14130.0440 </code></pre> <p>Using the summary above, I create a second model that only includes variables that were significant to the first model, and then I compare the Incidence Rate Ratio 95% ranges.</p> <pre><code># Creating a second model, # removing variables proving non-significant from the first model mod2 &lt;- glm.nb(response ~ conditions + doses_per_month +1 WHITE #+2 HISPANIC +3 BLACK +4 ASIAN #+5 NATIVE HAWAIIAN #+6 AMERICAN INDIAN +7 OTHER #+8 DECLINED , data = dat) # Create a table that compares the Incidence Rate Ratios between models 1 and 2 conf_mod1 &lt;- data.frame(exp(confint(mod1))) conf_mod1<span class="math-container">$coefficient &lt;- exp(coefficients(mod1)) conf_mod1$</span>feature &lt;- row.names(exp(confint(mod1))) conf_mod1<span class="math-container">$lower2.5 &lt;- conf_mod1[,1]-1 conf_mod1$</span>upper2.5 &lt;- conf_mod1[,2]-1 conf_mod1<span class="math-container">$coeff_impact &lt;- conf_mod1$</span>coefficient-1 conf_mod1 &lt;- conf_mod1[c(&quot;feature&quot;, &quot;lower2.5&quot;, &quot;upper2.5&quot;, &quot;coeff_impact&quot;)] conf_mod1$model &lt;- &quot;mod1&quot; conf_mod2 &lt;- data.frame(exp(confint(mod2))) conf_mod2<span class="math-container">$coefficient &lt;- exp(coefficients(mod2)) conf_mod2$</span>feature &lt;- row.names(exp(confint(mod2))) conf_mod2<span class="math-container">$lower2.5 &lt;- conf_mod2[,1]-1 conf_mod2$</span>upper2.5 &lt;- conf_mod2[,2]-1 conf_mod2<span class="math-container">$coeff_impact &lt;- conf_mod2$</span>coefficient-1 conf_mod2 &lt;- conf_mod2[c(&quot;feature&quot;, &quot;lower2.5&quot;, &quot;upper2.5&quot;, &quot;coeff_impact&quot;)] conf_mod2$model &lt;- &quot;mod2&quot; # Combine IRRs for models 1 and 2 conf_mods &lt;- rbind(conf_mod1, conf_mod2) # Order variables for plot feature_order &lt;- conf_mod1 %&gt;% mutate(sig = ifelse((lower2.5 &lt; 0 &amp; upper2.5 &lt; 0) | (lower2.5 &gt; 0 &amp; upper2.5 &gt; 0), 1, 0)) %&gt;% arrange(desc(sig), desc(upper2.5)) conf_mods<span class="math-container">$feature &lt;- factor(conf_mods$</span>feature, levels = feature_order<span class="math-container">$feature) conf_mod1$</span>feature &lt;- factor(conf_mod1<span class="math-container">$feature, levels = feature_order$</span>feature) # Plot comparisons/outcomes ggplot(conf_mods) + geom_linerange(aes(x=feature, ymin=lower2.5, ymax=upper2.5, colour=model, group=model), size=6, position = position_dodge(.8)) + #geom_point(aes(x=feature, y=coeff_impact)) + geom_hline(yintercept=0, color=&quot;red&quot;, linetype=&quot;dashed&quot;) + theme_bw() + theme(axis.text.x=element_text(angle=45, hjust=1)) + labs(title = &quot;Negative Binomial Model Incident Rate Ratio Range (95%) Comparisons\nComplete Model (mod1) vs Reduced Model (mod2)&quot;, x=&quot;&quot;, y=&quot;Incidence Rate Ratio Range&quot;) </code></pre> https://stats.stackexchange.com/q/576360 1 Does a CNN always learn a latent space? mesllo https://stats.stackexchange.com/users/58785 2022-05-23T22:06:48Z 2022-05-24T01:29:08Z <p>In general, a latent space is a structure of reduced dimensionality than that of the input space where points on this space share resemblance the closer they are to each other.</p> <p>This <a href="https://towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d#:%7E:text=The%20latent%20space%20is%20simply,representations%20of%20data%20for%20analysis." rel="nofollow noreferrer">article</a> also refers to the layers of a convolutional neural network as a latent space (see the diagram). Some CNNs essentially squash an input image into a compressed representation too with appropriate use of convolutional and pooling layers.</p> <p>What I want to understand is: can we really look at the CNN's layers as the same kind of latent space representation as described in the former definition, i.e are the feature representations generated by these layers also like points on a latent space? I cannot seem to understand this.</p> <p>If so, where can I find some good literature where a latent space representation is explained like this on a more general level for CNNs?</p> https://stats.stackexchange.com/q/576289 -2 Performing Fisher exact test or prop.test with counts of non-integer numbers? Marwah Al-kaabi https://stats.stackexchange.com/users/338054 2022-05-23T12:00:15Z 2022-05-24T01:57:29Z <p>If most of my count data are have non-integer numbers like (45.5,88.5), can I still use prop.test or fisher test? how that affect the results? This is how I calculated the counts: Lets say I have 100 sample, 50 of them have X and 39 of them have Y and 11 of them have both X and Y. So the count of X= 55.5, Y= 44.5. Please I need you help.</p> https://stats.stackexchange.com/q/576282 0 How do you test if the average of a population is the same as the variance of the same population? Fanta https://stats.stackexchange.com/users/211596 2022-05-23T10:56:07Z 2022-05-24T01:27:06Z <p>What can be a statistical test to find out if a population has the mean equal to its own variance? I.e. Mean(X)=Var(X)?</p> <p>I am interested in it because Poisson regression makes the assumption that the mean of the predicted variable is the same as the variance of the predicted variable, i.e. lambda = E(Y) = Var(Y) where lambda is the parameter for the Poisson distribution for Y.</p> https://stats.stackexchange.com/q/576244 1 Measuring value of free users sachinruk https://stats.stackexchange.com/users/29537 2022-05-23T02:28:17Z 2022-05-24T01:02:34Z <p>I was wondering if the following scenario had a word for it (eg. Survival analysis...).</p> <p>Suppose we have a company that has pro users (pay$10/ month) and free users. I want to calculate the worth of the free users to the company. Despite not spending any money with the company, they make the product more popular and therefore bring in more revenue.</p> <p>Currently we frame the value of a user as a proportion of the <code>$10</code> as follows: <span class="math-container">$$value = 10 \frac{activeness}{max\_activeness}$$</span> where activeness is calculate as the number of logins per month. Logic being the more active they are the more valuable they are to the company.</p> <p>While this is a start, I was wondering if there was a more statistically rigorous way of defining long term value of free users. What other data would be necessary to frame this question better?</p> https://stats.stackexchange.com/q/565919 1 Plotting Log Likelihood Vinny https://stats.stackexchange.com/users/350629 2022-02-27T01:38:41Z 2022-05-24T02:06:15Z <p>It was suggested I ask this here instead of Stack Overflow.</p> <p>I am trying to plot the negative log likelihood of an exponential distribution. I am not getting how I am supposed to think of it. The equation for negative log likelihood is provided...</p> <p><span class="math-container">$$l(\lambda; x_1, \dots, x_n) = n \ln(\lambda) - \lambda \sum_{j=1}^n x_j$$</span></p> <p>I am not getting how I am supposed to plot it when there is only one output. How is this used in relation to gradient descent? Is this function performed multiple times and the rate parameter changes? I have looked online and cannot find anything I can really understand.</p> https://stats.stackexchange.com/q/544311 2 Why does Hutchinson's trace estimator reduce computation complexity? jzin https://stats.stackexchange.com/users/266255 2021-09-11T10:16:24Z 2022-05-24T02:05:58Z <p>Given a matrix <span class="math-container">$A</span>, we want to compute its trace, in which we can use a trick name <a href="https://blog.shakirm.com/2015/09/machine-learning-trick-of-the-day-3-hutchinsons-trick/" rel="nofollow noreferrer">Hutchinson's trace estimator</a> <span class="math-container">\begin{align} tr(A) = tr(A\mathbb{E}[\epsilon \epsilon^T])=\mathbb{E}[tr(A \epsilon \epsilon^T)]=\mathbb{E}[tr(\epsilon^T A \epsilon)]=\mathbb{E}[\epsilon^T A \epsilon], \end{align}</span> where <span class="math-container">\epsilon \sim \mathcal{N}(0,1)$</span> is a standard normal distribution. Then the trace of <span class="math-container">$A$</span> can be estimated using Monte Carlo sampling.</p> <p>Some literature said, by transforming the computation into quadratic form, the computation complexity of the trace calculators can be reduced. I can't understand it. Why can such a stochastic quadratic way reduce the complexity of trace computation?</p> https://stats.stackexchange.com/q/499423 7 Normalized Cross Entropy ved https://stats.stackexchange.com/users/29770 2020-12-05T03:09:35Z 2022-05-24T02:01:44Z <p>In this paper: <a href="http://quinonero.net/Publications/predicting-clicks-facebook.pdf" rel="nofollow noreferrer">http://quinonero.net/Publications/predicting-clicks-facebook.pdf</a>, the authors introduce a metric called Normalized Cross Entropy (NCE):</p> <p><span class="math-container">$$\text{NE} = \frac{-\frac{1}{N} \sum_{i=1}^n(y_i\log(p_i) + (1-y_i)\log(1-p_i))}{-(p\log(p) + (1-p)\log(1-p))}$$</span></p> <p>where <span class="math-container">$p_i$</span> is the estimated <span class="math-container">$P(y_i=1)$</span> and <span class="math-container">$p=\sum_i y_i/N$</span> is the &quot;average&quot; probability over the training set. Note that here, unlike the paper, I've assumed <span class="math-container">$y_i \in \{0,1\}$</span> to give the numerator the more familiar looking form of binary cross entropy.</p> <p>The authors claim that the normalization, i.e. dividing the cross entropy in the numerator by the cross entropy for a model that predicts <span class="math-container">$p$</span> for every example, is because the closer <span class="math-container">$p$</span> is to 0 or 1, the easier it is to achieve a better log loss (i.e. cross entropy, i.e. numerator). Can someone explain why this is true?</p> https://stats.stackexchange.com/q/459135 0 How to get odds ratio using glmnet? user1828605 https://stats.stackexchange.com/users/174870 2020-04-08T04:47:50Z 2022-05-24T03:05:10Z <p>I ran three regularization methods, lasso, ridge, and elastic net. Lasso was able to get the best accuracy, so I'm selecting it. Is there a way to calculate odds ratio from the coefficients? Does it make sense to do it in glmnet?</p> <p>I took the following steps: </p> <pre><code>train.control &lt;- trainControl(method = "repeatedcv", number = 10, repeats = 5, allowParallel = T, verboseIter = T) set.seed(1234) lasso_model &lt;- train(traget~ ., trainTransformed[,-2], trControl = train.control, method = "glmnet", tuneGrid = expand.grid(alpha = 1, lambda = seq(0.0001, 0.05, length = 5)), family = "binomial") </code></pre> <h3>Plot and predict the model</h3> <pre><code>plot(lasso_model$finalModel, xvar = "lambda", label = T) plot(lasso_model$finalModel, xvar = "dev", label = T) plot(varImp(lasso_model, scale = F)) p.lasso.pred &lt;- predict(lasso_model, testTransformed) p.lasso.pred.cm &lt;- confusionMatrix(p.lasso.pred, testTransformed$BMK_R_Derailment, mode = "prec_recall") </code></pre> <p>Now, all tutorials that I've read stops at this point. I'm really confused as to whether to stop here, or take the features from lasso with coefficients > 0 and run logistic again to get the odds ratio for the coefficients. </p> <p>And I also did that. However, most of the variables are not significant (which is fine). Then should I select the variables that are significant and do the regular (step-wise - not sure if I should do this) logistic regression? or leave the model as is because lasso produced those features?</p> https://stats.stackexchange.com/q/361928 2 identify level shifts in a time series user217451 https://stats.stackexchange.com/users/217451 2018-08-13T05:15:40Z 2022-05-24T02:21:09Z <p>I have a time series as follows:</p> <p><a href="https://i.stack.imgur.com/uisDJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uisDJ.png" alt="enter image description here"></a></p> <p>I want to identify the locations of level shifts in this time series. Are there R packages available to do the job?</p> https://stats.stackexchange.com/q/329639 3 Why does changing factor level order of a categorical predictor affect significance of continuous predictors in a linear model with interactions? user195995 https://stats.stackexchange.com/users/195995 2018-02-20T14:09:31Z 2022-05-24T00:25:18Z <p>Although I am asking this question using <code>R</code>, I imagine it is potentially applicable more broadly in linear modelling, so I think general explanations are also really helpful.</p> <p>I have a dataset with a binomial response, two continuous predictors, a categorical predictor, and a random effect of block. I have a model structure like this:</p> <p><code>response~cont1 + cont2 + cat + cont1:cont2 + cont1:cat + cont2:cat + (1|block)</code></p> <p>After running this model once, I decided that I wanted to reorganize the levels of the categorical predictor so that a different one was alphabetically first. <code>R</code> and <code>lme4</code> pick the alphabetically-first level of a categorical predictor to incorporate in the intercept, and for the second model, I needed the last one of my levels (alphabetically) to be first.</p> <p>In looking at the results of both of these models, I noticed that the significance of the main effects of my continuous predictors (as well as their slopes) changed when I altered the order of levels of my categorical factor. <strong>My question is: Why does that happen?</strong> I would expect that the intercept would change, as well as the pairwise comparisons between the levels of the categorical predictor, but I didn't expect anything to change with my continuous predictors.</p> <p>After doing some troubleshooting, I have discovered that removing the interaction terms erases any differences in the significance of the predictors between the two factor-level ordering schemes. This makes me think it is something about how the model is parsing the variation in the continuous predictors, but I really have no idea. I also tried the model with a simple <code>lm</code> style linear model, and get the same pattern of results, so I don't think it is specific to GLMMs.</p> <p>Any thoughts would be very much appreciated. Thanks!</p> <p>Also, a reproducible example is below. This is using a publicly-available dataset, and please ignore that the models may not be relevant to the data themselves, as I just wrote it quickly to replicate the issue that I am seeing in my data, and didn't pay attention to what the different columns of data actually are.</p> <pre><code>#Data for example can be downloaded here: #https://github.com/lme4/lme4/blob/master/inst/testdata/gopherdat2.RData load("gopherdat2.RData") #Needed libraries library(lme4) library(car) #I need a categorical predictor, so for the purposes of this model, I will use year as categorical Gdat$yFac&lt;- ifelse(Gdat$year==2004,"year1",ifelse(Gdat$year==2005,"year2","year3")) #Model with default organization of categorical predictor m1&lt;-glmer(shells~density+prev+yFac+density:yFac+prev:yFac+ (1|Site),data=Gdat,family="poisson") #Anova from the car package to get p-values for the different model terms. Anova(m1,type="III") #Making "year2" be the first one alphabetically. #I realize that the function stats::relevel does this in a better way, but this is what I do so #that I can keep track of which way I am parameterizing the model. Gdat$yFaca&lt;-ifelse(Gdat$yFac=="year2","ayear2",paste(Gdat$yFac)) #Model with the new organization of the categorical predictor m2&lt;-glmer(shells~density+prev+yFaca+density:yFaca+prev:yFaca+ (1|Site),data=Gdat,family="poisson") #Anova from the car package to get p-values for the different model terms Anova(m2,type="III") #Two models to show how it works without interactions m3&lt;-glmer(shells~density+prev+yFac+(1|Site),data=Gdat, family="poisson") #Original categorical order m4&lt;-glmer(shells~density+prev+yFaca+(1|Site),data=Gdat, family="poisson") #New categorical order #Running this shows that they are essentially the same. Anova(m3) Anova(m4) </code></pre> https://stats.stackexchange.com/q/139660 21 Detecting changes in time series (R example) mlee https://stats.stackexchange.com/users/69488 2015-02-27T20:58:56Z 2022-05-24T00:43:09Z <p>I would like to detect changes in time series data, which usually has the same shape. So far I've worked with the <code>changepoint</code> package for R and the <code>cpt.mean(), cpt.var()</code> and <code>cpt.meanvar()</code> functions. <code>cpt.mean()</code> with the PELT method works well when the data usually stays on one level. However I would also like to detect changes during descents. An example for a change, I would like to detect, is the section where the black curve suddenly drops while it actually should follow the examplary red dotted line. I've experimented with the cpt.var() function, however I couldn't get good results. Have you got any recommendations (those don't have to necessarily use R)?</p> <p><img src="https://i.stack.imgur.com/ZLETi.png" alt="Change curve"></p> <p>Here is the data with the change (as R object):</p> <pre><code>dat.change &lt;- c(12.013995263488, 11.8460207231808, 11.2845153487846, 11.7884417180764, 11.6865425802022, 11.4703118125303, 11.4677576899063, 11.0227199625084, 11.274775836817, 11.03073498338, 10.7771805591742, 10.7383206158923, 10.5847230134625, 10.2479315651441, 10.4196381241735, 10.467607842288, 10.3682422713283, 9.7834431752935, 9.76649842404295, 9.78257968297228, 9.87817694914062, 9.3449034905713, 9.56400153361727, 9.78120084558148, 9.3445162813738, 9.36767436354887, 9.12070987223648, 9.21909859069157, 8.85136359917466, 8.8814423003979, 8.61830163359642, 8.44796977628488, 8.06957847272046, 8.37999165387824, 7.98213210294954, 8.21977468333673, 7.683960439316, 7.73213584532496, 7.98956476021092, 7.83036046746187, 7.64496198988985, 4.49693528397253, 6.3459274845112, 5.86993447552116, 4.58301192892403, 5.63419551523625, 6.67847511602895, 7.2005344054883, 5.54970477623895, 6.00011922569104, 6.882667104467, 4.74057284230894, 6.2140437333397, 6.18511450451019, 5.83973575417525, 6.57271194428385, 5.36261938326723, 5.48948831338016, 4.93968645996861, 4.52598133247377, 4.56372558828803, 5.74515428123725, 5.45931581984165, 5.58701112949141, 6.00585679276365, 5.41639695946931, 4.55361875158434, 6.23720558202826, 6.19433060301002, 5.82989415940829, 5.69321394985076, 5.53585871082265, 5.42684812413063, 5.80887522466946, 5.56660158483312, 5.7284521523444, 5.25425775891636, 5.4227645808924, 5.34778016248718, 5.07084809927736, 5.324066161355, 5.03526881241705, 5.17387528516352, 5.29864121433813, 5.36894461582415, 5.07436929444317, 4.80619983525015, 4.42858947882894, 4.33623051506001, 4.33481791951228, 4.38041031792294, 3.90012900415342, 4.04262777674943, 4.34383842876647, 4.36984816425014, 4.11641092254315, 3.83985887104645, 3.81813419810962, 3.85174630901311, 3.66434598962311, 3.4281724860426, 2.99726515704766, 2.96694634792395, 2.94003031547181, 3.20892607367132, 3.03980832743458, 2.85952185077593, 2.70595278908964, 2.50931109659839, 2.1912274016859) </code></pre> https://stats.stackexchange.com/q/134934 1 using training data in final model output Grant McKinnon https://stats.stackexchange.com/users/58895 2015-01-26T04:33:49Z 2022-05-24T00:03:35Z <p>I have customer data for around 400,000 customers where 270,000 of them are current customers and 130,000 of them are past customers who churned, what I am doing is classifying them as 0 (non-churn) and 1 (churner) to come up with probabilities for likelihood of churning. I am using random forests in R. </p> <p>What I want to know is can I use the full training set (splitting 80/20 for train and test sets) then use the entire current customer list to output the probabilities or will using the same data as the training/testing data affect the final output? </p> <p>Should I instead take a sample of current and past customers and not include that in the final output of the model? I need to use some current customer data to train the model but can I still use that same data to output the churn risk? </p> https://stats.stackexchange.com/q/105530 5 Gibbs sampling for correlated random variables Till Hoffmann https://stats.stackexchange.com/users/17643 2014-07-02T12:51:52Z 2022-05-24T00:29:08Z <h1>Short summary</h1> <p>Suppose two latent variables of a hierarchical model are correlated. Let $1-\epsilon$ be the degree of correlation. As $\epsilon\rightarrow 0$ the variables become perfectly correlated and Gibbs sampling appears to only be possible if the two variables are considered as one variable (i.e. using block-Gibbs). Is this indeed the case or am I lacking understanding?</p> <p>Edit: <a href="http://en.wikipedia.org/wiki/Gibbs_sampling#Failure_modes" rel="nofollow noreferrer">Wikipedia</a> discusses this problem in their section on "Failure modes". However, there is no reference or resolution to the problem.</p> <h1>Complete question with example</h1> <p>Consider a population whose members have two binary attributes: age $X$ and health $Y$. We encode the variables as $$X=\begin{cases}0&amp;\text{if young}\\1&amp;\text{if old}\end{cases}$$ and $$Y=\begin{cases}0&amp;\text{if healthy}\\1&amp;\text{if sick}\end{cases}.$$ Let the joint probability distribution be given by $$P\left(X=x\cap Y=y\right)\equiv p_{xy}=\frac{1}{2}\left(\begin{array}{cc}1-\epsilon&amp;\epsilon\\\epsilon&amp;1-\epsilon\end{array}\right)$$ such that $\sum_{xy}p_{xy}=1$. Let us expand the probability distribution such that $$P\left(X=x\cap Y=y\right) = P\left(Y=y|X=x\right)P\left(X=x\right),$$ where \begin{align} P\left(X=x\right)\equiv p_x &amp;=\sum_y p_{xy}=\frac{1}{2}\quad\forall x\\ P\left(Y=y|X=x\right)\equiv p_{y|x}&amp;=\frac{p_{xy}}{p_x}=\begin{cases}1-\epsilon&amp;\text{if }x=y\\\epsilon&amp;\text{if }x\neq y\end{cases} \end{align}</p> <p>Suppose we make a noisy observation of the state of health an individual $\hat{Y}$ with error rate $\delta$ such that $P\left(\hat{y}=y|Y=y\right)=1-\delta$. We construct a hierarchical model as shown in the figure below. <img src="https://i.stack.imgur.com/Ujls0.png" width="110"/> </p> <p>To use a Gibbs sampler, we construct the conditional distributions (assuming a flat prior on $X$) $$P\left(X=x|Y=y\right)=\begin{cases}1-\epsilon&amp;\text{if }x=y\\\epsilon&amp;\text{if }x\neq y\end{cases}$$ $$P\left(Y=y|X=x\cap\hat{Y}=\hat{y}\right)\propto\begin{cases}1-\epsilon&amp;\text{if }y=x\\\epsilon&amp;\text{if }y\neq x\end{cases}\times\begin{cases}1-\delta&amp;\text{if }y=\hat{y}\\\delta&amp;\text{if }y\neq \hat{y}\end{cases}.$$</p> <p>However, in the limit $\epsilon\rightarrow 0$ knowledge of $x$ gives us full knowledge of $y$ and Gibbs sampling is no longer possible because the sampler gets stuck: If $X=0$, then $P\left(Y=y|X=0\cap\hat{Y}=\hat{y}\right)=\delta_{y0}\quad\forall\hat{y}$ and vice versa. What is the best way to deal with this situation?</p> https://stats.stackexchange.com/q/82651 2 Treatment of missing values introduced by padding lagged variables Luke https://stats.stackexchange.com/users/32018 2014-01-18T01:13:20Z 2022-05-24T02:34:35Z <p>In the case of a linear regression with lagged independent variables, what are the techniques for dealing with the NA values introduced by padding lagged variables (since t &lt; 0 values do not exist)?</p> <p>I ask because I'm implementing a random forest with lags and have to decide what to do about these missing values. </p> <p>Replacing them with 0 seems misguided, but imputing backwards doesn't make much sense either. I suppose I could insert a time mean, or an overall mean for the variable if I have panel data...</p> <p>Edit 1: Looking around more, SAS lets you choose between replacing NA with a time mean, overall mean, or zero: <a href="http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_panel_sect015.htm" rel="nofollow">http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_panel_sect015.htm</a> Any idea on the costs and benefits of these different approaches?</p> https://stats.stackexchange.com/q/37737 0 Using z-standardization to account for covariate jokel https://stats.stackexchange.com/users/10389 2012-09-21T11:23:32Z 2022-05-24T01:04:40Z <p>I would like to know whether z-standardization is an appropriate way to account for a covariate. Please consider the following dummy example (I am not interested in the interpretation of the result but only in the statistical plausibility):</p> <p>In a sample of 100 subjects relationship between IQ and head size is investigated (one measurement per subject). However IQ has been estimated with IQ Test 'A' in some subjects and with IQ test 'B' in the other subjects. Also there is a significant difference in IQ measured by 'A' as compared to IQ measured by 'B'. Therefore 'zIQ' is obtained by z-standardization of all IQ values obtained by test 'A' as well as z-standardization of all IQ values obtained by 'B'. Is it appropriate to use zIQ to investigate its relationship with head size independently of which test ('A' or 'B') has been used?</p> https://stats.stackexchange.com/q/16953 7 How to detect structural change in a timeseries Dail https://stats.stackexchange.com/users/5405 2011-10-13T11:46:03Z 2022-05-24T02:27:18Z <p>Is there a specific method to detect change points(structural breaks) in a timeseries? (stocks prices).</p> https://stats.stackexchange.com/q/6033 13 Detect changes in time series stephanos https://stats.stackexchange.com/users/2667 2011-01-06T13:53:16Z 2022-05-24T01:23:14Z <p>I came across a picture of an application prototype that finds significant changes ("trends" - not spikes/outliers) in traffic data:</p> <p><img src="https://i.stack.imgur.com/sGYeF.png" alt="alt text"></p> <p>I want to write a program (Java, optionally R) that is able to do the same - but because my statistic skills are a little rusty, I need to dig into this topic again.</p> <p><em>What approach/algorithm should I use/research therefore?</em> </p>