All Questions

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

How do I appropriately control for a limiting/maximum value in regression?

I have a dataset where one variable is limited by the value of another. It is a study of participants with a particular disease; therefore age of disease onset, A, is limited by age of registration ...
0
votes
0answers
30 views

How does weights argument in gam() to handle the heterogeneous variance issues

In my case there were multiple observations per Group(random effect) in a single Year. So I aggregate these cases by calculating ...
1
vote
0answers
23 views

Penalize the MSE of half of predicted values more than the other half

I'm using MSE loss for an multi-layer perceptron that learns to approximate the target feature vector $[\hat{x_1}, \dots, \hat{x_N}, \hat{y_1}, \dots, \hat{y_N}]^\intercal$. The catch here is that I'd ...
1
vote
0answers
139 views

Including a baseline risk for each participant to consider having a different number of data points in Cox model

We used the Cox model to associate environmental exposures with time to heart failure using cohort data. The following question has been raised: When using the extended Cox model, did you include a ...
-3
votes
0answers
67 views

Statistics question please help [on hold]

The value of the Margin of Error E that corresponds to a 95% confidence level (z subscript bevelled alpha over 2 end subscript space = 1.96) for a random sample of size 694 and sample proportion p ...
1
vote
0answers
11 views

Testing significant difference between two independent means

I have two groups within a dataset. One group completed one 10-point Likert, the other completed a similar but different 10-point Likert. Beyond simple descriptives, I'd like to identify whether one ...
1
vote
0answers
20 views

Differences between cumulative link models (ordinal) and multinom (nnet) for fitting multinomial data

I'm trying to understand cumulative link models and how they differ from multinom models in R. Here's a simple example of a multinom model and plot output using the nnet package: ...
0
votes
0answers
11 views

Correlation and skewness

I have a data set on which i want to run a machine learning algorithm , some of the columns are skewed. If I apply a transformation (lets say log) to those columns and I want to display the matrix of ...
0
votes
0answers
28 views

form of the model when using backshift operator

Be $Y_t=X_t + \epsilon_{1,t}$, in which $X_t = X_{t-1} + \epsilon_{2,t}$ and $E[\epsilon_{1,t}\epsilon_{2,s}] = 0 \forall t,s$. How could I say why this process is related with a model on the form $(1-...
0
votes
1answer
18 views

Skewness and correlation [duplicate]

I have a data set on which I want to run a machine learning algorithm. Some of the columns are skewed. If I apply a transformation (let's say log) to those columns and I want to display the matrix ...
0
votes
0answers
13 views

Correlation between two variables from nested, non-normally distributed, semi-continous / count data

I am interested in correlating the physical distribution of cell types and cellular material within tissue that has been imaged as a large grid of images (each grid is a datapoint). My data comprises ...
1
vote
0answers
8 views

Reason for Precision and Recall terminology?

Why are the statistical indicators Precision and Recall named that way? Is there a canonical interpretation?
0
votes
0answers
7 views

Worse outputs when use short input for XLnetLMhead model but improved by adding extra random tokens? Using huggingface pytorch-transformers

My question arose from following links from huggingface github. https://github.com/huggingface/transformers/issues/846 Q1) can someone give some more insight what @thomwolf explaining about? ''' ...
0
votes
0answers
16 views

How to compare two within-subject conditions if there are multiple trials per subject? [on hold]

I have the following experiment design: I am comparing two conditions tested across 30 subjects. Each subject is tested for both conditions, so the condition is a within subject factor. Also, each ...
0
votes
1answer
12 views

Determine dimension of mapping function Phi from form of Kernel

If I have a kernel of the form $k(x, y) = (x^Ty)^n$ where $x$ and $y$ are d-dimensional, how can I determine the dimension of the mapping function $\phi(x)$, in terms of n and d, without explicitly ...
0
votes
0answers
5 views

How do you understand an LDA score?

I'm reading a research paper involving the human gut microbiome. LDA analysis was performed to determine species richness in one study group, but I don't know how to interpret LDA scores. For example, ...
0
votes
1answer
12 views

Does the Vargha-Delaney A effect size stand as a standardized effect size?

Effect sizes are categorized into 'Standardized effect sizes' and 'Unstandardized effect sizes'. My question is, Vargha-Delaney A falls in which of the categorizes? does it cound as standardized ...
2
votes
1answer
23 views

How to test whether two binomially distributed variables come from the same distribution?

I have data where I have a set of pairs of integer, each binomially distributed, and I want to be able to test whether the observations came from different distributions, though I don't care what ...
0
votes
1answer
27 views

Statistics book for Machine Learning [duplicate]

Hello I am a mathematician currently studying Data Science and I would like to ask for a book that can give me a variety of statistical tools for having more solid conclusions in my analysis of a ...
2
votes
0answers
15 views

Mean of repeated samples

Let's assume we have random variables $X_1\sim N(\mu_1,1),\cdots,X_n\sim N(\mu_n,1)$. Now we take one sample from each and get $X_1 = x_1,\cdots,X_n = x_n$. We order them and calculate the mean of top ...
4
votes
1answer
43 views

In a linear regression hypothesis equation, what does each symbol represent?

So I've been watching Andrew Ng's machine learning lectures, and I'm on a video about univariate linear regression. He was talking about how a Hypothesis takes an input and predicts an output, like a ...
0
votes
0answers
15 views

Multiple correspondents analysis, indicator matrix, why do we double count for $I_{k\neq k'}$ [on hold]

From the text : "Exploratory Multivariate Analysis by Example Using R", Page 134 has the following: One way of comparing these two categories is to count the individuals which select both ...
0
votes
0answers
22 views

R code of t-test with 1% level of significance? [on hold]

The manufacturer of a certain make of electric bulbs claims that his bulbs have a mean life of 25 months with a standard deviation of 5 months. Random samples of 6 such bulbs have the following values:...
0
votes
1answer
23 views

Data Mining/Statistical Methods to find trends, peaks, etc

currently I am working on a project for my final exam. The data is coming from a streaming plattform. The data I am using are some logging data (data when customers have problems with the streaming ...
0
votes
0answers
14 views

Compressing an image using neural network

I've recently given several attempts to compress an image using neural network. The approch is to treat a trained neural network itself as a compressed data. I was expecting that, given x,y as input, ...
0
votes
0answers
23 views

Finding optimal cutoff point [duplicate]

.Hello,everyone. I am studying the influence of one biomarker on multiple disease characteristics, and I would like to calculate its cutoff point. I created univariate ROC curves to investigate the ...
0
votes
0answers
21 views

Covariance matrix as a sum of two covariance matrices

Suppose that a random vector $\mathbf{n}$, $$\mathbf{n} = \mathbf{n}_A + \mathbf{n}_B \ , \tag{1}$$ can be written as a sum of two random vectors $\mathbf{n}_A$ and $\mathbf{n}_B$, that are ...
0
votes
0answers
14 views

Implementation of 2SLS in regression with AR errors?

Consider the following simple example: $Y_t=\beta X_{t-1}+\varepsilon_t$ $X_t=\gamma Y_t + Z_t +u$, where $\varepsilon_t=\alpha\varepsilon_{t-1}+\eta$, and $E[Z_t\varepsilon_t']=E[uu']=E[\eta\...
0
votes
0answers
16 views

Basic results on convergence in distribution

Let $\{X_i\}_{i=1}^n$ be independent zero mean random variables with finite variance and $\{r_n\},\{d_n\}$ positive monotone increasing real sequences. Assume that $$\frac{\sum_{i=1}^{r_n} X_i}{Var(\...
0
votes
0answers
5 views

How to test goodness of fit for AFT Survival Models?

I am testing a few survival models in the parametric family and would like to test the goodness of fit of the models to the data I have. Which tests would be useful and is it even necessary to carry ...
0
votes
0answers
11 views

Forward model selection: optimize on training or test dataset (within a nested framework)? [on hold]

I am working on a binary classification problem where I have a large number of features (~100 features) on a small dataset (n=100 in each class). I am using a double cross-validation to try and find ...
1
vote
1answer
25 views

Range of integration for joint and conditional densities

Did I mess up the range of integration in my solution to the following problem ? Consider an experiment for which, conditioned on $\theta,$ the density of $X$ is \begin{align*} f_{\theta}(x) = \...
2
votes
1answer
22 views

Rule of Thumb for Sensitivity and Specificity

What are rule of thumb cutoffs for sensitivity and specificity? What is high, good, fair, low?
1
vote
0answers
27 views

Unusual DV Odds Ratio for multiple Binary Logistic Regression

I am attempting to diagnose issues with the DV odds ratio and resulting 95% CI for the final step of my logistic regression. As you can see in the below image, when the "Continuous F" variable is ...
0
votes
0answers
11 views

Role of $\beta_i$'s slack variables in getting the dual form of soft margin SVM using L1 regression? [on hold]

In this Pg 70 of this document, when getting to the dual form of SVM problem with a soft margin that is L1 regularized, they consider another set of slack variables $\beta_i$'s other than the general $...
2
votes
0answers
28 views

If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal ...
0
votes
1answer
20 views

From Baysian Network To Correlation Matrix

I have a bayesian network that the edges are likelihood estimations from features {x1,...,xn}. how can to estimate a covariance matrix from bayesian net between x? Normally, we use from a correlation ...
0
votes
1answer
49 views

Linear regression $y_i=\beta_0 + \beta_1x_i + \epsilon_i$ covariance between $\bar{y}$ and $\hat{\beta}_1$

I am currently reading through slides from Georgia Tech on linear regression and came across a section that has confused me. It states for $$ y_i=\beta_0+\beta_1x_i+\epsilon_i $$ where $\epsilon_i \...
0
votes
0answers
10 views

How to create matrix of all 2^n binary sequences of length n using recursion in R? [migrated]

I know I can use expand.grid for this, but I am trying to learn actual programming. My goal is to take what I have below and use a recursion to get all 2^n binary sequences of length n. I can do this ...
0
votes
0answers
22 views

Covariance equation [on hold]

we have the function for marginal rate of substitution as M_t+1. log of that is m_t+1 = ln B -γln((c_t+1)/(c_t)) We also have another equation. cov(ln((c_t+1)/(c_t)),lnRe_t+1) = 0.03 B = e^-0.03 In ...
4
votes
1answer
45 views

Simulate multivariate outliers that are hidden in 2D scatterplots

How could I simulate multivariate outliers that are "hidden" in all pairwise 2D scatterplots between the variables? By "hidden" I mean that they can't be seen (as obvious outliers) or detected ...
1
vote
1answer
37 views

Is this time series stationary? What would be your approach to forecasting it?

I've been working on the time series prediction of a signal and came across a small misunderstanding. The signal is depicted below: Apparently it looks like there are several stationary local areas ...
0
votes
0answers
6 views

Difference lemmatizing/stemming when preprocessing text to organize abstracts looking for document insights?

I'm working with R text2vec in order to apply an LDA on a 230k text data that I have on hand. I tried both stemming and lemmatizing separately but I am not completely aware of which gives out the ...
4
votes
1answer
46 views

Density estimation as an optimization problem

Density estimation is the estimation of a probability density function from observed data. Can some of the common approaches to density estimation, such as kernel density estimation, be formulated as ...
0
votes
2answers
27 views

performance of a regression model

I am doing random forest regression for multiple features. Now I want to know my model performance. But there is no confusion matrix or accuracy matrix for checking regression model performance. So ...
0
votes
1answer
29 views

Intutitive meaning behind the formal definition of sufficient statistic

According to the definition of sufficiency, A statistic is sufficient for a parameter if the conditional distribution of X given a value of statistic does not depend upon the parameter. What I am ...
0
votes
1answer
22 views

Notation for Pearson residual vs Pearson correlation coefficient

From my book, the Pearson residual is defined as: $R = \frac{Observed - Expected}{\sqrt{Expected}}$ And from wikipedia, the Pearson correlation coefficient is defined as: $r_{xy}={\frac {\sum _{i=1}...
1
vote
1answer
44 views

Maximum Likelihood with dependent observations

In the context of a normally distributed dependent variable, the likelihood for a single observation is given as: $L(y_i|x,β,σ2)=\displaystyle\cfrac{1}{√2πσ^2}e^{\displaystyle\cfrac{(y_i−x_iβ)}{2σ^2}}...
1
vote
0answers
9 views

XGBoost model in Spark --> Missing value treatment [on hold]

Unlike python, where missing value is handled internally by the XGBoost algorithm, While building XGBoost model in SPARK, the missing values are implicitly converted to 0.0(float?!). Is this okay ? ...
0
votes
1answer
20 views

Low propensity score among treated and high among the control units

Is it conceivable for the treated units to have propensity scores systematically lower than those estimated for the control units? Thanks in advance.

15 30 50 per page