Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [theory]

For questions about statistical theory. Always include a more specific tag as well.

0
votes
0answers
11 views

Studying the relationship between non-inependent variables

What are some approaches to studying the relationship between two non-independent variables? When doing linear regression to study the relationship between two variables, it is assumed that your "x" ...
0
votes
0answers
5 views

References for generalization bounds?

I'm looking for references (books, papers, lecture notes etc) on generalization bounds and their proofs. Specifically, I'm looking to fully understand the technique of defining a hypothesis class (or ...
2
votes
0answers
24 views

Why we say that a probability measure $P$ is defined on $(\Omega, \mathcal{F})$?

I was wondering why we say that a probability measure $P$ is defined on $(\Omega, \mathcal{F})$, being $\Omega$ the sample space and $\mathcal{F}$ one sigma-algebra, when actually we have that $P$ is ...
0
votes
0answers
77 views

How to manually calculate odds ratio for continuous variables?

In school, long before learning about logistic models, I've been taught how to calculate odds ratios by hand. Formula was based on a contingency table, just like this: This is very easy to ...
1
vote
0answers
14 views

Clarification on the concept of “General formulation of the problem of statistical inference” - Wald

I'm looking for clarification on one part of this definition, but also some feedback on my interpretation of the whole concept. The definition comes from the end of the first chapter of On the ...
1
vote
0answers
51 views

Combining subjoint distributions to create a larger joint distribution

I am trying to construct large joint distributions through smaller joint distributions and I'm not sure how to approach the literature. I am curious if there exists a function which can take n ...
2
votes
1answer
44 views

What is the point of a permutation F-test when all you need is one F-test for one-way ANOVA?

Say you have three groups, and each group has 5 observations. You can figure out if there is a significant difference between means with a simple one-way ANOVA. I read in my nonparametric book, one ...
0
votes
0answers
13 views

Regarding a Proof in ''Robustness and Generalization'' by Xu. and Mannor

I am trying to understand the mathematics of the paper "Robustness and Generalization". The problem that I am facing is probably very simple but I am just not able to wrap my head around it. There ...
5
votes
1answer
86 views

Find the values of $ a$ so that A is positive definite (p.d)

Let $A=(1-a)I_n + a J_n$ Find the values of $~a~$ so the Matrix is p.d ? Note:$~I_n~$ is the identity matrix and $~J_n$ is the $1's$ matrix. I know that $~~A$ is p.d $~iff~λ_i >0$ so, I need to ...
1
vote
1answer
52 views

Based on the ideas of Parameter Estimation and Fitting Probability Distributions, what stops us from making any function be a PDF(PMF)?

Currently I am doing an introduction to parameter estimation and fitting probability distributions to sets of data. So in a small synopsis what I understand the whole process to be like is the ...
1
vote
0answers
57 views

Does every loss function correspond to MLE/MAP

Many of the losses used in regression/classification tasks correspond to maximum likelihood estimation (MLE) or maximum aposteriori (MAP) under a specific data likelihood distribution $p(\mathbf{y}|X,\...
0
votes
1answer
33 views

Why isn't the least squares predictor $\Phi(\Phi^\top\Phi)^{-1}\Phi^\top$ simply the identity matrix? [duplicate]

Given target vector $y$. Want to predict it using linear regression $h(w) = w^Tx$ Let $\Phi$ be the least squares matrix, i.e., $\Phi = \begin{bmatrix} x_1^\top \\ \vdots\\ x_n^\top \end{bmatrix}$ ...
2
votes
0answers
35 views

Operator Theoretic Perspectives on Bayesian Inference

On the Wikipedia page for conjugate prior there is a section "Analogy with Eigenfunctions", which describes a metaphor where observation of data is seen as a "linear operator" on the space of ...
0
votes
0answers
31 views

multivariate linear regression without b_0 [duplicate]

I created a multivariate regression following the scheme $$y = \beta_0 + \sum^n_{i=1}\beta_i*x_i$$ and got an average deviation ofaround 5%. When I tried the regression without the $\beta_0$ I got a ...
6
votes
2answers
330 views

ELI5: The Logic Behind Coefficient Estimation in OLS Regression

Like a lot of people, I understand how to run a linear regression, I understand how to interpret its output, and I understand its limitations. My understanding of the mathematical underpinnings of ...
0
votes
1answer
60 views

Type I and Type II errors in Hypothesis Testing

I am confused about the last highlighted sentence regarding finding a subset S, for which BOTH Type I and Type II error probabilities are 0. For $P_\theta S$, which is the probability with which we ...
0
votes
1answer
95 views

R: how is the Pr(>|z|) in the results of glm.fit calculated and why?

I've been searching but I can't find anywhere an explanation of how the Pr(>|z|) column is calculated in the results of R's glm.fit function. I would really appreciate: a) an explanation so I can ...
0
votes
1answer
35 views

How to prove the distribution of Generalised Instrumental Variables Estimator

$\hat { \beta } _ { GI V } = ( X ^ { \prime } Z ( Z ^ { \prime } Z ) ^ { - 1 } Z ^ { \prime } X ) ^ { - 1 } X ^ { \prime } Z ( Z ^ { \prime } Z ) ^ { - 1 } Z^ { \prime }y\hspace{35pt}(a)$ $(a)$ is ...
2
votes
0answers
17 views

Correcting for sample overlap (loss of statistical independence) in two z-tests

Short form: How can I correct two p-values computed from normal distributions to account for the fact that part of the samples overlap? Long form: I have a sequence of i.i.d RVs of length $n$. The ...
4
votes
1answer
145 views

Uniqueness of Reproducing Kernel Hilbert Spaces

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]: Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in ...
0
votes
0answers
33 views

Aikaike Information Criterion: derivation in original paper

I have been reading AIC paper 'Information theory and an extension of the maximum likelihood principle' by Akaike (1974). I have been able to understand up to the third section of the paper, but I am ...
1
vote
0answers
107 views

Is it true that normalizing the output of a ReLu feedforward Neural Network that its Rademacher Complexity becomes a constant?

I was trying to understand what happened with the Rademacher Complexity: $$ R_S(F) = \frac{1}{m} \mathbb E_{\sigma} [\sup_{f \in F}\sum^m_{i=1} \sigma_i f(z_i)] $$ or $$ R_{P,m} = \mathbb E_{s \...
0
votes
0answers
24 views

pooled GEV return level estimation

Let’s suppose I have 4 AMAX time series of 40 years each one, they could be eg wind speeds from 4 meteorological stations in a given region. The time series have been normalised. I want to fit a GEV ...
2
votes
0answers
30 views

Where can I read about the theory behind GLMs in their most general form?

I was following along with the MIT Opencourseware "Statistics for Applications" and their analysis of GLMs only covers discussion of GLMs whose dependent variable $y$ is distributed according to a ...
0
votes
0answers
12 views

Bounding Classification Loss Function

I am having trouble to establish some bounds. An exercise asks to me to bound: $$ L(\theta) - L(\tilde{\theta}) \leq C E[\min((X'(\theta-\tilde{\theta})^2,1)] $$ where $L(\theta) = E[(Y-\phi(\theta'...
0
votes
2answers
92 views

How to handle 'unfairness' of missing data in machine learning?

Let me explain what I mean by unfairness. Let's say I have a multi-class classification problem where I am trying to predict the 'best drug' (among multiple candidate drugs) for each patient. So ...
2
votes
1answer
1k views

XGBoost implementation for unbalanced data using scale_pos_weight parameter

I have a confusion regarding how cost sensitive custom metric can be used for training of unbalanced dataset (two class 0 and 1) in XGBoost. Metric: Cost = 10*#of false positives + 500*# of false ...
1
vote
0answers
51 views

Downsampling features: how to select the most optimal features to recapitulate clustering [closed]

I've performed single cell analysis in which each gene represents a feature to cluster upon; there are about 20,000 genes expressed across all the cells in the dataset. I use the top 1500 or so ...
1
vote
0answers
24 views

Adverserial Verification of an XGboost Classifier

This paper proposes an algorithmic framework, predictor-verifier training, to train neural networks that are verifiable, i.e., networks that provably satisfy some desired input-output properties. The ...
2
votes
1answer
75 views

Why fixed design analysis for observational data

Why do we use a fixed design analysis of regression coefficients, even for observational data, where the design is not fixed? For instance: $Var[\hat \beta]=(X'X)^{-1}\sigma^2$ is conditional on $X$. ...
1
vote
2answers
332 views

Binomial Approximation to Hypergeometric Probability

I am trying to understand how to apply the binomial distribution to a simple probability problem. I can solve the problem directly via the classical definition of probability but, when trying to ...
0
votes
0answers
91 views

Prospective studies and variable updates

In a cohort study, I'm measuring the apparition of a disease over time and confronting it against a variable, which we'll call X. This is a very straight-forward analysis where I have the baseline ...
1
vote
2answers
142 views

How does the realizability assumption imply that there is always a hypothesis that gets zero train error?

I was reading Shai-Shawrts and Shai Ben-David's Understanding Machine Learning book and it said after defintion 2.1 something like this: The realizability assumption (i.e. that $E_{p_{x,y}} [loss(h^...
1
vote
1answer
22 views

Using largest gradient at every iteration: good strategy?

Assume you have access to an oracle, which, given a set of labeled data $D = \{(x_1, y_1), ..., (x_n, y_n))\}$ returns a single data point $d^j_{max} = (x_i, y_i)$ with the property that its gradient ...
0
votes
0answers
6 views

Reference request for parametric bootstrapping theory [duplicate]

Where is a good reference for the theory behind parametric bootstrapping (use MLE estimates as parameters of a distribution,then simulate from that distribution using those estimates as true values)? ...
0
votes
0answers
46 views

What is the main contribution of “Donoho and Johnstone” school in Statistics?

This is a question about the history of Theoretical Statistics. I recently read the book "Asymptotics in Statistics" by L. Le Cam and G. Yang. In the preface, they mentioned the book did not cover the ...
0
votes
0answers
55 views

Latent Class Analysis Bias Generation Among Missing Data

Our university is conducting disease research on patient data: 4 variables among 5,191 cases. There is a significant amount of missing data (approximately 52% of the data set). We are attempting to ...
2
votes
1answer
54 views

Can the Vapnik-Chervonenkis inequality be generalised to non-zero-one error functions?

I learned about the VC inequality of the bound of difference between training errors and generalisation errors. The two places (Stanford CS229 notes and Wikipedia) where I read about the theorem both ...
4
votes
1answer
64 views

Linear combo of normals is normal; how about other distributions?

We know that univariate normal distributions are independent only if their every linear combination is itself normal: $$\tag{1} Z_i\sim N(\mu_i,\sigma_i^2)\,(\forall i) \implies\\ \biggl( \textrm{...
2
votes
2answers
59 views

High-level time-series question: How does one study a series' trend?

I want to understand a series' trend, not the deviations from the trend. I would like to do analysis on the trend, such as run a multivariate regression, but every time-series source I read online ...
2
votes
1answer
33 views

Statistical theory proof intuition (UMVU estimators)

I've been working through this problem in Theoretical Statistics by Keener, but could not solve it. I looked up the answer and I do understand why it's correct, but I don't understand what intuition ...
1
vote
0answers
19 views

Process for finding UMVU estimator

I've been working on a problem from Theoretical Statistics: Topics for a Core Course by Keener. I spent a few hours on it making very little progress before caving and looking up the solution. I don't ...
1
vote
0answers
158 views

Optimal value for the intercept term in SVM

(note that this problem is different from this one, since the latter considers primal's Lagrangian) Hi, I am trying to figure out the SVM's dual problem. The primal problem is $${\displaystyle {\...
2
votes
1answer
70 views

Bidirectional effects?

When testing relationships between variables, we are often trying to establish a finding that Variable A causes (or is connected with) an increase in Variable B, or that Variable A causes a decrease ...
1
vote
0answers
44 views

Supervised learning when there is a true underlying model vs. supervised learning when there is no underlying model?

(I found a more concise way of expressing my question - I'm leaving the original wording below for reference purposes) Is there a difference in the way we approach supervised learning problems when ...
1
vote
1answer
285 views

Feedforward neural network for sinusoidal prediction

This is 100% curiosity, so I apologize if the question is under-constrained. If so, please comment with where my thinking is misguided! Can a feedforward neural network predict a sinusoidal ...
3
votes
1answer
65 views

The ratio of a maximized likelihood and a marginal likelihood

I stumbled upon the following quantity and I'm wondering if anyone knows of anywhere it has appeared in the stats literature previously. Here's the setting: Suppose you will observe data $y\in \...
17
votes
4answers
738 views

How do Bayesian Statistics handle the absence of priors?

This question was inspired by two recent interactions I had, one here in CV, the other over at economics.se. There, I had posted an answer to the well-known "Envelope Paradox" (mind you, not as the "...
3
votes
1answer
87 views

What are the main inequalities used in statistical proofs? [closed]

I know Jensen's is used everywhere. What other inequalities are used in statistical proofs (please state the inequality rigorously and formally)? As a bonus, you could cite a paper/theorem where the ...
1
vote
1answer
57 views

uniform convergence and model selection question

I am working on the Question 1(c) of problem set 3 from cs229. A screenshot of the question is I am not sure how this formula came to be \begin{align*} \left|\epsilon(\hat h_j) - \hat \epsilon_{...