Newest questions tagged taylor-series - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-17T17:16:11Z https://stats.stackexchange.com/feeds/tag?tagnames=taylor-series&sort=newest http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/416244 0 Quadratic Approximation of the binary logistic regression Annalise Azzopardi https://stats.stackexchange.com/users/137487 2019-07-06T09:10:22Z 2019-07-06T09:10:22Z <p>I am using <a href="https://web.stanford.edu/~hastie/Papers/glmnet.pdf" rel="nofollow noreferrer">https://web.stanford.edu/~hastie/Papers/glmnet.pdf</a> package to solve my optimization problem for the Binary Logistic Regression. </p> <p>On page 10 it is stated that the quadratic approximation of the log likelihood function is used in the algorithm which is understandable due to speed. I am trying to get from eq.14 to eq.15 in this paper using the Taylor expansion however got stuck. Below is what I have achieved so far:</p> <p>argmin {<span class="math-container">$\frac{1}{n}$</span> <span class="math-container">$\sum_{i=1}^n$</span>-(<span class="math-container">$\bf x_i^T$</span> <span class="math-container">$\bf{\beta}$</span>) <span class="math-container">$y_i$</span> <span class="math-container">$+ log(1+exp($</span>(<span class="math-container">$\bf x_i^T$</span> <span class="math-container">$\bf{\beta}$</span>)) <span class="math-container">$+{\lambda}||{\beta}||_1$</span>} <span class="math-container">$...(1)$</span></p> <p><span class="math-container">$\frac{-\partial\ell}{\partial\beta}$</span>= <span class="math-container">$-\mathbf x_i^T y_i+\pi_i \mathbf x_i^T$</span></p> <p><span class="math-container">$\frac{-\partial^2\ell}{\partial\beta^2}$</span>= <span class="math-container">$-\mathbf x_i^T \pi_i (1-\pi_i) \mathbf x_i^T$</span></p> <p>I have ahieved the 1st and 2nd derivative but when am trying to get to the quadrative approximation of <span class="math-container">$(1)$</span> am not getting the same as in eq.15 of the above mentioned paper. Can someone help me please?</p> https://stats.stackexchange.com/q/406769 2 Moment Generating Function of Beta ( Taylor series) GAGA https://stats.stackexchange.com/users/240699 2019-05-06T02:12:08Z 2019-05-07T16:45:05Z <p>Suppose X is a random variable with a Beta ( a =<span class="math-container">$\frac{1}{2}$</span> , b=1) distribution and x in (0,1)</p> <p>Then the moment generating function is calculated as below</p> <p><span class="math-container">$M_X(t)$</span> = <span class="math-container">$\mathbb{E}[e^{tX}]$</span> =<span class="math-container">$\frac{\Gamma(\frac{1}{2} +1)}{\Gamma(\frac{1}{2} ) +\Gamma(1)} \int_0^1 e^{tX} x^{\frac{1}{2}-1} (1-x)^{1-1}\ dx$</span>= <span class="math-container">{\frac{1}{2}}$$\sum_{k=0}^\infty</span> <span class="math-container">\int_0^1 \frac{{t^kx}^k}{k!}</span> <span class="math-container">x^{-\frac{1}{2}} \ dx</span> =<br> <span class="math-container">\frac{1}{2}</span> <span class="math-container"> \sum_{k=0}^\infty \frac{t^k}{k!}</span> <span class="math-container">\int_0^1 {}</span> <span class="math-container">x^{k-\frac{1}{2}} \ dx</span> = <span class="math-container"> \sum_{k=0}^\infty \frac{t^k}{k! (2k+1)}</span> </p> <blockquote> <p>After - Using Taylor series expansion &amp; interchanging the summation and integration of the taylor series</p> </blockquote> <p><span class="math-container"> M_X(t) </span> = <span class="math-container"> \sum_{k=0}^\infty \frac{t^k}{k! (2k+1)}</span> </p> <p>where k is non negative integer and <span class="math-container">t \in \mathbb{R}</span></p> <p>How can I prove that this last sum is finite or infinite , is there a theorem ? (my math background is limited)</p> https://stats.stackexchange.com/q/401749 1 Multivariate Taylor series for moments of a random variable c.uent https://stats.stackexchange.com/users/91277 2019-04-08T04:06:28Z 2019-04-08T04:06:28Z <p>In the <a href="https://tminka.github.io/papers/aspect/minka-aspect.pdf" rel="nofollow noreferrer">expectation propagation for the generative aspect model</a>, Minka uses Taylor series for the parameter estimation of the topics <span class="math-container">p(w\mid a)</span> eq 31.</p> <p>I am a little confused in the last equation. He expresses the expectation of a function in terms of Taylor expansion as follows (eq 40), <span class="math-container">Var(\lambda)</span> is the covariance matrix of <span class="math-container">\lambda</span>:</p> <p><span class="math-container">\begin{equation} \mathbb{E}\left[f(\boldsymbol\lambda)\right] \approx f(\mathbb{E}\left[\boldsymbol\lambda\right]) + \frac{1}{2} Tr\left(f''(\mathbb{E}\left[\boldsymbol\lambda\right]) Var(\boldsymbol\lambda)\right) \end{equation}</span></p> <p>However, in <a href="https://stats.stackexchange.com/questions/306866/taylor-approximation-of-expected-value-of-multivariate-function">another post</a> I found the following derivation for multivariate Taylor expansion: </p> <p><span class="math-container">\begin{equation} \mathbb{E}[f(\lambda)] \approx f(\mathbb{E}\lambda) + \frac{1}{2} \sum_{i=1}^n H_f(\mathbb{E}\lambda)_{ii} Var(\lambda_i). \end{equation}</span></p> <p>The only difference is that in the first approximation Minka gets the product of the hessian and the covariance matrix inside the trace operation. This involves the interaction terms <span class="math-container">Cov(\lambda_i,\lambda_j)</span>. However, Michał Stolarczyk in the stats exchange post gets the trace of the diagonal of the hessian and the diagonal of the covariance matrix; for instance no interaction terms.</p> <p>Using the interactions terms of the covariance matrix, I get the expression (eq 33) by Minka in his paper:</p> <p><span class="math-container">\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}}{(\sum_bp(w\mid b)m_{iab})^2}-1 \end{equation}</span></p> <p>However, using Michał's expression directs me to the following expression:</p> <p><span class="math-container">\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}-\sum_bp(w\mid b)^2m_{iab}^2}{(\sum_bp(w\mid b)m_{iab})^2} \end{equation}</span></p> <p>Minka's result uses the interaction terms and the one shown comes from the following expression</p> <p><span class="math-container">\begin{equation} (\sum_bp(w\mid b)m_{iab})^2=\sum_bp(w\mid b)^2m_{iab}^2 + \sum_{k\neq j}p(w\mid b=i)p(w\mid b=j)m_{iak}m_{iaj} \end{equation}</span></p> <p>However, Michał's derivation makes sense to me. So, I am confused about the expression of multivariate Taylor expansion for the moments of functions of random variables. Which one is correct or when should I use either one? </p> https://stats.stackexchange.com/q/398730 1 Taylor Series Power Function in R Ash https://stats.stackexchange.com/users/241959 2019-03-21T15:41:07Z 2019-03-21T15:41:07Z <p>I'm trying to recreate a compartmental model from Yu et al (2017). In this model there is a expanded taylor series power function that is used that I am not understanding how to code. <a href="https://i.stack.imgur.com/4och9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4och9.png" alt="powerfunction"></a> where W(t) = 0, K = 1350, and N = 0. W(t) will increase in number as it is recent mortality based on other equations. </p> <p><a href="https://i.stack.imgur.com/it9JI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/it9JI.png" alt="enter image description here"></a></p> <p>The model above is the kind I am trying to create. where the W compartment is the recent mortality (which is where W(t) comes form). I am aware of how to code the other compartments but the effect B(W(t);k;N) has on S is where I am struggling. </p> <p>B(W(t);k;N) is a behavioural function: "This function models peoples decrease in exposure to infectious individuals in response to reported influenza mortality"</p> <p>I was looking into the taylor function and but I'm not understanding how to use it. The basic model outline I have is this:</p> <pre><code>sir.model.closed &lt;- function (t, x, params) { #here we begin a function with three arguments S &lt;- x #create local variable S, the first element of x I &lt;- x #create local variable I R &lt;- x #create local variable R D &lt;- x #create a local variable for D W &lt;- x #create a local variable for W V &lt;- x #create a local variable for V with( #we can simplify code using "with" as.list(params), #this argument to "with" lets us use the variable names { #the system of rate equations dS &lt;- dI &lt;- dR &lt;- dD &lt;- dW &lt;- dV &lt;- dx &lt;- c(dS,dI,dR,dD,dW,dV) #combine results into a single vector dx list(dx) #return result as a list } ) } </code></pre> <p>Any guidance in this area would be appreciated.</p> <p>Yu D, Lin Q, Chiu AP, He D (2017) Effects of reactive social distancing on the 1918 influenza pandemic. PLoS ONE 12(7): e0180545. <a href="https://doi.org/10.1371/journal.pone.0180545" rel="nofollow noreferrer">https://doi.org/10.1371/journal.pone.0180545</a></p> https://stats.stackexchange.com/q/398072 1 The question of Taylor expansion of loss function in XGBoost [duplicate] Bowen Peng https://stats.stackexchange.com/users/239780 2019-03-18T02:32:31Z 2019-03-18T15:33:13Z <div class="question-status question-originals-of-duplicate"> <p>This question already has an answer here:</p> <ul> <li> <a href="/questions/396807/xgboost-tayler-expansion-detail" dir="ltr">xgboost tayler expansion detail [duplicate]</a> <span class="question-originals-answer-count"> 1 answer </span> </li> </ul> </div> <p>I am learning <code>XGBoost</code> from <a href="https://xgboost.readthedocs.io/en/latest/tutorials/model.html#" rel="nofollow noreferrer">documentation</a>, but there are a few questions in the derivation of it.</p> <p>In the part of <code>Additive Training</code> of <code>Tree Boosting</code>, they say we take the Taylor expansion of the loss function up to the second order in general case, but I get some questions in derivation from:</p> <p><span class="math-container">\text{obj}^{(t)} = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t) + \mathrm{constant}</span></p> <p>to:</p> <p><span class="math-container">\text{obj}^{(t)} = \sum_{i=1}^n [l(y_i, \hat{y}_i^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t) + \mathrm{constant}</span></p> <p>where the <span class="math-container">g_i</span> and <span class="math-container">h_i</span> are defined as</p> <p><span class="math-container">\begin{split}g_i &amp;= \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)})\\ h_i &amp;= \partial_{\hat{y}_i^{(t-1)}}^2 l(y_i, \hat{y}_i^{(t-1)})\end{split}</span></p> <p><br> I mean I know how to make the Taylor series expanded to second order:</p> <p><span class="math-container">f(x) = f(x_k) + (x - x_k)f^{'}(x^{k}) + \frac{1}{2!}(x-x_k)^2f^{''}(x_k) + o^n</span></p> <p>And I assume <span class="math-container">f(x) = l(y_i, x)</span>, <span class="math-container">x = \hat{y}^{(t-1)} + f_t(x_i)</span> and <span class="math-container">x_k = \hat{y}^{(t-1)}</span>, then use <span class="math-container">\partial\hat{y}^{(t-1)}</span> in the Talyor series, so we get the right result as mentioned above.</p> <p>But I don't know that is a right derivation or not and even if it is right, I still feel it is hard to understand why they choose to expand it in this way. </p> <p>I would appreciate it if anyone could help me.</p> https://stats.stackexchange.com/q/396807 1 xgboost tayler expansion detail [duplicate] monk https://stats.stackexchange.com/users/239805 2019-03-11T08:54:18Z 2019-03-13T07:59:16Z <div class="question-status question-originals-of-duplicate"> <p>This question already has an answer here:</p> <ul> <li> <a href="/questions/202858/xgboost-loss-function-approximation-with-taylor-expansion" dir="ltr">XGBoost Loss function Approximation With Taylor Expansion</a> <span class="question-originals-answer-count"> 1 answer </span> </li> </ul> </div> <p><a href="https://i.stack.imgur.com/2phJY.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2phJY.jpg" alt="enter image description here"></a></p> <p>This is the objective function for Xgboost. <br/> I have no idea where <span class="math-container">g_{i}</span> and <span class="math-container">h_{i}</span> came from <br/> is some one explain how this two terms came form? <br/> or direct me to the related tutorial page then I will search for it</p> https://stats.stackexchange.com/q/395473 0 Derive taylor series expansion of df Dominic Joseph https://stats.stackexchange.com/users/189610 2019-03-04T08:25:39Z 2019-03-04T08:25:39Z <p>I was trying to understand ito's lemma. When I came across the taylor series expansion of df(x).</p> <p>df(x) = f'(x) dx + (1/2!) f''(x) (dx)^2 + ...</p> <p>I searched everywhere for the derivation of this but couldn't find. So I tried it on my own.</p> <p>f(x)= f(a) + f'(a)(x-a) + (1/2!) [f''(a)*(x-a)^2] +...</p> <p>so if Itake d/dx</p> <p>df(x)/dx = f'(a) + (1/2!) [f''(a)d(x-a)^2/dx] + ...</p> <p>Making it</p> <p>df(x)/dx = f'(a) + (1/2!) [f''(a) 2(x-a)] + (1/3!) [f''(a) 3(x-a)^2] +...</p> <p>Can someone help me with how to proceed from here?</p> https://stats.stackexchange.com/q/392536 0 How to use derivatives of a function to better estimate its variance over the domain? MInner https://stats.stackexchange.com/users/41167 2019-02-14T18:57:08Z 2019-02-15T17:39:30Z <p>How to use derivatives of a function to better estimate its variance over the domain?</p> <p>I have a scalar smooth function <span class="math-container">f(x)</span> and a multivariate random variable <span class="math-container">x</span> with known distribution (e.g. multivariate standard normal with large diagonal sigma). I know that the function f is very peaky in a sense that for majority of x'es its value rarely deviate from the mean, but on a small subset of x'es it attains extreme values. I need a finite sample estimate of the variance of this function over the domain of X. I can sample x, compute f(x) and use a standard formula, but it seems to be not very efficient since the domain of X is large and high dimensional (hundreds of dimensions) and I know that it is mostly constant. </p> <p>Intuitively, if I encounter a region with low curvature and "close to the mean" value of f(x), I can "mark" it as "not interesting" and explore the rest of X. </p> <p>Is there a formal way of doing this? Assume for simplicity that f is antisimmetric so f(x) = -f(-x) so we only need to look for maximas of x. One could probably do something like a variational approximation q such that x' ~ q(x'|x) maximizes expectation of f(x') while being not too far from the standard normal, but this way we have a significant chance of greatly underestimating the variance. </p> https://stats.stackexchange.com/q/380417 0 Iterated estimation of Taylor series luchonacho https://stats.stackexchange.com/users/100369 2018-12-05T10:24:06Z 2018-12-11T11:41:29Z <p>Say your data generating process is given by the function <span class="math-container">y=f(x|\theta)</span>, where <span class="math-container">y</span> and <span class="math-container">x</span> represent variables (data) and <span class="math-container">\theta</span> represent parameter(s). For convergence reasons (e.g. <span class="math-container">f(\cdot)</span> is highly non-linear on parameters and a GMM estimator does not converge), you decide to estimate a Taylor series expansion of <span class="math-container">f(\cdot)</span> around <span class="math-container">\theta=\theta_0</span>. Let's denote this approximated function as <span class="math-container">y \approx g(x|\theta)_{\theta_0}</span>.</p> <p>Say you estimate <span class="math-container">\theta</span> in <span class="math-container">g(\cdot)</span> based on a random sample of <span class="math-container">\{y,x\}</span>, and you get <span class="math-container">\hat\theta_1</span>. Then, you recompute the Taylor series approximation around this point estimate (keeping the Taylor series order constant), and produce <span class="math-container">y \approx g(x|\theta)_{\hat\theta_1}</span>. Then, you estimate again, yielding <span class="math-container">\hat\theta_2</span>. You iterate until</p> <p><span class="math-container">$$ (\hat\theta_n - \hat\theta_{n+1})^2 &lt; \epsilon $$</span></p> <p>for an arbitrary threshold <span class="math-container">\epsilon &gt; 0</span>.</p> <p>Convergence (in terms of the optimisation criterion above) is of course of paramount importance. Notice that for an arbitrarily large <span class="math-container">\epsilon</span> there is always a solution, as long as <span class="math-container">\hat\theta</span> can be computed, which itself depends on the properties of <span class="math-container">g(\cdot)</span>, e.g. on the order of the Taylor expansion; a linear model is always estimable, beyond trivial issues like multicolinearity. </p> <p>My question is, <strong>is the method above a thing?</strong> I've searched for "iterated estimation of taylor series" on Google, in this forum and in Math.SE and cannot find anything about this. Maybe the method is just plainly wrong, e.g. convergence is not assured by any known theorem.</p> <hr> <p><strong>More details on the method</strong></p> <p>For instance, consider a <a href="https://en.wikipedia.org/wiki/Constant_elasticity_of_substitution" rel="nofollow noreferrer">CES production function</a>:</p> <p><span class="math-container">$$ Y = \left(\alpha K^\theta+ (1-\alpha)L^\theta\right)^{1/\theta} $$</span></p> <p>where Y, L and K are variables, and <span class="math-container">\alpha</span> and <span class="math-container">\theta</span> are parameters. Assume we are particularly interested in estimating <span class="math-container">\theta</span>.</p> <p>So, you produce a first order Taylor series expansion of the <strong>log</strong> of <span class="math-container">Y</span>, around <span class="math-container">\theta= \theta_0</span>. The new formula (which is equivalent to the so-called <a href="https://en.wikipedia.org/wiki/Cobb%E2%80%93Douglas_production_function#Translog_production_function" rel="nofollow noreferrer">translog production function</a> when <span class="math-container">\theta_0 = 0</span>) is:</p> <p><span class="math-container">$$ln(Y) \approx \frac{1}{\theta_0} ln\left(\alpha K^\theta_0+ (1-\alpha)L^\theta_0\right) + (\theta - \theta_0)\left[\frac{1}{\theta^2} ln\left(\alpha K^\theta_0+ (1-\alpha)L^\theta_0\right) + \frac{1}{\theta_0}\frac{\left(\alpha K^\theta_0 ln(K)+ (1-\alpha)L^\theta_0 ln(L)\right)}{\alpha K^\theta_0+ (1-\alpha)L^\theta_0} \right] $$</span></p> <p>So, you estimate the above equation with a random sample of <span class="math-container">\{Y,L,K\}</span>, using e.g. non-linear least squares, for a given arbitrary <span class="math-container">\theta_0</span>. <strong>Importantly</strong>, <span class="math-container">\theta_0 \neq 0</span>, because otherwise the equation above changes completely (see translog function in link). From this estimation, you obtain an estimate of <span class="math-container">\theta</span>, <span class="math-container">\hat\theta_1</span>. Then, you re-estimate the model assuming <span class="math-container">\theta_0 = \hat\theta_1</span> (so, a new Taylor series around a different value). Then, estimate the new equation, obtaining <span class="math-container">\hat\theta_2</span>. Iterate until some convergence criterion is fulfilled.</p> https://stats.stackexchange.com/q/378775 2 Taylor expansion for random variables Hercules https://stats.stackexchange.com/users/228115 2018-11-26T07:04:30Z 2018-11-26T14:52:18Z <p>Let <span class="math-container">X_n</span> and <span class="math-container">Y_n</span> be random variables such that <span class="math-container">X_n-Y_n\overset{p}{\longrightarrow}0</span> as <span class="math-container">n\rightarrow\infty</span>. Let <span class="math-container">f(.)</span> is a differentiable function. Is the following correct?</p> <p><span class="math-container">f(X_n) = f(Y_n) + f'(Y_n)(X_n-Y_n) + o_p(X_n-Y_n)</span> as <span class="math-container">n\rightarrow\infty</span></p> https://stats.stackexchange.com/q/362575 4 Taylor Series Expansion of Unconditional Expectation Amrit Prasad https://stats.stackexchange.com/users/203347 2018-08-16T23:02:23Z 2018-08-20T11:50:37Z <p>We know that the best 1st order approximation of an unconditional expectation is the following-</p> <p>$$E(y|x)=(E(y)-\beta E(x))+\beta x$$</p> <p>where \beta=\frac{\operatorname{Cov}(y,x)}{\operatorname{Var}(x)}=\frac{E(xy)-E(x)E(y)}{E(x^2)-[E(x)]^2}.</p> <p>Is there an a general expression available that would expand the above to include the higher orders of x analogous to the Taylor's Series Expansion?</p> <p>$$E(y|x)=a_0+a_1x+a_2x^2+...\infty$$</p> https://stats.stackexchange.com/q/359541 2 Is there any sort of higher-order SVD (quadratic and above) for dimensionality reduction? user650261 https://stats.stackexchange.com/users/30776 2018-07-28T21:53:19Z 2018-07-28T21:53:19Z <p>X-Posted on math.stackexchange, apologies, though I thought this was equally relevant to both communities.</p> <p>I'm wondering if there exists any higher-order SVD for dimensionality reduction. Note that I do not mean <a href="https://en.wikipedia.org/wiki/Multilinear_principal_component_analysis" rel="nofollow noreferrer">multilinear PCA</a>, which operates on data tensors, but some form of SVD which can produce, say, a quadratic approximation of a dataset.</p> <p>Intuitively, SVD takes in a set of vectors as a datset, and computes the ranked eigenvalues and eigenvectors which correspond to a reduced subspace and a "weight" representing their importance. Given a decomposition:</p> <p>$$A = U\Sigma V^T</p> <p>Where A is a matrix where the columns are data vectors, the truncated matrix U (keeping only the leftmost n columns; call this truncated version \mathcal{U}) provides a reduced approximate basis for the original dataset A. However, this approximation is purely linear. It can thus be thought of as a linear approximation of the data on some n reduced variable set. For a particular column of a_i of A, a_i \approx \mathcal{U} x_i for some dimensionality-reduced vector x_i.</p> <p>I am wondering if it is possible to do something akin to a Taylor expansion here, and recover a higher-order approximation of the data. For example, for a quadratic system of order n, I'd like to generate the \mathcal{U} matrix, but also a third-order tensor \mathcal{W} such that a "quadratic" approximation of the data could be written, as, say: </p> <p>a_i \approx \mathcal{U} x_i + \frac{1}{2} x_i^T (\mathcal{W} ~ \vdots ~ x_i)</p> <p>where \vdots is a tensor-vector contraction. (This is a form I made up now for discussion's sake; it is possible that the true form of some quadratic approximation is slightly different). Here, the best x_i the quadratic approximation will be better than the linear one.</p> <p>Ideally, I'd be searching for some general theory which would allow for arbitrary order approximation.</p> <p>Is there any general established method for this (or something similar)? If so, is it tractable? If not, is there a mathematically grounded reason why not?</p> https://stats.stackexchange.com/q/350463 0 Exponential Kth Moment Derivation cdDC https://stats.stackexchange.com/users/211037 2018-06-08T15:26:17Z 2018-06-08T15:26:17Z <p>I have essentially a mathematical question, relating to deriving the formula for the kth moment of an exponential. I can't seem to work out how we get from the 2nd line to the 3rd line; i.e. the fraction to the expansion: \begin{align}%\label{} \nonumber M_X(s)&amp;=\frac{\lambda}{\lambda-s}\\ \nonumber &amp;=\frac{1}{1-\frac{s}{\lambda}}\\ \nonumber &amp;=\sum_{k=0}^{\infty} \left(\frac{s}{\lambda}\right)^k, \hspace{10pt} \textrm{for }\left|\frac{s}{\lambda}\right|&lt;1\\ \nonumber &amp;=\sum_{k=0}^{\infty} \frac{k!}{\lambda^k} \frac{s^k}{k!}. \end{align} My maths is quite rusty so this is probably something simple but I would appreciate if someone could take a minute to help me out. Cheers!</p> https://stats.stackexchange.com/q/329370 2 Can we use backpropagation to fit other models? Edv Beq https://stats.stackexchange.com/users/87106 2018-02-19T03:27:03Z 2018-03-02T13:26:30Z <p>It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials: </p> <p> f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n $$</p> <p>We can represent it in a graph and take derivatives backwards the same way as everything is differentiable including the center. </p> <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href="https://i.stack.imgur.com/fHTi0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fHTi0.png" alt="Taylor Graph"></a></p> <p>Any thoughts? Thanks</p> https://stats.stackexchange.com/q/317781 1 Understanding a Taylor expansion for the bias of local polynomial regression Epiousios https://stats.stackexchange.com/users/166428 2017-12-08T12:49:23Z 2017-12-08T13:09:25Z <p>I'm trying to understand the proof of an expression for the asymptotic bias in local polynomial regression of degree p\ge0.</p> <p>Specifically, I'm distraught with equation (3.59) on <a href="https://books.google.de/books?id=BM1ckQKCXP8C&amp;printsec=frontcover&amp;dq=local%20polynomial%20modelling%20and%20its%20applications%20theorem%203.1&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwiagoCe0vfXAhVJI1AKHVEeAqUQ6AEIKTAA#v=snippet&amp;q=%22by%20using%20the%20taylor%20expansion%20the%20conditional%20bias%22&amp;f=false" rel="nofollow noreferrer">page 102 of this book</a>. This is part of the proof of Theorem 3.1 on <a href="https://books.google.de/books?id=BM1ckQKCXP8C&amp;printsec=frontcover&amp;dq=local%20polynomial%20modelling%20and%20its%20applications%20theorem%203.1&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwiagoCe0vfXAhVJI1AKHVEeAqUQ6AEIKTAA#v=snippet&amp;q=%22the%20asymptotic%20conditional%20bias%20for%20p%22&amp;f=false" rel="nofollow noreferrer">page 62</a>.</p> <p>Here's the setup. Let (X_1, Y_1),\ldots,(X_n,Y_n) be iid taking values in \mathbb R^2, let X_1 have density f. Let m(x)=E(Y|X=x) and let K be a symmetric kernel with bounded support, let K_h(t) = K(t/h)/h, where h is the bandwidth. Write$$\mathbf X=((X_i-x_0)^j)_{i=1,\ldots,n \atop j=0,\ldots,p}, \mathbf W = \operatorname{diag}(K_h(X_1-x_0),\ldots,K_h(X_n-x_0)),\\ \mathbf y=(Y_1\ldots,Y_n)^T, \mathbf m=(m(X_1),\ldots,m(X_n))^T.$$</p> <p>Then the conditional bias of the local polynomial estimator \hat\beta=(\mathbf X^T \mathbf W \mathbf X)^{-1}\mathbf X^T \mathbf W \mathbf y is$$\operatorname{Bias}(\hat\beta|(X_1,\ldots,X_n))=(\mathbf X^T \mathbf W \mathbf X)^{-1}\mathbf X^T \mathbf W \mathbf r =: S_n^{-1}\mathbf X^T \mathbf W \mathbf r,$$where \mathbf r = \mathbf m-\mathbf X \beta,\, \beta=(m(x_0),\ldots,m^{(p)}(x_0)/{p!}).</p> <p>Assume that m^{(p+1)}(\cdot) is continuous in a neighborhood of x_0. Fan writes on page 102:</p> <blockquote> <p>By using the Taylor expansion the conditional bias S_n^{-1}\mathbf X^T \mathbf W \mathbf r of \hat\beta can be written as$$S_n^{-1}\mathbf X^T \mathbf W \Bigl[\beta_{p+1}(X_i-x_0)^{p+1}+o_P\left\{(X_i-x_0)^{p+1}\right\}\Bigr]_{1\le i\le n}$$</p> </blockquote> <p>I don't understand what is meant by o_P\left\{(X_i-x_0)^{p+1}\right\} in this context. I know that the usual definition is that it's a term which converges in probability to zero even after dividing by (X_i-x_0)^{p+1}. But what converges in probability here? Is it meant that this holds as n\to\infty?</p> <p>I tried writing everything out but failed to understand what he means:$$ \begin{align} \mathbf r &amp;= \mathbf m-\mathbf X \beta \\ &amp;= \Biggl[\sum_{l=1}^{p+1} \frac{m^{(l)}(x_0)}{l!} (X_i-x_0)^{l} + (X_i-x_0)^{p+1}\frac{m^{(p+1)}(\xi_i)}{(p+1)!} \text{ (using Lagrange remainder)}\\ &amp;\quad\quad- \sum_{l=0}^{p} \frac{m^{(l)}(x_0)}{l!} (X_i-x_0)^{l}\Biggr]_{1\le i\le n}\\ &amp;= \left[\frac{m^{(p+1)}(x_0)}{(p+1)!} (X_i-x_0)^{p+1} + (X_i-x_0)^{p+1}\frac{m^{(l)}(\xi_i)}{(p+1)!}\right]_{1\le i\le n} \end{align} $$If I could show that in some sense$$ (X_i-x_0)^{p+1}\frac{m^{(l)}(\xi_i)}{(p+1)!} = o_P((X_i-x_0)^{p+1}) $$I would be done, but I'm not even sure in what sense he means this..</p> https://stats.stackexchange.com/q/310033 0 Priors on Taylor Expansion series Afshin https://stats.stackexchange.com/users/123131 2017-10-26T10:47:21Z 2017-10-26T12:09:21Z <p>I'm wondering what priors can i choose for a Taylor series as follows: \theta_{1}+\theta_{2} (y-\alpha) + \theta_{3} (y-\alpha)^2</p> <p>What priors should I use for updating these parameters (\theta_{1},\theta_{2},\theta_{3}) in a Gibbs sampler to accommodate the conditions on Taylor expansion? Any help would be appreciated! Thank you so much in advance</p> https://stats.stackexchange.com/q/309469 3 Proof that ML Estimator is asymptotically Normal Mario Migliaccio https://stats.stackexchange.com/users/180168 2017-10-23T15:44:59Z 2017-10-23T23:58:57Z <p>I'm trying to proof that the Maximum <strong>Likelihood Estimator</strong> is Asymptotically Normal distribuited. I'm stuck in the lasts steps. Here's what I've done:</p> <p>I do the Taylor's expansion of, that's the mean of the score function:$$\frac{1}{n}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}$$The Taylor's expansion around the true, unknown, value \theta_0 is:$$ \left.\frac{1}{n}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}\right\vert_{\theta_0}+ \left.\frac{1}{n}\sum \frac{\partial^2 log f(\underline{x}, \theta)}{\partial \theta^2}\right\vert_{\theta_0}(\theta-\theta_0) +R/n $$We know that the mean is an approximatio of the expected value thanks to Weak Law of Large Numbers. The first one goes to 0 and second goes to a -I_n(\theta) and the third goes to 0 for assumptions on the form of the remainder.</p> <p>Now my problem is that I was told to use the ML estimation \hat\theta and do again the Taylor's expansion, but I didn't get all the steps</p> <p>I know only that in the end we get this:$$ (\hat{\theta}-\theta_0)=\left[\frac{1}{\sqrt{n}}\sum \frac{\partial^2 log f(\underline{x}, \theta)}{\partial \theta^2}\right]^{-1}\left[\frac{1}{\sqrt{n}}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}+ R/n\right] $$</p> <p> \hat\theta-\theta_0 \sim N(0,I^{-1}(\theta_0)) </p> <p>I know that we have to use the Central Limit Theorem, but I'm quite confused and I don't know how to go on. I tried to get some information, but with no results. Can someone provide me a clear explanation on why the MLE goes to Normal asymptotically? Thank you.</p> https://stats.stackexchange.com/q/308752 1 Jaynes Probability theory 4.70 （Different answers with Jaynes when using Taylor power series.) hello_god https://stats.stackexchange.com/users/153910 2017-10-19T03:54:21Z 2017-10-26T20:59:41Z <p>I have read this derivation. <strong>$$L(f)\equiv{lng(f|DX)}=nln(f)+(N-n)ln(1-f)+const \;(4.69) $$expand L(f) in a power series about \hat{f}.The first terms as$$L(f) = L(\hat{f}) - \frac{(f-\hat{f})^2}{2\sigma^2}+... \;(4.70)$$, where$$\sigma^2 \equiv\frac{\hat{f}(1-\hat{f})}{N},\; where \;n = Nf$$</strong></p> <p><strong>But I can not get that result like 4.70.</strong> For$$L'(\hat{f})=\frac{(n-n\hat{f} - N\hat{f}+n\hat{f})}{\hat{f}(1-\hat{f})}=\frac{N(f-\hat{f})}{\hat{f}(1-\hat{f})}$$, So I will give my result with$$L(f)=L(\hat{f})+\frac{(f-\hat{f})^2}{\frac{\hat{f}(1-\hat{f})}{N}}+...\;(4.71)$$<strong>I don't think (4.71) is the same as 4.70.</strong> So i posted this thread. Where am i wrong? Thanks. </p> https://stats.stackexchange.com/q/304580 0 Taylor expansion in xgboost [duplicate] clueless_undergrad37 https://stats.stackexchange.com/users/176329 2017-09-23T02:41:31Z 2019-03-27T14:05:37Z <div class="question-status question-originals-of-duplicate"> <p>This question already has an answer here:</p> <ul> <li> <a href="/questions/202858/xgboost-loss-function-approximation-with-taylor-expansion" dir="ltr">XGBoost Loss function Approximation With Taylor Expansion</a> <span class="question-originals-answer-count"> 1 answer </span> </li> </ul> </div> <p>I'm reading through the math of xgboost:</p> <p><a href="https://xgboost.readthedocs.io/en/latest/model.html" rel="nofollow noreferrer">https://xgboost.readthedocs.io/en/latest/model.html</a> </p> <p>Under the ADDITIVE TRAINING section of the objective function, I saw that in the derivation of the objective at step t, a claim is made about how the GENERAL form of the objective function for any loss function (MSE, logistic loss, etc.) takes on the form of a 2nd order Taylor expansion. How do we know this is true in general? Is there a mathematical explanation of this that I'm just not seeing?</p> <p>I basically want to know how we are able to conclude that the following is the GENERAL form of the objective at step t:</p> <p>$$\text{obj}^{(t)} = \sum_{i=1}^n [l(y_i, \hat{y}_i^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t) + constant$$</p> <p>where the g_i and h_i are defined as:</p> <p>$$\begin{split}g_i &amp;= \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)})\\ h_i &amp;= \partial_{\hat{y}_i^{(t-1)}}^2 l(y_i, \hat{y}_i^{(t-1)}) \end{split}$$</p> https://stats.stackexchange.com/q/302780 0 Higher order delta / taylor series approximation relationship to normal distribution? Steve Kay https://stats.stackexchange.com/users/69771 2017-09-12T15:41:52Z 2017-09-12T15:41:52Z <p>For a normally distributed variable X, one can call on the delta method to provide an asymptotically normally distributed variable for a non-linear function of it, g(X). This is based on a linear taylor series approximation and uses Slutsky's theorem to justify the normality.</p> <p>Recently I programmed some routines in R that calculate the mean and variance of g(x) using higher order taylor series approximations. In my usual non-thinking mode, I never thought to consider that it might not be possible to just treat these new "improved" estimates as mean and variance of a normally distributed variable. Can someone please confirm that my suspicion is correct - ie it is unknown what asymptotic distribution is derived given the extra included random moment terms (except for limited case where second order taylor series is computed and g'(x)=0)? </p> <p>It just strikes me as bit of a shame if this is the case- we are left with "better" estimates of the mean and variance of g(x) but no way to compute confidence intervals from them? Given samples are always finite and the applicability of asymptotic theory to any sample result is always in doubt, it may well be that inserting such derived means and variances into a normal distribution may give better coverage than those obtained under standard delta technique? Don't know if anyone knows of any monte carlo simulation work around this issue?</p> <p>Thanks.</p> https://stats.stackexchange.com/q/299080 0 Estimating polymomial coefficient in R show_stopper https://stats.stackexchange.com/users/59397 2017-08-21T19:51:55Z 2017-08-21T21:07:20Z <p>I used a Taylor series to expand log(1 - ax) so I could estimate the value of parameter 'a'.</p> <p>The expansion becomes -ax - a^2*x^2/2 - a^3*x^3/3 . . .</p> <p>Now I need to estimate the parameter 'a' using regression and for simplicity I am only using the first 3 terms in the expansion.</p> <p>The equation to be estimated becomes y ~ ax + a^2*x^2/2 + a^3*x^3/3 [I have absorbed the negative sign on the left hand side of the equation]</p> <p>I wanted to ask if there is a way to estimate the coefficients a , a^2 and a^3 in the above equation, keeping in mind that all the three coefficients are powers of each other.</p> <p>Is there a package in R for this? </p> <p>Please do note that the Taylor series expansion was necessary as there are several other terms in the original equation which I haven't mentioned here. </p> <p>Edit:</p> <p>The original equation I have is:</p> <p>Y ~ (1 - aX)(B^b)(C^c)(D^d)</p> <p>In the above equation I have to estimate a,b,c,d, where a is to be estimated as aconstant while b,c and d as smooth splines. </p> <p>So I have taken log on both side, which makes it:</p> <p>log(Y) ~ log(1 - aX) + b<em>log(B) + c</em>log(C) + d*log(D) </p> <p>If there is a better way to approach the entire equation, do mention.</p> https://stats.stackexchange.com/q/289484 1 Expansion of Cumulant Generating Function of Negbin Daschin https://stats.stackexchange.com/users/164739 2017-07-08T10:02:04Z 2017-07-09T08:18:09Z <p>Let X \thicksim Negbin(r,p) where (0\lt p \lt 1) </p> <p>I want to derive skewness and kurtosis of X by getting the Cgf of X.</p> <p>First, since Followance of Negative Binomial equals to the distribution r-th repetition of Geo(p), such as X \equiv Z_1+Z_2+...+Z_r where Z_i \thicksim Geo(p), \forall i\in\{1,2,3,..r\} </p> <p>then since Mgf of Z_i = pe^t(1-e^t(1-p))^{-1}, Mgf of X = (pe^t(1-e^t(1-p))^{-1})^r</p> <p>Then Cgf of X, </p> <p>C(t) = \log(pe^t(1-e^t(1-p))^{-1})^r\\ = r[\log p + t - \log(1-e^t(1-p))]</p> <p>then since -\log(1-A) = A +1/2A^2+1/3A^3 .., </p> <p>C(t) = r[\log p + t + \sum_{i=1}^\infty\dfrac{[e^t(1-p)]^i}{i}]</p> <p>then I need to change above identity into the form of </p> <p>\sum_{i=1}^\infty \dfrac{c_i(0)t^i}{i!} so that I could easily assume the characteristic of distribution from the coefficients, such as mean/variance/skewness/kurtosis etc.</p> <p>But I can't figure out how could I rearrange the \sum_{i=1}^\infty\dfrac{[e^t(1-p)]^i}{i} into a form of t^i not (e^t)^i.</p> <p>Any hints? </p> https://stats.stackexchange.com/q/270069 1 What is the intuition behind neural networks? Dmitry Nalyvaiko https://stats.stackexchange.com/users/152382 2017-03-27T10:37:48Z 2017-03-27T11:20:37Z <p>Everywhere in the theory of neural networks, authors saying that idea came about by observing the work of the human brain. But I can not believe in that. I guess, everything is much simpler and neural networks is specific functions of math's series. Proof of this is the existence of the Weierstrass theorem and Taylor series, which says that every function can be approximated by certain polynomials. Am I right?</p> https://stats.stackexchange.com/q/269249 1 Estimate mean and variance of pdf from truncated taylor expansion of logarithm of pdf Heinz https://stats.stackexchange.com/users/154216 2017-03-23T00:26:01Z 2017-03-30T14:46:44Z <p>In a maximum likelihood fit, one estimates the parameter with the mode of the likelihood L, and the variance of this estimator with the second derivative of \log(L):$$ \bar\theta = \mathrm{Mode}[L]  \mathrm{Var}[\bar\theta] \approx -\left(\frac{\partial^2}{\partial \theta^2}\log(L(\theta))\Big|_{\bar\theta}\right)^{-1} $$From a bayesian point of view, it works because in the statistical limit the likelihood becomes normal and sharper than any regular enough prior, and this estimates are exact for a normal distribution. However, with not enough data, this estimate may become evidently biased.</p> <p>I would like to raffinate this approximation using successive derivatives of \log(L). My intent is to estimate at least the mean and variance of the posterior distribution (which is L times the prior, then normalised). Using derivatives of \log(p) has the advantage that it does not require normalisation:$$ \partial_x\log(N\cdot p(x))=\partial_x\big(\log(p(x))+N\big)=\partial_x\log(p(x)) $$</p> <p>I could not find anything about this in the literature, but I'm not in the field so I hope either I just didn't use the right words, or there exists a better technique.</p> <p>Trying to work out this, I got in trouble with the fact that odd derivatives in the expansion of \log(p) give a divergence at infinity on one side when exponentiating the truncated series, so I could not even try to integrate the approximate pdf to directly compute the moments as a function of expansion coefficients.</p> https://stats.stackexchange.com/q/255813 3 Characteristic function of distribution Surfer on the fall https://stats.stackexchange.com/users/143587 2017-01-12T10:29:01Z 2017-11-27T22:33:02Z <p>$$p(x)=e^{-2 |x|}$$with x in [-inf, +inf]. I've calculated the characteristic function as E[e^{ikx}]=\frac{1}{ik+2}-\frac{1}{ik-2}=\frac{4}{k^2+4}. Now i'd like the moments.. so I suppose I should manipulate this expression in order to reduce it to a sum of infinite moments, but I can't! Could you help me?</p> https://stats.stackexchange.com/q/241399 1 Manually compute linearized standard errors of mean in survey Mario https://stats.stackexchange.com/users/135549 2016-10-20T15:04:06Z 2016-10-20T15:08:30Z <p>I am would like to manually compute the Taylor-linearized standard error and 95% confidence interval for the mean of a variable <code>x</code> in a survey with 1-stage primary sampling Units <code>y</code> and strata <code>z</code>. I would like to obtain the same value that I get when using the following <code>Stata</code> command:</p> <pre><code>svyset Y [pweight=pweight], strata(z) svy: mean x </code></pre> <p>I would be grateful if you could show the complete formula and all the steps of the computation.</p> https://stats.stackexchange.com/q/230569 1 Delta method with mix of continuous and discrete variables user106927 https://stats.stackexchange.com/users/106927 2016-08-18T20:26:48Z 2016-08-19T07:59:42Z <p>This is my first question on Cross Validated so please bear with me if my question is lagging in any dimension. My question regards how to evaluate a Jacobian matrix when one variable is binary. I have googled quite extensively but found no answer. </p> <h1>Setup:</h1> <p>I have three variables p, r and s. p and r come from a copula and thus have uniform marginal distributions and covariance E[(p-\bar p)(r-\bar p )]=\sigma_{pr}. The binary variable s\in\{0,1\} is independent of p and r and E[s]=w.</p> <p>I define the vector \mathbf{x} with mean \mathbf{\bar x} and variance\mathbf{\Sigma}_\mathbf{x} :$$ \mathbf{x}=\begin{pmatrix} p\\ r\\ s \end{pmatrix}, \mathbf{\bar x}=\begin{pmatrix} \bar p\\ \bar r\\ w \end{pmatrix},\mathbf{\Sigma}_\mathbf{x}=\begin{pmatrix} \sigma_p^2&amp;\sigma_{pr}&amp;0\\ \sigma_{pr}&amp;\sigma_p^2&amp;0\\ 0 &amp;0&amp;w(1-w) \end{pmatrix} $$</p> <p>The variables r and s are used to construct a new variable with the function \Lambda(r,s) defined as:$$ \Lambda(r,s)=wG\left(s{G}^{-1}(r)+(1-s){F}^{-1}(r)\right)+(1-w)F\left(s{G}^{-1}(r)+(1-s){F}^{-1}(r)\right) $$F and G are continuous distributions with support on the positive real line.</p> <p>I want to know the covariance between p and the new variable created by \Lambda(r,s). My approach has been to use the Delta method.</p> <p>I define a new vector:$$ \mathbf{y}=q(\mathbf{x})=\begin{pmatrix} p\\ \Lambda(r,s) \end{pmatrix} $$In the case where all variables where continuous I would use the delta method to find an expression for the covariance matrix of \mathbf{y}:$$ \mathbf{\Sigma}_\mathbf{y}=\mathbf{D}\mathbf{\Sigma}_\mathbf{x}\mathbf{D}^T $$where \mathbf{D} is the Jacobian evaluated at \mathbf{\bar x}, \mathbf{D}=\mathbf{J}\Big|_{\mathbf{x}=\mathbf{\bar x}}.</p> <h1>Question</h1> <p>The last part is my central issue: <em>how (if possible) can I calculate \mathbf{D} when s is a binary variable?</em></p> <h3>My own Suggestion</h3> <p>I have been toying around with central differences. But I don't know if this approach works for the delta method. </p> <p>The Jacobian of q(\mathbf{x}) in this case could be defined as:$$ \mathbf{J}=\begin{pmatrix} 1&amp;0 &amp;0\\ 0 &amp;\frac{\partial\Lambda(r,s)}{\partial r} &amp;\frac{\Lambda(r,1)-\Lambda(r,0)}{2} \end{pmatrix} $$But how do I get from \mathbf{J} to \mathbf{D}? It is not obvious to me how to evaluate the \frac{\partial\Lambda(r,s)}{\partial r} and \frac{\Lambda(r,1)-\Lambda(r,0)}{2}, which is necessary for knowing \mathbf{D}.</p> https://stats.stackexchange.com/q/222110 5 Are Maximum Likelihood Estimators asymptotically unbiased? BloXX https://stats.stackexchange.com/users/122158 2016-07-04T18:16:13Z 2016-11-09T19:11:58Z <p>I can follow the proofs in which the asymptotic normal-distribution of a maximum likelihood estimator \tilde{\theta}_n is derived. </p> <p>however, does this already imply that the maximum likelihood estimates are asymptotically unbiased, i.e. do we have$$E(\tilde{\theta}_n) \to \theta \text{ as } n \to \infty?$$</p> <p>Since I am aware that in general it is not true that convergence in distribution implies convergence in moments, an explanation would be nice.</p> <p>The result is often stated e.g. <a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation#Higher-order_properties" rel="noreferrer">in Wikipedia</a> as "...it means that the bias of the maximum likelihood estimator is equal to zero up to the order n^{-1/2}")</p> <p>1.) Are some kind of regularity conditions from the mle theory used to establish this result?</p> <p>2.) Or is a \sqrt{n}-convergence of an estimator (to a normal distribution) in general already enough to establish convergence of its moments?</p> <p><strong>Note:</strong> the Wikipedia article mentions Cox, David R.; Snell, E. Joyce (1968). <em><a href="https://www.jstor.org/stable/2984505?seq=4" rel="noreferrer">A general definition of residuals</a></em> , as a source where the order of the bias is derived (formula (12) or (20)). </p> <p>However in this paper I can't follow the arguments 100%, since their Taylor approximation of L'(\widehat{\beta}) is lacking the remainder term. What is the argument used here to discard it completely?</p> https://stats.stackexchange.com/q/205555 5 Alternative Confidence interval for Odds Ratio \hat{p}\over{1-\hat{p}} from Logistic Regression? StatsStudent https://stats.stackexchange.com/users/7962 2016-04-05T07:01:41Z 2016-04-07T20:41:46Z <p>I have a question about confidence interval calculations for the odds ratio \hat{p}\over{1-\hat{p}} from a logistic regression model (perhaps obtained from the method of Generalized Estimating Equations, but that's of secondary importance). It's best asked after a bit of background and an example:</p> <p>Let's assume we have a simple logistic regression model with a single binary independent variable (X={0,1} ). For X=1, we can estimate the log odds by:</p> <p>\log({\hat{p}\over{1-\hat{p}}})=\hat{\beta}_0+\hat{\beta}_1</p> <p>To obtain a 95% confidence interval (CI) for the odds ratio, textbooks that I've consulted indicate that you calculate the 95% CI for the odds-ratio, {\hat{p}\over{1-\hat{p}}}, by:</p> <p>Lower Bound= e^{[\hat{\beta}_0+\hat{\beta}_1-1.96Var(\hat{\beta}_0+\hat{\beta}_1)]}</p> <p>Upper Bound= e^{[\hat{\beta}_0+\hat{\beta}_1+1.96Var(\hat{\beta}_0+\hat{\beta}_1)]}</p> <p><strong>What I am wondering is if it is appropriate to use the the Delta Method to obtain the variance of the odds ratio \hat{p}\over{1-\hat{p}} = \exp{(\hat{\beta}_0+\hat{\beta}_1)} instead?</strong> For example, could obtain a 95% confidence interval using the following method?:</p> <p>(1) First, exponentiate to obtain the point estimate of the odds ratio: \hat{p}\over{1-\hat{p}}$$=\exp(\hat{\beta}_0+\hat{\beta}_1)=$f(\hat{\beta}_0,\hat{\beta}_1)$.</p> <p>(2) Find the approximte large sample variance of this non-linear point estimate by the Delta methods by finding the gradient of $f$ as: $\nabla f =[\exp(\hat{\beta}_0+\hat{\beta}_1), \exp(\hat{\beta}_0+\hat{\beta}_1)]$</p> <p>(3) Calculate $Var({\hat{p}\over{1-\hat{p}}})$ $\approx$ $\nabla f \hat{\Sigma} \nabla f^T$. where $\hat{\Sigma}$ is the empirical covariance matrix of $\hat{\beta}$.</p> <p>(4) Find the approximate standard error, $SE$ by taking the square root of the approximate variance estimate $SE = \sqrt{\nabla f \Sigma \nabla f^T}$</p> <p>(5) Finally, obtain the 95% CI by $\exp(\hat{\beta}_0+\hat{\beta}_1)\pm Z_{1-.05/2}SE$.</p> <p>Is the delta method I have described appropriate to find a 95% confidence interval, and if it is, why is it never described in textbooks on logistic regression? If it isn't, why not, when it seems you can use it to linearize it for similar problems.</p> <p><strong>UPDATE:</strong> I updated the step 3 to make it clear that I'm using the estimated, empirical covariance matrix.</p> <p><strong>UPDATE 2</strong></p> <p>I still haven't received a very satisfactory answer to this question. It has even made me wonder why people don't use the Delta Method to compute confidence intervals around estimates of just $\hat{p}$ now, instead of talking the inverse of the logit link of the upper and lower confidence intervals. Maybe @FrankHarrel can shed some light as he seems especially knowledgeable about these types of calculations?</p> https://stats.stackexchange.com/q/203154 1 Characteristic function issue user146925 https://stats.stackexchange.com/users/76536 2016-03-23T03:00:03Z 2016-03-23T06:25:35Z <p>As mentioned in a previous post, I've been trying to work through ALL of the problems in Jacod and Protter's <em>Probability Essentials</em>. The following problem has been giving me issues:</p> <blockquote> <p>Let $Z \sim N(0,1)$. Show that $E[X^{2n+1}]=0$ and $E[X^{2n}]=\frac{(2n)!}{2^n n!}$</p> </blockquote> <p>The issue I'm having is proceeding form my Taylor Expansion for $\varphi_{_Z}(u)$. Here's my work:</p> <blockquote> <p>Since we're dealing with $N(0,1)$, we have $$\varphi_{_Z}(u)=e^{-\frac{1}{2}u^2}$$ Recalling that the Taylor expansion of $e^x$ wrapped around 0, we have $$\varphi_{_Z}(u)=e^{-\frac{1}{2}u^2}=\sum_{k\geq0}(-\frac{u^2}{2})^k\cdot\frac{1}{k!}=\sum_{k\geq 0}(-1)^k\frac{u^{2k}}{2^kk!}$$ $$\implies \varphi_{_Z}(u)=\sum_{k\geq 0}(-1)^k\frac{u^{2k}}{2^kk!}$$ Next, recall by a theorem in the book that $$\frac{d^m}{du^m}\varphi_{_Z}(0)=i^mE[Z^m]$$ where $i\in \mathbb{C}$. It follows that $$\frac{d^{2k}}{du^{2k}}\varphi_{_Z}(0)=\sum_{k\geq0}(-1)^k\frac{(2k)!}{2^k k!}=i^{2k}E[Z^{2k}]$$ and $$\frac{d^{2k+1}}{du^{2k+1}}\varphi_{_Z}(0)=\sum_{k\geq0}(-1)^k(0)=0$$ Since $(-1)^k=i^{2k}$, it follows that we have $$\sum_{k\geq0}i^{2k}\frac{(2k)!}{2^k k!}=i^{2k}E[Z^{2k}]$$ $$\implies E[X^{2k}]=\frac{(2k)!}{2^k k!}$$</p> </blockquote> <p>Is it justified to get rid of the sum? I'm basically at the answer, but keeping the sum is killing me. Any suggestions? </p>