Questions tagged [numerics]

Also known as Numerical Analysis, Numerics aims to provide methods and algorithms for numerical computations.

Filter by
Sorted by
Tagged with
1
vote
0answers
31 views

Why did the log likelihood decrease with additional parameters?

I'm trying to decide the effect of some factors on the time for an event to happen . Specifically, I am looking at how long it takes for the subject to pass a test (recognize the stimulus) when ...
7
votes
4answers
399 views

Expectation of $\ln(1 + e^x)$, where $x$ is normally distributed

I need to evaluate the following integral: $$\int_{-\infty}^\infty\mathrm d x \exp\left(-\frac{(x-\mu)^2}{2\nu}\right) \ln(1+e^x)$$ where $\mu$ is a finite real number and $\nu > 0$. This is just ...
0
votes
0answers
13 views

How do you handle large numbers that give infinity in intermediate calculations? [duplicate]

For example, say you need to calculate $\ln \left( \alpha + \beta e^x \right)$ for potentially large $x$. The value itself might be small, but since you need to calculate $e^x$ first, it produces ...
0
votes
0answers
19 views

Random effects estimates using heuristic / numeric approaches

This is perhaps more a conceptual question. I'm using an heuristic algorithm (ABC: Artificial Bee Colony) to search solutions for a given model that can take additional factors such as numerical and ...
1
vote
0answers
8 views

How does the choice of norm affect the condition of a problem?

we know that for a differentiable problem, the absolute condition number is the norm of its jacobian i.e. ||J||. We also know that a well-conditioned problem typically has a small condition number. ...
1
vote
1answer
203 views

Inverse-normal CDF approximation in Excel, Python or R

I read that the implementations of Inverse-normal cumulative distribution function (CDF) /quantile / ppf in R, Python (scipy) and Excel give similar results. However, I can't find the very formulae ...
0
votes
0answers
8 views

How to measure changes in condition indices over time

I am trying to understand how adding data, one observation at a time, affects the condition indices of a model. A similar question is how adding individual observations affects the principal ...
0
votes
0answers
16 views

How to quantify rate of convergence in terms of number-of-observations instead of iterations?

I observe discrete points of data, and wish to compute an integral across those points. Since the data is quite sparse, I need to interpolate and extrapolate. There are various approaches in use (...
2
votes
0answers
104 views

Intuition about a coupon problem were we ask for the distribution of the unique coupons when the number of draws is fixed

Alternative viewpoint of the coupon collectors problem In the coupon collectors problem we draw from a collection of $n$ coupons, with replacement and ask the question how many draws $K$ it takes to ...
3
votes
0answers
79 views

Why is it much quicker to compute ridge regression than regular linear regression?

By my understanding, for a matrix with n samples and p features: Ridge regression using cholesky takes O(p^3) time Ordinary linear regression takes O(p^3) time Singular value decomposition if u, v ...
4
votes
1answer
143 views

How can we numerically compute the autocorrelation of a sample from a Markov chain generated by the Metropolis-Hastings algorithm?

Let $(X_n)_{n\in\mathbb N_0}$ denote a $\mathbb R^d$-valued Markov chain generated by the Metropolis-Hastings algorithm. Suppose I've run the algorithm on a computer and obtained a sample $x_0,\ldots,...
0
votes
0answers
51 views

Model failed to converge in lme4::glmer() when the a factor is centered or releveled

I'm running a mixed-effects model using glmer() function. The modeling works well with R's default dummy coding. But if I center or relevel a factor of 2 levels, the model failed to converge. I am ...
3
votes
2answers
253 views

adding a small constant to the diagonals of a matrix to stabilize

I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix....
1
vote
1answer
104 views

downsampling a kde / combining kde and histogram

I'm calculating a KDE of one parameter (y, particle density) in bins of another parameter (x, distance from the origin). At ...
1
vote
0answers
26 views

Model or numerical inaccuracy in R package distr

With the goal to have an "outlier"-aware normal distribution, I build a simple univariate mixing model of normal and uniform distributions: ...
1
vote
2answers
110 views

NUTS Drawing samples from slice sampler; how to keep bounds on log scale?

I'm currently working to adapt the No U-Turn Sampler from this paper for a model I'm working on. The No-U Turn sampler augments the typical hamiltonian system by incorporating a slice variable $u$ ...
1
vote
0answers
156 views

Bayesian Networks - Factor Graphs - Belief Propagation - Numerical stability

I am trying to do inference for a Bayesian Network with discrete probabilities. I converted the network to a factor graph and implemented the sum-product algorithm (belief propagation). My goal is ...
1
vote
0answers
77 views

Natural Splines and Smoother Matrix

In the context of smoothing splines, one can show that the Reinsch form is given by: $ \hat{y} = N (N^{T}N +\lambda \Omega)^{-1}N^{T} y = (I+ \lambda K)^{-1}y $ where (1) $K = (N^{T})^{-1}\Omega N^{-...
1
vote
0answers
114 views

Moment generating function of a Weibull distribution and root finding heavy and light tailed case

I consider the equation $M_x(v)=1+(1+\beta)\mu$ and I need to find the solution $v>0$ such that the equation is fulfilled. For this example I consider the moment generating function $M_X(v)$ of a ...
0
votes
2answers
322 views

Estimating correlation matrix using numeric likelihood maximization

I'm performing maximum likelihood estimation on jointly distributed data and I'm having some issues estimating the correlation terms. I am using an approach based on the Cholesky decomposition, but I ...
3
votes
1answer
58 views

Are analytical derivatives unambiguously superior to numerical derivatives in GMM?

I am estimating a non-linear GMM model. In both Stata and R, you need to specify the moment equations and the instruments, but there is no need need to provide analytical derivatives for the estimator ...
1
vote
0answers
77 views

Underflow when estimating marginal likelihood via bridge sampling

I try to use an iterative procedure to estimate the marginal likelihood in a Bayesian setting for model selection. In case you are interested in the specifics of bridge sampling in my application, see ...
3
votes
0answers
197 views

Optimization with/without an analytical gradient

A colleague is optimizing a function (e.g. trying to find the minimum of a function $f(x_1, x_2, \ldots)$). We know the analytical form and it is differentiable. I suggested calculating the ...
2
votes
1answer
603 views

Are mvrnorm() in MASS R package and rmvn() in mgcv R package equivalent?

I am carrying out posterior simulation with GAMs/SCAMs and was wondering if/how the rmvn() function differs in any way from the ...
1
vote
1answer
28 views

numberical implementation of linear regression with “loose variable”

I understand how to solve a linear system $X \beta = y$ the solution is $\beta = (X^{T}X)^{-1} X^{T} y$ The problem is I could have an entry $\beta_i$ where it has no exposure in $X$. i.e. $X$ has ...
3
votes
0answers
27 views

An urn problem: few red balls, many draws (with replacement)

So, this is a freshman probability problem and I am embarrassed to p[ost it, but I have been up for 35 hours and my brain is broken. I have an urn with 60,000 white balls and 6 red balls. From this ...
2
votes
0answers
57 views

Dealing with Numerical Issues from Computing Weighted Sample Covariance Matrix

I have vector-valued samples $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_N$ with normalized weights $w_1, \ldots, w_N$. I'm trying to compute a weighted sample covariance matrix: $$ \sum_i w_i (\...
0
votes
0answers
801 views

Formula for Inverse T-distribution

I am trying to formulate an expression to calculate the critical value of a T-distribution for a given degrees of freedom. I have done so already for the Normal Distribution by considering the TI-84 ...
5
votes
2answers
313 views

Parameter estimation without an explicit likelihood function

I have a parametric model, some data $y$, and I would like to find a maximum likelihood estimate for the model parameters $\theta$. My usual approach would be to write down the likelihood function $\...
4
votes
0answers
416 views

Choosing the basis functions in a linear regression

I have two random variables $X$ and $Y$ and I'm trying to model $\mathbb{E}[Y|X]$. To this end, I'd like to pick a collection of functions $f_1, f_2 \dots f_n : \mathbb{R} \to \mathbb{R}$ and then ...
2
votes
2answers
1k views

Deep Learning: Condition Number and Poor Conditioning

I am reading the following section of the book Deep Learning. Can you provide an intuitive explanation of the above section? I don't quite understand the statement "When this number is large, matrix ...
2
votes
0answers
99 views

Training an Artificial Neural Network with limited-memory Quasi-Newton

I would like to train a simple Artificial Neural Network implementing an algorithm of the class of limited-memory Quasi-Newton. I read the paper Modified quasi-Newton methods for training neural ...
5
votes
2answers
412 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
0
votes
0answers
96 views

Integral of probit likelihood

I'm currently trying to implement the Heckman method for estimating the dynamic probit panel model (Original paper can be found here). I'm trying to implement it according a paper of Stewart (2006), ...
2
votes
0answers
381 views

GMM Estimation and convergence problem

I try to minimize an unweighted moment function $G(\theta)$ given by $G(\theta) = \bar{g}(\theta)'\bar{g}(\theta) $. $g(\theta,x_i)$ contains the specified moment conditions, where we state $E(g(\...
2
votes
1answer
408 views

Moments of the truncated normal distribution (univariate) away from the mean

I need to compute the mean and variance of the truncated normal distribution. For simplicity, let us focus on a standard normal, since the general case can be reduced to this. The PDF is given by: $$...
1
vote
0answers
82 views

Is there a solution for Canonical Correlation Analysis on large sparse matrices?

I'm trying to run CCA over two views which are sparse matrices. The two views are very high dimensional (e.g. 300k, 400k) with 1m samples. CCA needs the input views to be zero mean but I won't be ...
31
votes
4answers
13k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
0
votes
1answer
52 views

Distribution of infinite sum $\sum_{t=0}^{\infty} \epsilon_t r^t $

In my current statistics course we're being taught about time series, and in this context we came across sums like this: $$\sum_{t=1}^{\infty} \epsilon_t r^t \quad \epsilon_t\sim \text{WN}(0,\sigma^2) ...
0
votes
1answer
256 views

Method to compute empirical derivative about some point

I have black-box access to some function and I want to compute the derivative about the point X. Is there a method that does this?
10
votes
1answer
4k views

Softmax overflow [closed]

Waiting the next course of Andrew Ng on Coursera, I'm trying to program on Python a classifier with the softmax function on the last layer to have the different probabilities. However, when I try to ...
4
votes
1answer
761 views

What is the best way to apply the log-sum-exp trick in this situation?

I am aware of the "log-sum-exp" trick for calculating the logarithm of sums that handles overflow and underflow issues. However, I would like to know more about how it works. In particular, I am ...
1
vote
1answer
41 views

Numerical method to find x [closed]

I am carrying out a simulation study using the cumulative distribution function $$ F(x)=\frac{4x\arctan\frac{x}{a}}{a^2(π-2)}$$ Now I need to get x subject in order to carry out the simulation study. ...
1
vote
0answers
56 views

Proving an equivalence relation by using numerical methods such as Gaussian quadrature

The background is residual useful life prediction. The following is my problem description. Degradation signal path: $r(t)=\phi+\theta t$ , where $\phi$ is assumed to be the same for all units,and $\...
5
votes
0answers
159 views

What is the fastest way to compute PC1 scores, without performing the whole PCA?

I want to compute only the first principal component's scores $t_1$ of a large number $n$ of data points x with a high dimensionality $p$. Assume the data has been centered about zero. Data points ...
2
votes
0answers
481 views

Comparison between analytic gradient and numerical gradient for multivariate normal distribution wrt mean and covariance

The analytic gradients of log multivariate normal distribution wrt mean and covariance matrix can be found at StackExchange post and The gradient of the log-likelihood of normal distributions. I ...
6
votes
1answer
239 views

Relationship between binomial regression link function and goodness-of-fit tests [now with link to R code]

Some background: A number of papers in the literature (various ones by Hosmer and Lemeshow; Copas; le Cessie and van Houwelingen; Cressie and Read; Osius and Rojek; J. R. Dale) discuss a family of ...
2
votes
0answers
1k views

Logistic regression and singular Hessian

I've been following along with Andrew Ng's excellent course on Machine Learning (CS 229), and have been working on Problem Set 1. I'm trying to fit a logistic regression model using Newton's Method. ...
0
votes
0answers
76 views

Accurate tail probabilities (p-values) in MASS package

An R package was reporting a p-value for my dataset to be zero. Digging into it, I found that the p-values were computed in MASS:::summary.loglm : ...
7
votes
2answers
1k views

Is it possible to have pearson correlation coefficient values < -1 or values > 1?

I am trying to calculate the Pearson correlation coefficient according this formula over a large dataset: Mostly, my values are between -1 and 1, but sometimes I get weird numbers like: ...