Using logistic regression to produce probability curve

Question

I have data for an experiment that aims to check how good a machine is in detecting flaws in structures.

The Probability of Detection is a function of the flaw size and some other environmental factors. I have 70 measurements points summarized in the table below, where I have the flaw size, and if the flaw was detected or not (0 or 1).

From this data, we can plot those points as shown in the figure below (triangular points). However, I am interested in getting the fitted curve from the data points. The probability of detection is a function of flaw size. According to the reference, I am reading from the probability density function can be written as below: where a is the size of the flaw, mu and sigma are the mean and the standared deviation respectively, B0 and B1 are functions of the mean and the standard deviation. Using logistic regression in R or minitab, B0 and B1 can be estimated.

I am interested in working out this calculation myself and write my own code for that, in other words, I am interested in the algorithm that will allow me to estimate the parameters to draw the fitted curve.

Answer 1

If you are looking for a way of estimating the coefficients of a logistic regression then gradient descent is the simplest way, for instance take a look at secrion 2 from https://machinelearningmastery.com/implement-logistic-regression-stochastic-gradient-descent-scratch-python/ to see how it can be implemented in python.

Answer 2

The logistic regression model is defined as

$$ \begin{align} \eta &= \boldsymbol{X}\beta \\ E(Y|\boldsymbol{X}) &= \mu = g^{-1}(\eta) \\ Y &\sim \mathcal{B}(\mu) \end{align} $$

where $\eta$ is the linear combination, $g^{-1}$ is the inverse of link function and $\mathcal{B}$ is the Bernoulli distribution that serves as a likelihood function. The "default" choice of link function for logistic regression is the logit function

$$ g(z) = \frac{1}{1 + e^{-z}} $$

but in your example you are referring to another choice, the probit function.

To estimate the $\beta$ parameters you need to maximize the likelihood function

$$ \DeclareMathOperator*{\argmax}{arg\,max} \hat\beta = \argmax_\beta \;\mathcal{B}(g^{-1}(\boldsymbol{X}\beta)) $$

In real-life applications this is usually done using algorithms like iteratively reweighted least squares. However if you want to code the algorithm by-hand to understand how does logistic regression work, you probably should try something simpler like gradient descent, you can find worked example of implementing it in Andrew Ng's Machine Learning course on Coursera.org (it's free).

To plot the predictions, you need to use the estimated parameters $\hat\beta$ and the values of $\boldsymbol{X}$ that you want to evaluate it on, next you just plug-in the values into the formula and get the predicted probabilities $\hat\mu$:

$$ \hat\mu = g^{-1}(\boldsymbol{X}\beta) $$

If you want to get the predicted values of $Y$, then you would need to decide for a decision rule like "if $\hat\mu > \alpha$ then predict $1$ otherwise predict $0$" for some decision boundary $\alpha$. The "default" choice of $\alpha$ is usually $0.5$, but this does not to have to be the optimal choice.