# Using logistic regression to produce probability curve

I have data for an experiment that aims to check how good a machine is in detecting flaws in structures.

The Probability of Detection is a function of the flaw size and some other environmental factors. I have 70 measurements points summarized in the table below, where I have the flaw size, and if the flaw was detected or not (0 or 1). From this data, we can plot those points as shown in the figure below (triangular points). However, I am interested in getting the fitted curve from the data points. The probability of detection is a function of flaw size. According to the reference, I am reading from the probability density function can be written as below: where a is the size of the flaw, mu and sigma are the mean and the standared deviation respectively, B0 and B1 are functions of the mean and the standard deviation. Using logistic regression in R or minitab, B0 and B1 can be estimated.

I am interested in working out this calculation myself and write my own code for that, in other words, I am interested in the algorithm that will allow me to estimate the parameters to draw the fitted curve.

• What do you mean by "getting the probability smooth curve from the data"? If you assume that your data should be aligned with the sigmoid curve, then you are wrong. The predicted probabilities will be aligned with it, but not the data since the data is only 0s and 1s. – Tim Sep 19 '17 at 9:01
• You'll want to begin by getting some software that can fit a logistic regression; it can be done in R, for example, which is free. In fact you can even use it via a browser (e.g. by using r-fiddle), but we can't teach you how to use packages here. If you have a specific question about logistic regression we can probably answer it (it may have already been answered), but books are probably a better place to start learning it. Both Introduction to Statistical Learning and The Elements of Statistical Learning (each freely available as pdf from the authors' websites) have a little on it. – Glen_b Sep 19 '17 at 9:02
• @Tim Yes I meant to get the fitted curve from the data points – HaneenSu Sep 19 '17 at 9:06
• @Glen_b Thanks! Yes I know it can be done through R or Minitab, but I am interested in understanding the algorithm behind it. – HaneenSu Sep 19 '17 at 9:14
• Your question doesn't ask about the algorithm the packages use. It asks about how to proceed to fit the curve (which would be done by using them). What is it you actually want to know? – Glen_b Sep 19 '17 at 9:20

If you are looking for a way of estimating the coefficients of a logistic regression then gradient descent is the simplest way, for instance take a look at secrion 2 from https://machinelearningmastery.com/implement-logistic-regression-stochastic-gradient-descent-scratch-python/ to see how it can be implemented in python.

The logistic regression model is defined as

\begin{align} \eta &= \boldsymbol{X}\beta \\ E(Y|\boldsymbol{X}) &= \mu = g^{-1}(\eta) \\ Y &\sim \mathcal{B}(\mu) \end{align}

where $\eta$ is the linear combination, $g^{-1}$ is the inverse of link function and $\mathcal{B}$ is the Bernoulli distribution that serves as a likelihood function. The "default" choice of link function for logistic regression is the logit function

$$g(z) = \frac{1}{1 + e^{-z}}$$

but in your example you are referring to another choice, the probit function.

To estimate the $\beta$ parameters you need to maximize the likelihood function

$$\DeclareMathOperator*{\argmax}{arg\,max} \hat\beta = \argmax_\beta \;\mathcal{B}(g^{-1}(\boldsymbol{X}\beta))$$

In real-life applications this is usually done using algorithms like iteratively reweighted least squares. However if you want to code the algorithm by-hand to understand how does logistic regression work, you probably should try something simpler like gradient descent, you can find worked example of implementing it in Andrew Ng's Machine Learning course on Coursera.org (it's free).

To plot the predictions, you need to use the estimated parameters $\hat\beta$ and the values of $\boldsymbol{X}$ that you want to evaluate it on, next you just plug-in the values into the formula and get the predicted probabilities $\hat\mu$:

$$\hat\mu = g^{-1}(\boldsymbol{X}\beta)$$

If you want to get the predicted values of $Y$, then you would need to decide for a decision rule like "if $\hat\mu > \alpha$ then predict $1$ otherwise predict $0$" for some decision boundary $\alpha$. The "default" choice of $\alpha$ is usually $0.5$, but this does not to have to be the optimal choice.