Can I run a logistic regression with a small sample size and perfect convergence? I'm hoping someone will be able to help me with my question. I have a dataset with 71 records and 4 variables. Three of the four variables are independent, and one is dependent. All four variables are nominal with two levels, 0 or 1. The three independent variables, respectively, have 26, 17, and 9 "successes" (where the score is 1). The dependent variable has 44 "successes". However - the "successes" in the dependent variable ALWAYS overlap with one or more "successes" in the independent variable. A "success" in one independent variable means that the dependent variable was also a "success."
Given the small number of responses in my sample and the 1:1 overlap between "successes" in the independent and dependent variables, is it possible for me to run some variation of logistic regression on these variables to detect a relationship between the dependent and independent variables? 
Thanks in advance! 
 A: Yes, logistic regression is capable of learning a rule of the form "if any of the input variables are 1, then predict success." Such a model might look something like:
$$\hat{p} = \sigma(-10 + 20x_1 + 20x_2 + 20x_3)$$
where $\sigma$ is the sigmoid function. 10 is arbitrary and is chosen for concreteness: all that matter is the intercept is a large negative number and the coefficients are even larger positive numbers. When all three IVs are zero, this is $\sigma(-10) = \epsilon$ (where $\epsilon$ is a very small number) and if even one IV is a 1 then it will be at least $\sigma(10) =1 - \epsilon$. If two or more IVs are 1, it simply gets closer to 1. Because the decision threshold for for logistic regression is usually 0.5, every dependent class will be predicted with perfect accuracy.
Notice that this rule is of the form $x_1=1 \cup x_2=1 \cup x_3=1 $, in other word it is an "OR" expression. Logistic regression can also learn "AND" expressions. Interesting enough, however, it cannot learn "XOR" rules unless you provide interaction terms in the design matrix. And in general it cannot learn arbitrary logical expressions. Compare this to say, a decision tree, which can. This limitation was one of the early motivations for exploring the multilayered perceptrons which eventually led to deep learning. But I digress.
If the rule is true for literally every row in your data set, we say the DV classes are "linearly separable." Try to fit a vanilla logistic regression to linearly separable data with your favorite statistical software and the parameters will not converge: the fit keeps getting better as the parameters get arbitrarily bigger so it will "blow up." That is to say, eventually the program will hit a maximum number of iterations or the limits of its floating point representation and quit with an error message. These numerical issues can easily be fixed by adding a regularization term to the model. Regularization is widely supported typically is as easy as checking a box or passing an optional argument into a function. 
