Fixed effect logit not converging on continuous predictor

Question

I have an unbalanced panel dataset with weekly observations on various products. N, the number of products, is about 6300 and is very large as compared to T, the number of observations per product, and T varies by product, with a median of 53 observations per product.

When a new version of each product is released, the product is removed from the sample. There is then a dummy, upgrade, taking the value 1 on the week the upgrade is released (aka the last observation of the old product), and 0 before. Now I want to investigate whether the occurrence of an upgrade depends on past demand movements. To that end, I used various probability models, regressing the binary response upgrade on multiples variables.

My fixed effects logits - clogit and bife on R - failed. After trying out different predictor combinations, I isolated the culprit: age, the variable giving the age of the products, in days since initial release. age isn't a demand-side variable but it's an important control, so I want it to work. Note that I'm not interested in the amount of time before upgrade, so survival analysis isn't what I'm doing here.

More specifically, when I regress upgrade on age alone with clogit, with products id as fixed effect:

fit_clogit=clogit(upgrade~age+strata(id),data=mydata,method="exact")

I get the following error:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  NA/NaN/Inf in foreign function call (arg 5)

Surmising the possibility that the small amount of observations for some products caused a separability problem, I tried the following:

-checking that age contains no NA, NaN or Inf

-removing products with few observations; I went as far as to removing products with less than 25 observations

-removing observations where age took on a value that was always associated with the same value of the binary response (aka no upgrade or always an upgrade for products of that age)

-replacing the continuous age by bins: "more than 6 months", "6 months to one year"... and so on

-replacing age by a natural spline of itself (with 4 or 3 degrees of freedom), to reduce the weight of outliers

-likewise, removing the tails of the distribution of age

-and converting age to z-scores, computed over the period

I also tried the other available values of method for clogit... All to no avail. In some cases, I even got an additional error message:

In addition: Warning message:
In fitter(X, Y, strats, offset, init, control, weights = weights,  :
  Ran out of iterations and did not converge

With bife, running:

fit_bife=bife(upgrade~age | id,data=mydata,model="logit",bias_corr = "ana")

Either the algorithm failed to converge or it converged but yielded estimates with a p-value of 1. I tried the same remedies as before, they didn't work. Using bias_corr = "no" didn't solve anything.

I have two final remarks. First, fixed effect models work with other predictors; so, the large amount of fixed effects doesn't seem to be an absolute impediment. Moreover, in logits with no fixed effects, and time-invariant categorical variables instead, things run well and age always has a significant effect.

So, the problem seems to come from the interaction of age with the fixed effects but at this point, I'm at a loss. What's wrong with my data?

Thanks in advance for your help.

EDIT: here is the summary of age:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    184     487     919    1364    1632   30765

With 91st, 96th and 100th percentiles at, respectively: 2832 ; 4095 ; 7670.18.

And here's an histogram of the distribution, as well as a boxplot by value of the response:

This extremely long upper tail is indeed a problem. For example, by removing those observations with age in the vicinity of 30 000, the algorithm runs with age as the single variable. Adding another degree of age, or adding some more variables, causes it to fail.

Traceback:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  NA/NaN/Inf in foreign function call (arg 5)

5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 566183L), upgrade) ~ age + strata(id), 
       data = mydata, method = "exact")
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ age + strata(id), data = mydata, method = "exact")

For a more workable sample, I removed all products with age observations in the top 1% of the age distribution (a bit more actually), as well as products with few observations. This yielded the following distribution of age:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    184     510     924    1230    1585    6689  

With 91st, 96th and 100th percentiles at, respectively: 2604 ; 3336 ; 4879.

Now the algorithm doesn't fail at once with age as the sole predictor. However, adding an additional variable in the model, such as age squared (from previous investigations, the effect appears concave, better to account for that), breaks it down (no convergence). Meanwhile, the model still works with all predictors excepted age.

Traceback with age and age^2 as predictors, and number of iterations increased by a factor of 100 as compared to default:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  (converted from warning) Ran out of iterations and did not converge

10: doWithOneRestart(return(expr), restart)
9: withOneRestart(expr, restarts[[1L]])
8: withRestarts({
       .Internal(.signalCondition(simpleWarning(msg, call), msg, 
           call))
       .Internal(.dfltWarn(msg, call))
   }, muffleWarning = function() NULL)
7: .signalSimpleWarning("Ran out of iterations and did not converge", 
       quote(fitter(X, Y, strats, offset, init, control, weights = weights, 
           method = method, row.names(mf))))
6: warning("Ran out of iterations and did not converge")
5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 514218L), upgrade) ~ age + I(age^2) + 
       strata(id), data = adj_data, method = "exact", iter.max = 2000, 
       outer.max = 1000)
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ age + I(age^2) + strata(id), data = adj_data, 
       method = "exact", iter.max = 2000, outer.max = 1000)

I'm using survival 2.41.3.

Also, using a factor variable agegroup with 6 levels (between 183 and 365 days, 1 to 2 years, 2 to 3 years, 3 to 4 years, 4 to 5 years and 5 years or more), I get the following (on the downsized sample):

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  (converted from warning) Loglik converged before variable  1,2,3,4,5 ; beta may be infinite. 

10: doWithOneRestart(return(expr), restart)
9: withOneRestart(expr, restarts[[1L]])
8: withRestarts({
       .Internal(.signalCondition(simpleWarning(msg, call), msg, 
           call))
       .Internal(.dfltWarn(msg, call))
   }, muffleWarning = function() NULL)
7: .signalSimpleWarning("Loglik converged before variable  1,2,3,4,5 ; beta may be infinite. ", 
       quote(fitter(X, Y, strats, offset, init, control, weights = weights, 
           method = method, row.names(mf))))
6: warning(paste("Loglik converged before variable ", paste((1:nvar)[infs], 
       collapse = ","), "; beta may be infinite. "))
5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 514218L), upgrade) ~ agegroup + 
       strata(id), data = adj_data, method = "exact", iter.max = 2000, 
       outer.max = 1000)
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ agegroup + strata(id), data = adj_data, 
       method = "exact", iter.max = 2000, outer.max = 1000)

Answer 1

I'm not familiar with either package, but my guess is either 1) a bug in the package or 2) model misspecificiation.

Since the models are being fit with exact log likelihood and these functions are convex, global convergence should be guaranteed for any feasible data set, it just might take prohibitively long amount of time, depending on the fitting algorithm. You mentioned normalization of regressor variables doesn't help. Just how many data points do you have? P-values of 1 again suggest misspecification, perhaps due to under-/overfitting. As far as I understand, in the conditional logistic regression, you are trying to find a single mean/slope pair (for age) that will maximize the likelihood of >6000 conditional distributions. If there is a great deal of dissimilarity between groups with similar ages, the resulting likelihood may be very flat, convergence very slow and the final fit very poor. Even with more regressors, there might not be enough compared to the number of strata. The bife documentation says it uses a 'unconditional model's which I assume means uses it uses explicit parameters for each value of id, but in this case you might be running into a lack of data to estimate 6000+ variables with much accuracy.

Does the problem occur with synthetic data? Can you provide a data sample or a minimal reproducible example? That error is probably coming from the underlying optimization routine called by clogit, which isn't very good error propagation/interpretation on the author's part.