0
$\begingroup$

I have an unbalanced panel dataset with weekly observations on various products. N, the number of products, is about 6300 and is very large as compared to T, the number of observations per product, and T varies by product, with a median of 53 observations per product.

When a new version of each product is released, the product is removed from the sample. There is then a dummy, upgrade, taking the value 1 on the week the upgrade is released (aka the last observation of the old product), and 0 before. Now I want to investigate whether the occurrence of an upgrade depends on past demand movements. To that end, I used various probability models, regressing the binary response upgrade on multiples variables.

My fixed effects logits - clogit and bife on R - failed. After trying out different predictor combinations, I isolated the culprit: age, the variable giving the age of the products, in days since initial release. age isn't a demand-side variable but it's an important control, so I want it to work. Note that I'm not interested in the amount of time before upgrade, so survival analysis isn't what I'm doing here.

More specifically, when I regress upgrade on age alone with clogit, with products id as fixed effect:

fit_clogit=clogit(upgrade~age+strata(id),data=mydata,method="exact")

I get the following error:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  NA/NaN/Inf in foreign function call (arg 5)

Surmising the possibility that the small amount of observations for some products caused a separability problem, I tried the following:

-checking that age contains no NA, NaN or Inf

-removing products with few observations; I went as far as to removing products with less than 25 observations

-removing observations where age took on a value that was always associated with the same value of the binary response (aka no upgrade or always an upgrade for products of that age)

-replacing the continuous age by bins: "more than 6 months", "6 months to one year"... and so on

-replacing age by a natural spline of itself (with 4 or 3 degrees of freedom), to reduce the weight of outliers

-likewise, removing the tails of the distribution of age

-and converting age to z-scores, computed over the period

I also tried the other available values of method for clogit... All to no avail. In some cases, I even got an additional error message:

In addition: Warning message:
In fitter(X, Y, strats, offset, init, control, weights = weights,  :
  Ran out of iterations and did not converge

With bife, running:

fit_bife=bife(upgrade~age | id,data=mydata,model="logit",bias_corr = "ana")

Either the algorithm failed to converge or it converged but yielded estimates with a p-value of 1. I tried the same remedies as before, they didn't work. Using bias_corr = "no" didn't solve anything.

I have two final remarks. First, fixed effect models work with other predictors; so, the large amount of fixed effects doesn't seem to be an absolute impediment. Moreover, in logits with no fixed effects, and time-invariant categorical variables instead, things run well and age always has a significant effect.

So, the problem seems to come from the interaction of age with the fixed effects but at this point, I'm at a loss. What's wrong with my data?

Thanks in advance for your help.

EDIT: here is the summary of age:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    184     487     919    1364    1632   30765

With 91st, 96th and 100th percentiles at, respectively: 2832 ; 4095 ; 7670.18.

And here's an histogram of the distribution, as well as a boxplot by value of the response: enter image description here enter image description here

This extremely long upper tail is indeed a problem. For example, by removing those observations with age in the vicinity of 30 000, the algorithm runs with age as the single variable. Adding another degree of age, or adding some more variables, causes it to fail.

Traceback:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  NA/NaN/Inf in foreign function call (arg 5)

5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 566183L), upgrade) ~ age + strata(id), 
       data = mydata, method = "exact")
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ age + strata(id), data = mydata, method = "exact")

For a more workable sample, I removed all products with age observations in the top 1% of the age distribution (a bit more actually), as well as products with few observations. This yielded the following distribution of age:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    184     510     924    1230    1585    6689  

With 91st, 96th and 100th percentiles at, respectively: 2604 ; 3336 ; 4879. enter image description here enter image description here

Now the algorithm doesn't fail at once with age as the sole predictor. However, adding an additional variable in the model, such as age squared (from previous investigations, the effect appears concave, better to account for that), breaks it down (no convergence). Meanwhile, the model still works with all predictors excepted age.

Traceback with age and age^2 as predictors, and number of iterations increased by a factor of 100 as compared to default:

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  (converted from warning) Ran out of iterations and did not converge

10: doWithOneRestart(return(expr), restart)
9: withOneRestart(expr, restarts[[1L]])
8: withRestarts({
       .Internal(.signalCondition(simpleWarning(msg, call), msg, 
           call))
       .Internal(.dfltWarn(msg, call))
   }, muffleWarning = function() NULL)
7: .signalSimpleWarning("Ran out of iterations and did not converge", 
       quote(fitter(X, Y, strats, offset, init, control, weights = weights, 
           method = method, row.names(mf))))
6: warning("Ran out of iterations and did not converge")
5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 514218L), upgrade) ~ age + I(age^2) + 
       strata(id), data = adj_data, method = "exact", iter.max = 2000, 
       outer.max = 1000)
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ age + I(age^2) + strata(id), data = adj_data, 
       method = "exact", iter.max = 2000, outer.max = 1000)

I'm using survival 2.41.3.

Also, using a factor variable agegroup with 6 levels (between 183 and 365 days, 1 to 2 years, 2 to 3 years, 3 to 4 years, 4 to 5 years and 5 years or more), I get the following (on the downsized sample):

Error in fitter(X, Y, strats, offset, init, control, weights = weights,  : 
  (converted from warning) Loglik converged before variable  1,2,3,4,5 ; beta may be infinite. 

10: doWithOneRestart(return(expr), restart)
9: withOneRestart(expr, restarts[[1L]])
8: withRestarts({
       .Internal(.signalCondition(simpleWarning(msg, call), msg, 
           call))
       .Internal(.dfltWarn(msg, call))
   }, muffleWarning = function() NULL)
7: .signalSimpleWarning("Loglik converged before variable  1,2,3,4,5 ; beta may be infinite. ", 
       quote(fitter(X, Y, strats, offset, init, control, weights = weights, 
           method = method, row.names(mf))))
6: warning(paste("Loglik converged before variable ", paste((1:nvar)[infs], 
       collapse = ","), "; beta may be infinite. "))
5: fitter(X, Y, strats, offset, init, control, weights = weights, 
       method = method, row.names(mf))
4: coxph(formula = Surv(rep(1, 514218L), upgrade) ~ agegroup + 
       strata(id), data = adj_data, method = "exact", iter.max = 2000, 
       outer.max = 1000)
3: eval(coxcall, sys.frame(sys.parent()))
2: eval(coxcall, sys.frame(sys.parent()))
1: clogit(upgrade ~ agegroup + strata(id), data = adj_data, 
       method = "exact", iter.max = 2000, outer.max = 1000)
$\endgroup$
1
$\begingroup$

I'm not familiar with either package, but my guess is either 1) a bug in the package or 2) model misspecificiation.

Since the models are being fit with exact log likelihood and these functions are convex, global convergence should be guaranteed for any feasible data set, it just might take prohibitively long amount of time, depending on the fitting algorithm. You mentioned normalization of regressor variables doesn't help. Just how many data points do you have? P-values of 1 again suggest misspecification, perhaps due to under-/overfitting. As far as I understand, in the conditional logistic regression, you are trying to find a single mean/slope pair (for age) that will maximize the likelihood of >6000 conditional distributions. If there is a great deal of dissimilarity between groups with similar ages, the resulting likelihood may be very flat, convergence very slow and the final fit very poor. Even with more regressors, there might not be enough compared to the number of strata. The bife documentation says it uses a 'unconditional model's which I assume means uses it uses explicit parameters for each value of id, but in this case you might be running into a lack of data to estimate 6000+ variables with much accuracy.

Does the problem occur with synthetic data? Can you provide a data sample or a minimal reproducible example? That error is probably coming from the underlying optimization routine called by clogit, which isn't very good error propagation/interpretation on the author's part.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer. Unfortunately I'm not allowed to share this data, but I can provide summary statistics and plots. I have 2388 "goods" and 443219 "bads". I'm aware separability could be an issue here, I've tried to track it down. What I really don't get is why clogit works just fine with any combination of predictors excluding age. In particular, you would also expect to see dissimilar stratas in the same month-of-the-year. If anything, shouldn't that be more a problem than for age? And yet, month dummies work fine, and so do week-of-the-month dummies. $\endgroup$ – Magean Apr 4 '18 at 23:15
  • $\begingroup$ Hmm. It's sounding more like a bug to me. Have you tried with synthetic age data? The only way the algorithm might fail to convergence is if the coefficient grow too large (but this is also convergence, to probability 0 or 1). That however should be handled accordingly, and otherwise prevented. Even if the age relationship is highly non-linear, this shouldn't break the model, convexity ensures that. Could you post the full stack trace? Also, silly things like do your variables have the correct types? $\endgroup$ – deasmhumnha Apr 5 '18 at 3:30
  • $\begingroup$ Traceback() does not return anything more than what was in the error message Error in fitter..., along with initial calls. Is that the stack trace you were referring to? Regarding synthetic age data, yes I tried binning age into factors; I also checked that age was numeric and not diffdate. Anyway, it seems the algorithm is extremely sensitive to which values of age are included: I managed to get clogit to work with age as the only predictor (but not with other powers of age or other variables) by excluding the most extreme outliers; excluding slightly less extreme outliers breaks it down. $\endgroup$ – Magean Apr 5 '18 at 11:25
  • $\begingroup$ Yeah that is the stack trace, but there should be a numbered list that also includes internal function calls. Could you just share the print out any way. Also try flagging warnings as errors with options(warn=2). By synthetic data, I meant completely fake data, say draws from rnorm(0,1). $\endgroup$ – deasmhumnha Apr 5 '18 at 19:35
  • $\begingroup$ Played around with some fake data with similar summaries as your dataset. Using survival v2.41.3Could not reproduce any errors: NA, factors, large values. Do you mind posting a box plot or histogram of age? What version of the packages/R are you using? Do you have access to another computer with R? Could be a problem with your current build. $\endgroup$ – deasmhumnha Apr 6 '18 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.