The Aalen-Johansen approach used for multi-state modeling in the R survival package is quite flexible. The (startTime, stopTime, event) way of presenting the data to the functions automatically assumes a multi-state model if event is a factor variable (the only requirement is that reference state of event represents censoring at stopTime), and the software inspects properly formatted data to identify the numbers and directions of transitions, numbers of subjects having each type of transition, etc. See the survcheck() function illustrated in Section 3.4.2 of the main survival package vignette.
Although the worked-through examples in the multi-state vignette all seem to have unidirectional transitions, the approach allows for transitions in either direction between any two states, illustrated at the bottom right of Figure 1 of that vignette. As I interpret section 2.3 of the multi-state vignette, you can even model a transition to the same state (although such situations most likely come from a data-coding error). So coxph() analysis of a multi-state model thus provides time-dependent multi-state modeling without assuming parametric forms of baseline hazards.
With that flexibility, however, comes much responsibility. As the vignettes emphasize, just getting the data into the correct format is often a challenge. Then you have to make serious decisions about how to model.
You show 4 states with 6 transitions. Should all transitions depend on the same covariates? Should covariate coefficients be allowed to differ among transitions, or should some coefficient values be "common" among some transitions? With larger numbers of transitions having independent coefficient values, the number of transitions needed to model without overfitting grows accordingly. Should baseline hazards differ among transitions (like strata in a standard Cox model) or should baseline hazards be "shared" among some transitions? (The "common" coefficient values and "shared" baseline hazard are specified in a list presented to coxph() as the formula for a multi-state model, as shown in both vignettes linked above.)
The (startTime, stopTime, event) format is also what is used for time-varying covariates, so in principle there is no problem incorporating them. There are at least two practical problems. First, if there is a new row representing a change in a time-varying covariate for an individual who doesn't change state, I understand that you need to make sure that the event is shown as a censoring. Otherwise, I think that the system will try to model it as a transition to the same state. Second, there is a danger that the time-varying covariates in practice will have been measured after the transition rather than before, while the modeling needs to use the values in place infinitesimally before the transition.
I am not completely sure about delayed entry. What you describe is left truncation/right censoring or event, which is the default interpretation of (startTime, stopTime, event). That data format in some situations allows for discontinuous intervals of risk for an individual; see Section 3.7.2 of Therneau and Grambsch. With multi-state modeling, however, "individuals are not allowed to have gaps in the middle of their time line" (Section 2.3 of the multi-state vignette). One might interpret that prohibition to mean that delayed entry is OK provided there are no further gaps, but I haven't tried that myself. Test models with and without delayed entry on some sample data.
