If the functional relationship between the exposure and average response is not an S-shaped logistic curve, there are still reasons why we might consider an S-shaped logistic curve as a meaningful summary of those data.
As an example, we might have omitted a prognostic factor from a model, meaning that the true marginal relationship between the exposure and the outcome is not logistic, but a complicated semi-logistic function that averages up risks across several conditional logistic curves. This is the principal of non-collapsibility in logistic regression.
Basically, we can rarely ever be certain that the S-shaped logistic trend is, in fact, the "right" one... but it is a useful one! All models are wrong, some models are useful.
Kenji is right that when we try to approximate an S-shaped trend, and the data show strong distributional violations, there may be some sensitivity analyses to consider like testing for higher order polynomial effects. Another type of test to consider is breakpoints, adjusting for "knots" so that trends can change direction. These approaches are hybridized in splines and made even more general by using LOESS curves to explore general non-linear relationships between exposures and outcomes.
Nonetheless, you may revert to the original question: you may say "I want to summarize these data using a single logistic curve whose intercept represents log-odds of the outcome for exposure=0 and whose slope is the log-odds ratio as a measure of association between an exposure and an outcome." The desire then is to obtain a robust error estimate that is unbiased and consistent. The S-curve then is taken to summarize a first order trend in the data, which you can think of as a rule of thumb: does the risk tend to increase or decrease as the exposure goes up, and by how much? To do this, you need only to apply sandwich based standard errors. This can be done using Generalized Estimating Equations with working independence covariance structure, logistic link, and binomial variance structure.