Consider the case where more than the minimal number of vectors lie on the lines (or hyperplanes in higher dimensions) defined by the margin found by the SVM algorithm. For example, see the image below where a third redundant vector lies on the margin. Do we have any guarantees on the selection of support vectors in this case? For example, will all three points be stored as support vectors or just the minimal number necessary? If it's the former, is there any structure in their associated dual coefficients?
No, quite often you end up with redundant support vectors that lie on the margin, but the decision surface can be constructed with only a subset of the identified support vectors. The example you give with three co-linear support vectors is a good example of that. Vapnik gave a name to the support vectors without which you can't construct the decision surface, but I can't remember what it is off hand.
Whether the training algorithm will identify all of the support vectors or just those necessary to represent the decision surface is likely to vary from one algorithm to another. In this example, an algorithm that is based on identifying the closest points between convex hulls of the examples belonging to each class may well only identify two of the three support vectors. One based on Sequential Minimal Optimisation would probably depend on the way it identifies the next dual parameter to optimise, and something like interior point may identify all three?