Welcome to Cross Validated!
I'll start with that I'm not 100% certain I understand the question. It may be that you are going about this in reverse. I would suggest you start out with fitting your data to your model (
lm) and then using graphical methods to explore your fits. You can do this with ggplot.
Unfortunately, it is not possible to extract the information about the fits from
ggplot. However, if you do the
lm regression outside of
ggplot you will get the same information.
Also, I notice that you did not specify a formula in
geom_smooth, so I think it will default to
y~x. If this is what you want, then perhaps you would be best to start out with a model such as
fit = lm(Yvals ~ Xvals + ColorVals + Xvals:ColorVals, data=all_sum)
I would suggest exploring the examples in the documentation. This blog post gives a nice function you can use to plot the results of a given fit with many useful annotations. Also, this question sounds a bit similar to yours and does have an accepted answer.
I hope this helps!