I think I have a good dataset to apply machine learning.
I have a bunch of scatter plots and I generated a loess for each. Each loess is stored as a series of Y coordinates that correspond to X coordinates in a dataframe. All X coordinates are identical (1-34) Some of the loess curves are "desirable" - or what I would want to filter out, some are not.
So I looked through and found about 50 "ideal" loess curves. Each is a dataframe containing an X and Y column with a third column containing the Identifier.
in this case, CpG is the identifier, SmoothRange is the loess Y and AVGMOrder is the loess X.
# A tibble: 34 x 3 CpG SmoothRange AVGMOrder <chr> <dbl> <int> 1 cg21119901 0.0898 1 2 cg21119901 0.0898 2 3 cg21119901 0.0898 3 4 cg21119901 0.0898 4 5 cg21119901 0.0898 5 6 cg21119901 0.0898 6 7 cg21119901 0.0898 7 8 cg21119901 0.0676 8 9 cg21119901 0.0676 9 10 cg21119901 0.0676 10
I have about 50 of these to use for training and about 20,000 to filter through to find "similar" loess curves.
I started by trying to adapt this dataset to a simple caret vignette: https://machinelearningmastery.com/machine-learning-in-r-step-by-step/
The problem is, I get stuck very quickly.
the "createDataPartition" step is proving to be difficult to these "sets of loess curves".
So I have two questions here really,
1) Will this Vignette help me filter out the loess curves that best matches my dataset? 2) How do I make validation and training datasets for this type of data?
# create a list of 80% of the rows in the original dataset we can use for training validation_index <- createDataPartition(unique(GoodCpGs$CpG), p=0.80, list=FALSE) # select 20% of the data for validation validation <- unique(GoodCpGs[-validation_index,]) # use the remaining 80% of data to training and testing the models dataset <- GoodCpGs[validation_index,]
in case you are confused, I have a few photos that might help explain.
TL;DR: I have a lot of curves. Some look like the first, some look like the second. I need to figure out how to only select the curves that look like the first photo. I have a dataframe that contains all of the coordinates to graph. Is there a machine learning vignette or book that can help me out?
Photo 1: a few examples of Good "Preferred" loess curves Photo 2: One example of a loess curve to be filtered out