# Machine Learning to filter similar loess curves from a large dataset

I think I have a good dataset to apply machine learning.

I have a bunch of scatter plots and I generated a loess for each. Each loess is stored as a series of Y coordinates that correspond to X coordinates in a dataframe. All X coordinates are identical (1-34) Some of the loess curves are "desirable" - or what I would want to filter out, some are not.

So I looked through and found about 50 "ideal" loess curves. Each is a dataframe containing an X and Y column with a third column containing the Identifier.

in this case, CpG is the identifier, SmoothRange is the loess Y and AVGMOrder is the loess X.

# A tibble: 34 x 3
CpG        SmoothRange AVGMOrder
<chr>            <dbl>     <int>
1 cg21119901      0.0898         1
2 cg21119901      0.0898         2
3 cg21119901      0.0898         3
4 cg21119901      0.0898         4
5 cg21119901      0.0898         5
6 cg21119901      0.0898         6
7 cg21119901      0.0898         7
8 cg21119901      0.0676         8
9 cg21119901      0.0676         9
10 cg21119901      0.0676        10


I have about 50 of these to use for training and about 20,000 to filter through to find "similar" loess curves.

I started by trying to adapt this dataset to a simple caret vignette: https://machinelearningmastery.com/machine-learning-in-r-step-by-step/

The problem is, I get stuck very quickly.

the "createDataPartition" step is proving to be difficult to these "sets of loess curves".

So I have two questions here really,

1) Will this Vignette help me filter out the loess curves that best matches my dataset? 2) How do I make validation and training datasets for this type of data?

# create a list of 80% of the rows in the original dataset we can use for training
validation_index <- createDataPartition(unique(GoodCpGs\$CpG), p=0.80, list=FALSE)

# select 20% of the data for validation
validation <- unique(GoodCpGs[-validation_index,])

# use the remaining 80% of data to training and testing the models
dataset <- GoodCpGs[validation_index,]


in case you are confused, I have a few photos that might help explain.

TL;DR: I have a lot of curves. Some look like the first, some look like the second. I need to figure out how to only select the curves that look like the first photo. I have a dataframe that contains all of the coordinates to graph. Is there a machine learning vignette or book that can help me out?

Photo 1: a few examples of Good "Preferred" loess curves Photo 2: One example of a loess curve to be filtered out

• Please show your own thoughts and include a TL;DR. – From Review – Jim May 8 '18 at 20:33