I think I have a good dataset to apply machine learning.

I have a bunch of scatter plots and I generated a loess for each. Each loess is stored as a series of Y coordinates that correspond to X coordinates in a dataframe. All X coordinates are identical (1-34) Some of the loess curves are "desirable" - or what I would want to filter out, some are not.

So I looked through and found about 50 "ideal" loess curves. Each is a dataframe containing an X and Y column with a third column containing the Identifier.

in this case, CpG is the identifier, SmoothRange is the loess Y and AVGMOrder is the loess X.

# A tibble: 34 x 3
   CpG        SmoothRange AVGMOrder
   <chr>            <dbl>     <int>
 1 cg21119901      0.0898         1
 2 cg21119901      0.0898         2
 3 cg21119901      0.0898         3
 4 cg21119901      0.0898         4
 5 cg21119901      0.0898         5
 6 cg21119901      0.0898         6
 7 cg21119901      0.0898         7
 8 cg21119901      0.0676         8
 9 cg21119901      0.0676         9
10 cg21119901      0.0676        10

I have about 50 of these to use for training and about 20,000 to filter through to find "similar" loess curves.

I started by trying to adapt this dataset to a simple caret vignette: https://machinelearningmastery.com/machine-learning-in-r-step-by-step/

The problem is, I get stuck very quickly.

the "createDataPartition" step is proving to be difficult to these "sets of loess curves".

So I have two questions here really,

1) Will this Vignette help me filter out the loess curves that best matches my dataset? 2) How do I make validation and training datasets for this type of data?

# create a list of 80% of the rows in the original dataset we can use for training
validation_index <- createDataPartition(unique(GoodCpGs$CpG), p=0.80, list=FALSE)

# select 20% of the data for validation
validation <- unique(GoodCpGs[-validation_index,])

# use the remaining 80% of data to training and testing the models
dataset <- GoodCpGs[validation_index,]

in case you are confused, I have a few photos that might help explain.

TL;DR: I have a lot of curves. Some look like the first, some look like the second. I need to figure out how to only select the curves that look like the first photo. I have a dataframe that contains all of the coordinates to graph. Is there a machine learning vignette or book that can help me out?

Photo 1: a few examples of Good "Preferred" loess curves Photo 2: One example of a loess curve to be filtered out

Example of a Good "Preferred" loess curve Example of a Bad "To be filtered out" loess curve

  • $\begingroup$ Please show your own thoughts and include a TL;DR. – From Review $\endgroup$
    – Jim
    May 8, 2018 at 20:33

1 Answer 1


EDIT: UPDATE: It worked. I am retraining the model to optimize this. K nearest neighbor seems to work the best. I hope this might help someone one day.

I have come up with a way to approach this issue that is compatible with caret.

The line will be split into 5 groups (see photos above). A slope will be determined for each group. The groups will be the observations for each CpG.

From there, the slopes per each group are used to define wether the line is "preferred" or "not preferred".

I will be trying to set this up over the next few days and will post my results.

Idea of how I will approach machine learning of a non-parametric curve


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.