3
$\begingroup$

I have a question about process, and I’m a relative noob when it comes to Machine Learning.

Let’s say I have dataset with X features to train a model. During development of my model I drop a few features columns, delete rows with NaN, perhaps standardize values, etc, leaving me with fewer than the original X features. I save the model with joblib (I'm using sklearn).

When I give this model to another user with their own dataset with X features won’t they have to do the same data transformations (dropping data/columns/standardize etc) on the data to be able to use the model they loaded?

If so, how can this be easily accomplished for the new user? It doesn’t seem possible for me to bundle code with joblib, so do I write a set of separate functions for the data transformation and send it along with the saved model? How, a standalone .py file, or a notebook (I’ve been doing most of my work in Colab this far)? What’s best practice/pragmatic here?

Sorry if this is a very basic question, I see a lot about developing models, not so much about how to share models. In this case I just want to evaluate a bunch of different models (each with possibly their own sets of data transformations, though everyone started out with the same data set). I have access to the original data with the X features that everyone else started out with.

$\endgroup$
4
$\begingroup$

I would recommend on preparing a pipeline for your users. The pipeline will scale, impute, rebalance data and it is even possible to programm it with different parameters or several estimators from a family. They have to do the same transformation, but with the pipeline you can be sure that it will be done in the same way for everyone. You should also make yourself comfortable with random_state so that your users get the same train test splits every time in your pipeline.

A simple example can be found here: https://towardsdatascience.com/a-simple-example-of-pipeline-in-machine-learning-with-scikit-learn-e726ffbb6976

doc from sklearn: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline https://scikit-learn.org/stable/modules/compose.html#combining-estimators

$\endgroup$
2
  • $\begingroup$ Thanks .. I assume I'd be using the pipeline in conjunction with ColumnTransformers to help in the various transformations, does that sound right? $\endgroup$ – Levon Mar 15 at 13:34
  • $\begingroup$ If you for example think of scaling your column , than look at the 1st link, you can import the StandardScaler class and use it in the pipeline for example like that: steps = [('scaler', StandardScaler()), if you want to transform columns e.g. relabeling them than there is a LabelEncoder, You are free to do anything you want. If the answer satisfies you, in general, I would be happy, if you would gave the answer an upvote and mark it as correct as it would help other users $\endgroup$ – Patrick Bormann Mar 15 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.