Being a software developer, I am coming from an object-oriented programming background, and I think in terms of classes and objects.

I am currently working on a Spark notebook to perform calculations on a dataset, for which I am following a more procedural approach (i.e. adding new columns to a dataframe as I perform the calculations.)

My question is if it is common among data scientists to use classes in their Spark notebooks to encapsulate functionality as opposed to following a more procedural approach.


In general you are writing pipelines and you relay on fact that "you run n+1 cell only if previous n completed with success" and you don't have to rely on super structurized code. On the other hand you want to have some functionalities closed in classes/functions and then run them with different parameters, but don't stick too much to classes since 90% of work is prototyping. After all if you written your code in propper way making it as class shouldn't be much of a hassle, so code should be clean and clear but structurizing it makes iterations very slow.

So rule of a thumb you close code into class only when you want to reuse it and you are sure it will remain like this for very long time

  • $\begingroup$ Thanks very much for those guidelines. $\endgroup$ Oct 16 '19 at 16:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.