1
$\begingroup$

I have made some sales propensity models for my company in R. My code is a function oriented script fetching data from databases, reshaping and engineering data and then building models giving final predictions.

I have to run this every month. How do I do this? What infrastructure do I require to automate this process where sales leads predicted out of my model are directly given out to the leads management platform ( or say any internal tool ) . Is it done using some tools? I have completely no clue . How is a framework here created? Is this called coding in production environment?

To give context of tools . We do not have any Hadoop environment. We have a sql server databases, r server with high capacity and Tableau.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For any automation, shell programming in Unix like system / batch programming in windows are needed. This means you will need a script to run the model and generate a result file instead of using an IDE (such as R studio) to run it.

To do the shell / batch programming.

  1. You should be able to run your code via rscript in command line. Treat the model execution as a "application to run" and generate the results in file format, remove all the interactive or debugging components.

  2. Using rscript, you can write results to a csv file, and another application (python, Java, etc.) can pick up the results in csv produced by R. (Or you can use R to write to SQL server using RODBC connector.)

  3. Weekly execution can be done using cron job in Unix like system, and schtasks in Windows.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ +1. I would only add to this to use a service like AWS where you can make the necessary instances on say, EC2 or EMR to run the scripts and then use something like AWS DataPipeline or AirBnBs Airflow to automate the jobs. $\endgroup$
    – Alex R.
    Sep 19, 2017 at 19:08
  • $\begingroup$ @AlexR. You have a high reputation on mathematics and know technology this well? what do you do in your job :) ? $\endgroup$
    – Haitao Du
    Sep 19, 2017 at 19:18
  • $\begingroup$ @ hxd1011: I did a PhD in math, but ended up in industry, working at startups. Great way to learn the 99% of stuff I never learned in academia :). $\endgroup$
    – Alex R.
    Sep 19, 2017 at 19:59
  • $\begingroup$ @AlexR. startup is the place to learn (this is why you mention AWS. I think. ) But I never had chance to work there. In your work, do you still invent new models use math? or use mature packages such as tensorflow? $\endgroup$
    – Haitao Du
    Sep 19, 2017 at 20:04
  • $\begingroup$ @hxd1011 -This is helpful ! Two further questions : 1) I have no knowledge on Batch programming , any learning sources to Productinize R scripts through this way ? As i understand the only tool used here is Command ( cmd ) - apart from R , no other tool is required right ? 2) As Alex pointed out - i don't have really a big data issue , my function overall takes 1 hour to run and is manageable on a 32 gb desktop , 3) Any learning source for a beginner with an example of an R code productionized through this batch programming ? $\endgroup$
    – Pb89
    Sep 20, 2017 at 1:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.