I am looking for best way to deploy my Deep Learning application which does the following:

  1. Fetches data from BigQuery
  2. Finds anomaly (based on trained model)
  3. Throws the output in a mysql database. I want this program to run everyday on a specific time like a cronjob.

Any lead on how can I achieve this deployment, other than using a cronjob because it will be unnecessary to pay for the time and resources when it will not be in use.

Thanks in Advance.

  • $\begingroup$ This is an engineering question, not a stats question. I think it belongs on a different forum. $\endgroup$ – Dan Jul 31 '18 at 10:01

If you want to schedule a task, you will necessary have to pay for a compute ressource, but you do not necessary pay for a ressource which can run your task.

To be more clear: take a small remote instance (full time) to a cloud service using a cronjob which will create an ephemeral instance (bigger) where your task will be executed.

If you want a tool to help you, take a look at Airflow


Not the answer you're looking for? Browse other questions tagged or ask your own question.