Lecture: Distributing Machine Learning Tasks with Python
Training machine learning models to fulfil a business need is becoming increasingly common. At The MoonVision GmbH we built our training api and machine learning models using python. This talk will present how we are able to train machine learning models through an API in a robust, scaleable manner while using almost exclusively python.
I will present how we use python and Django to power our computer vision api and then use dask to distribute the workload to a cluster. The challenges we ran into and how we overcame them will be explored. We will start from our architecture diagram to understand the system as a whole and then dive into the individual components. The use of Kubernetes to simplify deploying and monitoring our python applications will also be discussed.