Trade-Offs

This section deals with pitfalls of Metaflow and other technologies that help solve the same problem.

What Are the Pitfalls of Gaining More Productivity With Metaflow?

• Data storage can become huge if bad design choices are taken when making a flow.
• Distributed model training over multiple machines is not possible using Metaflow. Even though the library has technical capabilities to support that there, structuring a flow like that will not ensure idempotency in tasks. But it can severely reduce the compute time of very large deep learning models.
  • Libraries like Torch support distributed data parallel training and recent works in distributed training have shown very fast training with scaling out GPUS. Fast AI have showcased this by training on ImageNet with 93% accuracy under 18 minutes
• Doesn't support dynamic DAGs
• No inbuilt Support for Spark.
  • Support can be leveraged via abstractions from other modules.

Similar Technologies and Related Works

Flyte by Lyft

Architecture of Flyte

Flyte by Lyft which was open-sourced in Jan 2020 has the same architecture with native support for Spark. Even though Flyte and Metaflow are really similar libraries, their design choices have drifted them apart quite distinctly.

Flyte has more strictly typed decorators and input conventions. For example: Flyte takes strict input parameters according to dataflow from steps. Metaflow stores artifacts generated from a step in a flow internally as part of the object. It does so assuming that access to properties will happen with respect to the dataflow paradigm. This design difference makes Metaflow more human-friendly for adoption.

Task Definition on Flyte

Workflow definition from the Tasks.

Flyte also has its own suite of services it deploys on Kubernetes. It is not self-isolated like Metaflow. Unlike Metaflow, Flyte has very good support for BigQuery and other data processing frameworks like Spark and Kubernetes.

Kubeflow