One of the typical constructions, that have been developed by data engineers and data scientists for data processing are the data pipelines. Pipelines – consist of simple steps connected to each other by passing data from the dataset along the connection direction. The purpose of the functions inside the pipeline may vary widely: from the restoring of missing values to split the data to the train, test and validation samples, and learning complex neural-network-based models. Design of such pipelines can be automated using advanced language models given the description of the steps in plain English.
The goal of the project is the collection of data that contains typical steps for data processing pipelines. Each step item includes the description in English, the corresponding snippets in Python and some high-level tagging information that would help identifying similar tasks. Such collection should be extensible and accessible via API.
Code4ML: a Large-scale Dataset of annotated Machine Learning Code
Taxonomy class to Python Translator
![Whiteboard-[ibJrGtamRb-XDh-kVnXwLA]-01.png](https://s3-us-west-2.amazonaws.com/secure.notion-static.com/fe47b385-7c52-42a3-9ebb-90aa0495324a/Whiteboard-ibJrGtamRb-XDh-kVnXwLA-01.png)