Bonjour!

I'm Julien, freelance data engineer based in Geneva 🇨🇭.

Every week, I research and share ideas about the data engineering craft.

Not subscribed yet?

👨🏽‍💻 echo {YOUR_INBOX} >>

Software has always been a matter of abstraction.

Over the years, the industry has constructed layers upon layers to develop increasingly complex software.

The same trend is happening in the data world.

More and more tools are emerging to standardize the construction of data pipelines, pushing towards a declarative paradigm.

Engineers spend less and less time coding and more and more parametrizing and coordinating functional building blocks.

In the first version of this post (co-written with Benoît Pimpaud), we highlighted the early signs of this trend:

We called it provocatively: From Data Engineer to YAML Engineer.

One year later, the movement has only accelerated.

So, let’s keep the exploration going with:

1- Declarative Data Ingestion: dlt

ELT too,s have always had a declarative flavor—you define your connector and target destination and let the tool handle the rest.

And for common sources with well-supported, battle-tested connectors, this works beautifully.