Over the past weeks, I've had the pleasure of diving back into the world of WebAssembly. I had some opportunities to get the taste of WASM at Asquera years ago, but I never got a serious chance to dig in. That time seems to have come though, as Vector is adding WASM plugin support. Thankfully, it would seem that WASM is a bit like a fine wine, and gets better with age. Today, let's do some wine tasting.

In this post we'll take a look at WASM outside of the browser where it's solving real world problems. We'll dig into some industrial use cases, then we'll explore the Lucet WASM compiler & runtime, how to tame the Foreign Function Interface, and what an ergonomic API looks like for both sides.

The frontier is everywhere

Vector isn't some laboratory research project or some toy experiment. It's a practical tool for real people to use to solve real problems. It's designed to be simple, practical, and adaptable.

That means, of course, we had a plugin/module/extension system to it.

There are a number of reasons for Vector to need to be extensible:

• As Vector adds more and more components, our build graph and binary size grow. This leads to increasingly more complex build environments and increasingly higher builder requirements. Supporting plugins would allow us to migrate functionality out of Vector core and into plugins.
• Some of the formats Vector needs to speak, such as Protobufs, are slow, or difficult to read without knowing the format of the message ahead of time. While Rust has amazing support for Protobufs through Prost, that support can only be built in at compile time. Supporting plugins would mean we could let folks build modules that used Prost or other libraries to decode events.
• We desire to support our users writing their own components in their favourite languages, and then let them leverage Vector to do the rest of the 'dirty' work for them. Supporting a plugin system will spurn us further in this direction, forcing us to build a nice external API.

As an observability router, we have some technical constraints, desires, and expectations:

we measured a roughly 4 μs overhead