All we wanted was a simple A/B test

Our team needed to run custom user experiments across our platform - a travel experiences marketplace. We were running a Next.js + React application with varying degrees of well-written and...uhhh..less than well-written views and components. The platform was sprawling, but worked well and heavily utilizes Next.js-flavored Server Side Rendering (SSR).

To implement experiments, we started with NPM (of course) and implemented react-ab-test. POW! We built our fist set of experiments, set up tracking in Mixpanel, and deployed it to production in less than a couple of days. Success!

We were excited about the build and quick turnaround! That was until the user complaints started coming in 😫. Some of our A/B experiments were breaking styling! After the requisite conversations with our product owner, we pulled the experiments off production and got to work investigating. We read the documentation inside and out, examined our SSR use-case, and staged countless iterations of the experiments - all to no avail. The library was simply not playing nice in our Next.js / SSR production environment.

We opened an issue with react-ab-test, explored forks of it and similar libraries, and worked to create PRs in other third-party libraries - none of it worked. At best these libraries included SSR as an afterthought and at worst they wouldn't run in any of our environments. We were now very overscoped on the first set of experiments and had more experiments waiting in the product roadmap. We were stuck and had no custom A/B experiments on our platform.

So we built nextjs-ab-test:

We still had no answers from the community, so we built our own in-house A/B testing library. The goals: follow best practices, keep it minimal, and make it SSR-first. And after a lot of tweaking, reviewing, and bug-fixing...it worked!

We stayed away from outdated event emitters, borrowed best practice algorithms for variant selection, and kept things minimal. The library simply uses the component's internal state to pick which variant to render. It also has simple hooks to make it play nice with newer react components.

Does it work?

nextjs-ab-test is fully tested. Our product team loves TDD and made sure it would be reliable across our platform and as a standalone library. The project uses a test suite built on Enzyme and Jest.

Sweet, how does it work?

I mean sure! The full documentation is on github and NPM and there are also examples there (since we open sourced it 😉 ). Here's a breakdown:

Component based usage

Let's say we want to show 2 different Variants. One that displays a coffee and one that displays an iced tea and you want to show only one to 70% of your users. You would do it like:

import { Experiment, Variant } from "nextjs-ab-test";

<Experiement name="coffeeOrIcedTea" weights={[70, 30]}>
	<Variant name="coffee">
		<p className="drink">French Vanilla Coffee</p>
	</Variant>
	<Variant name="tea">
		<p className="drink">Lemon Iced Tea</p>
	</Variant>
</Experiment>

So we create the experiment by giving it a unique name which is coffeeOrIcecTea this helps differentiate between different experiments. The weights prop works from the first listed variant to the last listed variant. In this case, it would be the coffee variant to the tea variant. The coffee variant would show 70 percent of the time while the tea variant would show 30 percent of the time.

Now if you wanted to make sure that a variant shows ALL the time, you can just set the activeVariant. Looks a little like this:

<Experiment name="coffeeOrIcedTea" activeVariant="coffee">
	// Variant Code here
</Experiment>