A/B testing 101 with React.js and Meteor

Cover for A/B testing 101 with React.js and Meteor

We use React.js and other Javascript frameworks to make better user experiences. But if you are not using metrics to measure the success of your work, you will never know if it makes a real difference.

A/B testing is very easy to implement into any codebase and will probably have a high ROI. React.js has a few A/B testing libraries so it is just a matter of storing data somewhere (in your own database or using a service such as Mixpanel or Segment).

Sometimes there are obvious flaws and tweaks to make to an application better. But most of the time improvements are not that easy to find and A/B tests can really help.

As for meteor there's only one small package but is not meant to work with React.js. For this article, let’s use a Meteor application that is fairly popular: Telescope. There's a new version in development and it’s using React.js for the frontend. A perfect example for A/B testing!

If you don't know Meteor, don't worry! Most of the code is about React.js so any React.js developer can understand the article.

Getting Started

Start by installing the new version by following the readme. When you open the application into your browser, you should see Telescope running:

Telescope Nova Homepage

You can see a newsletter form. This is probably one of the most important features in the application from a business perspective because it is the way users will remember the product exists and have news about it. You want to get the most subscribers possible. A/B testing improvements to this form will give you a way to measure success of your changes. Let's be scientific about improving apps instead of relying on our intuitions!

So let's create an experiment with two variants, A and B. The default is probably not going to convert well but what will. So A and B are going to be improved texts compared to the default text so we will be able to know what is the better text. Each time someone subscribe to the newsletter is counted as a success for the displayed variant.

A/B tests are meaningful only if you get at least a 100 successes for each variant within 6 weeks. If you don’t have enough traffic to get to this number, don’t use A/B tests and focus more on SEO and content marketing to grow your traffic.

Adding a database collection

Let’s create a new package in packages/abtests with the following package.js:

// packages/abtests/package.js
Package.describe({
  name: "abtests",
  summary: "Telescope abtests package",
  version: "0.26.0-nova",
  git: "https://github.com/TelescopeJS/telescope.git"
});

Package.onUse(function (api) {

  api.versionsFrom(['METEOR@1.0']);

  api.use([
    'nova:core@0.26.0-nova',
    'nova:base-components@0.26.0-nova'
  ]);

  api.addFiles([
    'lib/components.js',
    'lib/collection.js'
  ], ['client', 'server']);

  api.addFiles([
    'lib/server/publications.js'
  ], ['server']);

  api.export('ABTests');

});

Then we are going to create the collection in collection.js:

// packages/abtests/lib/collection.js
ABTests = new Mongo.Collection('abtests');

ABTests.schema = new SimpleSchema({
  _id: {
    type: String,
    optional: true,
    publish: true
  },

  name: {
    type: String,
    publish: true
  },

  variant: {
    type: String,
    publish: true
  },

  plays: {
    type: Number,
    publish: true,
    defaultValue: 0
  },

  wins: {
    type: Number,
    publish: true,
    defaultValue: 0
  }
});

Meteor.methods({
  'abtests.play': function (name, variant) {
    ABTests.upsert({name: name, variant: variant}, {$inc: {plays: 1}});
  },

  'abtests.win': function (name, variant) {
    ABTests.update({name: name, variant: variant}, {$inc: {wins: 1}});
  }
});

The schema of the collection is pretty straightforward. We have the name of the experiment, the variant, the display/play count, and success/win count. After that we have methods to increment plays and wins for a specified experiment and variant.

Add react-ab-test to a component

I am going to use react-ab-test to implement A/B tests in the component managing the newsletter subscription. This component can be found in packages/nova-base-components/lib/components/NewsletterForm.jsx and is used in packages/nova-base-components/lib/components/Layout.jsx.

npm install --save react-ab-test

We create a CustomNewsletterForm component in our abtests package that will override NewsletterForm:

// packages/abtests/lib/components.js
import React, { PropTypes } from 'react';

class CustomNewsletterForm extends Telescope.components.NewsletterForm {

}

Telescope.components.NewsletterForm = CustomNewsletterForm;

Then we can now start adding react-ab-test to the form, first by importing the components and other objects we need.

import Experiment from 'react-ab-test/lib/Experiment';
import Variant from 'react-ab-test/lib/Variant';
import emitter from 'react-ab-test/lib/emitter';
import experimentDebugger from 'react-ab-test/lib/debugger';

And make a new render based on the original render from NewsletterForm:

class CustomNewsletterForm extends Telescope.components.NewsletterForm {
  renderComponent(headerText) {
    return (
      <div className="newsletter">
        <h4 className="newsletter-tagline">{headerText}</h4>
        {this.context.currentUser ? this.renderButton() : this.renderForm()}
        <a onClick={this.dismissBanner} className="newsletter-close"><Icon name="close"/></a>
      </div>
    );
  }

  render() {
    ({Icon} = Telescope.components);

    // We don't generate the form on the server to keep things simpler.
    // It is possible but would only make this article uselessly complicated.
    if (Meteor.isClient && this.state.showBanner) {
      return (
        <Experiment ref="experiment" name="Newsletter Form Experiment">
          <Variant name="A">
            {this.renderComponent('Get the best design news by email each week')}
          </Variant>
          <Variant name="B">
            {this.renderComponent('Receive the latest design news in your inbox')}
          </Variant>
        </Experiment>
      );
    } else {
      return null;
    }
  }
}

We need to emit wins manually when a user subscribe. To do that we need to rewrite the subscription methods subscribeEmail and subscribeUser.

// Add these import declarations below the others.
import { Actions } from 'meteor/nova:base-components';

import Core from 'meteor/nova:core';
const Messages = Core.Messages;

// Add these methods inside the component class.
class CustomNewsletterForm extends Telescope.components.NewsletterForm {
  emitWin() {
    this.refs.experiment.win();
  }

  subscribeEmail(data) {
    Actions.call("addEmailToMailChimpList", data.email, (error, result) => {
      if (error) {
        console.log(error)
        Messages.flash(error.message, "error");
      } else {
        Messages.flash(this.props.successMessage, "success");
        this.emitWin(); // emit win when subscription is successful
        this.dismissBanner();
      }
    });
  }

  subscribeUser() {
    Actions.call("addCurrentUserToMailChimpList", (error, result) => {
      if (error) {
        console.log(error)
        Messages.flash(error.message, "error");
      } else {
        Messages.flash(this.props.successMessage, "success");
        this.emitWin(); // emit win when subscription is successful
        this.dismissBanner();
      }
    });
  }
}

Finally we listen to play and win events to store the results:

// after import declarations but outside the component class.
emitter.addPlayListener(function(experimentName, variantName) {
  console.log(`Start experiment ‘${experimentName}’ variant ‘${variantName}’`);
  Meteor.call('abtests.play', experimentName, variantName);
});

emitter.addWinListener(function(experimentName, variantName) {
  console.log(`Variant ‘${variantName}’ of experiment ‘${experimentName}’ was clicked`);
  Meteor.call('abtests.win', experimentName, variantName);
});

The experiment is now in place and it has been pretty straightforward: a couple of components, a couple lines to store the results, and that’s it. react-ab-test has a debugger to switch variants easily.

// Add this line below the event listeners.
experimentDebugger.enable();

React.js experiment debug

Making the variants more different

A/B Testing is normally about making one change, that’s what we did here. But really changing a bit of text might not make sense because it doesn’t change the experience of the form that much. In my opinion, the variant to the default form should customize all the different texts and change the color of the button to make it stand out. In a way, it is still just changing one component.

See the whole A/B testing example for React.js and Meteor on GitHub

Getting the results

To get the results I made a server publication:

// packages/abtests/lib/server/publications.js
ABTests._ensureIndex({name: 1});

Meteor.publish('abtests.get', function (name) {
  return ABTests.find({name: name});
});

So when you want to see the results of the experiment, you just open the developer console of your browser then:

Meteor.subscribe('abtests.get', 'Newsletter Form Experiment');
ABTests.find().fetch();

Use a calculator to get the success rate of each variant or extend my code to add more information to the ABTests collection.

Conclusion

You can setup A/B tests in 15 minutes. If you have the budget, just use Mixpanel to store data, it has many great features and will require less maintenance.

Analytics are the only way to justify the value you bring to the product. Because improved UX happens at the same time as performance improvements, new features, better sales, and better marketing, you can't know what value your work have brought unless you measure it. I think it's crucial for developers to realize this.