Words Matter: Testing Copy With Shakespeare

The Backstory

  1. Copy others?
  2. Go with whatever we hear about most on Twitter?
  3. Hire an expert and let them decide?
  4. Let leadership make the call?
  5. Vote on what we think will work best?

Introducing Shakespeare

  • Every test required non-trivial engineering support.
  • Tests weren’t easy to set up, configure or clean up afterwards.
  • Tests took days to weeks to deploy.
  • We didn’t have a uniform way to test across platforms.
  • There was no easy way to test localized or transcreated copy.
  • We lacked the infrastructure for continuous explore/exploit tests (for which real-time analysis detects winning versions early and then allocates more users to those cells vs. traditional, equally distributed a/b tests).
  • Engineering dependencies are greatly reduced.
  • Tests are easier to set up, configure and clean up.
  • We have the ability to make real-time copy updates.
  • We’re able to consistently test across platforms.
  • Localized or transcreated copy can be tested independent from the English source.
  • We have the option for continuous explore/exploit.

Building Shakespeare

String storage

String fetching

Improving the Process

Setting up a Shakespeare test

Establishing test copy with a/b test-cell mappings

How strings are fetched with Shakespeare

Continuous Integration (CI) and Continuous Delivery/Deployment (CD)

In summary

  • The Shakespeare Web UI makes it easy to enter copy variants.
  • A rules engine extracts test-cell copy mapping logic.
  • A data-subscription service handles rules distribution.
  • Our proprietary tool ABlaze allocates tests.
  • Shakespeare returns real-time user test-cell examination and copy override.
  • Continuous Integration and Continuous Delivery/Deployment provides easy integration and real-time deployment.

The Content Design Perspective

  • Word choice for microcopy. Even the smallest change can have a huge impact.
  • Tone. Our voice attributes are Helpful, Warm, Playful, Relevant and Provocative. When should we lean into different tones? Shakespeare is helping us find out.
  • Global relevance. Sometimes a language hypothesis created in Silicon Valley or L.A. doesn’t resonate in other areas of the world and feels more natural when it’s customized to the market.
  • UX best practices. By adding step numbers to the copy in our onboarding flow, we were able to increase the completion rate because people knew how many steps to expect.
  • Style. We avoid all caps because they can feel shouty, but is there ever an exception to this rule?
  • Clarity. Is there a simpler, more intuitive or more inclusive way to explain something?
  • Context awareness. When the coronavirus pandemic began, we were able to quickly modify the text in our sign-up flow, since the “before” version felt tone-deaf in light of travel restrictions and more time spent at home.

Beyond the Metrics

--

--

--

Learn more about how Netflix designs, builds, and operates our systems and engineering organizations

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What to expect in your first 2 weeks at Perlego

Bery Data Joined Secretworld Metaverse

Is it Time for Go Programming?

CloudAnswers Hackathon Highlights: Shipment Scanner

Why social distancing is important to fight Corona virus — A Network analysis of Corona Virus in R

WSO2 log monitoring with Fluend

A Simple Golang Application via Kubernetes

Interactive Butterfly Flocking

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Netflix Technology Blog

Netflix Technology Blog

Learn more about how Netflix designs, builds, and operates our systems and engineering organizations

More from Medium

Slack: Prototyping a way to humanize the out-of-office reply

How to effectively combine premium and freemium service: A product UX implementation.

Strategy one (1) — showing three (3) touchpoint to payment on how to effectively onboard users into a product payment model.

In 2022, we may still not correctly understand the value of UX Design

Solving LeanIX Case Study — Improving EAM Matrix Report