A/B Testing Campaigns: How to Test Prizes, Formats and Forms

Is a scratch card better than a wheel? Does a bigger prize bring more sign-ups, or just more prize hunters? Should you ask for a phone number or not? A/B testing campaigns is the most reliable way to find out. Instead of guessing, you show two versions to comparable groups of people and let the results decide.

This guide explains what to test first, how many people you need, how to set up tests for prizes, formats and forms, and how to read the results without fooling yourself.

A/B testing campaigns: the basics

An A/B test compares two versions of something that differ in one way. Version A is your current campaign (the control). Version B changes one thing, such as the prize. Everything else stays the same: the audience, the timing, the channel and the design.

Three rules make a test worth running:

1. Change one thing at a time. If you change the prize and the format, you won't know which one made the difference.
2. Split the audience at random. The two groups must be comparable. Desktop vs mobile, or Monday vs Friday, is not a random split.
3. Decide the metric before you start. Pick the one number that decides the winner, for example sign-up rate (sign-ups ÷ visits).

How to split traffic for a gamified campaign

Scratcher doesn't have a built-in A/B split, so you run the test by making two copies of the campaign and splitting the traffic in the channel that sends people there. This is often the cleanest set-up anyway:

• Newsletter. Most email tools can split a send at random. Put the link to campaign A in one version and campaign B in the other.
• Paid social and search. Create two ads or ad sets with the same targeting and budget, each linking to its own campaign. Many ad platforms have their own split-test feature for this.
• QR codes and print. Use two QR codes on alternate posters, flyers or receipts in the same stores.
• Website pop-ups. You can target pop-ups by page, domain and device, so the cleanest test is between comparable pages or sections. Treat the result with a little more caution than a random split.

If none of these are possible, you can run A and B in consecutive periods. It's the weakest option, because seasons, weekdays and campaigns elsewhere affect the result, so use it only for big differences.

What to test first

You can test almost anything, but some changes matter much more than others. Start with the decisions that affect the most people and cost the most money:

1. The prize. What people can win is usually the biggest driver of participation.
2. The form. The number of fields decides how many players become sign-ups.
3. The format. A quick game of luck vs a quiz or a skill game.
4. The invitation. The headline, button text or email subject line that gets people to start.
5. Timing and placement for pop-ups: after a delay, on scroll, on exit intent or on click.

Leave colours and small copy changes until later. They rarely move the result enough to measure with campaign-sized audiences.

Sample size basics

The most common testing mistake is calling a winner too early. With small numbers, random noise can easily make one version look better.

How many people you need depends on three things:

• Your current conversion rate. The rarer the action, the more people you need.
• The smallest difference you care about. Detecting a small lift takes far more visitors than detecting a big one.
• How sure you want to be. Most tests use 95% confidence.

You don't need to do the maths by hand. A free tool such as Evan Miller's sample size calculator tells you how many visitors you need per version. Put in your baseline conversion rate and the smallest lift you'd act on.

If the answer is more traffic than you'll get, don't run a fuzzy test. Test a bigger change instead, or combine the test across several campaigns.

Testing prize vs no prize

Before testing which prize works best, it can be worth testing whether the prize matters at all for your audience. For example:

• A: A wheel where everyone wins a small discount code
• B: The same wheel with a grand prize draw for all participants, and no discount code

Look at more than sign-ups. A prize that attracts a lot of people who only want the prize can give you a long list with low engagement afterwards. Follow both groups for a few weeks after the campaign and compare:

• Email open and click rates
• Unsubscribes
• Purchases or code redemptions

In Scratcher you control the prizes in detail: instant wins, draws, unique discount codes and barcodes, how many of each can be won and when. That makes it easy to keep everything else identical between A and B.

Prize ideas worth testing against each other:

• Your own product vs a gift card
• One big prize vs many small ones
• A discount code for everyone vs a draw for a few
• A consolation prize vs no consolation prize

Testing format

Format affects how long people spend with your brand and what you learn from them. Typical comparisons:

• Scratch card vs spin the wheel. Both are quick games of luck. A good first test if you want more sign-ups.
• Game of luck vs quiz. A quiz takes longer but teaches people about your products, and the answers tell you what they like.
• Single game vs calendar. A calendar brings people back many times. Compare total sign-ups and repeat visits, not just day-one numbers.

When you test formats, keep the prize, form and invitation identical. Choose from 40+ formats, and use the same template style for both versions so design doesn't skew the result.

Testing form length

Every field you add costs some participants. But some fields, like a phone number or a birthday, may be worth the drop. Testing tells you whether they are.

Example test:

• A: Name and email
• B: Name, email and phone number

Measure both:

• Sign-up rate (sign-ups ÷ players)
• Value per sign-up. If a phone number lets you sell by SMS, B may win on value even if it loses on volume.

Keep the consent text the same in both versions, so the only difference is the extra field.

Reading the results

When the test has reached the sample size you planned, compare the two versions on the metric you chose up front.

• Check significance. Use the same calculator or a significance calculator to see whether the difference is likely to be real.
• Look at the full funnel. Visits → players → sign-ups → follow-up actions. A version can win at one step and lose at the next.
• Check for surprises. Did one ad get more budget? Did one email land in more spam folders? If the split wasn't fair, the result isn't either.
• Write down what you learned, even when there's no winner. "The prize made no difference" is a useful result.

Scratcher's statistics show visits, players, sign-ups and winners for each campaign in near real time, along with where people came from. Because A and B are separate campaigns, you can compare them side by side. You can also follow both in Google Analytics or your ad platform. For the full list of metrics worth tracking, see our post on gamification marketing KPIs.

Test log template

Keep a simple log of every test, so the team builds knowledge over time instead of repeating the same tests. Copy this structure into a spreadsheet:

Start with one test

You don't need a testing programme to start. Pick your next campaign, choose the one decision you're least sure about, usually the prize, and split the traffic in your newsletter or ads. One good test teaches you more than ten guesses.

Want more ideas for converting website visitors? Read our guide to high-converting pop-ups, see how game pop-ups work, or try Scratcher free for 14 days and build version A and B today.