Holdout Test: Experimenting on Meta

. 2 min read

Have you ever wondered if your Meta advertising dollars are actually driving revenue? If customers didn’t see or click on your ads, would these individuals be buying your products anyway?

You can test this hypothesis through Meta Experiments, a feature that let’s you quickly set up and analyze tests using the platforms experimentation technology.

Holdout Tests will specifically let marketers measure the true value of their advertising by comparing people who had the opportunity to see their Meta ads with those in a holdout group.

At Jump, we’ve recently run a holdout test for a client to test Meta's Halo effect on new purchasers by holding out certain geographic locations. This test resulted in 9.87% increase in CVR and 13.54% increase in New Purchasers.

The Hack

For beginners conducting a holdout test, it's important to note that Meta will generally withhold your advertising from 10% of your targeted audience. The results of that audience are then compared with the rest of the people to whom they showed your campaigns.

To create a holdout test, go to the Experiments tool and click Get Started under the Holdout Test section.

For a holdout test, you can choose from these options:

  • Account—test all of the ad campaigns you’re running
  • Campaign Group—test a few of your campaigns that you group together
  • Campaign—test one selected campaign

Scroll down and select the various conversion events you want your holdout test to measure.

You can pick up to 10 conversion events for which Meta will report conversion lifts on the Results tab.

Select conversion events that align with your business objectives, think eComm or Lead Generation.

Be mindful that your test can start showing results once there’s at least 100 conversion events observed. However, Meta recommends waiting until your test has completed and results are finalized to judge your test findings and act on them.