How to Run a Head-to-head Test with Incrementality

As incrementality provides the most accurate reflection of the added value gained from app retargeting, app marketers want to optimize their campaigns towards the best performing retargeting providers.

Comparing different retargeting vendors is usually done with head-to-head tests as a way to judge their performance. Though running these tests can become costly, having the right goal and setup will benefit clients in the long run.

There are different setups one might want to choose depending on the goal and scenario in mind. The two most frequent goals and their corresponding setups are:

Question: “What is the absolute performance and scale of each partner?”

Setup: Partners should run on isolated user lists to prevent last click stealing and have objective KPIs in performance and total scale. The isolation can be easily achieved by giving different partners a different device id range.

For example: Partner A gets IDFAs/GAIDs ending in digits: 0-7, while Partner B gets 8-F for a 50/50 split (device IDs use hexa-decimal numbers 0-9 and A-F with 16 total values which are randomly distributed). Partners should also run their respective incrementality measurement products as long as they have compatible methodologies.

KPIs: with this goal in mind, one can (and should) use both non-incremental KPIs (like CPA of attributed conversions) and incremental KPIs (like cost per incremental conversion: ICPA)

Question: “How much incremental value (revenue/conversions) does each partner bring on top of other partners and other marketing activities?”

Setup: Partners should run on the same user segments and run their own incrementality measurement product, respectively. The control group of each partner will then include both organic behavior and the behavior caused by other marketing or traction channels. Running with the same user segments ensures that the incremental results will show what each partner is bringing on top of the other channels.

KPIs: with this setup, one should only look at incremental KPIs (incremental revenue, incremental conversions, iROAS, and iCPA). Looking at non-incremental KPIs like CPA (cost per attributed conversions) would incentivize the partners to fight for the last click to get the attribution, consequently spending more budget than necessary on clicks. Non-incremental KPIs will ultimately hurt the incremental performance KPIs.

In either case, it’s important to make sure that the uplift testing methodologies of all partners are compatible before running any tests. i.e.:

• All setups are similar - i.e. segmentation and budget allocation are similar
• Test and control group assignment is independent from each other and is truly random
• All partners provide compatible and verifiable reports through raw data exports

Head-to-head tests are crucial for the long-term success of any re-engagement strategy. With incrementality measurement added into the mix, the tests provide insights that steer marketers towards better performance.

Remerge recommends to use the second scenario, since it ensures the client can achieve maximum scale in incremental revenues by potentially running multiple partners (as long as results show positive incrementality).