N-testing overview

• What is N-testing and who can use it?
  • N-testing essentials
• How different teams use N-testing
  • How our team uses N-testing
  • How our community of studios uses N-testing
• Creating new A/B tests
• Why run N-tests?
  • Top ways to use N-testing
  • Some successful examples of N-testing tests
• N-testing page guides

What is N-testing and who can use it?

N-testing is a module on Homa Lab which enables game teams to A/B test new player behaviour by defining in-game parameters to change (configs), and the audience the changes apply to. Then, statistical tests are run to determine if the differences between the resulting groups are significant. Once a clear win or loss is found, the result is displayed alongside that config, which can be rolled out to the entire game audience to increase engagement or LTV.

N-testing essentials

• A/B tests are run only on new players - to ensure that all players are using the same game build and the spotted differences are relevant
• A proportion of players is selected for the test (e.g. 10% of players) and filters can be used to ensure the audience comparisons are relevant. For instance, for an LTV-based test it’s recommended to use one platform and one country (e.g. iOS, US) so that differences between groups are not caused by the groups containing users from different countries where LTV values vary significantly.
• It’s possible to configure the exact parameters changed in the game for each test - e.g. tutorial_on or tutorial_off
• The winners are selected by running statistical testing on the groups and it’s important to wait for a clear win/loss result before applying a config to avoid false positives/negatives
• Once a clear conclusion is reached, it’s possible to apply that configuration to all new players

How different teams use N-testing

How our team uses N-testing

Team members that frequently run A/B tests are: Publishing Managers, the Live Ops teams, internal game designers or producers. They use it to:

• Create A/B tests to compare results between multiple values of the same parameters - e.g. button_red or button_green. N-testing typically starts when a title has enough players, around the time of the launch, and from there continue throughout the game’s life.
• Estimate the number of users added to a test. When selecting the audience, Homa Lab will calculate and show how many players are expected to be added.
• Select the audience for the N-test by share (%) and key parameters (OS and geo). For instance, to get results quickly, a larger % of users could be targeted, or generic changes expected to increase playtime or retention across the board could target all geos.
• Deciding which config to roll out after the test - e.g. if winning config was rolled out to 10% of users for the test, you can then roll it out to 100% of new players without needing to push a new build. However, when there is a likely improvement rather than a certain improvement, the game team may choose to wait to get more conclusive results.
• Test the modification of parameters on a local device or from a specific IP to make sure they work well.

How our community of studios uses N-testing

Studios or game devs using Homa Lab can use N-testing to:

• See the results of different A/B tests run for their game. Right now these tests are typically started by publishing managers. However, studios can also propose new tests and review what has worked well in the past.
• Review the parameters which have been set up in the games. These parameters are configured in the build by the game team, but with many tests created over time, it can be useful to see them all in one place.

Creating new A/B tests

Right now, new tests have to be reviewed and run by a Homa team member. This is due to the fact that rolling out configs changes the game experience and with any change there is a risk of introducing errors or bugs.

However, to make this process easier and enable even more game teams to make data-driven decisions and improve their games with A/B tests, we’re in process of setting up a friendly tiering system to allow advanced users to configure and run their own tests.

Why run N-tests?

The goal of N-testing is to drive up LTV. This can be done directly, for instance by configuring the number of ads show, or testing new placements, or indirectly by aiming to increase engagement metrics such as retention or playtime.

One of the key benefits of N-testing is that it allows game teams to configure the experience the player received without having to deliver different builds to the app store, or to require users to update their game. This can save a significant amount of time. For instance, you can test two Christmas campaigns in parallel, and then roll out the one that performs the best to all users, with the certainty that all users who installed from a certain date onward will see it, even if they don’t update their game before the holidays.

Top ways to use N-testing

Some popular and successful ways to use N-testing have been to test:

• Different ad setups, placements, or ad pressure (ad frequency) to see if they change engagement and grow LTV. For instance, you could set ad_pressure_low - 60 or ad_pressure_high - 75 to show an ad every 60 or 75 seconds and determine if LTV changes.
• Different UI/UX elements - e.g. menu designs, button colours, CTAs
• Progression variations - e.g. adding a variable delay between offering a rewarded video or skipping without watching to proceed to next level
• Economy changes - e.g. changing the number of coins offered at the end of each level
• Game design edits - e.g. making the game more or less difficult to determine if it results in more engagement or conversions

Some successful examples of N-testing test

N-testing page guides

List of all the steps to use N-testing

1. Get your game ready for N-testing
2. How to setup/launch an A/B test
3. How to read the results of an A/B test
4. How to test a configuration on your local device 
5. How to apply new parameters value from finished A/B test
6. How to have the list of parameters of a game