N-Testing
      Back to home
      1. Knowledge Base
      2. N-Testing

      How to apply new parameters value from finished A/B test

      Table of Contents

      Notes: This part is handled by our team. If you are part of a studio, get in touch with your publishing manager to assist in deploying winning configs to new players.

      • Homa team members can modify parameter values and apply these modification into scope (OS, country).
      • Studio teams can only see the value parameters for their game.

      When can we say that a test is done?

      In the previous page we showed how to create and launch an A/B Test. Homa publishing managers (in collaboration studio teams) can decided to conclude a test or continue to collect more data. All test viewers however can see the results. Some reasons why it may be a good idea to let a test run for longer after you've obtained some initial results are:

      • Reviewing longer term metrics. For instance, maybe Retention D3 has +10% but maybe Retention D14 could be -5%
      • Allowing more data to collect and potentially find more conclusive results for different metrics

      Capture d’écran 2022-11-16 à 14.34.48

      Otherwise, here are some general guidelines for ending tests:

      • Tests should run for a minimum of 7 days to evaluate if they can be stopped
      • A significant number of players/installs should be included in the test (at least 2000-3000 players in the defined scope) to have enough relevant data for engagement metrics

      Step 1: How to end a test

      Capture d’écran 2022-10-20 à 14.56.36

      Note: By ending a test, the value of parameters is not updated in the “production” configuration yet.

      Step 2: Update the default parameters if a winner configuration was found

      A separate result will be shown for each configuration. The winner of the test however, is selected from all clear or possible wins, based on having the highest overall positive impact. By clicking the Roll out button the winner will be distributed to all new players. However, any of the variants can be rolled out from their individual contextual menus. 

      Capture d’écran 2022-11-17 à 11.30.18

      After clicking the Roll out button, a pop-up will display where you will be able to choose the scope for the roll-out and also optionally end the test. 

      Capture d’écran 2022-11-17 à 11.31.42

      Note:

      • The new value will impact only the NEW players of the game instantly. To apply it to existing players, you will need to override the value (see step 3 below).
      • If you want to roll out manually you can check step 3 on this page: Get your game ready for N-testing
      • The roll out in the tab parameter tab will be available in the next 5 minutes

      Step 3: (optional) Set up an override to also affect existing users

      An override is a set of Parameters that is applied to all Players (new players or existing players who have already opened the game). The overrides must be applied on the scope level

      Notes:

      • If the A/B testing has a successful result, an override is not applied automatically, only the Homa publishing manager (in discussion with the studio) can decide if the values should be overridden and so apply for existing players

      The list of parameters you want to override is saved under a version. Each time you need to change this list, you need to save the new list under a new version.

      If needed you can revert the version to come back to a previous one

      • In the N-testing module → App configurations
      • Click on your game
      • Click on APPLY ALL PLAYERS

      Capture d’écran 2023-05-29 à 18.05.09

      • Select the scope that you want to made the modification
      • Click on Create new version - a new version will be created

      Capture d’écran 2023-05-29 à 18.05.41

      • Click on ADD PARAM

      Capture d’écran 2023-05-29 à 18.06.13

      • Enter the Parameters with the values that you want to override. You can add multiple parameters by clicking on ADD PARAM.

      Capture d’écran 2023-05-29 à 18.07.16

      • Click on SAVE & APPLY. You can see that the parameters are active in the selected scope

      Capture d’écran 2023-05-29 à 18.07.47

      • If you want to come back to a previous version (or an other version), you can click on ADD OVERRIDE, select the version that you want and click on the button UPDATE.

      Capture d’écran 2023-05-29 à 18.08.44

      Notes:

      • If you want to override values in multiple scopes (like different countries), you need to repeat the operation in all the concerned scopes.