NewThe guide: 11 paywall examples

10 minutes

Mobile experimentation: our method for optimizing a B2C app

Mobile experimentation runs through seven steps: data, problem, hypothesis, prioritization, experiment, measurement, and learning. The goal is to learn faster, not to multiply tests.

Clément Parot

Mobile experimentation: our method for optimizing a B2C app

Article summary

  • Mobile experimentation runs through seven steps: data, problem, hypothesis, prioritization, experiment, measurement, and learning.
  • Testing at random means optimizing a solution before you have identified the problem.
  • Changing the paywall, the pricing, the onboarding, and the acquisition at the same time makes it impossible to know what produced the result.
  • The speed of learning depends on the quality of the hypotheses, the speed of execution, and the quality of the measurement.
  • Tests cover ASO, onboarding, the paywall, pricing, retention, and notifications, not only the payment screen.

Summarize this article with

A good mobile experimentation strategy rests on 7 steps:

StepRole
01. DataUnderstand what is happening
02. ProblemIdentify where it gets stuck
03. HypothesisUnderstand why
04. PrioritizationChoose what to test
05. ExperimentBuild the test
06. MeasurementMeasure the real impact
07. LearningTurn the result into the next test

The goal is not to run more tests. The goal is to learn faster.

Two mobile experimentation mistakes

❌ 01 - They test at random

A competitor ships a new paywall → we copy it.

A conversion drops → we change the pricing.

A designer proposes a new version → we test it.

⚠️ Warning : We optimize a solution before having identified the problem.

❌ 02 - They test everything at the same time

New paywall.
New pricing.
New onboarding.
New acquisition.

Then: Revenue +18%

⚠️ Warning : We may get a result. But we do not know what caused it.

AB TEST EXAMPLE
AB TEST EXAMPLE

Newsletter

Once a week.

Last issue

We opened 10 apps to see how they sell a subscription.

+534 people read us each week

What is mobile experimentation?

Mobile experimentation means using data, hypotheses, and controlled experiments to gradually improve an app.

It can include A/B testing, but not only that.

An experiment can cover:

  • ASO;
  • onboarding;
  • the paywall;
  • pricing;
  • offers;
  • conversion;
  • retention;
  • notifications;
  • the post-payment journey.

The principle stays the same:

Observe → understand → hypothesize → experiment → measure → learn.

The goal is not simply to find a variant that converts better.

The goal is to understand why it converts better, and to use that learning to decide the next test.

The result when it is done well:

Before / After firstapp support
Before / After firstapp support

Our mobile experimentation framework

At firstapp, we apply this loop on every B2C app we optimize. Our method can be summed up in seven steps:

DATA → PROBLEM → HYPOTHESIS → PRIORITIZATION → EXPERIMENT → MEASUREMENT → LEARNING

Then the loop starts again. Each experiment should reduce part of the uncertainty we have about the product.

1 - Instrument the app properly

Before testing anything, you have to be able to measure what is happening.

A monetization funnel can look like this, for example:

StepEvent
Installinstall
Onboarding startedonboarding_started
Onboarding completedonboarding_completed
Paywall viewedpaywall_viewed
Trial startedtrial_started
Subscription startedsubscription_started
Renewalrenewal

Each step has to be instrumented correctly. But events alone are not enough.

We also want to be able to analyze users along different dimensions:

  • acquisition source;
  • country;
  • platform;
  • campaign;
  • cohort;
  • subscription type;
  • behavior;
  • tenure.

An overall conversion of 4% can look interesting.

💡 Key data : If users coming from TikTok convert at 1.5% and those coming from organic at 7%, we probably do not have a single problem. We have several segments with different behaviors.

Good experimentation therefore starts with good instrumentation.

2 - Identify the problem

A metric tells us where to look.

It does not necessarily tell us why.

Take an example: only 20% of users see the paywall.

The instinctive reaction could be:

"The paywall needs to be shown earlier."

But we have already jumped to a solution. We first need to understand what is happening.

To do that, we can cross quantitative data with:

  • App Store reviews;
  • user interviews;
  • support tickets;
  • recorded sessions;
  • feedback;
  • cancellation reasons.

The goal is to turn a quantitative anomaly into an understandable problem.

StepExample
ObservationUsers drop off massively from onboarding at step 5.
InvestigationStep 5 asks for account creation before the user has really understood the app's value.
Potential problemAccount creation comes too early.

We now have a question we can test.

3 - Formulate a hypothesis

We use a simple structure:

💡 Template : We believe that [problem] prevents [segment] from [action].
We are going to test [change].
We believe this will increase [metric].

Example:

We believe that account creation too early in onboarding creates friction before the user understands the value.

We are going to move account creation to after the first moment of value.

We believe this will increase the onboarding completion rate.

The difference can seem subtle. But it is fundamental.

We are not testing:

"a new onboarding".

We are testing:

the hypothesis that a specific friction prevents activation.

4 - Prioritize the experiments

A team can easily have 30 or 50 test ideas.

It probably does not have the capacity to launch 50 of them. So we have to prioritize.

A simple method is to use:

ICE = Impact × Confidence / Effort

CriterionQuestion
ImpactWhat would the business impact be if the hypothesis is correct?
ConfidenceHow much data supports this hypothesis?
EffortHow long does it take to design, build, and ship the experiment?

This avoids a classic trap: spending three weeks building a test whose potential impact is low, while another experiment could be launched in two days.

The speed of experimentation also depends on the ability to choose the right tests.

5 - Design and launch the experiment

An experiment has to be defined before it is built. We define in particular:

ElementRole
ControlThe current experience.
TreatmentThe new experience.
AudienceThe users who will be eligible for the test.
Primary metricThe main metric that will let us evaluate the hypothesis.
Secondary metricsThe complementary metrics.
GuardrailsThe metrics that must not get worse.

For example, for a new paywall:

FieldExample
HypothesisA clearer value proposition will increase conversion to trial.
Primary metricPaywall view → Trial start
Secondary metricsInstall → Trial
Trial → Paid
Revenue per install
GuardrailsRefund rate
D7 retention
Cancellation rate

This avoids treating as a winner an experiment that lifts an intermediate metric but hurts revenue or retention.

6 - Measure the right impact

An A/B test should not only measure what happens on the screen being tested.

Take a paywall. Variant B increases the trial-start rate by 25%. That looks excellent.

⚠️ Warning : If users of this variant then convert less well into paid subscriptions, the result can be negative for the business.

So we want to look at the full chain:

Paywall view → Trial start → Trial to paid → Revenue per install → LTV

Depending on the context, we can track:

  • conversion;
  • revenue per user;
  • revenue per install;
  • ARPU;
  • retention;
  • churn;
  • renewal;
  • LTV.

The right metric is the one that matches the business goal of the experiment.

7 - Turn results into learnings

A test should never end with:

B wins +12%.

We want to know:

Why?
ResultWhat we do with it
The test winsUnderstand what caused the improvement, then formulate a new hypothesis.
The test losesWe have potentially invalidated a hypothesis. That is information too.
InconclusivePart of our understanding of the problem is still insufficient.

In all three cases:

Result → learning → new hypothesis → new test

This is the loop that creates cumulative progress.

What can you test on a mobile app?

Experimentation is not limited to the paywall. We generally look at several large zones of the funnel.

ZoneGoal
ASOImpression → visit → install
OnboardingInstall → activation → monetization
PaywallPaywall view → trial / paid
Pricing & packagingConversion, but above all revenue and LTV
RetentionRenewal, churn, LTV

ASO

The App Store is the first conversion step.

You can experiment with:

  • the icon;
  • screenshots;
  • screenshot order;
  • messaging;
  • the value proposition;
  • localization;
  • Product Pages.

Goal: impression → visit → install.

Onboarding

Onboarding has to turn an install into understanding, then into activation. To go further: how to optimize a mobile onboarding.

You can test:

  • number of screens;
  • order;
  • personalization;
  • quiz;
  • account creation;
  • the moment of the first "aha moment";
  • messaging;
  • paywall timing.

Goal: install → activation → monetization.

Paywall

The paywall is not just its design. See also how to optimize a mobile app paywall.

You can test:

  • placement;
  • timing;
  • structure;
  • messaging;
  • CTA;
  • benefits;
  • social proof;
  • number of plans;
  • offer;
  • personalization.

And above all:

⚠️ Warning : The best paywall for an app is not necessarily a competitor's.

Mobile app optimization
Mobile app optimization

Pricing & Packaging

The price itself is an experimentation variable.

You can test:

  • price;
  • monthly vs annual;
  • trial length;
  • presence or absence of a trial;
  • lifetime;
  • bundles;
  • entry offers;
  • pricing by country.

A price increase can lower the conversion rate while increasing revenue. That is why:

Conversion ≠ Revenue
Revenue ≠ LTV

You have to look at the metric that matches the real business goal.

A/B testing by firstapp
A/B testing by firstapp

Retention

Optimization does not stop at the first payment. On churn: how to reduce cancellations.

You can experiment with:

  • post-purchase onboarding;
  • activation;
  • notifications;
  • value reminders;
  • engagement;
  • cancellation flow;
  • retention offers;
  • win-back.

A user who cancels also gives us information.

Why did they leave? That answer can become the next hypothesis to test.

How to avoid the main experimentation biases

BiasWhat happensSolution
ConfirmationYou think variant B is going to win, and you read every signal as confirmation.Define the hypothesis and the success criteria before the test.
CompetitiveYou see a competitor do something and assume it works.Start from your own data first.
Multiple testingYou change five variables at the same time.Reduce the number of variables when you are trying to understand a cause.
Metric fixationYou optimize a metric because it is going up.Define a primary metric and guardrails tied to the business.
Vanity metricsYou look at a metric because it is easy to improve.Tie every experiment to a business goal: Revenue → Conversion → Retention → LTV.

The speed of learning is a competitive advantage

A large part of the subject can be summed up with a simple formula:

Speed of learning = quality of the hypotheses × speed of execution × quality of measurement

If you have excellent hypotheses but it takes three weeks to launch each test: you learn slowly.

If you ship very fast but test at random: you produce a lot of noise.

If you have good hypotheses and fast execution but poor instrumentation: you do not know what works.

The three have to work together.

Our experimentation method in one loop

In the end, everything comes back to this:

Measure → experiment → learn → improve

A B2C app should not be treated as a "finished" product.

It should gradually become a system able to measure, experiment, learn, and improve.

A roadmap should therefore not only answer:

"What are we going to build?"

It should also answer:

"What are we going to learn?"

That is the difference between building an app and building an app you know how to move forward.

Learn faster, not test more

A good mobile experimentation strategy is not about multiplying A/B tests.

It is about building a system where every experiment has a reason to exist.

The goal is not to test more. The goal is to learn faster what actually moves your app forward.

If you already have a B2C app and you do not know which levers to prioritize, ASO, onboarding, paywall, pricing, conversion, or retention, we can start by analyzing your funnel and identifying the first hypotheses to test.

FAQ - Mobile experimentation

What is mobile experimentation?

It is a method that uses data and tests to identify what actually improves a mobile app's performance.

What can you test on a B2C app?

You can experiment across almost the whole funnel: ASO, onboarding, paywall, pricing, offers, conversion, and retention.

Do you need a lot of users to do A/B testing?

Not necessarily. The volume you need depends on the metric, the conversion rate, and the impact you are looking for. With little traffic, some qualitative tests or targeted changes can be more relevant.

How long should an A/B test last?

There is no universal duration. It depends in particular on traffic volume and on how fast users reach the primary metric. What matters is defining the analysis criteria before launching the test.

Should you test only one thing at a time?

When the goal is to identify a cause-and-effect relationship, isolating the variables makes interpretation much easier. Testing too many changes at once can make the results hard to attribute.

What is the best place to start?

With the funnel. First identify the step that represents your biggest improvement potential, then try to understand why users drop off there.

Newsletter

Once a week.

Last issue

We opened 10 apps to see how they sell a subscription.

+534 people read us each week