The trap: picking 'subscription conversions.' Conversions will almost certainly go up with a tighter paywall. But total revenue might drop if the reduced-readership side (fewer pageviews, lower ad impressions, lower brand exposure) offsets gains.

The right metric: revenue per visitor (ARPU) over a 90-day window, holding all visitors constant.

Why:

Captures both subscription revenue (growing) and ad revenue (potentially shrinking with tighter paywall).
Normalized per visitor, so it's not contaminated by acquisition changes.
Over 90 days, smooths out novelty effects.

What else to monitor (guardrails):

Total site visits. A paywall that suppresses visits hurts brand, long-term acquisition, and Google rankings.
Share of referrals. Fewer articles read → fewer social shares → less organic growth.
Churn of existing subscribers. A change that annoys current subs isn't worth the new conversions.
Email signups (soft funnel). Users hitting the paywall who sign up for a newsletter are still in the funnel even if they don't pay.

The structure of the analysis:

At day 90:

Revenue per visitor: total_revenue / unique_visitors — up or down?
Decomposition: (subscriber_revenue + ad_revenue + other) / unique_visitors. Which piece moved?
Segment by visitor type: casual readers (most affected), loyal readers (least).

A likely outcome:

"Revenue per visitor is up 8% at day 90. Subscription conversions are up 40% (tighter paywall pushes more users to subscribe). Ad revenue per visitor down 12% (fewer casual visits). Net positive, but the ad-side decline needs monitoring — if it accelerates, we'll need to tune the paywall up or look at the mix of content behind it."

What senior candidates add:

Consider segment-level analysis: some visitor segments (local readers, students) respond very differently. One blanket paywall rule vs. dynamic segmentation.
Reference NYT and FT methodology — both have publicly discussed paywall strategy. NYT famously loosened their paywall during COVID because acquisition value of free reading exceeded the short-term subscription gain.

A news site is launching a paywall. What metric do you track?

Draft your answer

Stuck? Peek at a hint

More case studies questions