Lots of e-commerce managers have had the experience of putting an extra ‘sign up for this’ or ‘buy this’ prompt in their checkout flow and seen HUGE lift.

You’re a web user in 2019, I don’t have to tell you how many SAAS and e-commerce sites have a nag popup on their landing page.

Nagging works, but it has diminishing returns. The first time you annoy somebody, they give you what you ask for. The third time, they put you in time-out (if you’re nagging a parent) or close the tab (if you’re talking about users of websites).

Nobody is measuring the long-term effect on users of constant nagging, and as a consequence the internet is in a death spiral of shitty and annoying callouts.

  1. Everyone measures the high, nobody measures the hangover
  2. Goodhart’s law
  3. Metrics-driven management = short-termism
  4. What to try

Everyone measures the high, nobody measures the hangover

As a typical coffee-drinker, I’ve wondered if coffee only perks me up because it lessens my caffeine withdrawal, and I’d be better off quitting. But your first ever cup of coffee is awesome.

So with nag popups or adding a button to a clean UI. Nobody measures how UX / conversion flows perform with 0 pieces of garbage vs 5 – they just measure the marginal impact of each new widget.

I’m not saying that every popup or nag feature has a backlash, but it’s critical to think about long-term feedback effects, try to measure them, and prefer product changes that don’t punish your users or sacrifice their convenience.

Goodhart’s law

Is an economics thing which says (paraphrased) that when you make a metric a KPI, if that metric is a proxy for the health of your business rather than a direct measurement, you’re fooling yourself.

Even cash can be a proxy. Quarterly profit, for example, can be optimized by selling all your factory equipment at the end of the quarter.

Volume is another proxy that can deceive. Comedy writer Robert Asprin trained me to love a good ‘make it up on volume’ joke. Moviepass’s great gift to the world is that investors can now ask themselves ‘is this moviepass’ when looking at something with awesome growth and oh-so-slight marginal losses.

Metrics-driven management = short-termism

Modern major-generals at successful shops like google want to have data underneath every button.

This has always felt to me like tech’s passive-aggressive way of seizing power from other talent silos. It’s not a complete argument. Someone is still using judgment to decide which questions to ask, and like most forms of optimization this is local; it doesn’t support the big decisions that create real change.

Here’s what’s more concerning – the duration of these experiments is pretty short. Nobody is doing the experiment ‘let’s value a user’s time and not fuck with them for like two years and see if they’re more loyal than that poor meat popsicle whom we tease and torment’.

You can improve some numbers by 10 percent while over the long term creating an unusable experience. If you deal with snowballs by driving them up the hill and leaving them there, you’ll have an avalanche.

If something feels horrible to insiders the odds are it’s horrible for users as well. (Unless your product is specific and atrocious dogfood – like the teletubbies or non-prime mortgages).

What to try

  • Develop some metrics that you won’t try to influence, i.e. non-target metrics. Watch for when they diverge from the yes-target metrics where you’re placing pressure
  • If you’re doing a lot of sequential short-term experiments of similar types, look for diminishing returns on older cohorts
  • Experiment with not experimenting on a slice of each user cohort (for their whole life as users)