Decide the rollback before the test
Instagram is walking back its full-screen feed after a week of public anger and weak usage numbers. An experiment can only tell you what it measured, so the rollback line should be written before it starts.
Yesterday Instagram said it would walk back its changes. The full-screen, video-first feed it had been testing will be phased out over the next week or two, and the share of recommended posts in people’s feeds will drop for now. Earlier in the week, Kylie Jenner and Kim Kardashian had both shared a “Make Instagram Instagram again” post.
Adam Mosseri gave two reasons, in this order: “people are frustrated and the usage data isn’t great.” He also said, “I’m glad we took a risk.” Casey Newton reads it as a pause in a longer move toward video, one that started well before this test.
Two kinds of evidence
What interests me is that Instagram had two kinds of evidence, and this time they agreed. Loud users said they hated it. The numbers said they weren’t using it the way Instagram hoped.
They won’t always agree. Sometimes the loudest people hate a change that the numbers say is working, and sometimes everyone is polite about a change that’s quietly costing you. If a team hasn’t decided in advance which signal counts, it’ll end up picking whichever one supports what it already wanted to do.
An experiment can only tell you what it measured, so decide before it starts what would make you roll it back.
Write the guardrail down
At Uber, a new feature went out behind a flag, to a few people first, with an experiment measuring what it did. The part of that system I’d copy anywhere is the guardrail: a metric with a limit, written down before launch. If the change pushes the metric past the limit, the owner gets paged. Nobody has to win an argument in the middle of a bad week.
For a change like Instagram’s, I’d want a guardrail on something people came for, such as time spent with posts from people they follow, alongside the metric the change was meant to move. If the new feed grows video watching and shrinks time with friends past an agreed line, it rolls back, whatever anyone tweets.
The cost of being public
A test this visible is hard to run cleanly. When the people in it can post about it, the test becomes the news, and the news changes how people use the app. That’s a reason to keep big changes small at first, and a reason to have the rollback line written down. A team that has agreed on its limits in advance can roll back calmly instead of in public.