Measurable does not mean important On using data to drive design decisions

Using data to design sounds objective. I'm trying to prove otherwise in this article.

Metrics != intent

You can look at metrics and build your own story around them. Say on average people drop off after a few minutes on a page, you can't sensibly measure for everyone what was the cause unless you're talking to users or viewing each session independently. Drop offs are interpreted negatively when it could just mean that someone has found what they needed, got distracted, or yes, they were frustrated and quit.

If you look at it objectively, the thing you actually know is that they dropped off, but the best you can do is guess intent. We keep asking numbers to tell us things they inherently are unable to.

Winning the wrong game

Having a red CTA will probably perform successfully when it comes to click through rate metrics. It still wouldn't make for good design cause it doesn't fit into a page's hierarchy. If that leads you to decide against redesigning that CTA to fit the hierarchy because doing so would probably "tank metrics", then what you ended up measuring here is your ability to attract attention.

When that becomes a habit and you start designing around metrics, you're essentially just becoming better at producing the behavior you chose to measure rather than actually improving the product. There's always going to be another way to design that number to go up by making things bigger, moving it higher, adding a notification, interrupting the user. But prominent design isn't necessarily good design.

AB testing

Say you want to AB test an agent drawer on the left vs the right side of an interface. If the left side gets more usage, that still does not mean left = better design. The more people use something could mean it was helpful, or, it could mean the exact opposite, that it wasn't helpful and it required more interaction. On the other hand, someone whose intent is to use an agent will find it regardless as long as it's not purposefully hidden. It really can be that simple and it doesn't depend on if the agent is on the left or right. I think we generally ask bigger questions than what AB tests can solve.

Letting data decide on design questions also comes at the big cost of taking away the product's stability. You have to trust your judgment to give those answers. (assuming you've done the design reps)

Courage and vision

I really do think the biggest danger of data driven design is that it compounds. An AB test is going to tell you to make something more prominent, a different data point tells you to use the wrong pattern cause it performs better, and another tells you to move a page into the wrong navigation. You'll see that each metric is doing better in isolation but the product as a whole becomes louder and less predictable.

Data is not going to be able to tell you what makes an experience good. You have to have the courage to make them using your own intuition and judgment.

Designing something worth using

I don't design to control what people do. I want to inform people and design intuitive experiences so that they can make decisions for themselves. I'm not against influencing behavior to an extent, I'm against designing around the extraction of a specific behavior. Especially when behaviors are so difficult to interpret accurately in the first place. I'd rather spend that energy building genuine features that people choose to use.