I love working at KA because we can test our assumptions at scale.
For one small example, we recently experimented with the layout of math subjects in the menu.
Here’s one version we tried, where the grade levels appear in a row at the bottom:
Here’s another alternative, where the grade levels appear in a column alongside the other subjects:
Which one is better?
They’re so similar, perhaps, that you have no opinion! Maybe you like the first because of the row’s aesthetic. Or, you might like the second because it emphasizes grade levels. (“I’m in 8th grade, so I want to look at 8th grade stuff!” vs “I’m interested in all of Geometry, give me all of it!”)
If we were a static textbook, we’d make a decision without knowing whether it was the right one. Maybe some feedback would come from an opinionated reader, but more likely not. Thankfully, we are not in the business of making textbooks, and we inform our decisions with data.
With 1 – 3 % gains in return visits and completing different types of content, the second alternative won. Yay!