Is it possible to have too much of a good thing? Too much call to action? You decide!
During our weekly company updates, we read out testimonials from students, teachers, and parents, and I’m reminded of our work’s impact. Even so, there’s a distance between them and me (literally!).
That distance shrunk to 0 at the Grace Hopper Celebration career fair a couple weeks ago, where Pamela, Kayla, May-Li, and I represented KA. We met over 800 of the 8000 GHC attendees over the course of three days, and so many of them came up simply to say:
I love what you’re doing. Keep it up!
I didn’t understand my professor, and then I found Khan Academy.
You helped me pass Calculus
I’m not looking for a job yet, I just wanted to come by and say thank you.
Coming up on four years at KA, I’m somehow still surprised when I meet someone in real life who learned something with KA. It’s nice!
I love working at KA because we can test our assumptions at scale.
For one small example, we recently experimented with the layout of math subjects in the menu.
Here’s one version we tried, where the grade levels appear in a row at the bottom:
Here’s another alternative, where the grade levels appear in a column alongside the other subjects:
Which one is better?
They’re so similar, perhaps, that you have no opinion! Maybe you like the first because of the row’s aesthetic. Or, you might like the second because it emphasizes grade levels. (“I’m in 8th grade, so I want to look at 8th grade stuff!” vs “I’m interested in all of Geometry, give me all of it!”)
If we were a static textbook, we’d make a decision without knowing whether it was the right one. Maybe some feedback would come from an opinionated reader, but more likely not. Thankfully, we are not in the business of making textbooks, and we inform our decisions with data.
With 1 – 3 % gains in return visits and completing different types of content, the second alternative won. Yay!
Every time new devs join the team, they get a mentor, they get some onboarding docs, they ramp up, and they become wonderfully productive people.
Considering that we’ve had ~50+ (?) people flow through some sort of onboarding (counting our ~25 person dev team and 3 classes of summer interns), we might think we’re done with “working on onboarding”. Our onboarding is reasonable, with docs about the dev environment, code review, unit testing, style guides, what to expect on your first day / week / month…
…but we don’t update them as much as we should. There are pages that mention workflows that have completely changed, eg post-commit code reviews with mercurial/kiln when we’ve long switched to pre-commit code reviews with git/phabricator. I looked around today, and some pages literally make no sense given our “new way of doing things” – but how would a new dev know that? (Thankfully they have mentors to guide the way, but we shouldn’t have confused them in the first place!)
Our docs have a bunch of potholes that our brave new devs encounter, and though each blip might take just a little bit of time, they can certainly add up to a whole afternoon (or gasp! a day!) of slogging through boring mind-numbing things that literally tens of other people have already done. [very dramatic]
Instead of each person tripping over the same obstacle, why not clear it? Give them the onboarding that will help them thrive, not onboarding that they will thrive in spite of.
We’re about to absorb ~15 interns for the summer, so there’s no time like now to update our docs! Both for all the interns who will undoubtedly ask certain questions, and for their mentors who will be there to answer them. Summer 2014, here we come!
We recently made a bunch of perf fixes to reduce homepage load times (by 30% in some cases!).
Sounds too simple to be true, but we reduced time on both server and client by doing less stuff:
1) Remove unnecessary RPCs
2) Load less css / js
After these fixes were deployed, we noticed a small trickle of errors on homepage load:
TypeError: '_BaseValue' object is not subscriptable
The errors seemed non deterministic, and they seemed to happen more often when hitting non default versions of our app. After some sleuthing, we figured out what was going on, and this inspired the following quiz.
Happy half-birthday to you, little dashboard! Since then, the core idea has remained the same (a good sign!), which lets us iterate and refine (also good!).
Can you spot the differences?
Originally, we launched the dashboard with one mission: “The World of Math.” In December, we introduced more missions, and you can see that I’m focusing on 6th grade right now.
Here are the missions we have so far:
Since launch, we’ve also refined the dashboard’s appearance. We grouped logical components together – like the mastery challenge with the user’s list of upcoming practice tasks. We organized the mission progress by topic, so users can better see and understand the context of their progress. We’ve iterated on color and layout, and all of this combines to make the dashboard feel cleaner and better.
As you explore the dashboard, you might notice a few new, possibly experimental, features. You might see a video in the dashboard (which typically only had exercises). You can decline tasks that you’re not interested in. When you master a mission, you’re greeted with a celebration of your achievements.
And that’s not all! There are experiments for how you’re first introduced to the dashboard, and under the hood, we’re tweaking how we suggest exercises to users (which exercises to suggest and when to suggest them). The list of iterations we’ve made and will make go on and on.
With most (if not all) of these changes, we listen to our users’ reactions. Quantitatively, we measure the impact on their retention and engagement through a/b testing. Do they come back more often? Do they practice more? Are they getting more problems correct/incorrect? Qualitatively, we listen to all the different ways they can communicate their delight or distaste. Is there a huge uptick in bug reports because they hate the new layout?
Can’t wait to see how the dashboard grows in the next six months – by then it’ll be a toddler.
In March, we launched a redesign of Khan Academy where we gave the world the gift of color! 
As you meander about Khan Academy, you’ll see that each domain’s color scheme permeates all the way through. For example, “Discoveries and projects“ falls under science, so it’s pink.
Instead of copying and pasting domains + colors everywhere, we used LESS variables and mixins.
More specifically, we did something like this gist 
As a fun demo, open some science page and then do this in the console:
and you’ll see the theme switch just like that!
 – Where Jason and Tabitha both blogged about it
 – I should figure out how to embed gists some time. Or move off of wordpress.
approach one: make a custom property that converts between ndb and db keys.
this is OK, but db validation will complain when you try to build queries that filter by ndb key. this approach does work in the bare-bones case where you have a db entity handy, and then get the referenced ndb entities via the custom property. (thank goodness for ben ko for asking me to test that!)
approach two: store the urlsafe version of the ndb entity’s key as a db.StringProperty.
this is OK, but annoying in that you have to remember to convert from string to ndb.Key and then get the entity. and do the reverse when you assign.
approach three: rewrite your entire codebase to use ndb.
this is OK, but only if your codebase is two lines. oh ndb.KeyProperty!
is there an approach four that is not a compromise at all?
these were the things i looked for when i was naively job-searching 2.5 years ago:
- mission oriented
- i wanted to work on something that “mattered”, something that was directly doing good. i always had an idealistic streak, and i think it’s crazy that so many of us (particularly in tech and in the bay area) live in such fortune without thinking beyond our own existence. <insert caveat about how many products do tons of good unintentionally or indirectly>
- team size of < 15
- i wanted to be part of a team full of cohesion and trust, where the only focus was building the product. big teams can fulfill these criteria, but it’s all too easy to get sidetracked with politics, unnecessary process, and asking for permission. small teams cultivate seamless cooperation more easily, plus there’s more ownership per person!
- young product
- just like people, products are more malleable when they’re young. i wanted to explore (with aforementioned mission-oriented small team) all the possibilities of growing the product. i wanted to see the product take its first baby steps, to make its first stumbles, and even to go through its angsty teenage phase.
- i wanted to help the world out, and selfishly, i wanted to help myself out too. i wanted to become a better software engineer, better person, better _________. it was important that i would be able to learn from my team, and hopefully that would go both ways.
and the rest is history! suffice to say, i’m lucky + glad that i matched ka’s criteria too.
p.s. maybe you’ll find what you’re looking for at khanacademy.org/careers ?
p.p.s. i might have just scooped part of a podcast that will be released. in some form. by jason and me. shhhh!
learning is fun
along the way
as we ‘grow up’
learning becomes a chore.
at ka, we think learning is
and we try to
restore that original
and sometimes we might be silly
on a whim
to remind you
not to take anything
p.s. thanks to alpert
for his .gif factory.
p.p.s. sorry to ie8 users.
your browser has