See the impact of your work, in person!

24th October 2014

During our weekly company updates, we read out testimonials from students, teachers, and parents, and I’m reminded of our work’s impact. Even so, there’s a distance between them and me (literally!).

That distance shrunk to 0 at the Grace Hopper Celebration career fair a couple weeks ago, where Pamela, Kayla, May-Li, and I represented KA. We met over 800 of the 8000 GHC attendees over the course of three days, and so many of them came up simply to say:

I love what you’re doing. Keep it up!

I didn’t understand my professor, and then I found Khan Academy.

You helped me pass Calculus
Physics
Chemistry
Econ

I’m not looking for a job yet, I just wanted to come by and say thank you.

Coming up on four years at KA, I’m somehow still surprised when I meet someone in real life who learned something with KA. It’s nice!

Show and tell: testing assumptions

6th August 2014

I love working at KA because we can test our assumptions at scale.

For one small example, we recently experimented with the layout of math subjects in the menu.

Here’s one version we tried, where the grade levels appear in a row at the bottom:

control

Here’s another alternative, where the grade levels appear in a column alongside the other subjects:

column

 

Which one is better?

They’re so similar, perhaps, that you have no opinion! Maybe you like the first because of the row’s aesthetic. Or, you might like the second because it emphasizes grade levels. (“I’m in 8th grade, so I want to look at 8th grade stuff!” vs “I’m interested in all of Geometry, give me all of it!”)

If we were a static textbook, we’d make a decision without knowing whether it was the right one. Maybe some feedback would come from an opinionated reader, but more likely not. Thankfully, we are not in the business of making textbooks, and we inform our decisions with data.

With 1 – 3 % gains in return visits and completing different types of content, the second alternative won. Yay!

Invest in onboarding (continuously)

6th May 2014

Every time new devs join the team, they get a mentor, they get some onboarding docs, they ramp up, and they become wonderfully productive people. 

Considering that we’ve had ~50+ (?) people flow through some sort of onboarding (counting our ~25 person dev team and 3 classes of summer interns), we might think we’re done with “working on onboarding”. Our onboarding is reasonable, with docs about the dev environment, code review, unit testing, style guides, what to expect on your first day / week / month…

…but we don’t update them as much as we should. There are pages that mention workflows that have completely changed, eg post-commit code reviews with mercurial/kiln when we’ve long switched to pre-commit code reviews with git/phabricator. I looked around today, and some pages literally make no sense given our “new way of doing things” – but how would a new dev know that? (Thankfully they have mentors to guide the way, but we shouldn’t have confused them in the first place!)

Our docs have a bunch of potholes that our brave new devs encounter, and though each blip might take just a little bit of time, they can certainly add up to a whole afternoon (or gasp! a day!) of slogging through boring mind-numbing things that literally tens of other people have already done. [very dramatic]

Instead of each person tripping over the same obstacle, why not clear it? Give them the onboarding that will help them thrive, not onboarding that they will thrive in spite of.

We’re about to absorb ~15 interns for the summer, so there’s no time like now to update our docs! Both for all the interns who will undoubtedly ask certain questions, and for their mentors who will be there to answer them. Summer 2014, here we come!

non deterministic bug or ndb?

17th March 2014

We recently made a bunch of perf fixes to reduce homepage load times (by 30% in some cases!).

Sounds too simple to be true, but we reduced time on both server and client by doing less stuff:

1) Remove unnecessary RPCs

2) Load less css / js

After these fixes were deployed, we noticed a small trickle of errors on homepage load:

TypeError: '_BaseValue' object is not subscriptable

The errors seemed non deterministic, and they seemed to happen more often when hitting non default versions of our app. After some sleuthing, we figured out what was going on, and this inspired the following quiz.

non-deterministic-bug-or-ndb

Enjoy!

Dashboard: six months later

20th February 2014

Happy half-birthday to you, little dashboard! Since then, the core idea has remained the same (a good sign!), which lets us iterate and refine (also good!).

Six months ago: original-dashboard

Today:small-blue

Can you spot the differences?

Originally, we launched the dashboard with one mission: “The World of Math.” In December, we introduced more missions, and you can see that I’m focusing on 6th grade right now.

Here are the missions we have so far:

mission-picker

Since launch, we’ve also refined the dashboard’s appearance. We grouped logical components together – like the mastery challenge with the user’s list of upcoming practice tasks. We organized the mission progress by topic, so users can better see and understand the context of their progress. We’ve iterated on color and layout, and all of this combines to make the dashboard feel cleaner and better.

As you explore the dashboard, you might notice a few new, possibly experimental, features. You might see a video in the dashboard (which typically only had exercises). You can decline tasks that you’re not interested in. When you master a mission, you’re greeted with a celebration of your achievements.

avatar-party

And that’s not all! There are experiments for how you’re first introduced to the dashboard, and under the hood, we’re tweaking how we suggest exercises to users (which exercises to suggest and when to suggest them). The list of iterations we’ve made and will make go on and on.

With most (if not all) of these changes, we listen to our users’ reactions. Quantitatively, we measure the impact on their retention and engagement through a/b testing. Do they come back more often? Do they practice more? Are they getting more problems correct/incorrect? Qualitatively, we listen to all the different ways they can communicate their delight or distaste. Is there a huge uptick in bug reports because they hate the new layout?

Can’t wait to see how the dashboard grows in the next six months – by then it’ll be a toddler.

Implementing color palettes with LESS mixins

16th October 2013

In March, we launched a redesign of Khan Academy where we gave the world the gift of color! [1]

Screen Shot 2013-10-16 at 7.28.53 PM Screen Shot 2013-10-16 at 7.29.03 PM Screen Shot 2013-10-16 at 7.29.08 PM Screen Shot 2013-10-16 at 7.29.15 PM

As you meander about Khan Academy, you’ll see that each domain’s color scheme permeates all the way through. For example, “Discoveries and projects“ falls under science, so it’s pink.

science

Instead of copying and pasting domains + colors everywhere, we used LESS variables and mixins.

More specifically, we did something like this gist [2]

As a fun demo, open some science page and then do this in the console:

$(“.science”).addClass(“math”).removeClass(“science”)

and you’ll see the theme switch just like that!

blue science

[1] – Where Jason and Tabitha both blogged about it

http://shipordie.com/post/45235932174/khanacademy-org-gets-a-makeover-bug-fixes-and-ux

http://tappitha.tumblr.com/post/46001133750/athena-behind-the-scenes

[2] – I should figure out how to embed gists some time. Or move off of wordpress.

tales of app engine: how to reference an ndb entity from a db entity

24th September 2013

approach one: make a custom property that converts between ndb and db keys.

https://gist.github.com/marcia/6695509

this is OK, but db validation will complain when you try to build queries that filter by  ndb key. this approach does work in the bare-bones case where you have a db entity handy, and then get the referenced ndb entities via the custom property. (thank goodness for ben ko for asking me to test that!)

approach two: store the urlsafe version of the ndb entity’s key as a db.StringProperty.

this is OK, but annoying in that you have to remember to convert from string to ndb.Key and then get the entity. and do the reverse when you assign.

approach three: rewrite your entire codebase to use ndb.

this is OK, but only if your codebase is two lines. oh ndb.KeyProperty!

is there an approach four that is not a compromise at all?

why i joined khan academy

18th September 2013

these were the things i looked for when i was naively job-searching 2.5 years ago:

  • mission oriented
    • i wanted to work on something that “mattered”, something that was directly doing good. i always had an idealistic streak, and i think it’s crazy that so many of us (particularly in tech and in the bay area) live in such fortune without thinking beyond our own existence. <insert caveat about how many products do tons of good unintentionally or indirectly>
  • team size of < 15
    • i wanted to be part of a team full of cohesion and trust, where the only focus was building the product. big teams can fulfill these criteria, but it’s all too easy to get sidetracked with politics, unnecessary process, and asking for permission. small teams cultivate seamless cooperation more easily, plus there’s more ownership per person!
  • young product
    • just like people, products are more malleable when they’re young. i wanted to explore (with aforementioned mission-oriented small team) all the possibilities of growing the product. i wanted to see the product take its first baby steps, to make its first stumbles, and even to go through its angsty teenage phase.
  • mentorship
    • i wanted to help the world out, and selfishly, i wanted to help myself out too. i wanted to become a better software engineer, better person, better _________. it was important that i would be able to learn from my team, and hopefully that would go both ways.

and the rest is history! suffice to say, i’m lucky + glad that i matched ka’s criteria too.

p.s. maybe you’ll find what you’re looking for at khanacademy.org/careers ?

p.p.s. i might have just scooped part of a podcast that will be released. in some form. by jason and me. shhhh!

show and tell: silly things

9th September 2013

learning is fun
but
somehow
along the way
as we ‘grow up’
learning becomes a chore.

at ka, we think learning is
AWESOME!
and we try to
restore that original
thrilling
feeling of
wonder.

and sometimes we might be silly
on a whim
to remind you
(and us)
not to take anything
too seriously.

wiggly

bouncy

peekaboo

p.s. thanks to alpert
for his .gif factory.

p.p.s. sorry to ie8 users.
your browser has
no capacity
for silliness.

ka: from cool library to cool tutor

22nd August 2013

just last week, we launched a brand new learning dashboard on khan academy, and it illustrates a cool shift in the role khan academy can play in a learner’s journey.

prior to the learning dashboard, khan academy was like a very cool library. with 4000+ videos and 400+ exercises organized in some fashion, the burden was on the user to find the right content to consume. the user must be self aware and know 1) what topic she wants to learn, and perhaps more challenging, 2) what topic she is ready to learn. we had a few mechanics that made the experience less-library-esque, but they weren’t baked into the core experience.

:: enter, the learning dashboard ::

the learning dashboard was designed and built with the “cool tutor” idea in mind. we try to understand what you understand, and then we suggest the exercises that will be the most helpful.  we aim to suggest things that are *just right*. not too hard as to be impossible, but not so easy that it’s pointless. the more you do these exercises, the more we learn about your learning. a virtuous cycle!

if you happen to have a coach on khan academy, she can step into that cool tutor role and recommend exercises to you too! and if you don’t like any of the suggestions, and you wish you could just get lost in the metaphorical bookshelves of khan academy… you can still do that too. :)

more improvements, soon!