Buy My Book, "The Manager's Path," Available March 2017!

Thursday, May 5, 2016

Thoughts on Take Home Interviews

There is a movement now in tech to really think about what it would take to improve our interview process. This is a movement a long time coming. White board coding interviews are clearly a strange way to measure a person's ability to actually do the day to day work of a modern software engineer. And we know that we tend to have a lot of bias in our interview processes that takes what we wish were an objective evaluation of skills and turns it into something very, very subjective.

Recently, my friend Julia Grace wrote about the interview process at Slack, to grant more transparency into what it takes to become an engineer at one of the hottest companies around. While this is a recruiting tactic it's also great that they are helping people understand what to expect and how to apply. Slack is taking pains to try to avoid bias, by having people complete a take-home technical exercise.
This varies by position, but generally you’ll have a week to complete a technical exercise and submit the code and working solution back to us.
Since we don’t do any whiteboard coding during the onsite interview, the technical exercise is one of the best ways we’ve found to evaluate programming competency.
The exercise is graded against a rigorous set of over 30 predetermined criteria. We’re looking for code that is clean, readable, performant, and maintainable. We put significant effort into developing these criteria to ensure that, regardless of who grades the exercise, the score is an accurate reflection of the quality of the work. We do this to limit bias. If there are clear criteria, variations that might impact score but have nothing to do with the candidate (such as if the grader is having a good day) are less likely to influence the outcome.
On twitter, a discussion ensued about whether asking people to spend time at home doing exercises didn't itself cause bias, against those who did not have a lot of spare time to be doing take-home exercises. Julia mentioned that they expect it to take 2-4 hours, but admitted that some people got really into the project and spent far longer than that.

This brings up three good questions that I want to address:

1) Are take-home exercises on the balance good?
2) Is it reasonable to expect people to spend 2-4 hours of their own time on a take-home exercise?
3) What about the people who will spend more time?

On the issue of 1, I think that yes, take-home exercises can help a lot to address the bias that happens when you know the person who wrote the code. They have the potential to be the blind audition step that the tech industry needs.

On the issue of 2, I actually ALSO think it is ok to ask candidates to spend a few hours on these exercises. This comes with two caveats. First, you should genuinely believe that the exercise can be done in the number of hours you expect by a candidate qualified for that level (so, measure how long candidates report spending on it). Second, these exercises only take "a few hours," not "tens of hours."

I feel ok, within bounds, to say that you should find a few hours to do a coding assignment, especially if it reduces the time you would need to spend onsite. The benefits of getting rid of whiteboard coding and that particular bit of evaluation bias outweigh the possible inconvenience. If you're looking for a job, you have to budget time to interview. So take half a day off if you can to do this project. Just because the exercise says take home doesn't mean you actually have to do it in your off hours. If you're unable to take time off of work to interview at all, unfortunately, you're going to have a problem getting a new job anywhere.

Which brings us to issue 3, what about the people who will spend more time? This is the most interesting part.

Julia said that candidates get really into solving the problem, and spend more time on it because they're excited about what they're building. That is awesome, but it also makes my blood run cold. At this point you're talking about something that is both a test of your programming abilities but also a creative project. Let's contrast that to the description that Foursquare gives of their take-home portion:

Instead we give out a take-home exercise that takes about three hours. The exercise consists of three questions:
1. A single-function coding question
2. A slightly more complicated coding question that involves creating a data structure
3. A short design doc (less than a page) on how to implement a specific service and its endpoints.
Every question we use is based on a real problem we’ve had to solve and has a preamble explaining the reason we need to solve this problem. If there is an obvious solution with a poor running time we mention it since we can’t help course-correct when the work isn’t being done live. We also provide scaffolding for the coding questions to save the candidate time.
This appears to be designed as an exercise that will only require a lot more time if you are struggling with the solution. Sure you can go nuts creating a crazy creative data structure or design doc, but this is a pretty clinical test. There's few places to add bells and whistles and they're unlikely to get you any brownie points with the interviewers.

These are two processes with the same goal, to reduce bias in our interviewing process, but slightly different tactics in the take-home. I would guess that Slack's take-home is fun and appealing to a set of people, those who enjoy tinkering or creating cool new projects. And it will find those people and make them shine, and probably serve as a good bit of recruiting to make them even more excited about the possibility of working there.

But I can tell you that for someone like me, I would hate being given the "creative" take-home coding problem. I'm happy to write some code to show you that I can. But I don't like to tinker, and I prefer that my creative work be collaborative. It feels like you are wasting my time and instead of making me more excited, it makes me far less excited.

The creative take-home also seems likely to select for those with free time, because if it is really an exercise that some people want to overdo, they will overdo it and you will have a hard time not rewarding that enthusiasm (why shouldn't you!). And while it's ok to ask for a few hours, building something that rewards those who can spend far longer is likely to bias against those who have, say, kids to take care of after work and on weekends, or other activities that limit their free time.

On the other hand, Foursquare's test is the first take-home I ever read about where I thought, "yes, that I would do." It is respectful of my time, and gives me something constrained to complete. We can elaborate on creative topics in the in-person interview, I do much of my best creative thinking with other people, elaborating on ideas together. Of course, I am a strong in-person communicator, and having a process that pushes the creativity into the in-person interaction interviews selects for people like me.

In the end, you probably want to hire a spectrum of people. When thinking about how to design your take-home tests, make sure that you are being considerate of the candidate's time, and decide if you really need this test to be a test of both technical skills and creativity, or merely a screening for technical skills. It may be that different roles need different screenings, and you may even want to offer both! That's a lot to ask for, but no one ever said that fixing hiring was going to be easy.