Thursday, May 5, 2016

Thoughts on Take Home Interviews

There is a movement now in tech to really think about what it would take to improve our interview process. This is a movement a long time coming. White board coding interviews are clearly a strange way to measure a person's ability to actually do the day to day work of a modern software engineer. And we know that we tend to have a lot of bias in our interview processes that takes what we wish were an objective evaluation of skills and turns it into something very, very subjective.

Recently, my friend Julia Grace wrote about the interview process at Slack, to grant more transparency into what it takes to become an engineer at one of the hottest companies around. While this is a recruiting tactic it's also great that they are helping people understand what to expect and how to apply. Slack is taking pains to try to avoid bias, by having people complete a take-home technical exercise.
This varies by position, but generally you’ll have a week to complete a technical exercise and submit the code and working solution back to us.
Since we don’t do any whiteboard coding during the onsite interview, the technical exercise is one of the best ways we’ve found to evaluate programming competency.
The exercise is graded against a rigorous set of over 30 predetermined criteria. We’re looking for code that is clean, readable, performant, and maintainable. We put significant effort into developing these criteria to ensure that, regardless of who grades the exercise, the score is an accurate reflection of the quality of the work. We do this to limit bias. If there are clear criteria, variations that might impact score but have nothing to do with the candidate (such as if the grader is having a good day) are less likely to influence the outcome.
On twitter, a discussion ensued about whether asking people to spend time at home doing exercises didn't itself cause bias, against those who did not have a lot of spare time to be doing take-home exercises. Julia mentioned that they expect it to take 2-4 hours, but admitted that some people got really into the project and spent far longer than that.

This brings up three good questions that I want to address:

1) Are take-home exercises on the balance good?
2) Is it reasonable to expect people to spend 2-4 hours of their own time on a take-home exercise?
3) What about the people who will spend more time?


On the issue of 1, I think that yes, take-home exercises can help a lot to address the bias that happens when you know the person who wrote the code. They have the potential to be the blind audition step that the tech industry needs.

On the issue of 2, I actually ALSO think it is ok to ask candidates to spend a few hours on these exercises. This comes with two caveats. First, you should genuinely believe that the exercise can be done in the number of hours you expect by a candidate qualified for that level (so, measure how long candidates report spending on it). Second, these exercises only take "a few hours," not "tens of hours."

I feel ok, within bounds, to say that you should find a few hours to do a coding assignment, especially if it reduces the time you would need to spend onsite. The benefits of getting rid of whiteboard coding and that particular bit of evaluation bias outweigh the possible inconvenience. If you're looking for a job, you have to budget time to interview. So take half a day off if you can to do this project. Just because the exercise says take home doesn't mean you actually have to do it in your off hours. If you're unable to take time off of work to interview at all, unfortunately, you're going to have a problem getting a new job anywhere.

Which brings us to issue 3, what about the people who will spend more time? This is the most interesting part.

Julia said that candidates get really into solving the problem, and spend more time on it because they're excited about what they're building. That is awesome, but it also makes my blood run cold. At this point you're talking about something that is both a test of your programming abilities but also a creative project. Let's contrast that to the description that Foursquare gives of their take-home portion:

Instead we give out a take-home exercise that takes about three hours. The exercise consists of three questions:
1. A single-function coding question
2. A slightly more complicated coding question that involves creating a data structure
3. A short design doc (less than a page) on how to implement a specific service and its endpoints.
Every question we use is based on a real problem we’ve had to solve and has a preamble explaining the reason we need to solve this problem. If there is an obvious solution with a poor running time we mention it since we can’t help course-correct when the work isn’t being done live. We also provide scaffolding for the coding questions to save the candidate time.
This appears to be designed as an exercise that will only require a lot more time if you are struggling with the solution. Sure you can go nuts creating a crazy creative data structure or design doc, but this is a pretty clinical test. There's few places to add bells and whistles and they're unlikely to get you any brownie points with the interviewers.

These are two processes with the same goal, to reduce bias in our interviewing process, but slightly different tactics in the take-home. I would guess that Slack's take-home is fun and appealing to a set of people, those who enjoy tinkering or creating cool new projects. And it will find those people and make them shine, and probably serve as a good bit of recruiting to make them even more excited about the possibility of working there.

But I can tell you that for someone like me, I would hate being given the "creative" take-home coding problem. I'm happy to write some code to show you that I can. But I don't like to tinker, and I prefer that my creative work be collaborative. It feels like you are wasting my time and instead of making me more excited, it makes me far less excited.

The creative take-home also seems likely to select for those with free time, because if it is really an exercise that some people want to overdo, they will overdo it and you will have a hard time not rewarding that enthusiasm (why shouldn't you!). And while it's ok to ask for a few hours, building something that rewards those who can spend far longer is likely to bias against those who have, say, kids to take care of after work and on weekends, or other activities that limit their free time.

On the other hand, Foursquare's test is the first take-home I ever read about where I thought, "yes, that I would do." It is respectful of my time, and gives me something constrained to complete. We can elaborate on creative topics in the in-person interview, I do much of my best creative thinking with other people, elaborating on ideas together. Of course, I am a strong in-person communicator, and having a process that pushes the creativity into the in-person interaction interviews selects for people like me.

In the end, you probably want to hire a spectrum of people. When thinking about how to design your take-home tests, make sure that you are being considerate of the candidate's time, and decide if you really need this test to be a test of both technical skills and creativity, or merely a screening for technical skills. It may be that different roles need different screenings, and you may even want to offer both! That's a lot to ask for, but no one ever said that fixing hiring was going to be easy.

6 comments:

  1. Hey Camille, stimulating post, thank you.

    I wonder if my teams have focused too much on testing coding skills in the interviewing process. If we look, albeit with the benefit of retrospect, why most startups fail it's not because they hired devs with inadequate coding skills, it's because cross-functional teams for some reason built the wrong thing.

    Maybe attributes like communication skills, the ability to collaborate effectively, to empathize with customers, and the ability and willingness to be civil to one's colleagues, are even more important, can we come up with ways to measure? E.g. situational interviewing?

    @CartyBoston

    ReplyDelete
  2. This is super-fascinating to me, especially since I just completed a Slack take-home and did indeed end up spending WAY more than the allocated time on it, partly because it ended up being a lot more fun than I had intended. But I was coming from a very different place - I haven't worked as a developer for over 10 years and have been trying to get back into it. So there was no way I was going to be able to get it done in 2-4 hours because of re-ramping time. So getting back into the actual fun of it was part of what drove me to spend more time (once I got over the frustrating bits). But I'm definitely at an advantage in that I'm not currently working, so I HAD that time.

    That said, the worst part of it was, as you say, the isolation. "But I don't like to tinker, and I prefer that my creative work be collaborative." I would never spend the time I did on a project like that in my Real Life, because when I was bashing my head against some specific obstacle, I would go talk to another developer about it! But in an interview/application process, that felt like it would be cheating. I also strongly prefer collaboration over tinkering/optimizing my OWN stuff and if I hadn't been working out some 'rustiness' and had the time available, I would have probably just walked away from it.

    The description of the Foursquare one appeals to me as well! I'm so excited to see that companies are pushing back against the what-is-now-somehow-suprisingly-called "standard"* type of interview and working on making hiring more effective!


    *(The puzzle/whiteboard type of interview wasn't standard when I started in the industry, and in companies that aren't the Big Ones (Google, Amazon, Microsoft, etc) it's still not common. I'm often surprised when I talk to younger developers who have never heard of interviews that aren't done that way :)

    ReplyDelete
  3. Here is the thing. if as an employer, I am aiming for the very strongest candidates who will deliver the strongest results, then I am fine biasing for those who choose to devote more freetime to an activity like this.... We pay our folks (a fixed amount) for this step in our process, but basically we are biasing for those who choose to go the extra mile. We have then committing to github, so we can see who spent more time... but generally you are scored for your total result. if you dotted your eyes and crossed your tees, then it probably helped you.

    ReplyDelete
    Replies
    1. In short... I am fine missing some who would be a great employee, if in trade I rarely make a hire that is not awesome. ... maybe not the answer you wanted to hear, but I thought I would add a shout from the other side of the interviewing table.

      Delete
    2. It's great that you're paying people. If you don't care about bias in your process, fine. As a person who has also been an employer, I do care about bias in these processes, which is why I wrote about it. It is funny that you think that showing empathy for the interviewees means that I am unfamiliar with the point of view of the interviewer, though.

      Delete
  4. The critiquing qualitative research is important method for the data but it should be carefully done. They will respect your requirements and perform their critique in the format that you require.

    ReplyDelete

Note: Only a member of this blog may post a comment.