Buy My Book, "The Manager's Path," Available March 2017!

Thursday, January 28, 2016

Qualitative or Quantitative but always Analytical

I did a panel at Etsy's engineering leadership offsite today, which was amusing. One of the topics of the panel was:
Challenges of balancing data-light product bets vs purely data driven incremental improvements.

All three of the panelists agreed, although each of us phrased it a different way. The first panelist (Dan McKinley) spoke about the need, even for products that are not purely A/B test driven, to drill down on the goals and try to find something to measure about how a project will actually achieve a goal. If the project is part of a larger goal to increase revenue by 25% this year, what way is it contributing to that? How do we measure its success? Even when you are driving decisions by "vision" there is some quantitative goal you are trying to achieve, so state what it is.

The second panelist (Albert Wenger) spoke of the importance of balancing the quantitative with the qualitative. Some things cannot be purely quantitatively measured, and there are qualitative measures that are incredibly valuable to the process of discovering product market fit. User testing, user research, watching how people actually engage with a product are all essential to creating something great, beyond simply finding numbers to measure and trying to increase them.

My perspective on the issue is that qualitative methods are important, but qualitative is still analytical. You may not be able to use data-driven reasoning because you're starting something new, and there are no numbers. It is hard to do quantitative analysis without data, and new things only have secondary data about potential and markets, they do not have primary data about the actual user engagement with the unbuilt product that you can measure. Furthermore, even when the thing is released, you probably have nothing but "small" data for a while. If you only have a thousand people engaging with something, it is hard to do interesting and statistically significant A/B tests unless you change things drastically and cause massive behavioral changes.

So, instinct and guesses are necessary. But we needn't lose our analytical approach just because we don't have data. When you build something, you have a hypothesis about the person you are building it for. You have a guess as to what they will like, and most of the time you have a reason for that guess. When you're trying to build a business, you need a chain of events that you expect could happen that would indicate a product is successful. You have a sense of what to start measuring once the thing is released that will show whether it is working. Answering the questions of who is my customer, why would she use this, and what will signs of interest and engagement look like is essential to going from vague instinct to thoughtful first product.

Data can't make all our decisions for us because data isn't there to get us from 0->1. We have to use our powers of observation, of empathy with our customers, and of deduction, to create smart hypothesis in a qualitative way. Qualitative analysis will always have a role in product development, and customer empathy is likely to drive us more quickly to great products than simply relying on numbers alone.

Thanks to Etsy for hosting and my apologies to my fellow panelists if I misrepresented your views!

Monday, January 25, 2016

Hiring Engineering Managers: Screening for Potential

I have been in many discussions on the best way to interview and hire engineering managers. Here are my thoughts, having done this both successfully and less successfully, including some things that I have looked for in the past when being interviewed. 

First: Be thoughtful about what you are looking for. There are people out there who are looking for the opportunity to go into management. Lots of them, in my experience. These are good people to hire, because many of them will turn out to be good, thoughtful managers. Most of the time I hired this sort of person into a senior engineering role that quickly became a tech lead role. I was explicit at hiring that we didn't have a team to give them immediately but that I would be expecting that as the overall team grew they would move into such a role when it became available.

Now, I should be clear, this exchange requires a lot of trust on both parties, as well as the possibly unspoken assumption that they still have to prove themselves capable of doing the job once on the team. I have turned down jobs that asked me to make this bet, and I would not hold it against anyone who felt that it was too risky. But on the flip side, if you really want to grow your management from within, hiring people who both want to be eventually moving to leadership roles but are comfortable coming in as individual contributors first is pretty essential to making that work. If you want to grow management from within but don't bother to look for people who want to grow into management roles when you're hiring, you may end up with mostly people who just want to code, and that is not good for anyone.

OK, so you're hiring a specific management position. What do you look for?

There are two camps here. The first camp is that you look for incredibly smart tech lead-level developers who may have had limited management experience and put them in the role. They will easily pass whatever technical screens you put in front of them, but hiring is a risk because they may not really be good managers. If this is the way you want to go about hiring managers, you need to spend a good deal of time focused on screening for management potential. Screening for potential is different than screening for experience. Right now, I'm going to talk about screening for potential; in part 2, I'll talk about experience.

Questions to tease out potential: 

Tell me about a project where you have acted as the tech lead. What was your role like, as tech lead? What were things you did that were different from the rest of the team? How did you ensure that the project was successful?

What you are looking for: Someone who answers with more than just "I designed the architecture, chose the libraries, and wrote the most technically challenging pieces of the code." They should have taken an active role in the project management, even if there was another person explicitly or implicitly in that role. They should have contributed to predicting problems with the delivery and working with the people on the team or cross-team to ensure success.

When you bring a new team member onto your team, what kinds of things do you personally do as part of their onboarding? Have you ever been a mentor to a new hire or intern? What was that like, what did you learn from it?

What you are looking for: Someone who is actively engaged in the work of bringing on new people, and thoughtful about making that process better. Someone who respected the work of mentoring, who isn't just trying to shed human interactions quickly to get back to code.

Tell me about some things you have done to make the process of writing or delivering code in your organization better.

What you are looking for: Some strong examples of seeing process/people/systems problems and raising their hand to suggest improvements.

One suggestion I have heard, but not tried, is to do a role-playing exercise with the candidate. Set up a circumstance that might happen, such as, an employee is unhappy that they did not get promoted. Provide an overview to the candidate and give the interviewer some details that may include information that the manager doesn't necessarily know. Have a third party observe the interaction, and then at the end of the role play all three talk about what went well and didn't. This could be an interesting tool for hiring either the experienced manager or the potential-driven manager, but I also suspect it is easy on the flip side for it to go poorly if the interviewer isn't good at guiding the roleplay and knowing what to look for. (Credit to Marc Hedlund for this idea!)

Finally, when hiring this type of candidate, you are probably going to get (and maybe even should be intentionally seeking out) someone who is going to not only manage people but drive technical decision-making. Make sure that the people who currently drive technical direction (if they exist) are aligned on that front with the candidate. If the candidate is gung-ho about functional programming and microservices and you are happy in a more conservative technical space, you may end up with someone who wants to make big technical changes that you might not agree with. The team should feel a general alignment to their technical perspective, and ideally people are excited about working for this person because they feel that they will learn something, but don't just hire them as a manager because people are excited about their technical chops.

Overall, when screening for potential, look for signs of stepping up, caring about the people on the team, and thoughtfulness about the processes. In general you should also be looking for excellent communication skills, and everyone who interviews the person should feel pretty comfortable with the idea of working for them. Be wise to potential bias here. The best managers are often overlooked because they don't pattern match to certain characteristics, whether they are "tall and male" or "forceful personality" or whatever. Talk to the team before the interview and give them examples of great managers who sit outside of those stereotypes to reset their stereotype bias a little bit.

When you mishire on potential, it tends to come from overvaluing technical skills + pattern matching on "what a leader looks like", and the failure mode is managers who just don't deliver coherent teams and/or can't deliver projects effectively. Never ever hire a person you feel is overly ego-driven.

One final word on this candidate: If you hire someone for potential, be prepared to train them. They are going to need help. Whether it is formal training or a lot of mentorship from you or a mix of both, no one is born knowing how to manage. I think it is worth sometimes taking a risk on people like this (I'm glad someone did that for me once!) but they will struggle if they have to do it all alone.