Buy My Book, "The Manager's Path," Available March 2017!

Friday, November 27, 2015

The Manager as Debugger

I have observed that the best engineering managers I know are often also great debuggers. Why would this be? What is it about these two tasks that has such an overlapping skill set?

A great debugger is relentless in their pursuit of the “why” for a bug. This is simple when we are looking for errors in application logic, but we all know that bugs can go many layers deep, particularly in complex systems that involve many separate parts operating over time-delayed networks. A sign of a poor debugger is a person who, when they add a log statement to a piece of concurrent code to attempt to find an error and see that the error can’t be reproduced, assumes that they have therefore fixed the problem. It’s a lazy habit but a common one. Sometimes there are problems that seem impossible to determine, and many people don’t have the patience to dig through layers of code, theirs and others’, log files, system settings, and whatever else is needed to get to the bottom of something that only happened once. I can’t blame them. Obsessive debugging of one-off issues is not always a great use of your time, but it does show a mind that cannot be satisfied with the unknown, especially when that unknown might cause them to be paged at two o’clock in the morning.

What does this have to do with management? Managing teams is a series of complex, black boxes interacting with other complex, black boxes. These black boxes have inputs and outputs that can be observed, but when the outputs aren’t as expected, figuring out why requires trying to open up the black box and see what is going on inside. And, just as sometimes you don’t have the source code, or the source code is in a language you don’t understand, or the log files aren’t readable, the black boxes of teams can resist yielding their inner workings. 

Teams also share the characteristic of another famous box, the one containing Schrödinger’s Cat. The point of that experiment is to show that the act of observing changes the outcome, or rather, causes an outcome to happen. You can’t go into a team and not change the behavior of that team by being around them, sitting in their meetings, watching their standups. Your presence changes the team behavior and may hide the problem you are trying to find, in the same way that a log statement can cause a concurrency issue to be magically erased, at least for some time.

To debug a system properly you have to be able to have a reasonable hypothesis that explains how the system got into the failed state, preferably one that you can reproduce, so that you can fix the bug (or at least prevent it from happening in the future). To debug a team, you also want to attempt to get a hypothesis for why the team was having problems. You want to do this in as minimally-invasive way as possible, to prevent your meddling from obscuring the problems. You have the added challenge of team problems not generally being single failures but more like performance issues, the system is running but it seems to slow down from time to time, the machines are ok except occasionally they crash, people seem happy but attrition is too high. 

Let’s work through an example. We have a team that feels slow. You have heard complaints from their business partners and product manager that they are slow, and you agree that the team just seems to lack the same energy as your other teams. How do we figure this out?

Debugging this deserves the same rigor you would apply to debugging a serious systems issue. When I debug a systems issue, the first thing I look at is generally log files and any other record of system state from the time of the incident. When talking about a team that is not producing work fast enough, look at the records. Look at the team chats and emails, look at the tickets, look at the repository code reviews and checkins. What do you see? Are there production incidents happening that are taking too much time? Are a bunch of people sick? Are they bickering over coding style constantly in their code review comments? Are the tickets that are being written vague, too big, too small? Does the team seem upbeat in their communication style, sharing fun things as well as important work in chat, or are they purely business? Look at their calendars. Is the team spending many hours a week in meetings? Is their manager doing 1-1s? None of these things is necessarily a smoking gun, but they may point to an area to address.

Perhaps everything seems ok in all of these indicators, but the team still just is not performing as well as you believe they should. You know the talent is there, the team is happy, they’re not overburdened by production support. What is happening? Now is the time to start doing some potentially destructive investigations. Sit in their meetings. Are they boring to you? Is the team bored? Who is speaking most of the time? Are there regular meetings with the whole team where the vast majority of the time is spent listening to the manager or product lead talk? Boring meetings are a sign. They may be a sign of inefficient planning on the part of the organizers. There may be too many meetings happening for the information covered. They may be a sign that the team members don’t feel that they can actually help set the direction of the team, choose the work that will happen. Boring meetings are a sure sign of time wasted, if not bigger leadership problems.

Ask the team what their goals are, can they tell you? Do they understand why those are the goals? If they don’t understand the goals of their work, their leaders (manager, tech lead, product manager) are not doing a good job engaging the team in the purpose of the work. In almost every model of motivation, people need to feel an understanding and connection with the purpose of their work. Who are they building these systems for, what is the potential impact on the customer, the business, the team? Did they have any part in deciding these goals, and the projects that they’re doing to achieve them? If not, why not? When you see a team spending all of their time on engineering-sponsored projects and neglecting product/business projects, it’s likely that the team does not appreciate or understand the value of the product/business projects that they are supposed to be working on, and therefore they are lacking in motivation to tackle them.

Finally, you might take a look at the actual team dynamics. Do people like each other? Are they friendly? Do they collaborate on projects or is every person working on something independently? Is there banter in the chat room, in emails? Do they have a good working relationship with other adjacent departments, and with their product managers? These are little things but even very professional groups tend to have a degree of personal connection between the members. A bunch of people who never talk to each other and are always working on independent projects are not really working as a team. There would be nothing wrong with that, if the team were performing well, but given that they are not, this may be contributing to your problem.

Sometimes managers of managers choose to take such problems as something that the manager of the team just needs to fix. You measure the manager on the output of their team, after all, and it is their responsibility to fix it if things are not going well. This is true, but just as I sometimes jump in and help debug complex system outages even though I rarely write code, it is ok to jump in and help debug team issues as you see them, particularly when the manager in question is struggling. It can be an opportunity to teach the manager and help them grow. It can also reveal more foundational problems with the organization, such as a lack of senior business leadership that even the best managers can’t identify or resolve alone. 


The pursuit of why when it comes to organizational problems is the thing that gives you patterns to match on, and lessons to lead with. We get better at debugging by doing it often, and learn which areas tend to break first and which indicators provide the most value for understanding issues. We become better leaders by pushing ourselves and our management teams to really get to the bottom of organizational issues, searching for why so that we can more quickly resolve such issues in the future. Without the drive to understand why, we rely on charm and luck to see us through our management career, we hire and fire based on charm and luck, and we have a huge blindspot when it comes to truly learning from our mistakes.

Friday, November 6, 2015

Truth and Consequences of the Technical Track

About a year ago, Dan McKinley aka McFunley wrote a bit on the technical track. The piece has stuck in the back of my mind for quite a while, and continues to influence certain points I make when discussing career progressions of engineers. Let's get them out.

I think that Dan's take is cynical, but not entirely incorrect. We need managers. As companies grow, at some point, you need people whose job it is to organize other people. I think the idealistic goal of management is to help teams live up to their talent potential. The realistic experience of management is often an abstraction layer between upstream and downstream, to help flow information, each layer helping to make a finer-grained set of decisions and adjustments to keep the overall company moving in the right direction.

Let's talk first about why I personally went into management. I was successfully performing in one of those technical track positions, at a company where I felt pretty confident I would be able to continue to be promoted in the technical track positions up to the highest level if I put the work in and continued to perform well. I was working on interesting things, paid very well, and on paper, everything was perfect. And yet, I quit to move to a smaller company and go into a management position. Why?

I realized pretty clearly that the scope of my ambitions was greater than the scope of my ability to produce things independently. I could not write enough code alone to achieve the kind of impact that I wanted to achieve on the company I was working for. Even with a small team, I felt that my ability to make a difference would be limited. I would have to make extremely good choices as to what to guide the team to do, and my effectiveness would always be limited by what I now realize is actually "product market fit." If I made a great product that no one else in engineering wanted to use (I was working in infrastructure engineering at the time), I would not have much of an impact.

So, instead, I gave up some degree of hands-on technical time in order to have a broader ability to focus people on projects where I felt their impact would make the most difference on the company. Ultimately, that is a core part of any technical leadership job, not doing all of the work yourself, but helping to both see what needs to be done and focus people on getting it done in the right way.

Now, you can imagine a situation where you have an engineering manager who deals with all the "people" side of things, paired with a technical specialist who makes all the decisions about what needs to be done and how it should be done, as per their technical expertise. Why doesn't this situation exist? Well, imagine putting yourself into the shoes of the engineering manager. You are basically a glorified project manager with little decision-making power. Except... all of the people report to you. You have hiring and firing power. You write their reviews. Do you really have no decision-making power? If you disagree with the technical specialist about what needs to be done, why would the technical specialist have final say? It is unrealistic to believe that the person with management authority over the people would have no influence over what they do, even with best intentions assumed.

The technical specialist, with no management authority, is limited. They are limited in what they can produce by the scope of resources (people, time, support from other departments, computers, etc) they have available to them. A people manager may give them a team to focus on their ideas, but that will always be balanced by their ability to produce valuable output as defined by whatever that people manager is being measured on. Miss an alignment with whatever the manager (or THEIR manager more likely) sees as valuable, and you risk losing those resources or having them siphoned off for "more important/pressing" work.

If a company has mostly reluctant/unhappy managers who would truly rather be coding, they have created a culture where code is more valued than management. Think about that. I created a team where the managers, by and large, were happy to be managing, and in the few times when they were not happy to be doing it, they went back to technical focus and we adjusted the teams. I was accused at least once of undervaluing technical contribution, so you should feel free to take this with a grain of salt, but reluctant management is a sign of a broken system, not a necessary an expected outcome of a growing engineering organization.

Dan mentions in his piece that one solution is that we could be promoting people more aggressively on the technical track. I agree with this, if the problem we are solving is giving productive individual contributors more money/bigger titles. Here is the alternate cynical perspective: the technical track exists so that we won't lose people who do good work and have valuable institutional knowledge, but the impact that those people have is often not equivalent to the impact that managers have (for good and for evil). Most companies don't actually need nearly as many people at senior levels of the technical track. We don't want to lose all of them, but there are diminishing returns for people who are possibly just incrementally better at building software. That's why you start to see language about scope of impact in staff engineering job ladders. As leaders, we want to encourage people to actually show results at broader scopes before we promote them. This just takes longer if you have to do it alone or with a small team. At a functional company, people aren't usually promoted as managers until they have shown they can do the next job by managing larger teams and actually doing the work to show the results. Why should the technical track be any different?

What about superpowers that could be given to technical specialists to make up for the authority given to managers? I would personally observe that often, senior technical staff work less hard than senior managers. The superpower of a senior technical staffer is that they get to work on harder intellectual problems and/or more speculative work, they get the luxury of a calendar that is far freer of obligatory meetings and tasks, and they are pressured far less than management for results on tight timelines. In short, by taking the technical path, they have traded off getting to continue to focus on what many engineers consider to be the fun stuff in exchange for a lot less clarity on how to progress and make an outsized meaningful difference. They don't have to work as hard as managers, they often don't work as hard, but on the flip side when they want to work harder to accomplish more it's less clear where to put that effort to gain value.

We all make tradeoffs. Engineering managers are making the tradeoff of getting farther away from the immediate reward (and, I will note, general tech cultural cachet) of day-to-day coding, as well as giving up control of their time, for a clearer career path and possibly more authority. Those who choose to stay technical should expect a likely slower career progression, and they will have to put in the difficult work to learn how to influence without management authority, but they should have more freedom from other people's demands and more control over when they put in the hard work to grow. Make no mistake, growth is hard no matter which way you go. You may have a clearer workout plan as a manager, but you still have to hit the gym and get sore. If you're not a little uncomfortable, what makes you think you're really growing into a new role?