Buy My Book, "The Manager's Path," Available March 2017!

Tuesday, March 6, 2018

Are you out of alignment?

Alignment, in the teamwork sense, means “a position of agreement or alliance.” It is one of the critical qualities that determines success in an organization, particularly at higher levels.

Many individual contributors (ICs) get stuck at a certain point in their career because they can’t see that they are out of alignment with their company, and they don’t realize that the way alignment is achieved has changed for them. In the earlier part of your career, you can be relatively well-aligned just by doing a pretty good job on the things you are asked to do. You get an assignment, presumably from someone who knows what is important to do, and you do it well. As you grow as an engineer, maybe the technical implementation details of those assignments get more challenging, the tasks get bigger, but as long as you keep conquering them, you are fine.

That is until some point you realize that you’re still good at conquering the tasks, getting those hard problems solved, but you aren’t really getting promoted any more.

Sometimes this comes on slowly, and it looks like boredom. There are fewer and fewer of those meaty big technical problems for you to work on. You assume that they don’t need your skills. You get jealous that people aren’t asking for your advice before they build things, because clearly you know better!

Or it comes on all at once. You are told explicitly that you should find the next thing you want to work on, and you look around and don’t know what to look for. There’s no obvious problem that breaks down into what you know how to do. Or you find something, and you dig in. You get a prototype finished, and everyone yawns. Maybe it gets used for something small, but instead of changing the world for your team, people don’t seem all that impressed, and the things they are impressed by seem so trivial, boring, and technically easy by comparison!

My friend, you have discovered that you are out of alignment.

There are a lot of obvious ways people are misaligned to their companies. The classic “not a cultural fit” is one. If you fight against every process, decision, person around you, even if you’re right, that alignment mismatch will hurt you.

The more subtle way that most of us get out of alignment is by being unwilling to admit that we don’t understand what’s important to the company. We maybe don’t want to understand. We want to do things that are fun and challenging and interesting to us and have that work out to be the important thing. But for the fun things to be the important thing effortlessly is pretty rare. At a certain level, creating the alignment between projects that are interesting for you and what is needed by the company is a big part of your job.

Getting into alignment with the company is often a challenge for senior ICs because it requires a major change in focus. The hardest part is now identifying the right problem to solve, instead of solving the hardest problem. If you want to be able to find interesting work and also work on important things, you generally have to go find the interesting important things yourself. This requires that you to talk to a lot of people and listen to their problems, and then place a bet on a solution to one of these problems that will actually both be feasible but will also be seen as important. Your manager might help identify people that you could talk to, but you must take responsibility for doing the legwork and making the final choice in problems to address.

The alternative way to resolve this alignment issue is to force yourself to address problems that you don’t want to solve, problems that are perhaps not as glamorous to you but are clearly important for the success of the company. Taking responsibility for cleaning up an area that feels intractably broken is not the job for every engineer, but for some people it is an easier and more obvious path to alignment and success. The risk you take here is that you either burn out fighting to fix a situation that is really about people and organizational issues that are beyond your scope, or you pick a broken area that no one really cares about fixing.

In both of these scenarios, you are taking more risk than you used to in the past, and that is where the second part of the quest for alignment comes in. It’s not just about being willing to find the work to do, it’s about finding the most important work that is right for you to do. The more attuned you are to the needs of the company, the kinds of work that is highly valued, the more likely you are to put your energy into something that will pay off well for you.

I often say that at more senior levels you get promoted for making wise bets, and really what that means is that you are smart enough to know the payout for what you’re pursing. Whether it’s knowing which fire to put out first, or having an idea for the thing to build to make your whole team more effective, the better-aligned you are to the needs and values of the organization, the more likely you are to be successful.

Enjoy this post? You might like my book, The Manager’s Path, available on Amazon and Safari Online!

Saturday, January 13, 2018

Stop answering your own questions

I have a bad habit. I noticed it today as I was leaving a comment in a strategy document. I’d highlighted some text that I found unclear and commented:
“Do you mean X or Y? Because I don’t think it is reasonable for us to do Y”
This is one of my bad management habits. I jump to conclusions. I pretend to ask a question but then make it clear that only one answer can be right.
It is probably obvious why this is bad. While I do want to know more, in this framing, I am talking to hear myself speak, rather than genuinely asking for more information that would help me understand the plan and allow me to give better feedback. By stating my preference up-front I cut off discussion. What’s worse, I make the receiver unlikely to honestly answer my question; unless, that is, they feel up to the task of debating me.

Get Curious

In writing you can review what you have said and edit your phrasing to eliminate this kind of thing, but it’s significantly more difficult when you’re in verbal conversation. When you’re talking you don’t have a chance to see yourself poisoning the well and cutting off opinions before they can be explained. This is one of the many reasons I have tried to embrace the mantra get curious. I use this phrase to remind myself that I have to make room for people, that my first reaction is not always the right one, and that when I hear something that doesn’t sound right I need to listen more rather than jump all over it.
If you are (or were) a highly opinionated engineer, practicing making space for information rather than quickly jumping in and sharing your conclusions is a must for leadership growth. The more senior you become, the harder it is for people to feel comfortable disagreeing with you openly. That is not a sign of weakness on their part! Most people with any sense of self-preservation know that saying the wrong thing to the wrong person can have negative consequences. Lucky for you, as the manager, people are going to listen to what you have to say. Unlucky for you, as the manager, if you don’t make space for them to say things that you disagree with, you are unlikely to hear important details.
Making good decisions requires you to get as much information as possible, to understand the nuances of the scenario from all angles. It is very difficult to do that if people are afraid to contradict you. One of the less-obvious ways we make people afraid is by offering our opinions too early, without taking the time to get the rest of the information. So the next time you’re tempted to ask a question and answer it yourself, stop. Get curious. You already know there’s something you don’t totally understand, so hold on stating your opinions until you’re sure you’ve gotten as much information as possible.
Now, to go edit that comment…

Wednesday, September 6, 2017

How do managers* get stuck?

*May also apply to senior ICs

Earlier this year, I wrote a piece called “How do Individual Contributors Get Stuck?” This was an attempt to help ICs provide constructive feedback to their peers, by identifying common challenges that I have seen developers struggle to overcome. 

This piece is a little bit different. I want to answer the question that I often hear from first-line managers, that is, managers who manage only individual contributors: “How do I get promoted to the next level of management? How do I prove I’m ready to manage managers?” 

Managers often believe that if they are handling the demands of managing their team, they should naturally be promoted to manage more people, bigger teams, teams of teams, as quickly as such an opportunity comes about. And yet, just as often as those opportunities come about, someone else is chosen, a person is hired from outside the company, and the eager manager is passed over. You’re stuck. When you find yourself “stuck” in terms of management career progression, what might really be happening?

Usually, getting stuck as a manager falls into one or more of the key areas of management: Failing to manage down, failing to manage sideways, and failing to manage up.

Scenario One: You aren’t actually scaling yourself effectively, aka, failing to manage down

You may think that you’re handling your team well, but when you look at your schedule, you’re working nights and weekends and then some to juggle all of the new tasks that managing a team entails. Sure, there are some companies which expect that from everyone, but it’s rarely a sign that you’re using your time effectively. Look at your team. Is it a well-oiled machine? Do you feel like the team is able to operate independently, get things done, without you micromanaging every detail? If not, you’re probably stuck on the basic needs of your current job. Some examples of this include:
  1. Can’t delegate. Look at all hands-on tasks you own, and ask yourself whether you are the only person who could be completing these tasks, or whether you could assign them to another senior engineer. If you are spending a lot of your time doing hands-on work that someone else could be doing, you probably aren’t delegating effectively.
  2. Not training your team. If there are too many tasks that only you can complete, you have made yourself a key dependency for your team. Who are your potential successors, and have you spent time training them on the things only you can do?
  3. Not enough attention to the process. Is your team drowning in alerts with no end in sight? Why haven’t you spent the time to allocate people to fix that? Have you spent time paying attention to the way work is assigned in your team? Do you actively participate in the planning process? When was the last time you tried changing it to see how it could improve? Process is part of your life now, and you need to tend to it.
  4. Won’t say no. If your team is completely overwhelmed with work, well, it’s partially your fault. You are the manager, and you are the person who is responsible for pushing back on the work commitments for the team. 

Scenario Two: You haven’t shown that you can expand beyond your team, aka, failing to manage up

Maybe your team is running well enough, but that’s all you’re doing. Opportunities for advancement are usually given to people who show up for those opportunities. You can easily get stuck by just getting comfortable in the place that you’re sitting. Someone who is failing to manage up often exhibits one of the following problems:
  1. Doesn’t attend to the details. Everything from clearly communicating the things that your team has accomplished, and sharing challenges or setbacks, to keeping your manager in the loop about major design decisions or roadmap changes, these details all matter. The best managers push information up without being asked and are quick to provide more details as necessary. Your manager wants to know that you are paying attention to what is going on.
  2. Complains a lot about things that are not working well, but never volunteers to fix them. If you are free with your criticism about the way things work, but don’t feel the need to do anything more than complain, you are holding yourself back. Instead of complaining, volunteer to lead the initiative to fix that team-wide problem. Bring problems and solutions.
  3. Drags her feet when given a clear task that is outside of her comfort zone. I have seen so many managers fail at the simple act of taking a clear assignment and seeing it through to completion. When your manager asks you to do something, either do it, or say you can’t/don’t really want to. But don’t just drag your feet and fail to do it.
  4. Doesn’t show a professional face to more senior managers. Do you openly look bored, distracted, or impatient in meetings? Do you write emails that communicate clearly? Do you think your manager would be comfortable having you present to her peers, alone? Your verbal, written, and body language communication is more and more important the more senior you become, and if you are lazy here it can hold you back.

Scenario Three: Fails to show peer leadership, aka, failing to manage sideways

Some people have well-running teams, they jump on fresh assignments, and yet they still get stuck. This is often due to the fact that your manager knows that she cannot put you as the manager to any of your existing peers. You are stuck because you haven’t shown enough peer/relationship management. This can sometimes look like:
  1. Doesn’t build strong peer relationships. If you spend most of your time focused downward or upward, you’re missing a step. When was the last time you helped out one of your peers? How often do you spend time with your peers, 1–1? Do you seek out feedback from your peers on your ideas, or ask them for help with your challenges? Having peers who trust and respect you, and more to the point, who might want to work for you if they had to, is needed for successful growth.
  2. Doesn’t look for additional tasks. You should be looking for opportunities to lead projects or initiatives outside of your team. How can you help your peers? You may be the best person to lead your area, but if you rarely push yourself outside of that area to talk to others and see places you could volunteer to improve, you’re missing a critical element of leadership.
  3. Doesn’t create a compelling vision or strategy that others want to buy into. You might have a clear roadmap for your team, but how much have you thought beyond your team? Have you ever shared any ideas you have for the larger group with your peers, and gotten their buy-in? Many people think that strategic thinking starts and stops with forming the strategy itself, but getting people around you excited by your ideas is critical to achieving them. 
  4. Doesn’t seem like someone a manager would want to report to. Ultimately, if you want to manage managers, a manager should want to work for you. That means that you are going to help train them, help them grow their career. You’re not going to be spending all of your time telling them exactly what to do, when to do it, no questions asked. If one of your peer line managers wouldn’t want to work for you, you might be stuck on giving the impression that you’re looking to progress solely so that you can acquire more power and influence.

What about senior ICs?

As you may have noticed, this advice applies to more than first-line managers. Many senior individual contributors start to trip on these issues. They can crank out code for days, but communicating, getting buy in, and going outside of their comfort zone stops their progression. Few senior ICs get promoted beyond a point on the strength of their ideas and code alone. 

Getting Unstuck

How do you get out of your rut? It starts by noticing where you are stuck. I noticed something that I’m not doing well just writing this list! Be honest, which of these are you really doing well at, and which are you failing? If you brought this list to your manager, what would they say? There’s only one real way to find out, so think about it, ask for feedback, and start to formulate your plan of attack for the things that are holding you back.
For more ideas, check out my book, The Manager’s Path, which addresses many of these stuck points!

Friday, January 6, 2017

How Do Individual Contributors Get Stuck? A Primer

Occasionally, you may be asked to give constructive feedback on your peers, perhaps as part of review season. If you aren’t a naturally critical person but you want to give someone a valuable insight, you may find this task daunting. To that end, I suggest the following:

Pay attention to how they get stuck.

Everyone has at least one area that they tend to get stuck on. An activity that serves as an attractive sidetrack. A task they will do anything to avoid. With a bit of observation, you can start to see the places that your colleagues get stuck. This is a super power for many reasons, but at a baseline, it is great for when you need to write a review and want to provide useful constructive feedback.
How do people get sidetracked? How do people get stuck? Well, my friend, here are two incomplete lists to get you started:

Individual Contributors often get sidetracked by…

  1. Brainstorming/architecture: “I must have thought through all edge cases of all parts of everything before I can begin this project”
  2. Researching possible solutions forever (often accompanied by desire to do a “bakeoff” where they build prototypes in different platforms/languages/etc)
  3. Refactoring: “this code could be cleaner and everything would be just so much easier if we cleaned this up… and this up… and…”
  4. Helping other people instead of doing their assigned tasks
  5. Jumping on fires even when not on-call
  6. Working on side projects instead of the main project
  7. Excessive testing (rare)
  8. Excessive automation (rare)

Individual Contributors often get stuck when they need to…

  1. Finish the last 10–20% of a project
  2. Start a project completely from scratch
  3. Do project planning (You need me to write what now? A roadmap?)
  4. Work with unfamiliar code/libraries/systems
  5. Work with other teams (please don’t make me go sit with data engineering!!)
  6. Talk to other people (in engineering, or more commonly, outside of engineering)
  7. Ask for help (far beyond the point they realized they were stuck and needed help)
  8. Deal with surprises or unexpected setbacks
  9. Navigate bureaucracy
  10. Pull the trigger and going into prod
  11. Deal with vendors/external partners
  12. Say no, because they can’t seem to just say no (instead of saying no they just go into avoidance mode, or worse, always say yes)
“AHA! Wait! Camille is missing something! People don’t always get stuck!” This is true. While almost everyone has some areas that they get overly hung up on, some people also get sloppy instead of getting stuck. Sloppy looks like never getting sidetracked from the main project but never finishing anything completely, letting the finishing touches of the last project drop as you rush heedlessly into the next project.

Noticing how people get stuck is a super power, and one that many great tech leads (and yes, managers) rely on to get big things done. When you know how people get stuck, you can plan your projects to rely on people for their strengths and provide them help or even completely side-step their weaknesses. You know who is good to ask for which kinds of help, and who hates that particular challenge just as much as you do.

The secret is that all of us get stuck and sidetracked sometimes. There’s actually nothing particularly “bad” about this. Knowing the ways that you get hung up is good because you can choose to either a) get over the fears that are sticking you (lack of knowledge, skills, or confidence), b) avoid such tasks as much as possible, and/or c) be aware of your habits and use extra diligence when faced with tackling these areas.

Wednesday, January 4, 2017

Hey Diddle Diddle, Data to Fiddle

When I worked in finance ages ago, there was a system used by many (but not me!) that was basically a combination of a gigantic distributed database plus a scripting language that allowed you to run calculations over information in that database. One of the things that you could easily do, as far as I understand, was "diddle" a piece of information. The "diddle" would change that piece of data inside of a particular scope, so that you could quickly see different calculations over the graph, without necessarily persisting that data back to the larger system. This was a useful construct for exploring what might happen with changes to different input data and exploring different scenarios. (The first half of this blog post provides some insights into how the system might have worked).

Whether my understanding is exactly right or not is irrelevant except that this concept of "diddling" stuck with me. There are times when what you want to do is take persistent data, make a small change to it on the fly, and use the results of that change without necessarily persisting it back to the original data set.

I've often thought that this concept is particularly useful in places like personalization. Imagine the situation where you have a complex set of results that you wish to display to a user, like say, the google search results. We all know that the ranking system for google results is a complex beast, relying on a huge amount of precomputed data, for example, the links between pages. But now, to compute that graph with personalization taken into account? You're probably not calculating all personalization vectors on the fly, but when I go to search for "java" you're also probably not doing a lookup of pre-computed personalized results for "java + userid:camille." Instead, you're applying a "diddle" function to the top set of the overall graph, and showing me the results in the diddled order that makes sense for me.

There are two parts to the concept that make it powerful for me. The first is the idea that you are changing things temporarily. To serve large sets of results fast (or, in the case of google, to be able to function at all), you need to pre-calculate a huge amount of data. You're doing a complex piece of work that takes some time, you don't want to have to redo it for every request. However, you don't want to force yourself to store all the work for all possible scenarios up-front. But, diddling in my mind presents a second element: it is only applied to the set of data within a limited scope. You don't diddle across the entire search index. You diddle the first few results, the ones that matter to the user in question.

There can be a ton of technical complexity to implement such a concept in practice. One immediate challenge is that of "diddling" in such a way as to drop results from the top set, thus requiring a re-querying for additional responses to get enough data to satisfy the user. The purpose of this post is not to go into the technical details of how you might implement such a thing, but to show you that you can reframe your thinking on a problem like this through its phases. Just because you have a list of thousands as your first pass of results doesn't mean you need to personalize across that whole data set to get the best results for the end user. If your goal is to get the most personally relevant of the most generally relevant, you probably want to operate on the top of the generally-ordered list, not necessarily the whole list itself.

There's many ways to attack such problems, and I have no idea how companies like Google solve the challenge of personalizing results under the hood. I do know that, to me, the idea of mixing indexed and computed results in personalized querying is a sticky one, and it's an analogy I use frequently. It helps you remember that there is value in the underlying order of the results as provided by the source of truth, and that personalization is often an enhancement on an underlying set of computed data, not the fundamental computation itself. Pulling out to a larger picture, remember that when your data gets in front of a human end-user, they are going to operate on only the tiny surface area they, as a human, can process at any one time. So in cases where you're serving humans, you can apply different patterns on the fly to the human-visible surface area that would be too expensive to apply to the entire data set at large.

Tuesday, November 29, 2016

Building and Motivating Engineering Teams

I have agreed to give a guest lecture for a class at Yale, and they’ve asked me to speak about “building and motivating engineering teams” from the perspective of a smaller startup. The readings for my section include A Field Guide to Software Developers by Joel Spolsky. I remember reading it when it was first written. I admire Joel’s work, and the piece has many valuable takeaways.

However. The industry has changed a lot in the years since this piece was written. In 2007, the options for engineers here in NYC were far more limited. You worked for a bank, a media company, ad tech was starting to blossom, some e-commerce. Google or Fog Creek if you were lucky. There were plenty of jobs but there was far less of a classic “tech company” presence than there is today.

Over the past 9 years, we’ve seen a massive increase in the number of startups here in NYC. Every major tech company has an office here, the Google office alone has thousands of engineers. We’ve also seen an increase in the supply of engineers. Between students realizing the value of a tech degree, people changing careers, and bootcamps rapidly churning out graduates, the mix of types of people who write software has changed. Most of the people I knew writing software in 2007 had tech or tech-adjacent (math, physics) degrees. My team when I left Rent the Runway had a significant number of developers who had moved into tech from other areas, some without degrees at all.

All of this is to say, playing to 2007 nerd stereotypes is not always a good way to build a team here in NYC. What DO engineers want? I believe that you have 3 axes with which to twiddle. Depending on your company, you can lean more on one to cover problems with the others, but you have to find your balance.

Money. Joel hides this at the bottom of his piece. It is correct to say that engineers don’t care about money above all else, but far too many people delude themselves into thinking that this means that you can lag industry standard salaries and build a good team on the basis of some other factor.

Salaries have gone way up in the past 10 years. People know that they can get paid a certain amount of money, and if you try to hire people who can easily make 50% more somewhere else, they are almost certainly not going to accept your offer. Figure out what the market spread is. The top will be companies like Google, Facebook, some financial companies. The bottom might be non-profits or extremely early startups with significant equity grants. Even a startup with 10–20 people is going to struggle to hire without paying close to market rate. Engineers are expensive. New engineers are expensive. Senior engineers are really expensive. They know their value. You have to pay them.

You are building a company. You are not going to have a perfect, smooth ride. Things will go well, things will go poorly. Very few teams have a straight shot to greatness that is so clear it can cover all management woes. When you don’t pay people well enough, you contribute to undermining their resilience in the face of problems at work. Think of it as the baseline of Maslow’s Hierarchy. Money does not solve all problems for most people, but lack of money exacerbates all irritations.

Purpose. You’re building a company. You want to inspire people to work for you, so you sell them on the mission of the company, the product they’ll be working on. You have to do this because you are not building a company with a bunch of insanely hard tech problems. If you are, you can lean on that to hire people, but to be real, these days most of us are not. The days when everyone had hard technical scaling problems have ended. Scaling is not the same bold new frontier that it was even 5 years ago. Sure, there are always technical challenges to be found in organizations, but for many companies those technical challenges revolve around matching technology with the product and business. This means you have to work harder to sell the learning opportunities.

Leadership undermines their teams when they refuse to let engineers into the non-technical decision-making processes. In Joel’s article he talks about developers wanting to be allowed to make decisions within their own realm of expertise, which is certainly a bare minimum. I would encourage you to go further than that. If you are building a product-focused business, where the challenges are less purely technical and more about engaging a customer, your engineers need to feel connection to that business. They need to feel like they understand it, like they can have ideas about it. This is why I’m such an advocate for cross-functional product development teams. Put engineering, product management, marketing, operations together as a group and let them work as a team to solve problems. Don’t just throw work over the wall to engineering and expect them to implement it.

Respect. The undercurrent that I like least in Joel’s piece is the undercurrent that engineers need to be coddled and pampered. Giving them “toys” instead of “tools,” keeping them out of the politics. There are plenty of engineers who want to be given hard work to do and left alone, in their private offices, to think and code. But there are increasingly more engineers who want to build businesses, and they want to be treated like adults in the process.

Our engineering teams are not overgrown children. They are not idiot-savants who can produce software but must be given sufficient cookies to do so. They are highly paid professionals. Let’s treat them that way. You should expect your engineers to show up for your business. Respect is not pampering, it is not treating the team like the stars of the show. Rather, respect is challenging the team to show up and grow up. Respect is giving them clear, achievable goals and holding them accountable. My experience has been that most great engineers want to work somewhere that inspires them to achieve. Many of us stop at the idea of “hard technical problem” when we think about inspiring our engineering teams, but challenging them to partner with people who have different perspectives is another way you can help them grow.

Respect that engineers are smart individuals who often have more to add to your business than just their coding talents, and teach them to respect that the other parts of the business have equally valuable skills and perspectives. Engineers don’t need to feel like the company royalty to be inspired to do good work, but they do need the opportunity to be treated like a partner.

These days, you have a lot of competition for talent, but you also have a lot of talent to choose from. Understand your company’s positioning. If you can’t pay top of market, you will have to rely on a balance of finding undeveloped talent and giving engineers other reasons to want to work for your company. For most of us, that means giving them a voice beyond the purely technical, and challenging them to see and understand perspectives outside of engineering.

Friday, August 19, 2016

Microservices: Real Architectural Patterns

A dissection of our favorite folk architecture


I’m fascinated by the lore and mystery behind microservices. As a concept, microservices feels like one of the most interesting folk architectures of the modern era. It’s useful enough to be applied widely across different usage patterns and also vague enough to mean many different things.

I’ve been struggling for a while with understanding what people really mean when they discuss “microservices.” Despite deploying what I would consider to be a version of that pattern in my last gig, it’s quite clear to me that the architecture we used is not the same as the pattern that all other companies use. Recently I finally interrogated someone who has deployed the pattern in a very different way than I have, and so I decided it would be illustrative to compare and contrast the circumstances of our architectures for those in the larger technical audience.

This article is going to have two examples. The first is the rough way “microservices” was deployed in my last gig, and why I made the decisions I made in the architecture. The second is an example of an architecture that is much closer to the “beautiful dream” microservices as I have heard it preached, for architectures that are stream-focused.

Microservices Basics

I think that microservices as an architecture evolved due to a few factors.
  1. A bunch of startups in the late 2000s started on monoliths like rails, scaled their business and team quickly, and hit the wall on what could reasonably be done in that monolith
  2. The cloud made it significantly easier to get access to a new server instance to run software
  3. We all got much more comfortable with the idea that we were dealing with distributed systems and in particular got comfortable making network calls as part of our systems
This combination of factors — scaling woes, easy access to new hardware, distributed systems and network access — played a huge part in what I might call “microservices for CRUD.” If you have managed to scale a company to a certain level of success on a monolith but you are having trouble scaling the technology and/or the engineering team, breaking the monolith into a services-style architecture makes sense. This is a situation I encountered first-hand.

The arguments for microservices here look something like:
  1. Services allow for independent axes of scaling. If you have a part of the system with higher load or capacity requirements than other parts, you can scale to meet its needs. This is certainly doable in a monolith, but somewhat more complicated to reason about.
  2. Services allow for independent failure domains, to a more limited extent. Insofar as parts of your system are independently operable, you may want to allow for partial availability by splitting them out into services. For example, in a commerce app, if you can serve the checkout flow even when the product search flow is down, that might be considered a good thing. This is much more complicated in practice than it is in theory, and people make many silly claims about microservices that imply that any overlap in services means that they are not valuable. Independent failure domains are sometimes more of a “nice to have” than a necessity, and making the architecture truly account for this is not easy.
  3. Services allow for teams to work independently on parts of the system. Again, you can do this in a monolith. I have done this in a monolith. But the challenge with monolith (and a related challenge with services in a monorepo (single source repository)) is that humans struggle to tangibly understand domains that are theoretically separate when they are presented as colocated by the source code. If I can see all of the code and it all compiles together and feels like a single thing, my tendency is to want to use it as a single thing. Grab code from here to use there, grab data from there to use here, etc.
A few more notes. “Monolith” and “monorepo” often get tangled up when talking about this world. A monolithic application is one where you have a set of code that compiles into a single main server artifact (possibly with some additional client artifacts produced). You can use configuration to make monoliths do almost anything you can imagine, including all of the services-type things above, but the image produced tends to include most if not all of the code in the repository. This does get fuzzy because sometimes teams evolve their monoliths to compile to a couple of specialized server artifacts via a combination of build tooling and configuration. I would generally still call this a monolithic architecture.

Monorepo, or monolith repository, is the model where you have a single repository that holds all of the code for any system you are actively changing (so, possibly excluding the source code for your OSS/external dependencies). The repository itself contains source code that accounts for multiple artifacts that are run as separate applications, and which can be compiled/packaged and tested separately without using the entire repository. Often this is used to enable certain shared libraries to change across all of the services that use those libraries, so that developers who support shared libraries can more easily evolve them instead of having to wait for each dependent team to adopt the newest version. The biggest downside of the monorepo model is that there’s not much OSS tooling that supports this, because most OSS is not built this way, so large investments in tooling are usually needed to make this work.

Microservices for CRUD-based Applications

Before I get to how to evolve a CRUD monolith to microservices, let me further articulate the architecture needed to build your traditional mid-sized CRUD platform. This type of platform covers a use case that is pretty well-trod, that of “transactions” and “metadata.”

Transactions: User does an action that you want to persist, consistency of data is very valuable. The “Create, Update, Delete” of CRUD. Much less frequent than the “Read” actions of CRUD. 
Metadata: Information that describes things to the users, but is usually only modified by internal content creators, or rarely by external users (reviews, for example). Changes less frequently, often highly cacheable. Even more, can often tolerate a degree of temporary inconsistency (showing stale data).
Are there more things that CRUD-heavy companies want to do, especially in the analytical space here? Sure. You may want to adjust results frequently based on user behavior as the user is browsing the site, and other personalization actions. However, that is a hard thing to do real-time and you don’t always have the volume of data you need from the user to actually do that well, so it isn’t generally the first-order concern of the system.

The process for moving off of a monolith in this type of architecture is relatively straightforward:
  1. Identify independent entities. This paper by Pat Helland, “Life Beyond Txns”, has some useful and interesting definitions there. It’s better to go a little bit too big early than to go too small and end up having to implement significant distributed transactions. You probably want data-owning services for the major business objects(products, users, etc), and then sets of integration services that implement aggregations and logic over those objects.
  2. Pull out the logic into services entity by entity. Try not to change the data model as much as possible in this process. Redirect the monolith to call APIs in the new services as functionality is moved.
That’s basically it. You pull pieces out until you have enough to cover a particular set of user functionality in data and integration terms, then you can start to evolve that part of the user functionality to do new things in the services.

These services are not classic SOA, but nor are they teeny-tiny microservices. The services that own the data may be fairly sophisticated. You may not want to have too many services because you want to be able to satisfy requests from the user without having to make a ton of network hops, and ideally, without needing to do distributed transactions.

You are probably not making new services every day, and especially if you have a sub-50-person engineering team and a long product roadmap, you may not want to invest extensive engineering time into complex orchestration and tooling that enables people to dynamically add new services at the click of a button (nb: the products to support this are getting better all the time, and so at some point this will be worth doing even for that smaller team. It is unclear to me whether that time is now or not.).

The equation to apply for determining how much to invest in tooling is pretty straightforward: how much time does it cost devs to have a less automated process for adding a new service, vs how long does it take to implement and maintain the automation for doing it easily, and how many new services do you expect to want to deploy over time? You’re making a guess. Obviously, if you think there is value to enabling people to spin up tiny services fast and frequently, it is better to invest time and tooling into this. As with all engineering process optimization decisions, it’s not a matter of getting it perfectly right, but rather, of deciding for the foreseeable future and periodically re-evaluating.

There are many microservices “must-haves” in this instance that I have found to be anything but. I mentioned extensive orchestration above. Dynamic service discovery is also not needed if you are not automatically spinning up services or moving services around frequently (load balancers are pretty nice for doing this at a basic level).

Allowing teams to choose their ideal language, framework, and data store per service is also certainly not a must-have and in fact it’s likely to be far more of a headache than a boon to your team.
Having independent data stores for the services is also not a must-have, although it does mean that you will have a high-risk SPOF on the shared database. As I was writing this piece I discovered a section of some writing on microservices from 2015:
Create a Separate Data Store for Each Microservice
Do not use the the same back-end data store across microservices. … Moreover, with a single data store it’s too easy for microservices written by different teams to share database structures, perhaps in the name of reducing duplication of work. You end up with the situation where if one team updates a database structure, other services that also use that structure have to be changed too.
This is true, but for smaller teams you can prevent sharing of database structures by convention (process and code review, and automated testing and checking for such access if it is a huge worry). When you carefully define the data-owner services, it’s less likely this will happen. And the alternative is the next paragraph:
Breaking apart the data can make data management more complicated, because the separate storage systems can more easily get out sync or become inconsistent, and foreign keys can change unexpectedly. You need to add a tool that performs master data management (MDM) by operating in the background to find and fix inconsistencies. For example, it might examine every database that stores subscriber IDs, to verify that the same IDs exist in all of them (there aren’t missing or extra IDs in any one database). You can write your own tool or buy one. Many commercial relational database management systems (RDBMSs) do these kinds of checks, but they usually impose too many requirements for coupling, and so don’t scale.(original)
This paragraph probably leads to sighs of exhaustion from anyone with experience doing data reconciliation. It’s due to this overhead that I encourage those of you in smaller organizations to at least evaluate a convention-based approach before deciding to use entirely independent and individual data stores. This is a decision you can delay as needed.

This version of the microservices architecture is very compelling for the scaled CRUD world because it lets you do a rewrite piece by piece. You can do the whole system, or you can simply take out pieces that are most sensitive to scaling. You proactively engage with many of the bits of distributed systems complexity by thinking carefully about the data and where transactions on that data will be needed. You probably don’t need a ton of fancy data pipelines floating around. You know where the data will be modified.

Do you have to go to microservices to scale this? Probably not, but that doesn’t mean using microservices to scale such systems is a bad idea. However, going extreme with the microservices model may be a bad idea, because you really don’t want to slice your data up in a way that ends up in distributed transaction land.

Microservices For Data Stream Processing

Now, let’s talk about a very different use case. This use case is not your classic CRUD application, thick with business rules around transactionally-updated objects. Instead, this use case has a large pipeline of data. It has small bits of data flowing into it from many different sources, a very large volume of many bits of data. This large volume of input data sources also has many different services that will consume it, modify it, and pass it along for further processing.

The major concern of this application is ingesting large quantities of ever-changing data, processing it in various ways, and showing a view of it to customers. CRUD concerns are secondary to the larger concerns of keeping up with the data stream and recalculating information based on what is happening on that stream.

Let’s take a metrics-aggregating SaaS application, for example. This application has customers all over the world with various applications, services, and machines that are reporting out metrics to the aggregator. These customers only need to see their data, although the combined total of data for any one customer may be very large. Our aggregator needs to consume these metrics and send them off to the application that is going to show them to the customer. The customer-facing application may be operating on a combination of incoming metrics in real-time plus historical data that comes from cache or a backing storage system. A large part of the value of the data is in the moving-window of what is happening right now/recently.

This architecture from the start has considerations of volume that even our scaled CRUD world may not care about for a very, very long time. Additionally, the data itself is mostly a stream of updates over time. The notion of the “stateful” data that is transactionally updated is minimal, the most useful data is more like a timeseries or log of events. The transactional data, say, stored user views and user configuration, may be more like the “metadata” of our CRUD application in the first example, infrequently changed compared to the updates coming in from the stream. The majority of developer time is most likely spent not in dealing with these transactional changes but rather in managing the streams of inputs, providing new types of inputs, applying new calculations to the stream of inputs, and changing the calculations.

In this example, you can imagine a service that wants to run an experiment by doing a different calculation across a particular element on the stream. Instead of modifying the existing code, the experimental service listens to the data stream at the same point as the existing calculation, provides a new calculation value, and pushes that calculation value back into the data pipeline on a different channel. At some point an experiment service pulls this data out for the customers who are assigned to the experimental treatment and shows the results of that calculation instead of the standard calculation. In all of these places you need a record of what happened in order to do analysis of experiment success and debugging, but that record does not need to be strongly, transactionally related to the record of other events in the system at this time, even across related users.

In this example, it may very well be much more effective to spin up new services as needed, in order to run quick experiments, rather than changing existing services. Especially in cases where the service can do this without needing to worry about coordinating the data consumption or production with any existing service. This is the world of what I would like to call “stream-centric microservices.”

If there is enormous value to your business to manage real-time data streams, and you are going to have a lot of developers consuming those streams by creating new services to listen to them and produce results, then you absolutely must be willing to commit to the investment in tooling to make the process of creating services and putting them into production as easy as possible. You will probably use this for all of your services over time, once you have it, but realize that the clear value is that you have dynamic data that can be processed and manipulated and experimented on independently.

Cron Jobs as Microservices

I’d be remiss if I didn’t mention this pattern. When it becomes very easy to make anything a microservice, everything becomes a microservice, including things we would traditionally run as cron jobs.

But cron jobs are a nice concept, and not everything has to be a “service.” You can use CloudWatch Events from AWS for this purpose, or scheduled Lambda functions. Use Gearman, a queue and async job runner, to schedule cron jobs. Remember your cron jobs need to be idempotent (can be run twice on the same input without changing the outcome). If you have an easy way to spin up services and it’s easy to create tiny services that are basically cron jobs, no problem, but cron jobs in and of themselves are not a great reason to create a large, orchestrated services environment.


I hope that this has been a useful breakout across a few axes of the wild world of microservices. Going through the thought experiment was very useful for me, personally. It helped me understand how what seems obvious to people at one extreme, say those who spend most of their time focused on stream processing, doesn’t make as much sense for people who are more focused on the world of CRUD application scaling.

(This was originally published on medium)