The Siren Songs of Hack Day Projects

I love hack days. I think spending a day every now and then to just work on whatever inspires you is one of the best trends of the last five years. Whether you use it to try out a new idea, fix a problem that's been bothering you for months, or play with a new tool, it's a great use of engineering time.

However, we can't forget the seductive illusion of the hack. When you have one day to make something, and you decide to make something new, you make it however you can. You don't write tests (unless perhaps you are the most enlightened TDDer). You don't worry about edge cases. You don't plan for scale. You're playing! You're having fun! You're creating something beautiful and impressive that can bring new frontiers to your business!

What's the downside? A good hack day produces ideas and projects that you want to put into production. But when the time comes to actually take those projects and make them production-worthy it's often quite a letdown. At Rent the Runway we've seen this happen twice over our two hack days. On the first, the whole team worked on a project to rebuild "Rent the Runway" on a completely new platform. The story goes that they got 80% of the way through in one 24 hour binge. 80%! This incredible feat actually lead us to say that we would move the entire site off of our existing platform within the next 3-4 months. Heck, if we could get 80% of the site rewritten by the whole team working for a day, surely we could get the site rewritten completely in a few months.

I'm a bit embarrassed to admit that we all bought into the hype of the 80% completion for longer than we should've. In reality, the 80% completion was more like "80% of basic functionality completed", which is still very impressive but ignores the 50% of required functionality that goes well beyond basic. A 3-4 month rewrite timeline might be realistic if you could take the entire engineering team, build no new features, and eventually ship something similar but not quite the same. Of course, none of these things are ever true.

The most recent hack day had some of our engineers build a very cool project utilizing our reviews data. So cool, in fact, that our product team latched on to it and decided to make it into a full-fledged major launch. Success! And now... we're just finishing a couple of intense weeks of project planning for what is quickly turning into a man-year's worth of an engineering project. It will be cool when it launches, but I don't think any of us expected this cool thing that took a day to hack to turn into 4 engineers working for three full months just to get a polished v1.

The hack day siren song can also cause you to fall in love with your hack so much you don't see the cost to making it production-ready. I've known of engineers that went off on their own and neglected their other duties for projects that would never see the light of production due to their complexity and questionable business value. It's hard to complain when people are working on something they're passionate about, but it's important for a team to feel like they are all working toward the same goal, and that can be difficult when some members are spending most of their time tinkering with side projects.

I'm jaded and boring, so my most recent hack day project took some images that were stored in the database and put them on the CDN. It took me longer than anticipated, but I wrote tests and made it work in both production and staging and released it that day. (My expected 2 hour hack still took me closer to 6 hours end to end.) It also enabled other hack day projects, which was really the point. At the end of the day, I'm glad most of the engineers I work with use their hack days to shoot for the moon, even if the final results always take longer to achieve than we hope and maybe spill over past that one day. After all, what's the point in fast-loading images if no one ever views them?

On Yaks and Hacks

A Yak Story

Today I released a nice little feature into production that tests promoting a customer's "hearted" items to the top of their search results. I decided that it would be a good test after seeing some analytics data on users of this particular feature, and sold it to the head of analytics and myself as a quick feature to spin up, half a day's dev work for me. We already had the data, I just needed to play with the ordering of the products to display.

Of course, it didn't turn out to be quite that simple. When I opened the code and prepared to add the new feature, I discovered a few things. First, my clean checkout of the code base wouldn't even compile. Apparently in the time I had been away from it, the other devs had renamed a maven dependency, rtr_infra_common, to rtr_services_common, only to discover that they needed both. Instead of starting the new rtr_infra_common from, say, version 2, they started it back at version 1.0. My local maven was very, very confused. 

Having figured that out, I finally had a clean, compiling workspace, so time to code. I have all the data I need to do this right here, right? Except, this "score" field that claims to be in the DTO for our products is not in fact being serialized by the providing service. Oh and in my absence the package that holds that DTO has also gotten refactored into its own project. And rtr_services_common depends on that. And rtr_infra_common depends on THAT. And I depend on  rtr_infra_common. So just to get the new data, I have to update the product catalog service, the DTO, and the whole chain of dependent projects.

As an aside, I think maven is kind of nice, but dependency management systems have caused the biggest yaks I've ever shaved, and those couple of hours of annoyance are a small drop in the bucket. 

So, got my data. But now I find another problem. The framework I'm working in was designed to take all of this data and apply experiments to it before returning it to the front end. But we hadn't actually built out a notion of the "user" beyond a userId for logging and bucketing purposes. There was nowhere to put the personalized data I needed to use to enhance the results. I could just save the list of items I needed and add an additional argument to the experiment processor with that data, but I knew that we would want more and different signals in the near future. So I took another couple of hours, refactored all of the code to properly model a user, changed and enhanced all the tests, and finally, finished the job... Except it didn't work. I had forgotten to update the clone method in our DTO to copy the new data I had added. My colleague stepped in for that last bit of belly shaving, as I had now blown my estimate by a good day and my other work was piling up. After going through the irritation of changing 4 projects for 1 line of code, he understood my earlier cursing, and we had our feature finally ready.

A Hack Story

On Wednesday, I get a report that there's a snag in a release that is supposed to go out this week. We're taking some work that used to be done manually by our email marketing staff and automating it. The developer on the project has hit a bug. Apparently, our email service provider does not support UTF-8 encoding, and of course, being a fashion company, we have designer names with the occasional accent that we need to throw around. He's been struggling with this, trying to figure out why escaping isn't working, and finally discovering that only Latin-1 will suffice. Before he sits down to write this re-encoding of everything, we ask our email marketers how they did this in the past. "Oh, we just change them to plain text characters". How many characters are we talking about? "Just an é and a ë". Would it be easier to just convert them to plain text than to Latin-1? Yes. And so we did.

Yaks vs Hacks

I don't like hacks, but they can be expedient. Instead of spending time going back and forth with our ESP we will get this manual email automated and out the door, and worry about our text encoding problems later. Shaving this yak buys us nothing at this point. 

On the other hand, I thought my new feature would be a quick hack that could give us a bunch of interesting data, and it turned into a minor yak (or perhaps just a shaggy water buffalo) that took a bit too long but was not quite a full-blown project. I couldn't hack the dependency problems, and hacking the code restructuring would have resulted in threading an arbitrary parameter through a bunch of unrelated code just to avoid thinking about the right solution. The upside of the yak shaving was more data, a better code base for making such changes in the future, and a recognition of a serious dependency structure problem that we need to address. So today I celebrate Yak Shaving Day. I hope it only comes once a year!

Moneyball

The first thing they always do is quiz you. When big tech companies look for employee prospects, the first test is always basic knowledge. The Googles of the world have a checklist. "Coverage Areas" is what they call the talents they're checking for in a candidate. The big three are Coding, Algorithms and Analysis, Systems Design. Maybe they will also care about Communication skills, and Cultural Fit. They put you through the paces. Can she code? Does she know her Big-O analysis, her data structures? Can she design a distributed system to mimic Twitter? Tick, tick, tick. Pass enough of the checkboxes, and they'll make an offer. Nice salary, great benefits, the chance to work on big problems with these intimidatingly smart folks. When it comes to the deep pockets and well-oiled recruiting machine of a big tech company, it can be hard for a startup to compete for talent. 

I was thinking about this a lot as I watched Moneyball last night. It got me thinking. I wish there was a sabermetrics for developers. After all, what is a startup but a cash-strapped team trying to win against the Yankees? This is an imperfect analogy, but let me break it down.

First a step back for those of you that haven't watched the movie and don't give a flip about baseball. Sabermetrics is basically a system that tries to predict, through past performance, how well a player will do in the future on certain metrics. In contrast to tests of basic skills (running, throwing, fielding, hitting, power) that talent scouts look for, sabermetrics focuses on the statistics of a player's actual game performance. Instead of going broke on a couple of big-name players, as many teams would, or gambling on hyped up fresh "potential", the Oakland As used this system to draft cheap players that together ended up forming a competitive team against the A-list roster of the New York Yankees.

The thing about baseball that makes it a useful analogy for many startups is that baseball is a team sport where the size of the team is fixed and the potential of the team in aggregate matters. The same is true for a tech team. We see so many stories about how you should only hire "A" players, that they attract other A players, and they out-perform other developers 10-1 (or 50-1 or 100-1 depending on how grandiose you're feeling). But how do you find these A players? The Derek Jeters are probably settled, and cost more than you can afford. Google and its kin have deep enough pockets to hire anyone that ticks off all the "Coverage Areas", know that they will get lucky and find some natural As in that bunch, train up some more, and the rest will be good enough to take on the mountains of other work needed at such a huge company.

The thing that Google intentionally neglects, and the thing you absolutely can't afford to ignore, is past performance in the game. What is the tech equivalent of on-base percentage? Product shipped. This is why everyone drools over open source developers as hiring potential. You can see their work, you can know that they've shipped. Teasing this out of others is a tough process. We don't have visibility into the work that most people have done. Writing code for you proves that they have the tools to deliver, and is an essential part of the interview process, but it's not enough to show that they actually have delivered. All we can do is ask them, as a detailed part of their interview, what they have done. 

I realize that relying on reports of past experience as part of the hiring process is fraught. I would never suggest you neglect technical vetting in favor of experience. It's easy to get burned by someone whose personality you like and who sounds smart, only to find out they don't have the basic skills to succeed. And even with the best interviewers in the world, it's always possible that you will still hire someone that chokes at the plate. In the case where you find yourself with a player that isn't shipping, you have to take to heart the other thing that you can do more easily than Google: get rid of underperformers. It sucks, and no one enjoys letting people go, but a startup cannot afford to keep people that aren't delivering.

Will this result in your team being full of A-list all stars? Maybe. Or you may find yourself with a team where few people are exceptional standouts, but everyone delivers. When everyone in your roster can reliably get on base, you're winning. You're shipping. And that's what the game is about.

Code Reviews, Code Stories

Code reviews are a tool that incites a lot of strong emotions from developers. Those who love them write tweets like 

Code review is hard - both reviewing and being reviewed make people cranky. But I know of no better way to make great software. -- @yminsky

This tweet, even in its adoration, still captures some of what people hate about code reviews. They make people cranky, to put it mildly. I personally don't like code reviews very much. In my experience, they are just as likely used to bully, score points, or waste time on pedantic style notes as they are to produce great software. In fact, I think code reviews are absolutely necessary only in three kinds of situations:

1) You don't have good automated testing practices in place to catch most errors at or before checkin.
2) You work on a big project with a very geographically distant team of developers (most open source).
3) You work in a company of many developers and huge code bases where strict style conformity is very important.

In my company, we have one team that falls into slot number 1, and they do code reviews for all checkins. The rest of the team does ad-hoc code reviews and occasional code walkthroughs, and we rely on the heavy use of testing to ensure quality. In general I feel that this enables a good mix of code correctness and checkin velocity.

The one thing this misses, though, is the required opportunity for developers to learn from each other. I like pair programming, and encourage my developers to do it when they are first starting projects or figuring out tricky bugs. But a lot of the time my developers are working in parallel on different code bases, and I felt that we were missing that learning moment that code reviews and pairing can provide. So I started asking people in interviews how they encouraged their team to work together and learn from each other, and I got a great suggestion: encourage the developers to show off work that they're proud of to other team members. Thus the Power Hour was born.

"Power Hour" is an hour that we do every Friday morning with my team and our sister team of warehouse developers. In that hour every developer pairs up with someone else and they take turns sharing something they did that week. Show off something you're proud of, ask for help with a problem you're stuck on, or get feedback on the way something was or is being designed. The power and choice of what to show lies in the hands of the person showing it, which takes away the punitive nature of code reviews; all they are required to do is to share some code. Show off, get help/feedback, and then switch. It's not about code review, that line-by-line analysis and search for errors big and small. It's about code stories, whether they are war stories, success stories, or stories still being written.

There are a lot of features of code review that this Power Hour doesn't satisfy. It's not a quiet opportunity to evaluate someone else's code without them looking over your shoulder. It's driven by the person that wrote the code and they are encouraged to highlight the good parts and maybe even completely bypass the ugly parts. It's not possible for the other person to fully judge the quality of what they're seeing, because judging isn't the point at all. Power Hour is a learning and sharing opportunity, and it is explicitly not an occasion for judging.

This has turned out to be a smashing success. It's amazing how much you can teach and learn in a concentrated hour of working with another person. Last week I showed off a new Redis-based caching implementation I wrote, and got to see all the code and a demo of a new Shiro authentication and entitlement system. I caught a few things I still needed to clean up in the process of explaining my work, and so did my partner. If you, like me, shun formal code reviews, I encourage you to try out a weekly turn at code stories. It's a great way to build up teamwork, encourage excellence, and produce great software, without the crankiness.

There Is No Number

Is it time to quit your longtime job? It's pretty easy to tell. Have you gone out and interviewed? Gotten job offers? Seriously considered them? Then you should quit. It's that simple. Not quitting now is delaying the inevitable and robbing yourself of the opportunity to grow.

I've seen this time and again, and gone through it myself. It's hard to leave a longtime job, especially when you are treated holistically well. Great perks, good pay, a big network, lots of friends. You may hate how things are done but you know how to get them done. It's scary to think about leaving that comfort zone. And when you bring up the possibility of leaving, either speculatively or actively trying to resign, it's likely that your boss will promise you up and down that they will move mountains for you. A new project! More responsibility! Leadership opportunities! Your pick of teams! Maybe even the biggest enticement out there: more money.

Take a lesson from Peggy Olsen. For those Mad Men fans out there, you know that Don will never change. What made him a great boss for her, a mentor, a leader, was also keeping her boxed in. He wasn't changing, and she needed to grow. Your Don could be your actual manager, but more likely it's the company culture and dynamics that are beyond the ability for one person to change. Be real: will more money make you excited to get out of bed in the morning? Will it make you feel like your contributions are truly important? Will it make you feel like you're actually growing and learning new things?



Sometimes, there is no number.

If that isn't enough to get you moving, here's one more the thing about leaving a long-time job, especially one at a big company: if you are leaving on good terms, there's a very good chance you can go back if things don't work out. A friend of mine recently did just that. After leaving her job at a larger tech company due to malaise with the work and a bad fit with her team, she went to a very small startup to do mobile development in a related genre. When she realized that the startup was just not managed well enough to succeed she reached out to her old employer and was hired back, happier than ever having both learned some new tricks and gotten into a new team.

Throw back your drink, and do the deed. It will probably feel like breaking up with a long-term partner, and it may be one of the most stressful things you've ever done, but it's also a good bet that you will end up better off in the long run. If nothing else, when the deed is done, I bet you will leave with a smile.

Corporate culture: A lesson in unintended consequences

It's been a little more than six months since I left finance to join Rent the Runway. Upon hearing that I moved from a big investment bank to a small startup, many people comment on how different it must be. And it is, but in ways than you might not expect.

The day-to-day aspects of the job are not that different. On the technical front, I'm designing SOA-structured systems, thinking about building a messaging-based data bus, worrying about how to model business workflow into software and making sure we can process everything we need to process in time. I have more freedom to choose the technologies I want to use but I'm also more constrained by limited resources to manage the complexity of additional systems. The giant internal infrastructural support organization I had for hardware and level one support is replaced by a giant cloud vendor that does pretty much the same thing but with crappier hardware and faster turnaround. The people-related work is mostly unchanged. I still use the skills I learned for building consensus across teams, coaching people in their careers, and managing up, down, and laterally. I run plenty of meetings, make plenty of schedules, and continue my struggle balancing management and organization with architecture and coding. My schedule starts and ends slightly later in the day, but even my hours are not much different.

Unlike some that have left finance, I don't hold any major grudges against it. I learned a lot in my six plus years working there, about technology, leadership, and working with people. So given that the work is similar, why leave the better paycheck and general stability for the uncertainty of startup life? I'm much happier now than I was in my last years in finance, and the reason for that is not technical, but cultural. When I started in finance, the culture was surprisingly startup-like. It was more formal, but each team was dedicated to the business they served and we were able to work fairly autonomously in support of that business. Over time, though, the culture changed to become more centralized and thus bureaucratic. Instead of picking the right thing for your business area, you had to get approval from people that had nothing but a vested interest in seeing their pet technology promoted in order to get themselves promoted. That of course led to politically-driven technology which is an absolute nightmare for a person that cares deeply about building great systems.

Ironically, I believe this happened at least partly because the company was concerned with building a formal technical career path in order to promote and retain technical talent. At first, this resulted in senior technical people getting much-deserved recognition and it was great. I wanted nothing more than to reach that highest level myself. But soon the highest-level title ("technology fellow") had been awarded to the backlog of people that truly deserved it, and it became something that was expected to be awarded to more people every year; each group needed to have one, and it was a cheap way to recognize and retain people in years when salaries were stagnant since it didn't necessarily confer any increase in compensation. This growing group of people, many qualified by deep work in one small area, were now expected to help guide the technology decisions of the whole company. And so a bureaucracy was born. Decisions that would have been made via influence within a group and business area started to be dictated by committees far away from that group. The fellows started, unintentionally, to squash creativity of those below them on that same technical track they were supposed to be growing.

As bad as that was, in my opinion the biggest problem with this role was that it was not equivalent to the same level title for managers, "Managing Director". Because the role of fellow became more of an honorary recognition than an actual promotion, it came to pass that the savviest fellows would also get themselves promoted to Managing Director (MD), and thus become first among equals. No longer did the mere tech fellow have the strongest voice in the technical decision making; everyone started striving to become more than just a fellow, and in order to get that MD promotion they needed the backing of the other tech fellows that were also Managing Directors. As we all know, the best technologists are not necessarily the best politicians and decisions started being guided more and more by those better at politics than technology. In this way, an effort to promote and retain technologists ended up building a technical culture of decision making driven by politics and bureaucracy.

At the end of the day, finance is an industry that cares first about money (duh), and secondly about power, and that is true for every successful person that works there in any capacity. Technical qualifications will only ever be rewarded as they make or save money. If you don't find finance fascinating, or care about money above all other things, this cultural friction may grind you down. I've discovered the truth in the wisdom that you enjoy your job more when you're working on a product that you use and enjoy. Rent the Runway has a different goal than my old investment bank; yes, we want to make money, but we want to make it by delivering a great experience to our customer, an experience that I participate in and really love. And that is a goal that I know I can build a great technical culture around.

War Stories: Guava, Ehcache, Garbage Collection

We're in the process of moving all of our major business logic out of our clunky Drupal frontend to Java backend services, and we took another big step down that road this week by moving all of the logic for filtering of our product grids to our new integration service. This release was the culmination of months of work and planning that started at the beginning of the year, and it gets us over a major functionality hump. The results are looking good, we've saved almost 3s average page load time for this feature. Yes, that's right, three seconds per page load.

As you may guess from the title of this post, the release was not entirely smooth for our infrastructure team. The functionality got out successfully, but two hours after we released we started noticing slowness on the pages, and a quick audit showed frequent full GCs on the services. Some rogue caching was being exercised much more than we had seen during load testing. After some scrambling, we resized the machines and restarted the VMs with more memory. Fortunately the cache would only get so big, and we could quickly throw more memory onto the machines (thank the cloud!). Crisis averted, we set to fixing the caching so that we wouldn't hit slow FGCs.

The fix seemed fairly straightforward; take the cache, which was originally caching parameters mapped to objects, and instead just cache the object primary ids. So the project lead coded up the fix, and we pushed it out. 

Here's the fix. Notice anything wrong? I didn't. We're big fans of Guava and use List transformers all over the place in our code base. So we load test that again, and it looks ok for what our load tests are minimally worth, so we push it onto one of our prod boxes and give it a spin.

At first, it seemed just fine. It hummed along, seeming to take less memory, but slowly but surely the heap grew and grew, and garbage collected more and more. We took it out of the load balancer, forced a full GC, and it still had over 600m of active heap memory. What was going on?

I finally took a heap dump and put the damned thing into MAT. Squinting at it sideways showed me that the memory was being held by Ehcache. No big surprise, we knew we were caching things. But why, then, was it so big? After digging into the references via one of the worst user interfaces known to man, I finally got to the bottom of an element, and saw something strange. Instead of the cache element containing a string key and a list of strings as the value, it contained some other object. And inside that object was another list, and a reference to something called "function", that pointed to our base class. 

As it turns out, Lists.transform is a lazy function. Instead of applying the transformer to the list immediately and returning the results, you get back an object that acts like a list but only applies the transform on the objects as you retrieve them the first time. Which is great for saving a bit of time up front, but absolutely terrible if you're caching the result to save yourself memory. Now, to be fair, Guava tells you that this is lazy in the javadoc:

But not until you get to the third part of the doc, and we are even lazier than Guava in our evaluation. So, instead of caching the list as it is returned from Lists.transform, we call Lists.newArrayList on the result and cache that. Finally, problem solved.

The best part of this exercise was teaching other developers and our ops folks about the JVM monitoring tools I've mentioned before; without jstat -gc and jmap I would have been hard-pressed to diagnose and fix this problem as quickly as I did. Now at least one other member of my team understands some of the fundamentals of the garbage collector, and we've learned a hard lesson about Guava and caching that we won't soon forget.