Buy My Book, "The Manager's Path," Available March 2017!

Wednesday, January 25, 2012

Developer Joy

I've been thinking a lot about developer joy lately. Specifically, I've been thinking about ways I can increase it for the developers that I work with at Rent the Runway. While walking home today I realized that, while I might have general ideas for things to try to increase that joy, I haven't ever bothered thinking about what it actually IS. So I present a list of things that increase my own developer joy.

1. The opportunity to learn new tools and systems.
If I could have a list of projects worthy of the 3 major NoSQL styles out there (BigTable, Dynamo, Document Store), I would be in hog heaven.

2. Code bases that are easy to check out, compile, run tests on, and deploy.
I love code bases with "eclipse" targets that just make it go.

3. 3+ hour chunks of time devoted to coding.
You know you're getting too close to being a manager when you have to start hard blocking your calendar if you ever want to get anything done.

4. Automated builds that are kept in good shape.
It's a matter of respect for your code and your teammates.

5. Having a team to brainstorm with.
Bouncing ideas off of others almost always hardens them into fighting shape fastest.

6. Source code for all my libraries at my fingertips.
Nothing frustrates me more than being unable to see what the code I'm calling is actually doing.

7. The time to deliver a quality product.
I like to feel pride in my work, and delivering something that works and works well is integral in that pride

I'm sure I'm missing a ton of things, and my list is probably different from your list. Please tell me what I'm missing! Leave a comment or hit me up on twitter.

Edit 2/1: What did I miss? My favorite submissions include:
Being able to point to work done, even on the backend (@lucasjosh)
The ability to get hardware when you need it! (@abeppu)
Opportunities to showcase work outside of the company via public speaking, blogging, and contributing to open-source. (@dblockdotorg)
And this very eloquent comment from my long-time mentor, Mike, on my G+:
" Remember there's the joy of stylistic expression in code. After the architecture white-boarding, after the design arguments (if there were any), then it's you and your editor (and your 3+ available hours to code) - and the joy of just saying it in code. Saying it well has always been so important to me."

Wednesday, January 18, 2012

Keep it Simple, Dingus

Earlier this week, I took a few devs from my company over to visit dB had offered to talk front-end technology choices and techniques with us, and it ended up being a very useful overview of lessons that they learned in building out their incredibly beautiful UI.

Much of the content went over my head. As you may be able to tell from the ultra-boilerplate layout of this blog, I'm not much of a front-end developer. But one thing that dB said in passing really hit home. When asked why he had chosen to use (or not use) some technology, he made the point that he wanted to keep the set of different systems he was using very limited to begin with because he thought that the architectural complexity overhead would be bad for a small startup.

I've been thinking a lot about that idea this week as I build out a new back end system to host our product data. We've already decided to go with MongoDB for the storage layer. It's easy to set up, supported by Play modules, and our ops people know how to administer it. Most importantly, it works very well for the type of data that we are sticking into it. Everything about the project has gone smoothly, and so over the past couple of days I've been trying to figure out how to do generic text search over the data. I want to use something that has all the goodies of lucene built-in, and I don't think that the MongoDB text search is going to quite do it (although using Mongo to provide fast filtering has been a dream).

I quickly found myself down a rabbit hole. Solr couldn't immediately parse the bson that Mongo spit out for our documents (not surprising), and I didn't find any easy translators online. I started looking at what modules existed in Play to enable integration with a search system and came across elasticsearch, then fell further down the rabbit hole trying to get the module to support not only JPA objects but morphia-generated Mongo objects. In the process I re-read articles and notes on the idea of using solr as the storage layer with no other backing store, was pointed at SenseiDB, and generally began to feel a sense of despair at the complexity of it all.

I like systems. I like to learn systems, to understand their strengths and weaknesses, to read their code. In the 6 or so weeks I've been on this new job, I've had to resist the urge to push immediately moving to a Hadoop-based analytics platform, our own distributed file system for image stores, possibly a different nosql for our user data. I realize now that I need to fight that urge even more. Using the perfect tool for every job incurs an administrative overhead and attention thrashing that an infrastructure team of 3 cannot possibly hope to manage well.

So tomorrow I'm going to take a step back and think about how I can simplify my text search problem. Maybe the answer is to just do it in Mongo for now, and save the feature/complexity tradeoff for another day. One thing is certain: right now, it's more important that I become an expert in the systems I have than pick the perfect technology for each tiny problem.

Thursday, January 12, 2012

Framework Developers, Application Developers

I was chatting over drinks with a buddy of mine (All Things All Things, aka Joe Stein) the other day, and we both agreed that we were annoyed with open source frameworks that seemed like they were built by people that never had written applications using said frameworks, and sometimes by people that seemed to have never developed applications at all. I've been both an application developer and a framework developer, and I can say without question the worst job I've ever done with a codebase was the case of working on a framework that I never used and didn't originate myself. Why does this happen? I'm a good developer, but I'm not immune to the common pitfalls of framework/library development.

Pitfall 1: Never running a feature in a real application
I think this is a very common problem of frameworks developed by people that aren't actively using them. You think of a cool feature, or maybe some user asks you for one, and you spec it out and implement it. You hopefully write some good unit and integration tests, and everything seems to work. But of course, you neglected to test things like what happens when the whole system is rebooted and the state of this feature changes. Especially with certain kinds of features you can build it half right and have it silently fail for a long time before anyone notices. Quotas in ZooKeeper are an excellent example of this: a monitoring feature that worked until the quota was written to snapshot, and didn't seem to be used by any of the maintainers of the project. (cf this not very descriptive jira)

Pitfall 2: Never having to test application code that uses this framework
I'm hitting this a bit in my usage of the Play framework. It's a framework that did have a lot of testing features built into it but... they neglected to implement Filterable in their Junit runner, so you can't run a single test out of a class in your IDE. I submitted a fix for this feature a few weeks ago that has been withering on the vine, despite the fact that this is an incredibly annoying thing to overlook and a trivial thing to fix. The framework also doesn't support changing the http port on the command line when running tests automatically. Why would you need to, unless you happen to have a code base with several active branches in development that are also being automatically tested as I do right now. The framework developers may never get bitten by this, but it's definitely an annoyance as an application developer using the framework.

Pitfall 3: Throwing in everything and the kitchen sink
I recently saw a retweet asking why the hell Guava would add an Event Bus feature. Does that really belong in a collections framework? When your whole life is the framework you're developing, sometimes no feature seems too small or too unrelated. Unfortunately, putting in too much for the sake of completeness can make your code harder for application developers to fully grasp. If I have several different subtle variations of a method, with slightly different argument lists, I have to constantly check the javadoc and stop to think every time I try to use your library I'm likely to use it less, or just find one way to do it and always do it that way. I will, and have, rejected libraries in the past on the basis of being overly feature-laden. I don't always want or need complexity, and I'd frequently rather work around a small missing element than spend my life searching for exactly the method I want to call.

Pitfall 4: Making your library difficult to read and debug through
When you coat everything in layers upon layers of indirection, reflection, deeply nested interface hierarchies, and painful call graphs, it's hard for your users to figure out what the hell is actually going to happen, and painful to debug through the code when something goes wrong. I can't possibly be the only developer that learns libraries half by reading the documentation, and half by just calling the method that seems right and reading through the code when it doesn't work. This is largely why I absolutely despise Fluent-style development. When it is done perfectly and just works (as in perhaps the case of something like Mockito), it's verbose but acceptable. When it's in a place where there are lots of links in the chain where something could go wrong, it is an absolute nightmare to read and debug. I'm keeping the call stack of my own application in my head, please make your library as easy as possible for me to add to that mental complexity.

The best way to get over most of these pitfalls is to have at least one person on your framework team that actually uses the framework you're developing for something else. Barring that, listen to your users carefully. When they are confused by how to figure out what to call, frustrated by the difficult of debugging, or complaining about the difficulty of testing your framework, these aren't problems to treat lightly. Remember, your framework succeeds or fails based not on it's own internal merits, but on how many people actually use it to develop other code. Application developers are a framework developer's best friend.

Friday, January 6, 2012

(E)Git Pain, Git Joy

I've been using git in anger for about a week now, after we migrated our repos at work to github. I thought that after half a day of struggle, accidental bad merges, and confusion I finally managed to get the hang of EGit. Lots of right-clicks (hey, it's Eclipse so whatever), remember to commit, then push, great.

Today, a coworker and I do a little pairing, write the skeleton of some new features. I commit and push my changes and leave to go to a meeting that ends up going for 2 hours, only to return to a request that I add a new file that had gotten missed in the commit. Why, why, why do I need to ADD this file? I understand that commandline git (and svn) require "add" before commit, but using Subversive for the last few years I've gotten out of the habit because, seriously, if I highlight it to be checked in just add it for me.

I also seem to keep hitting problems with merging when I've made changes to files that have subsequently been changed. No matter how unrelated and auto-mergeable the changes are, EGit doesn't seem to let me pull them.

These may (probably) be user error but it is so tiresome to be yet again at a place where the tools have not caught up with the cool new thing and don't bother to streamline the common case.

On the other hand, we had a production issue tonight. Right now, the production release process for this particular code is "check out trunk, restart" (yeah, yeah, I'm working on it, it's only been a month ok?). At some point I realize that this change is bad enough that I just need to go back in time to where the code was more stable, and deploy from there. Despite a lack of tags or branches involved in the current process, this was quite easy to do.
git branch rollback_0106
git checkout rollback_0106
git reset --hard <checkin>

Then push the branch, and release from that branch. It's incredibly fast, and very easy. I also was easily able to create a branch with some failing tests for my coworker to look at, without much hassle or annoyance.

So the jury is still out for me, but really I suspect the problem is just that the tooling has not caught up to the technology. Sadly, it seems you only get a little time in the sweet spot of good tooling and good technology before being forced to move to the next hot thing. I guess this is why most people just stick with the command line...