Parallelism and the Limits of Languages

While we make heavy use of the core of Clojure, we don't use its concurrency primitives (atoms, refs, STM, etc.) because a function like pmap doesn't have enough fine grained control for our needs. We opt instead to build our own concurrency abstractions in Clojure on top of the outstanding java.util.concurrent package.

Aria Haghighi, Prismatic 

 Once upon a time, in the days when RAM was not quite as cheap as it is now, I worked on a team that had a giant in-memory cache of data that it used to do various financial calculations. This data had, with the growth of the financial markets it represented, grown to be too big to fit into the memory of a standard server without having GC absolutely wreck the process. Having exhausted all minor efforts to tune the data, and not wanting to buy an actual supercomputer, the team turned to Azul, who at that time sold boxes with enough memory to fit our process and a fairly nifty no-pause GC. Great! Except there was one catch: instead of having the super fast CPUs we were used to, it had hundreds of tiny little slow cores (with rather crappy fpus). And on these hundreds of tiny little cpus our code was very, very slow. So we set out to parallelize.

When you're forced into parallelism at that level of detail, you quickly realize how difficult it is to have a generic way of doing it. The fact that languages like scala and clojure will allow you to just say "apply this logic over the collection in parallel" is very cool, and a great way for developers to dip their toes in parallel processing of their collection data. But it breaks down quickly when you are talking about vastly different types of data running in vastly different types of computation. If you need to eke out every ounce of performance, you want every computation to be tuned to the right batch size and the right number of threads, and the size of the thread pool for task X may depend on whether it tends to run at the same time as task Y or task Z. Don't forget that task Z, if requested, must be prioritized above all else. The size of the batches themselves may depend on the type of object inside them, and how that affects the cache lines of the processor you're running on. Oh, and of course, you're not just tuning this for prod, it has to at least run to completion on your integration boxes and they are of course slightly different processors. And don't forget, your developers aren't writing code on Azul boxes, they're writing code on desktops with 2 cores each; threading even the test data to a pool of 50 doesn't make any sense, but they still need to run things in parallel or they'll never see the stupid threading bugs they added when they forgot to use concurrent data structures.

I've always thought that the next truly big leap in VMs and languages will be those that can do smart things with threading without making developers sweat too much. If we're really entering a world of more and more data over more but slower cores, we need a language and a VM that revolutionize our management of threads the way Java and the JVM have revolutionized our notion of memory management. The functional languages out there help, but you can't forget the underlying VM magic that will really make this possible. Look to the example of memory management: it's not Java itself that enables you to write code without worrying about deleting references only when they are truly not being used, but rather the VM that watches all objects and knows how to find the live ones and delete those that are no longer referenced, all without the developer needing to indicate a thing. For us to conquer the thread problem, we need a VM to observe the behavior of the system beyond just the number of cores and help us to appropriately occupy CPUs over collections of varying size and composition. Languages can force developers to think about applying logic over their data (and thus help with parallelism as a mindset), but they will never solve the problem of automated parallelism without the VM to tune it in the background.

There will always be people that need to tune their code down to exactly what instructions run on what core, just as there are still people that need to do their own memory management, but it shouldn't be a requirement for getting good mileage out of those cores. I can't wait for the day when calling submit feels as archaic as calling malloc.

Yes, and...

I have a terrible habit. When confronted with a new idea, any idea, my knee-jerk instinct is to find its flaws. Or, to put it more succinctly, I'm a "no, wait!" kind of person. You want to add a new feature into my library? No, wait! That won't maintain API backwards-compatibility! No, wait! No one but you really needs that, why should we put it in a library used by everyone? No, wait! The work that will take us to review overwhelms it's worth as a new feature.

There is a well-trod school of thought that says to maintain creative collaboration, anytime you want to say "no, wait!", "no, but" or even just "but", you should instead start the conversation with "Yes, and". Want to add a new feature into my library? Yes, and we will have to think about how to add it without breaking API backwards-compatibility. Yes, and we should see what other community members think about the usefulness of the feature. Yes, and we'll have to evaluate the work that it will take to correctly implement and review.

"Yes, and" is a great baby step on the path to thought leadership and productive collaboration. But let's face it, it's easy to get a bit snarky with our yeses. Yes, and six months later, we'll have another release. Yes, and you can be on hand to fix the bugs that come in. Yes, and why don't we rewrite the whole thing in Scala while we're at it? (Watch Peggy struggle with this on Mad Men[16:50], when Megan shows her work)

So how do you really take "yes, and" to heart? My boss, who is a natural at this sort of thing, taught me one of his favorite tactics. It's not just saying yes. When a new idea is presented, find at least one positive thing to say about it, and praise that quality first.

 "That is an interesting idea. People using our service to do complex state maintenance could really use this feature, it would save them having to write a lot of code themselves."

The goal of this exercise is not only to keep the flow of positive collaboration going. It is also to force yourself to acknowledge the problem that the idea is trying to solve. You may not think the problem is worth solving, but this is something that your collaborator wants to explore, perhaps because it is causing them pain, perhaps because they have a different set of priorities than you, perhaps because they're simply trying to contribute to the discussion.

I will admit that despite knowing this is the right thing to do, I am absolutely terrible at following this advice in person. My knee-jerk nerd comes out and she is very into scoring points by pointing out flaws. Among the tech community, I'm hardly unique. Read the comments on almost any Hacker News thread if you don't believe me. And yet, the easiest place to start with this is online. You have time to think before you hit send on that email, submit on that comment, post on that tweet. Why not take the time to acknowledge the idea fully before pointing out its downsides? Just because you didn't think of it doesn't mean you lose by making it better.

Java console monitoring basics: The "j" series

Think fast: It's 10pm, you have a production java application on a box you can only ssh into, and it's in distress. This is the third time this month it's happened. You didn't write the code, and the joker who did didn't bother to put in any metrics for you to grab. What do you do? After cursing, but before giving up, restarting the process, and promising to debug it in the morning, you might want to go a round with the Java command-line monitoring tools.

Perhaps you are already familiar with these tools, but I've found that despite their incredible usefulness many seasoned Java developers have never heard of this tool stack. They hide away in the jdk bin directory, but they can be your best friend when you are stuck with nothing but a console and a prayer.

First we have the lowly jps. It does what you would expect: shows you java processes running as that user (or all users, if you're root).

Moving up the stack, we have jinfo. Show me the vm version, jars, and all the flags this process is running with, or the value of a particular flag. You may have this information elsewhere, but it's nice to have a shortcut.

More useful is jstack. Yeah, I'm sure you know how to kill -QUIT but this is a nice way to teach the newbie how to get a stack trace without the risk that they'll accidentally kill the process. If you should be so lucky as to have an obvious deadlock, jstack will kindly point that out to you so you can go fix it. Stack traces are the bread and butter of figuring out what's wrong with a process. Take several, see how they change, or don't, and you'll come closer to finding your problem code.

My personal all-time favorite is jstat. More specifically, jstat -gc 3s. That is, jstat of the garbage collector printing new results every 3 seconds. Back in the days when I wrangled gigantic VMs, this tool was invaluable for spot-checking garbage collection. The output is admittedly hideous. For example:

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
37568.0 41984.0 6592.0  0.0   74816.0  38651.6   70272.0    39290.5   63616.0 55956.0     38    1.608   4      1.441    3.049

Yes, that unformatted barf is what you will see on a screen console output for jstat -gc. But that barf tells you a lot. First things first, the most useful stuff is at the very end. See that "4" followed by "1.441"? That shows the stop-the-world GC collections, and the total time they have taken. If your application is running particularly slow, or is frequently unresponsive, the FGC count will quite possibly be high, and the time related to it will be very high. Remember, your app is essentially dead when FGC is running, so a high number of FGCs is a bad sign.

This also shows you the various used and total sizes of the generations. I'm not going to go into details of Java Garbage Collection but it is useful to be able to see them broken out and growing or shrinking. One pattern of GC to watch out for is the case when you don't have enough survivor/eden space to handle all of the transient data you need to do a big chunk of work, and not enough old capacity to take it all, but just enough freeable state to keep slowly moving forward with your computation. The result will be neverending full-gcs while your process moves slightly forward, does a full GC, moves forward again, does a full GC, and moves forward again. This looks in jstat like an ever-increasing series of FGC where each one finishes but only causes a small amount of space to be freed in OU, while EU and the survivor spaces are constantly near-full. The details of the actual GC behavior may change with versions of the JVM (and the flags that you are using), but the ability to monitor the behavior easily in real-time is always useful.

Finally, for the tools I usually use, there's jmap. jmap -heap is a nice way to get a pretty-print of the heap info. jmap -histo and jmap -dump are heavier commands that you might want to hold off on doing until you're ready to restart your process anyway, because sometimes running them will result in bad results for the process. If you're producing a ton of garbage and you don't know why, jmap can show you where the memory is going. jmap -dump will product a file that you can push into something like MAT for analysis.

None of these tools are an answer for proper monitoring of your JVM, but they're great for a quick and dirty debug before you restart a troubled process, and something to make sure your whole team has in their toolbox.

One Feature in One Month or Ten in Six?

The best story I can tell to illustrate the difference between project planning and technical design at a big company vs a small startup happened today. I was talking to our head of analytics about a new project we are speccing out. This project will inevitably be the beachhead of a huge set of new features and systems that we'll want to write over the coming months. He sees this as an opportunity to start building something that will be able to accomplish all of those features in one multi-faceted system. And he's right, this is definitely a feature that we could easily implement in such a system.

However, it's also a feature that we can implement fairly cheaply using building blocks we already have. The cost difference in engineering this feature is pretty stark; my estimation is that to implement it in what we have takes roughly a month, most of which would be spent getting the presentation layer to look good. To implement it in the larger space would probably take closer to 6. If we implemented the larger system, we would get a lot more than just this feature. We would probably get 10 features out of it, and the ability to really start powering several different things.

I'm still going with the one month solution.

On the surface, this seems like a pretty foolish trade. I know that I'm probably going to build the bigger system sooner or later, I know that I would probably get more value for my time building it now, and not building it now means writing some code that will eventually be thrown away. If I were doing this at a big company, I would always choose to do the 6 month project. In fact, I would probably be busy thinking of all the other things we could do with another 6 months, and building up a plan and design that would emphasize the power and complexity of my solution. I'd be aiming for a system that people could use and grow for years to come.

But I don't work at a big company any more. There's no certainty that this larger feature set will actually move the needle on our business. We need to be able to see it in production and learn from the results before we commit resources to engineering a long-term solution. Priorities can change a lot in the span of a month, especially in a data-driven customer-facing business such as ours.

And ironically, I buy myself engineering time by going for the quick 1-month win. That bit about the majority of the time requiring front-end work is the trick. I can slap RESTful endpoints out to serve up and consume this data fairly quickly. While my front-end folks are busy making things pretty, we have the chance to catch our breath and evaluate, start laying groundwork, and eventually swap out the services without the front-end team ever having to know that things have changed. So while it seems like I'm doing one feature in one month, I'm really doing ten features in six and a half months. They may not be exactly the same ten features I would have if I started tomorrow, but that's half the point: By getting one feature out fast I give my business something concrete to evaluate, and I make it more likely that the ten features I eventually implement are the right ten for the job.

Why I'm Moving Away from the Play Framework

I've been using the Play framework since I started at RtR 3 months ago. Last week, I made the decision that no new services will be written in Play from that point forward. It started out as a great little framework that was pretty quick to learn and easy to use, but it's turned into something that I would not recommend anyone use for serious production applications. What happened?

First, I lost faith in the developers.

One of the first things that annoyed me about Play was the inability to run a single test from within a play test class inside your IDE. I suppose the thinking was that you will always run the play test app, or something, but I prefer to leave my IDE as little as possible when I'm working, and running an entire test class worked fine. So, being the good little open source programmer that I am, instead of bitching I rolled up my sleeves and fixed the bug. It was a pretty trivial fix. I even wrote a test case. Then I put in a pull request and waited.
After submitting the pull request, I commented on the pull request, commented on the ticket, and finally sent an email to the mailing list. And the response I got was basically that the team is too busy working on the next generation of the product to absorb fixes for the older generation. Having worked on open source projects myself, I understand what it's like to have limited bandwidth to look at changes. But if the project team's bandwidth is so limited that they can't even afford to look at small fixes like this, it seems like the project is basically abandoned. At that point I lost faith that I could rely on the community to support the 1.X branch of this product. Not necessarily a dealbreaker, but definitely a bad sign. 

Then, I lost faith in the platform.

We started to hit some serious bugs in the platform during a big push on a complex service. First, our developers that used Mac and Windows hit a bug similar to this, where they just simply couldn't get the app to work no matter what they did. It worked fine in linux, but even a clean checkout would fail to run for them. It was inexplicable, irritating, and we lost a couple of days of development work trying to get around it (rolling back checkins, pulling out modules, poring through stack traces). By this point, I had lost faith in the community, so I didn't see the point in going to them for help. Fortunately, we did finally get around it (it seemed to be a bug in the CRUD module), but we were all really frustrated and annoyed with the framwork after that experience.

Finally, I lost faith in my own ability to debug the framework.

The issues above were enough for me to want to move off of Play for new projects. The thing that caused me to move off of Play for projects that are already in development (but not in production) was this: At some point, we had written a migration job in Play for a major data migration. We discovered the strangest thing would happen. The job would run across several job threads, and at some point, one of the threads would hang. But it would not hang in a way that I have ever seen a JVM thread hang. The thread was in RUNNABLE state, and it was in a HashMap method (either get or put) and it was just sitting there. Not doing anything. No locks, no database or other IO, plenty of memory, plenty of resources, just sitting in that HashMap.get method, hanging out.
Now, maybe you've seen that before (and if you have, please leave a comment!). But I have seen a lot of JVM issues in my day, and this is a new one. There was no reason for this thread to be hung. And yet it was. I can debug just about anything you can throw at me in Java, but I had absolutely nowhere to start looking to debug this issue, except a vague suspicion that it was related to the way the framework was rewriting the classes under the covers. That is a dealbreaker, ladies and gents. I could've probably debugged why the module was causing the app to crap out for my developers, if given enough time. But I cannot say with any certainty that I could debug whatever the hell was causing that thread to hang.

If I felt the developers supporting play were committed to building a real community of support around the 1.X version, I might have stayed with it longer. It's a giant pain to find something else that is easy, lightweight, supports JPA and doesn't force me to write XML. But I can't use a product that I know has issues even I can't debug, and a team I don't trust to maintain the product to the standards my team needs to confidently use it in production.

Three Reasons You Should Be Training Your Successor

It's Friday afternoon of a long week. Despite wanting to pack up your bag and go home, you know you need to review the new hire's code. It's their first week so inevitably it will be some scary stuff for you to untangle. When you get in there, it is scary. But not in the way you expected. Instead of causing more bugs than they fixed, they've actually managed to fix every bug you gave them and in the process redesign a hairy part of the system into something rather... elegant. You want to find fault somewhere, but you can't. They're really good, so good that you're feeling threatened. I want to convince you that this threat is an opportunity. When you find that up-and-comer on your team, instead of trying to prove that you are better than them, train them to replace you. Why? Here are three major reasons:

1) Career Growth 
It's hard to grow yourself and your career within a company if you hoard the knowledge or ownership of a particular project. The people that succeed the best at growing into bigger roles are those that can point to a protégé waiting in the wings to fill the position they are leaving behind. This applies to us tech types that may not want to grow into managers, but want the freedom to move into new projects. It's easy to get stuck on the same codebase when you're the only expert in the area. Training other smart developers on the team lets you move on to new and exciting things as the opportunity arises, and with no guilt about leaving people in the lurch.
Even if you prefer to switch companies every few years, it is always great to be able show a track record of several successful projects per company, and it gives you the opportunity to get a good breadth of experience.

2) Guru Points
Teaching and mentoring others makes us learn what we know even better. I always find that any time I have to give a teaching talk on a topic, I learn that topic a little bit more. Explaining things to others formalizes what you know, and in watching them learn they will almost always teach you something as well. Do you want to grow into the guru of your team or company? Mentor others and teach them all you know, and before long you may find yourself the most trusted advisor of the team.

3) A Stronger Network
Helping others grows your network, and provides you a pool of talent that you can draw on throughout your career. In a big company, knowing people on other teams that trust and respect you makes cross-company collaboration much more effective. If you are the sort to move between small companies every few years, it's likely you will need to hire talent to work with you and make those companies successful, and who better than your former mentees? And if one of those that you mentor grows to become a great success? They are not likely to forget the people that helped to get them where they are, and can be a very valuable part of your professional network.

This post is dedicated to my mentor Mike, who taught me this rule and practically every other useful thing I know. Please leave a comment here or on twitter!

Networking woes in Java

The only major CS subject I never took a class in was networking. It's kind of ridiculous, looking back, that I took as many systems classes as I did but always eschewed networking. I do own a copy of UNIX Network Programming: Networking APIs: Sockets and XTI; Volume 1, bought at some point in the past when I knew I was going to be doing some distributed systems work and figured it would be a useful reference. But I can't say it's been my constant companion. For I have learned one thing in my years of Java systems coding:

Networking code is HARD.

Here's exhibit A: ZooKeeper monitoring misuses sockets. I spent a good chunk of time desperately trying to figure out why my monitoring commands were crapping out halfway through when run from NY to LN. Turns out, you can't safely expect to just close half a socket, leave the other half open, push some data to it and then close it while seeing all the data through to the other side. Not without a final handshake indicating the client has gotten all the data. Or at least, I think that's the case. The thing is, this will work well enough over a very fast network connection or with very little data. The guarantees around so_linger etc change kernel to kernel and my reading at the time led me to think that in fact the standard linux kernel behavior in this case may well have changed over the years that ZooKeeper has been around. So we need to completely rip out and redo the monitoring code if we want to have any hope of this working right for other big, global deployments in the future.

Exhibit B is my current debugging nightmare. Part of our release last week involved a new backend Play service that itself connects to a different backend Play service to prepare results for our storefront. We noticed, several hours after launch, that the service started to throw exceptions that were ultimately caused by java.io.IOException: Too many open files. I know enough about Java to know that running out of file descriptors is often a Bad Thing. 

So we're leaking sockets. Why? To date, we don't know. The underlying libraries are async-http-client and netty, but there's very little to indicate what is going on.¹ The sockets show up in netstat/lsof as ESTABLISHED TCP connections to the various storefront servers. But the storefront servers do not have most of these sockets open on their end. How are they ESTABLISHED with no partner? It's an ongoing mystery, one that we haven't been able to reproduce on any other machine (the current theory is bad network hardware/software at the lowest levels, but honestly that's just a shot in the dark and one that we can't verify without taking down a production service).

So, while I keep debugging, what are the takeaways here?

1) You shouldn't write your own socket handling code in Java. Really, no. Don't do it. Use Netty. It's very good. Of all the things not to reinvent yourself, I would put networking at the top of the list with a bullet. It's hard, and requires the kind of deep expertise that you can't fake. And, when you fake it, you may end up with something like our ZooKeeper monitoring, that seems to work for years while hiding small but significant bugs.

2) If you're a system architect writing any kind of web services/distributed system architecture, you should know your unix socket monitoring commands. lsof is obtuse but powerful. netstat is simpler and still quite useful. This article has a few others, like ss and iftop. Know how to up the ulimits for your processes in case you find yourself with a slow socket leak that you need time to debug.

Have an idea what my bug is? I'd love to hear it! Leave me a comment or hit me up on twitter!

Edit 2/27: Looks like our bug was indeed on the cloud vendor side; possibly a misconfigured firewall. Moving to a new box and rebuilding the box we were on solved it.

¹ Thank God Play is at least using good networking libraries, because the last time I tested ZooKeeper, when it runs out of sockets the service hard fails with almost no indication of what happened.