blog.mike.axiak.net

My girlfriend is into Operations Research. Like any boyfriend, I can’t help but get somewhat interested in what she’s doing. Over the summer, she let me read The Goal. Other than the fact that about half way through it got very repetitive, it was a good read. But they were trying to teach us something. (Repetitio est mater studiorum!)

In O.R. (Operations Research), one type of thing you learn a lot about a lot is the factory. Factories are complicated entities, what with all the complicated dependency chains, asynchronous processing, multiple failure points, and deadlines just to mention a few of the problems. I learned about this buzzword-process called the Theory of Constraints about six months ago, and have been pondering about the importance of factory analysis ever since.

Fast forward to today. At the company I work for, we have a problem with processing of logs. It’s complicated. There are at least 4 different sets of rules with their own systems. Every night every single entry in our >50 million rows of entries need to be scanned and analyzed. This isn’t the difficult part, though. We recently moved to a second (and third) data center. So of course, our problems became exponentially worse. Now, I had to start moving computation around (boy do I wish I had the infrastructure for Hadoop/MapReduce). Things start failing more and more often. (Hard disk error, network failure, etc.) How do we make this system more robust?

I made a realization: We’re undergoing the same set of problems that factories undergo. A small failure in one spot propagates and ruins the rest of the system. Our initial approach (try to make everything “work” all the time—I know, naïve) is just staring to crumble. After trying to make errors more visible and making us respond earlier, I finally realized what the problem was:

Our batch sizes are too large.

Maybe this is common sense, but I don’t think it is for most people. After all, this is a very fundamental realization in Operations Management (for stick figures, read the summary section of this page). Currently, we do all of the processing nightly. This is nice for a couple reasons:

1. Simple (thus less prone to breakage)
2. Processing runs during hours when the machines aren’t under load [1].

However, there’s one this this is not good for: recoverability [2]. As any system grows—as more and more dependencies get added on—the mean time between some failure will reduce. This is a given. How we handle those failures is what matters. Do we let it propagate to the end user? Do we force it to repetitively wake up our sysadmin every night at 2am? I believe we can answer both of these with a resounding NO. We should build the system to expect failure. To do this, though, we have to reduce the batch sizes. So my new system will contain two major changes: