James on Software

Memory Allocators 101

2013-05-15T08:37:02-07:00

For the last few weeks, I've been working on a couple of patches to tcmalloc, Google's super high performance memory allocator. I'm going to post about them soon, but first I thought it would be cool to give some background about what a memory allocator actually does. So, if you've ever wondered what happens when you call malloc or free, read on.

A memory allocator's responsibility is to manage free blocks of memory. If you've never read a malloc implementation, you may have assumed that calling free simply causes memory to be released to the operating system. But acquiring memory from the OS has a cost, so allocators tend to keep free chunks around for a while for possible re-use before deciding to release them.

Managing freed memory is an incredibly interesting and hard problem with two main concerns: performance and reducing heap fragmentation / waste:

How do we organize free blocks of memory such that we can quickly locate a sufficiently large block (or determine that we lack one) when someone calls malloc without making calls to free prohibitively expensive?
What can we do to reduce fragmentation and waste in the face of sometimes drastically changing allocation patterns over the lifetime of a (potentially long-running) program? It's worth noting that heap fragmentation can have a substantial impact on CPU cache efficiency.
As a bonus, there's also the matter of concurrency, but that's probably beyond the scope of this post.

The most fun part of this problem is that our two primary objectives are often in direct opposition. For example, keeping one linked list of free blocks per allocation size (say, rounded up to number of pages) can make calls to malloc best case constant time, but unless some waste is accepted and chunks are kept around long enough to be reused, the worst case path will be taken more often than not.

Of course, there also are a multitude of other issues to consider, such as how to decide to release memory to the operating system, and how to avoid becoming the bottleneck in a concurrent program (I'm looking at you, glibc). And the implementation details are interesting, too.

Implementation

A very basic malloc implementation might use the linux system call sbrk(2) to acquire memory from the operating system and a linked list to store free chunks. That would make calls to free constant time, but malloc would be O(n).

Of course, the allocator needs to store metadata about each chunk it manages, such as its size, free/in-use status, free-list pointer(s), etc. But since you can't exactly call malloc in an allocator, it's common to store metadata in a "header" that just precedes the address that is handed to the application. So, if the header is 16 bytes in size, then the header would start at ptr - 16. Pointer arithmetic galore.

In order to reduce fragmentation and promote memory reuse, it's common for malloc implementations to attempt to coalesce free blocks of memory with adjacent ones, if they happen to also be free. If metadata is being stored in a header, then it's easy to determine the size and status of the pointer to the right.

But the header doesn't provide any way of determining the size of the chunk to the left, so coalescing mallocs frequently put the size of each block in a footer, which is typically sized to fit a size_t. Then finding the chunk to the left would look something like this:

Things get even more complicated because subsequent invocations of system calls like sbrk or mmap aren't guaranteed to return contiguous virtual addresses. So, when looking for chunks to coalesce, care has to be taken to make sure that invalid pointers aren't dereferenced by adding to a pointer that's on the edge of what's being managed.

Typically this means creating and maintaining a separate data structure with which to keep track of the regions of virtual address space that the allocator is managing. Some allocators, such as tcmalloc, simply store their metadata in that data structure rather than in headers and footers, which avoids a lot of error-prone pointer arithmetic.

I could probably continue writing about this forever, but this seems like a good place to stop for now. If your interest is piqued and you'd like to learn more about memory allocators, I highly recommend diving in to writing your own malloc implementation. It's a challenging project, but it's fun and it'll give you a lot of insight in to an important part of how your computer works.

Soon I'll follow up on this post with one about the allocator work I've been doing lately. Also, if you're interested in this kind of stuff, check out my new podcast where Joe Damato and I talk about systems programming. We'll definitely be covering allocators in the next few weeks sometime.

Some allocator resources:

malloc(3) man page
http://www.cs.cmu.edu/afs/cs/academic/class/15213-f10/www/lectures/17-allocation-basic.pdf
tcmalloc docs
glibc malloc source — this is some random github repo, so I have no idea how much its been fucked with

Introducing The Real Talk Podcast

2013-04-28T21:26:42-07:00

[Joe Damato] and I have released the inaugural episode of our new, highly technical podcast realtalk.io.

We will be doing frequent technical deep dives and releasing our conversations raw and unedited with all errors, omissions, awkward pauses, and curse words intact.

Check out the website, soundcloud, and subscribe.

MRI's Method Caches

2013-04-14T09:01:14-07:00

tl;dr

Method resolution is expensive, so method caches are crucial to invocation performance.
Your Ruby code probably calls methods kind of often, so invocation performance matters.
MRI's method cache invalidation strategy is quite naive, leading to very low hit rates in most Ruby code.
I wrote some patches that substantially improve the situation.
This blog post is surprisingly uninflammatory.

The Long Version

One of MRI's big performance problems is that method cache expiry is global. That is, any time you make a change to any class anywhere, the entire VM's method caches get busted at the same time. This is why you'll frequently hear people saying that "calling Object#extend is bad".

Actually, it's not just Object#extend. Charlie Somerville put together what I believe to be an exhaustive list of things that clear MRI's method caches. Method cache busting is so pervasive that it's almost impossible to avoid using somebody's code that does it. Disaster.

Let's back up for a second, though. What is a method cache and why are they important?

Method Cache Basics

Take the following class hierarchy:

Internally, MRI stores methods in a hash table on the RClass struct. When you call an inherited method on a descendent, MRI has to walk up the class hierarchy to find it, checking the method table at each step to see if there's anything there to call.

So, in our above example, if we wanted to call hello on an instance of E, MRI would have to execute method lookups on E, D, C, B, only to finally find the method in A's method table.

It turns out that method resolution is actually quite expensive, which is why method caches exist. Rather than resolving a method each time we want to call it, we cache a reference to the method somewhere we can get to it cheaply, substantially reducing the cost of subsequent invocations.

But Ruby is dynamic, so those caches can't necessarily live forever. If we call String#gsub, for example, and then undef it without expiring the method cache, it'll still be reachable. Cache invalidation is hard, as we know, so MRI takes a somewhat brute force approach.

How MRI's Method Caches Work

Currently, MRI has two types of method caches. Ruby code is compiled down to MRI's instructions. Each instruction has some data associated with it. The send instruction has an iseq_inline_cache_entry, which acts as an inline method cache.

You can read the logic here. Basically, it works like this: look at the inline cache. If it's valid, use it. If not, go actually look up the method and cache it. Pretty much exactly what you'd expect.

In the case of an inline instruction cache miss, there's actually a secondary, global method cache. Oddly, though, the global method cache is limited to 2048 entries, and its semantics for deciding what to keep and what to dump are essentially random.

It's not as unlikely as you might hope for two methods in a tight loop to be clobbering each others' entries in the global method cache table.

Both caches' entries have a field that stores the ruby_vm_global_state_version from when they were filled. A cache entry is considered valid if it matches the klass pointer and the current ruby_vm_global_state_version. So, incrementing the global state version by 1 invalidates all of the inline instruction caches as well as the global method cache.

This has the effect of making invalidation very cheap, but far reaching. Whenever you make a change to any class, call extend, or do any of the other things detailed in Charlie's article, all of the method caches that have built up since your program started become invalid and you have to repay the cost of method resolution all over again.

The Numbers

After years of complaining about Ruby's method caching behaviour, I finally decided to instrument it a couple of weeks ago. I found that for our application, the method cache was being invalidated at least 20 times per request, and that around 10% of our request profile was spent performing method resolution. For our application, the cost of Ruby's global method cache invalidation was extremely high.

The average cost of each method resolution for our production application is around one microsecond, which doesn't sound like a lot but it adds up. We were seeing at least 8000 cache misses per request, totalling 8ms or more.

As part of my instrumentation patchset, I also created a mechanism that logs a stacktrace each time the method cache is invalidated. I found that the majority of invalidations in our app were from inside of ActiveRecord - some easier to fix than others. Many were also caused by random gems doing things like instantiating OpenStructs.

At this point, it started seeming somewhat impractical to go and patch rails and all these other gems that I use, so I decided to investigate the amount of effort that would be required to actually solve the problem in MRI.

Hierarchical Method Cache Invalidation

I'll be using class to mean class or module here, since they have the same backing structure in the VM.

Ruby's inheritance tree is a directed acyclic graph, and the semantics of method resolution mean that a change to a given class only affects it and its descendents. So, in an ideal scenario, we would only need to invalidate the method caches for those branches of the inheritance tree.

I've written a patch for MRI that implements such an algorithm, and it's currently serving 100% of our production traffic. We've seen around a 9% reduction in average latency with this patch. Others who've tried it haven't seen such big jumps. Your mileage may vary.

The algorithm is actually quite simple (credit to Charlie Nutter / JRuby for the idea). We have a 64 bit global (one per VM instance) sequence that is monotonically increasing. Every time we alloc a new RClass, we increment the sequence, and assign the class a unique value.

Method and inline cache entries are tagged with the class's sequence value when they're filled. When a class is modified, we traverse the class hierarchy downwards from the modification point, assigning each class a new sequence number.

A method cache entry is considered valid if its sequence number matches the current sequence of the class. So, if the class or one of its parents has been modified since the cache entry was created, its class will have a new sequence number, and it will have therefore been invalidated.

Performance

For our application, this cache invalidation strategy substantially reduces the number of method cache misses we see in production and has reduced request latency by ~8-9%, but there are tradeoffs involved. Since invalidation requires a graph traversal, it's a lot more expensive than the current strategy of merely incrementing an integer.

If your application makes frequent modifications to classes and modules which have a large number of descendents, the cost of invalidation may outweigh the increase in method cache hit rate. That said, I would imagine that such modifications are relatively uncommon and should be considered a bad practice either way.

It's also worth noting here that while this patch will likely improve the performance of apps that employ the strategy of extending arbitrary objects to implement DCI, that pattern is still a performance problem, because it creates tons of one-off metaclasses whose methods wind up being mostly uncacheable.

The Code

My patchset includes several things:

Subclass tracking: Class#subclasses, and Module#included_in. Rails implements this with an O(n) traversal of ObjectSpace. With my patches, that's no longer necessary.
Hierarchical method cache invalidation: the subject of this whole article.
Method cache instrumentation: RubyVM::MethodCache has several useful singleton methods you may want to track, including hits, misses, miss_time, invalidation_time, etc.

You can find the code in my branch or install it with rvm install jamesgolick.

I am planning to submit these patches back upstream, but I have to port them to Ruby 2.0 first, so I guess that's my next project. Huge thanks and credit to Aman Gupta, Charlie Somerville, Charles Nutter, and funny-falcon for all their code, help, and testing!

The Cost of Ruby 1.9.3's GC::Profiler

2012-11-19T12:13:49-08:00

This is a long one, and y'all are busy I'm sure so here's the tl;dr:

If you run ruby in production, you need to keep track of GC stats.
Ruby 1.9.3's GC::Profiler does a bunch of really weird shit.
- It keeps a 104 byte sample of every GC run since it was enabled forever.
- Calling GC::Profiler.total_time loops over every sample in memory to calculate the total.
- The space used to keep those samples in memory is never freed. However, it does get reused when you call GC::Profiler.clear.
- Therefore: if you are using GC::Profiler in production, and you're not calling GC::Profiler.clear regularly, you're leaking a substantial amount of memory (>1GB / machine for us), slowing down garbage collection somewhat, and the cost of retreiving the stats (GC::Profiler.total_time) will continue to increase unbounded until the process is restarted.
I am working on an alternative, low-overhead GC Profiler that is designed to be run in production. It's called GC::BasicProfiler. You can find the patch here and follow development here.
Also, you may want to check out this fork for some backports from ruby 2.0 — including the COW-friendly garbage collector. Good stuff.

The Long Version

Ruby's GC is a steaming pile of shit — but that's not news to anybody. If you're running ruby in production, tracking GC behaviour is essential so that you can minimize its effects on perceived performance (hence oob_gc, etc). Fortunately, Ruby 1.9.3 ships with GC::Profiler, which provides detailed instrumentation on GC runs.

Over the last month or so, I've been working on some rails performance tooling. Last night, I noticed that requests with my instrumentation enabled were taking around an order of magnitude longer than those without it. Weird. So, I installed perftools.rb and rack-perftools_profiler and got a really surprising result (irrelevant lines ommitted):

Apparently calling GC::Profiler.total_time is so slow that more than 50% of request time was spent in there? Is that actually possible? My instrumentation calls GC::Profiler.total_time frequently under the assumption that it's inexpensive, but obviously that was a faulty assumption unless perftools.rb is wrong. Let's take a look at the implementation. (the code in context is here)

There's a loop in total_time? What the fuck is going on here?

Turns out that if you have GC::Profiler enabled, the VM records a gc_profile_record every time the garbage collector runs.

Then, when you call total_time, it loops over all of the gc_profile_records that have been created in order to sum the total. According to my profile, total_time was responsible for more than 50% of request time. How many gc_profile_records could there actually be?

Oh. Well I guess that explains that. So — stupid question, but when do these things get freed? Apparently only in rb_objspace_free which only ever gets called when the VM terminates, so the answer is never. Cool.

Upon further investigation, it's pretty clear that this whole system was designed with the expectation that you'd call GC::Profiler.clear regularly. The profiler keeps its samples in an array at objspace->profile.record that it increases in size by 1000 every time it runs out of space.

If you don't call GC::Profiler.clear, that array keeps increasing in size forever. This is not documented. Obviously.

On our production systems, unicorn workers that have been running for a few hours had an objspace->profile.size of around 350000. On x86_64, sizeof(struct gc_profile_record) == 104, so around 35MB of overhead per process multiplied by 25 processes per machine for a total of nearly 1GB per machine — after only 3 hours. That will grow forever until the processes are restarted.

That's the bad news.

The good news: GC::BasicProfiler

Ultimately, GC::Profiler was designed to provide detailed information about every GC run — probably for the VM implementers to use when tuning the GC (haha yeah right). But seriously, somebody probably wants that, but it isn't me. For those of us who simply want to keep track of GC stats on our production applications, we need a less expensive implementation.

GC::BasicProfiler is a first step towards something like that. It has a very simple, low-overhead implementation, and GC::BasicProfiler.total_time works the way you might expect.

Enabling and disabling BasicProfiler works exactly the same as Profiler but you don't need to call clear to avoid leaking memory. In fact, there's no clear method at all.

If you're interested in following the development of this patch, it'll be here.

One more lol for the road

It's the little things. Don't worry, though — fixed in BasicProfiler.

Moving On

2012-09-05T11:33:09-07:00

Almost four years ago, I was speaking at a software engineering conference in Montreal. At the speakers lunches, I met up with one of the founders of the conference, and we immediately hit it off. He told me about his growing company, and a month later, the consulting firm I'd been running was closed, our office vacant, and I had joined BitLove (the company that runs FetLife — which was then known as Protose) as CTO. It's bittersweet to announce that as of a few weeks ago, I've decided to move on.

Over the last four years, I played a huge role in every part of running FetLife. In addition to being responsible for our technology, I made business and product decisions, helped design features, wrote copy, communicated with the community, worked on support stuff, and more. I've always had an interest in — and read about — all this stuff, but actually having the opportunity to participate in it, make real mistakes, and have real successes was incredible.

As a technologist, it's hard to imagine somewhere I could've grown more quickly. When I joined, I knew a few things about writing Rails apps. While I was there, I got the opportunity to do everything with every part of the stack. I learned how to make it all run in production for a big user base and a lot of traffic, with a tiny team.

When I joined, we had ~100k users (I got user ID 129315) and Rails was serving around 50 million requests a month. Since then, our user base grew to over 1.5 million users — our traffic to almost 500 million pageviews a month and over 1 billion Rails requests (not to mention requests to other services like chat). We did it with an engineering team that hovered around 2 people (including me).

I'm really proud of the engineering work I did at BitLove. We operated an extremely high throughput MySQL installation, and I was able to solve various InnoDB scalability limitations that we encountered. I implemented a web-based IM system (similar to Facebook Chat) that hundreds of thousands of people use to send tens of millions of messages every month. I built everything from the presence and routing implementation in Erlang to the UI in Javascript. I also designed and built an extremely stable and fast activity stream architecture, almost single-handedly ran operations for the ~40 machine cluster in around 2 hours a week, and perhaps most importantly, I dramatically improved my health in the process.

The work I did at BitLove certainly represents the biggest challenges and accomplishments of my life and career to date. It was a wild ride with many ups and downs — everything they promised a startup would be. So it was incredibly difficult to leave a growing and successful company that I had a big hand in building. But it's time for new challenges.

So what's next?

I've got a couple of really amazing opportunities on the table right now that I'm super excited about. Because I get so heavily invested in my work and like to stick around companies for many years, this is a very big decision, and I'm not taking it lightly. I'm certainly open to hearing about any opportunities you think I might be a fit for, so do get in touch!

Consulting

In the meantime, I'm available for consulting work. If your company needs help with any of the kinds of things I discussed above — especially performance and scalability stuff, we should chat. Email me.

InnoDB kernel_mutex Contention and Memory Allocators

2012-07-18T15:41:41-07:00

tl;dr: We found that in our case, contention for InnoDB's kernel_mutex was caused by contention for a malloc arena lock. We fixed it by moving to tcmalloc. Instructions on how to do that here.

We recently doubled the IO throughput capacity of our near-capacity MySQL master by adding a second RAID controller, and striping the two together. As we were climbing up to a record throughput peak the following weekend, there was a major db latency spike (>3x).

A look at SHOW ENGINE INNODB STATUS indicated quite a bit of contention for InnoDB's kernel_mutex.

Note: the contention I observed was actually considerably worse than what I pasted above, but I didn't save the output, so this is all I have to show.

The kernel_mutex has been removed in MySQL 5.6, but that's unfortunately not ready for production. As a workaround, the Percona guys suggest modifying innodb_sync_spin_loops, which had absolutely no effect for our workload. They also suggest lowering innodb_thread_concurrency, which did reduce contention, but it also reduced concurrency, which left us right back where we started.

I pulled out my poor man's profiler to see if I could figure out exactly what was holding the lock and what it was doing with it. Here are the stacks I got.

Immediately, we can see that lots of stuff is waiting on locks inside of malloc/free-related functions. After reading through the MySQL sources, it was clear that this thread was holding the kernel_mutex.

Note: all links to glibc code below are specifically to the version that we are using.

Reading through _int_free in the glibc sources seemed to indicate that there was only one lock (malloc_state->mutex) in there.

Our glibc was built with --enable-experimental-malloc, which is supposed to reduce contention by dividing the heap in to multiple arenas, each with their own lock (at least as far as I understand it — and I'm far from an expert).

tcmalloc is a malloc implementation from google-perftools that satisfies small malloc requests without locks by using a per-thread cache. Using tcmalloc should mean that the allocations inside the kernel_mutex are (at least mostly) lockless.

Here's how to LD_PRELOAD tcmalloc.

Put this in /usr/local/bin/mysqld_wrapper:

Put this fragment in my.cnf:

Since we moved to tcmalloc, all of the contention for the kernel_mutex has completely disappeared. We're also seeing better performance overall and using ~15% less memory in total. This fix probably isn't applicable in all cases, but if you're seeing kernel_mutex contention, it's worth using your poor man's profiler to see whether swapping allocators might help.

How to Lose 100 Pounds

2012-07-07T10:57:13-07:00

I've struggled with my weight for nearly my entire life. I went from being chubby in elementary school to overweight in high school to obese in university. At my biggest, I was almost 280 pounds (I'm 5'6"). Finally, around 5 years ago, I got a spark of inspiration that ultimately led to me dropping a total of 110 pounds (and counting). Here's how I did it.

But first, the obligatory before and after shots:

Motivation

Losing weight requires an enormous amount of motivation. You're going to have to change your lifestyle and make real sacrifices. It's going to be hard. Motivation will help you continue to justify the changes you've made, and prevent you from slipping back in to old habits.

Funny enough, I actually got my first seed of motivation from pneumonia. I was 278 pounds at the time. After three horrible, bed-ridden weeks, I was down to 258. It was painful, but it taught me the most important weight loss lesson of all: it's possible.

Like a lot of other kids from my generation, I grew up overweight. When you can't remember a time when you weren't, being fat is a part of your identity. So, silly as it sounds, I think there was a part of me that believed that weight loss was impossible on some level — or at least that the amount of weight I needed to lose was insurmountable.

If you only take one thing away from this article, let it be that. You can lose weight. No matter how fucked up your metabolism (more on that later), no matter how long you've been overweight, it is possible.

Strategies

I'm going to talk about a few of the strategies, diets, and other random things that I have tried because I think people will find them interesting. But I'll give you an easy way out of reading the rest of this article just in case you're already bored. Ready? Here it is.

STOP EATING PROCESSED FOOD. THAT INCLUDES SUGAR, WHEAT PRODUCTS, SUGAR REPLACEMENTS LIKE SUCRALOSE, ASPARTAME, ETC, AND EVERYTHING ELSE YOU'RE THINKING OF THAT MIGHT BE AN EXCEPTION. EXCEPT STEVIA. YOU CAN HAVE STEVIA.

Ok, so with that yelling out of the way, here's a bit about my journey.

Briefly On Exercise

I'm going to keep this short. Exercise has never helped me lose weight. For much of the time that I was grossly over weight, I was also extremely physically active, often whitewater kayaking or downhill skiing for several hours 4 or 5 days a week, and continuing to put on fat. Despite conventional wisdom to the contrary, exercise isn't an effective weight loss strategy for me.

Portion Control

After I lost the pneumonia weight, I was literally terrified that I might put it back on. So I decided to try eating less. I ate all the same things, but avoided going back for seconds. I ate pasta, pizza, and dessert until I was full, but not stuffed. I lost 20 more pounds over a few months. Then, it leveled off.

That, really, is the story of my weight loss effort. Strategies, and diets that work for a while and then plateau. Sometimes, it's possible to break through a plateau, but other times, you need to up your game with better eating.

I tried for another six or so months to break through the portion control plateau. It never happened. I was actually feeling pretty good about where I was, though, so I didn't really make much of an effort to progress for a few more months.

Lower Carb Diet

Shortly after moving from Montreal to Vancouver, I started seeing a personal trainer, hoping to accelerate my progress on the scale and in the gym. She had me keep a food journal, and immediately picked up on the amount of carbs that I was eating back then. I was vegetarian at the time, and I was eating tons of breads and pastas. She told me to eat more vegetables, and tofu, and watch my carb intake. I lost about 20 pounds before plateauing hard.

On this diet, I was still eating bread, pasta, and sugar, just less. And after a while, I found it impossible to continue losing weight. So I started looking for other solutions.

Eat to Live

Eat to Live is an all vegan diet designed by Dr. Joel Fuhrman. Only fruits, vegetables, legumes, nuts, and seeds are allowed; no oils, dairy, sugar, or even juices (bit of an exaggeration, but for our purposes this is accurate enough) are permitted. I had very mixed results on Eat to Live. I did lose about 15 pounds, but I had a very difficult time keeping it off, and found it very difficult to eat enough food to feel full for more than an hour at a time. I found that I was constantly eating, and still often feeling starvingly hungry.

With that said, I actually know a lot of people who've had great success on ETL including my ex-girlfriend, who I was living with at the time (so we were eating nearly identically, though me significantly more than her), and my good friend Giles, who actually introduced me to the book. Which brings me to another one of my weight loss conclusions.

Everybody's body is different. Some people have amazing success on a diet, while others are incapable of losing weight. I no longer believe that there's one perfect diet out there that suits everybody. Your mileage will vary with every approach.

The only consistent thing I've been able to identify across all my friends and family who've lost weight is avoiding processed foods.

Psoriasis and acne

An interesting aside here is that ETL led me discover that it's possible to control psoriasis with diet. The medical community doesn't seem to be aware of this, but I am completely psoriasis free after years of being covered in it.

At first, I thought that it was the greens that caused my skin to clear up, but since then, I've realized that it's a balance of factors. Greens do help, but merely avoiding processed foods is enough to keep me completely psoriasis free. That being said, I started drinking coffee again a little while ago, and noticed that a small amount of psoriasis came back. Upping my intake of greens seems to make it clear up. So, it's a bit of a balancing act.

Oh also, I'm extremely prone to acne, but I've found that avoiding high glycemic index foods keeps my face and body completely clear of pimples.

On Vegetarianism

I'm definitely going to get hate mail for this, but here goes anyway. I was vegetarian for most of my weight loss journey. My conclusion was ultimately that vegetarianism made it significantly more difficult to lose weight. Here's why.

At home, cooking my own meals from my own groceries, vegetarianism was perfectly fine. But, every time I ate in a restaurant, on the street, or even at a friend's place, my options were nearly invariably some combination of pasta, bread, and sugar. I probably have the shittiest metabolism in the world, but when I eat that stuff, I gain weight. Lots of it.

I really enjoy eating in restaurants, which made the whole thing all the more difficult. During the whole time that I was on Eat to Live, I would painstakingly lose 7 or 8 pounds by religiously sticking to the diet for a month, then travel to a conference for a week and gain 15. It was frustrating to say the least, which led me to the very difficult conclusion that I needed to at least try breaking my nearly ten years of vegetarianism.

My Current Diet

My current diet is really simple: no processed carbs (that includes 'carbless' sugar replacements except stevia). I go through periods where I eat a ton of fruits and vegetables, but lately, I've mostly been eating meat and fish.

Do I miss chocolate and ice cream? Definitely. But I eat guilt-free bacon or chicken wings whenever I feel like it, and seeing results makes the sacrifice more than worthwhile.

This diet means that when I go out to eat (which I do regularly), I can have a steak without feeling guilty. I tell people that I'm allergic to sugar and flour, which gives me a reasonable excuse for being the pain in the ass guy who has to ask the waiter about the ingredients in every dish on the menu. I'd encourage you to tell similar lies if they help you stick to a diet.

There've been a few periods over the last year and a half where I've started eating bread again and gained back a bunch of weight. In April of this year, though, I finally committed to this diet as a more permanent lifestyle, and have been ever since. I've dropped around 40 pounds since then, and I'm not stopping until I can see my abs.

Conclusions

Everybody's body is different. Your friends may have had success with diet X, but you may not. Don't let that discourage you. You'll find something that works.

The best diet is the one that you can stick to, even if the weight loss is slower. If a diet fights against your lifestyle, it's going to be that much harder to maintain. That was my problem with ETL. I love to eat out and I travel a lot, so I couldn't stick to it. And at the end of the day, I didn't lose weight. The less you have to change your lifestyle to accomplish your goals, the better your chances of success.

You can lose weight, still enjoy the food you eat, and even go out to restaurants while you do it. Obviously, you won't be able to eat everything you're eating now, because if you could, you'd already be thin. But, it'll be a sacrifice worth making. The best thing you've ever done.

Objectify: A Better Way to Build Rails Applications

2012-05-22T11:36:49-07:00

For almost 6 years, the dominant "best practice" for building rails applications has been skinny controller, fat model. In other words, put all of your business logic in to your models — keeping it out of your controllers. The result is typically a small number of bloated objects that are impossible to reason about or test in isolation ^[1].

That property is important. To understand why, let's take a quick and highly selective look at the origins of object oriented programming.

On Encapsulation

One of the early papers that emphasized the importance of encapsulation in software development was James H. Morris Jr.'s Protection in Programming Languages. He argued that if we were going to be able to write correct software, “programs” (probably most analogous to objects in the OOP world) needed “protection” from each other.

...hostility is not a necessary precondition for catastrophic interference between programs. An undebugged program, coexisting with other programs, might as well be regarded as having been written by a malicious enemy—even if all the programs have the same author!

We offer the following as a desideratum for a programming system: A programmer should be able to prove that his programs have various properties and do not malfunction, solely on the basis of what he can see from his private balliwick.

The idea is that if we can prove that components work in isolation, then we have a better chance of having a functioning system when we assemble them in to a larger whole.

Aside from being central to the thesis in a paper that heavily influenced the development of object oriented programming itself, it doesn't seem like a stretch to argue that components that are provably correct in isolation would make a good building block from which to build working systems ^[2].

We can derive a lot of good object oriented practices from this idea. Two that are relevant here:

Single responsibility principle (SRP): To be able to prove that an object works correctly in isolation, that object's behaviour has to be clearly defined. The more responsibilities an object has, the more complex its behaviour becomes, and is therefore more difficult to prove and reason about.

For example, if we put all of our business logic in ActiveRecord::Base subclasses, the burden of proving that our code is correct becomes immense because we expose ourselves to mistaken interactions with all of the behaviour we've inherited. ActiveRecord::Base already has a responsibility: persistence.
Dependency injection (DI): Many objects have dependencies — other objects that they collaborate with to accomplish their goals. If those dependencies aren't configurable in some way, then we can't test the behaviour of an object without implicitly testing its collaborators as well. By injecting our dependencies, we can easily replace them with alternative implementations. In our tests, we can inject fake objects in order to isolate the object in question.

In Search of Something Better

So, how should we organize our applications? There are a few approaches floating around.

One in particular is something I've been thinking about and refining for a while now ^[3]. In this approach, persistence objects remain extremely thin, and business logic is encapsulated in lots of very simple objects known as “services” and “policies”. Not all objects in this methodology will fit in to one of those two categories, but they are two of the most important concepts.

Service objects are responsible for retrieving and/or manipulating data — essentially any work that needs to be done that you might have put in a controller or model object before. They typically only have one method, which I like to define as “#call” (they're usually named something like PictureCreationService, so naming the method #create would seem redundant).

Policy objects are responsible for enforcing access control policies. We use them in before filters to guard controller actions, reuse them in views to conditionally display pieces of UI (example: a delete button that requires administrative privileges), and anywhere else policies need enforcing, like background jobs. Policy objects only ever have one public method: “#allowed?”.

Composability

Decomposing the behaviour of our rails application makes it extremely simple for us to prove that our objects work in isolation because we're adhering to SRP. As a bonus, since all of our unit tests inject test doubles — that don't actually do any real work — in place of real collaborators, our tests are extremely fast.

There's one other major benefit to this approach: that which has been decomposed can also be recomposed. In other words, because our behaviours are factored in to many focused objects instead of a small number of bloated ones, we can (and do) recompose them to create other things. If you set out to build a User object, all you'll ever have is a User object; if you build the pieces of a User object, then you get a User object and any number of other things you can build from its pieces.

Objectify

We've been following and refining this approach with a real app of nontrivial complexity in production for a few years now, and it's been very successful. However, certain things have always felt somewhat ad hoc, and since rails isn't designed to support this kind of structure, it's easy for people (myself included) to get confused about exactly how functionality should be structured.

So, I spent the last few weeks building a framework that sits on top of rails and exposes these abstractions (and a few others) as first-class citizens. It's called objectify. It's very far from being perfect or finished, but it's a start, and we're already using it to clean up sections of our code with great success. You can read more about objectify in the README.

If you're interested in the future of objectify and better OOP practices for rails apps (and elsewhere), fork the code, join the mailing list, come hang out in #objectify on freenode, or just hit me up on twitter.

P.S. If you're interested in working on this kind of stuff and/or on a rails app that has been committed to these practices literally for years, get in touch because we're hiring!

[1] Note that the skinny controller, fat model approach doesn't necessarily indicate that business logic belongs in ActiveRecord::Base subclasses, and even mentions presenter objects. The article wasn't necessarily was wrong, but peoples' implementation of it has been.
[2] Barbara Liskov credits Morris's paper as being one of the primary influences of her Turing Award-winning research on abstract data types (a predecessor to what we now think of as classes).
[3] Another option is to implement all of our functionality as a series of modules, which we then include in to our model classes (or extend our model objects with). It is possible to test these modules in isolation by creating fake objects to include them on. But a lot of care still needs to be taken to make sure that modules being included together don't interfere with each other, which means that these modules aren't encapsulated. Without encapsulation, the benefits of isolated testing are mostly negated. This approach is actually equivalent to putting everything in the same class, except for the fact that it's spread out over multiple source files.

Why RubyGems Needs Loren Segal

2011-06-01T23:21:27-07:00

Full disclosure before I get started here. Loren and I are friends. I'd like to think that this blog post is mostly unbiased, but I'll let you come to your own conclusions.

Maintaining a piece of core infrastructure for a growing community is hard. Even if the code isn't especially complex, getting the release management issues right and keeping everybody happy is incredibly challenging.

Sometimes, that means making concessions, like continuing to maintain an API you don't like — or code that gets in the way of a refactoring you want to do to make your own life easier. But that's the challenge of maintaining software that tens of thousands of people depend on every day.

On Legacy Code

Did you know that in the linux kernel project, every commit absolutely must build cleanly and run the tests successfully to be accepted? Kernel hackers go to great lengths to make every patch fit these requirements. It's an enormous pain in the ass for the developers, but it means that when something breaks, git-bisect will find it for them.

Going to that much trouble just to be able to use git-bisect is a huge headache for developers. But it's worth it because the linux kernel is an important project that gets used by millions of people.

The problem with RubyGems isn't that the tests don't pass every commit, but that APIs have been disappearing too quickly. It's an important enough project used by enough people that deprecations should be measured in years not months.

The current maintainers of RubyGems don't want to live in that kind of world. They want to move quickly to refactor the codebase, deprecating and removing APIs where it suits them. And who could blame them? I wouldn't want to maintain a huge pile of legacy code either. Most programmers wouldn't.

You'd have to be crazy...

Actually, Loren is just about the only guy I know who isn't bothered by this sort of thing in the slightest. In fact, he seems to enjoy maintaining legacy code. It's a sick pleasure, and I know a little something about sick pleasures...

In all seriousness, a guy who cares about release management as deeply as Loren is one in a million. The fact that he's also an extremely talented engineer makes him one in a hundred million. When he told me a couple of months ago that he'd be willing to maintain RubyGems, I gave him an "are you serious?" look. Turns out, he was.

So, here we've got a ridiculously talented programmer who wants to make all of our lives easier by living with a whole bunch of legacy code for us. It really is a no-brainer.

The RubyGems team should bring Loren on board to run the project. He's more than willing to put in the time and effort, and he's the best person I can imagine for the job. The proof is in the pudding: Loren (and team) have put together a fork of RubyGems (1.3.7) that maintains backwards compatibility, and backports all of the performance improvements made since them.

SlimGems

SlimGems is a really great project with a really horrible name. It's an effort to make a RubyGems with a stable API (the one from 1.3.7), a better code base, and faster gem installs. And there are a lot more exciting plans for the future. Check out Loren's blog post for more info.

More importantly than any of that, though, is how helpful and friendly Loren is when it comes to bug reports and pull requests. He did it with YARD, and now he's doing it with SlimGems. I've never heard anybody give him anything but rave reviews.

Until Loren is an official RubyGems maintainer, I'll be running SlimGems. I moved our 30 our so servers at work over too. If that sounds interesting to you, 'gem install slimgems' is all it takes. Oh, and if you want to revert back to your original RG install, just 'gem uninstall slimgems'. That's just how we roll.

If you have any trouble or feedback, jump in to #slimgems on freenode, or open a GitHub issue, and we'll be happy to help you out.

The Future

Forks are good for communities. They're a great place for new ideas to be proven (think rails/merb). SlimGems has already demonstrated that its goals are possible. They've already achieved many of them. I'm ready to see the code and teams merged. Until then, do yourself a favour and 'gem install slimgems'.

VERIFY_NONE

2011-02-15T19:52:01-08:00

A while back, it came to my attention that ruby's net/https implementation had an insecure default: not verifying TLS certificates (OpenSSL::SSL::VERIFY_NONE). I wrote an article about it for RubyInside, and helped @geemus fix the issue in his excon gem. Despite this being an incredibly serious security issue, nobody really seemed to care. Oh well.

Then today, one of the biggest names in the ruby community, Aaron Patterson (aka @tenderlove), posted a gist of a little campfire bot that he wrote that forced net/https in to this insecure mode.

Yes, a campfire bot is relatively unimportant security-wise (except that if there's a man-in-the-middle, he now has credentials to access your campfire room, which may or may not contain company secrets — but I digress). Eventually I remarked that despite the relative unimportance of a campfire bot, tenderlove is a leader in the ruby community, and leaders should set good examples.

A few other people also responded. @joedamato posted an admittedly less constructive response. And Ben Black, a somewhat snarky, but not particularly harsh suggestion. That's when the hate started pouring in.

Here's the thing: this is a very serious security issue, and nearly every rubygem that uses net/http is guilty of it (yes, even active_merchant, the thing that everybody uses to interact with payment gateways). Why? Because of the prevelance of copy and paste coding. Yes, I do it, and so do you.

And nearly every net/https example uses VERIFY_NONE. It's so common in example code that in the related links on the RubyInside article about the perils of VERIFY_NONE, there's a link to example code that uses it (lol?).

Aaron is one of a small group of people in the ruby community who actually has the power to do something about this problem. By setting the right example, people will copy and paste good code instead of bad code. That's more useful than a million tweets or blog posts.

Yes, this may all seem trivial to you. It's just a hack, after all. Obviously, Aaron wasn't being deliberately insecure. He was just hacking, which is perfectly fine. But, we all know that hacks have a way of ending up in production.

It probably won't be tenderlove's app; it might be some noob who found and modified his code. But sometimes one man's hack winds up (however indirectly) being another man's business.