Be Genius

Two months without Twitter

Fri, 10 May 2013 22:23:06 -0000

Two months ago, frustrated with inability to focus on the things I cared about in life and at work, I formed a theory:

Twitter (et al.) has slowly been rewiring my brain to encourage rapid context switching.

On average, it probably takes less than a second to read and process a tweet. There are often hundreds of new tweets between "hits," which means that during vulnerable times of the day (e.g. in bed before sleep), my brain is rapidly context switching hundreds of times in a matter of minutes.

For me, Twitter has always been about consuming information and learning. Despite the fear of "missing out" on interesting links or thoughts in my network[1], which is predominantly made of people I respect and trust, I decided that this rapid and varied consumption is not a healthy way to achieve this goal.

It's been two months since I uninstalled all Twitter (and Facebook) applications from all my devices. I still tweet occasionally (using iOS/Mac's Notification Center), but I make a point not to check the Twitter feed. I re-enabled email notifications for mentions and only log in to the web interface to reply to mentions/messages.

I love it. I want this to stick.

[1] Prismatic is a pretty great way of countering this. It usually cuts through the noise and gives me meal-sized versions of the best content, instead of bite-sized versions of irrelevancy.

A collaborative thought experiment on company environment, values, culture, and ethos

Tue, 05 Mar 2013 23:01:14 -0000

TL;DR: add your thoughts/experience on company values and culture to this repo.

Lately, I have been thinking a lot about the types of working environments and how they impact the productivity and happiness of the people who work in them.

After many discussions on and research into the topics of leadership, management, and company culture, I felt like I had to start taking notes of how everything pieced together. Specifically, I wanted to write down what the culture and environment look like of a company that I would love to work at and/or start.

Being a developer and contributor to open source software, it seemed clear to me that something like this could benefit immensely from a collaborative and open approach.

And so, here is the ever-evolving work and distillation of this process.

I'd love for people to contribute their input in the form of pull requests and discussions. And, where more appropriate, people or companies could fork the still-not-aptly-titled work to form the basis of a different set of values.

If I were to start a company right now, a living document like this would be available for all employees to see, discuss, dispute, and change.

Critiquing Leadership (Part 1 of N on Leadership)

Mon, 25 Feb 2013 14:46:43 -0000

This is part 1 of some unknown number of thoughts and essays on leadership. Your thoughts are welcomed in the comments.

If leaders (or managers, as it were) are not constantly soliciting critique from their followers, their harshest critics won’t feel safe being honest or speaking up at all.

This can create a bubble of false perception, by such leaders, that they are doing well. Instead of hearing the honest criticism (which, of course, still exists), it will spread laterally behind closed doors and manifest in alienation, resentment, and/or apathy.

This can only perpetuate the problem.

If you are a leader and you are not being openly critized by those you purport to lead, that is a warning sign. Note that, like all humans everywhere, you are not perfect; a lack of criticism is therefore a sign of something other than your successes. Likely, it’s a sign that you do not have a healthy relationship with your followers (or employees).

Do not wait for criticism — seek it out. Do not form a bubble of complacency that limits your perception of reality and your ability to handle the inevitable, your loss of followers (or employees).

Talk: SSH Can Do That?

Thu, 29 Nov 2012 15:23:36 -0000

Last night I gave a talk on the darker side of SSH at ChiSC meetup group.

Here's a video version (better quality one coming soon) for those who are interested:

Call Clojure function on a timer

Thu, 20 Sep 2012 16:07:00 -0000

In Clojure, I didn't see a nice way to simply call a function on a timer (e.g. to poll for changes in another service).

I didn't find something in clojure.core to achieve this readily (but clojure.core is quite big, so I may have missed something obvious — let me know), so I whipped up the following to put in my project's util.clj file:

(defn tick
  "Call f with args every ms. First call will be after ms"
  [ms f & args]

  (future
    (doseq [f (repeatedly #(apply f args))]
      (Thread/sleep ms)
      (f))))

(defn tick-now
  "Call f with args every ms. First call will be immediately (and blocking)"
  [ms f & args]

  (apply f args)
  (apply tick ms f args)

There are two variants. tick waits ms milliseconds and then calls f with args and repeats indefinitely. tick-now does the same thing except it calls f with args before starting the timer.

They are simple to use:

user=> (tick 500 #(println "hi"))
; 500ms delay
hi
; 500ms delay
hi
; 500ms delay
hi
...

In my project, I'm using them like so:

(defn start-fetchers
  [api-token]

  (future
    (let [minutes (partial * 60 1000)]
      (tick-now (minutes 60)
                update-project-list!
                api-token)

      (tick-now (minutes 5)
                fetch-milestones!
                api-token
                projects-to-fetch
                milestones-by-project))))

Pure Evil

Fri, 07 Sep 2012 04:31:19 -0000

I love Vim. I take that back — I love modal editing.

Actually, I think very few people like Vim, it just happens to be the best kid on the block at modal editing. But, after using Vim (and, God, does it take a while to learn to use it), you are reduced to a pathetic teary wreck every time [you] have to use a text editor that isn't vim.

And why is it so debilitating to use something other than Vim for those who have honed their modal editing skills? Frankly, magic sauce of Vim is (mostly) made up from a few quite simple ingredients: motions, text objects, operators. There are plenty of cool tricks hidden inside Vim, but those three main things are the things you really miss, consistently, when using something else.

Every now and then, when forced outside of the comforting (yet somehow cold) embrace of Vim, the Vimmer flirts with Sublime Text 2’s vintage mode or IdeaVIM. We may even be impressed — but, eventually, something falls apart and the abstraction leaks.

Usually, this failure happens because the Vi(m) emulation layer often doesn’t implement motions, text objects, and operators. They just implement some keystrokes that appear to do the same thing in the same way — until you try something the developer didn’t preempt.

If I could have an editor that did these things properly, I would have no real attachment to Vim. I have other things about Vim that I love and would hate to lose but they aren’t unique and other editors (can) have them, too.

Luckily for me, I’ve been doing too much Clojure to ignore Emacs forever and 2 weeks ago, I switched. Here's how it played out:

Nope — not for me... But, wait! Let us try this Evil Mode so we can mock their pitiful attempt at modal editing (*evil laugh here*). What’s this‽ It’s actually good!

Inner dialogs aside, I’m not kidding; Evil Mode is stunning. It doesn’t have everything — but everything that it does have is built up using the same ~~functions~~ macros that you can use to extend it. Those are: (evil-define-state) (“mode” is an existing term in Emacs so what Vim calls modes, Evil Mode calls “states”), (evil-define-motion), (evil-define-operator), (evil-define-text-object), and a few others.

Evil Mode correctly implements the plumbing to wire these things together so that the Vim “language” which becomes natural and reflexive (e.g. di” -> “delete inside double quotes”, vt -> “visually select till space`) is completely preserved. If you wanted to define a new text object (e.g. a Ruby block), it is trivial to do so and all of the motions and operators will work flawlessly with it immediately. Frankly, it’s not even that nice in Vim.

At the end of the day, I’ve always hated things like VimScript and Vim’s single-threadedness but I could never walk away from a truly powerful (and mostly extensible) modal editing system. Now, that has changed; I get ELisp (which, for all its faults, is exponentially better than VimScript), a saner package system, and a level of extensibility that, by design, can never be matched by Vim. All that and I don’t have to give up modal editing? Hell yes!

Bonus points: one of the “states” that Evil implements is an Emacs state; this leaves Emacs and all it’s default behavior and keymappings 100% intact, meaning you can pair fearlessly with Emacsen, or slowly learn more about the Emacs way of doing things, if desirable.

So far, I’ve rarely noticed I’m not in Vim and am constantly surprised when I employ Vim reflexes in Emacs and they work exactly as expected or better (seriously, try :%s/foo/bar/g in Evil mode and tell me you don’t squeal with joy).

I refuse to pick a single side, so I’m picking both. I’m going to edit modally in Emacs and I’m going to be happy doing it! I don’t care if I simultaneously piss off Vim users and Emacs users — hell, they need more things in common anyway...

Happier as a Hacker

Thu, 06 Sep 2012 00:22:42 -0000

I’ve grown a lot in the last few years, personally and, specifically, in my career. My values have changed and it’s affected the way I think and work.

There has been a gradual progression towards caring more and more about the Right Way™ to do things. For example, more than ever before, I value simplicity and clarity over ease and cleverness (an epidemic that greatly afflicts the Ruby community, in my opinion).

But, these values have come at a cost to me. Somewhere along the way, I complacently let getting it right become far more important than getting it done.” Don’t get me wrong, getting things right is very important — where I went wrong was valuing getting it right the first time.

It may not plague others to the extent that it has me, but this misplaced obsession has gotten in the way of progress. It’s caused a type of analysis paralysis not unlike writers’ block; I discard ideas and directions too early because they aren’t “good enough.” But exploring a potentially bad direction is still being better than having no direction — which is the only place you end up when you discard ideas before they have a chance to blossom.

In the eternal words of Little Red Ridinghood:

Isn't it nice to know a lot?
And a little bit not.

I was happier as a hacker, blissfully ignorant as I was, because I got things done.

It’s time to become a hacker again; it’s time to be creative and explorative, and work towards approximating the Right Way™, iteratively. Truly, this is one of the great realizations of Agile — focus on progress instead of planning and pre-empting every possible outcome.

I can’t forget that.

Motivate your lazy sequences

Mon, 03 Sep 2012 18:00:53 -0000

I love Clojure’s laziness.

Recently, I’ve been using lazy-seq to consume remote collections via APIs, fetching pages of data transparently and only as needed. Gary Fredericks, Mike Busch, and I applied this to a project of ours that had to crunch tens of thousands of records from Salesforce. I’m also doing something similar with a personal project that has to fetch a lot of Pivotal Tracker stories to reduce them.

In cases like these, consuming (potentially unbounded) resources in a lazy manner allows one to start processing data earlier and to make as few requests as possible to get only the data you need.

Mostly Lazy

I want to talk about a neat little thing I did in my project to get a nice little performance boost on top of this laziness, without having to think about any low-level concurrency concerns.

Laziness

Here’s a piece of code that provides an “infinite” lazy sequence. In this case, it is of tweets:

(require '[clj-http.client :as http])

(defn tweets-for
  ([user] (tweets-for user nil))
  ([user last-tweet-id]
     (lazy-seq
      (let [url "http://api.twitter.com/1/statuses/user_timeline.json"
            params {:limit 10 :screen_name user}
            params (if last-tweet-id (assoc params :max_id last-tweet-id) params)
            response (http/get url {:query-params params :as :json})
            tweets (:body response)]

        (when (not-empty tweets)
          (concat tweets 
                  (tweets-for user (:id (last tweets)))))))))

So that’s cool. Now, note the following performance characteristics when contemplating the next section:

(def my-tweets (tweets-for "bjeanes"))

;; The following returns after a delay while we fetch the first page:
(first my-tweets) ;=> {:text "Tweet 0" ...}

;; This returns instantly because our `tweets-for` function fetches 10 tweets per page:
(nth my-tweets 9) ;=> {:text "Tweet 9" ...}

;; This returns after a delay because this tweet is on the next (still lazily unfetched) page:
(nth my-tweets 10) ;=> {:text "Tweet 10" ...}

Motivation

So laziness is pretty cool. But, sometimes, things can improve if you are ever so slightly less lazy. What if we could remove that little pause between the 9th and the 10th items in the list where we are just waiting around for the network request to Twitter to complete? We could be using our time to do more CPU-melting tweet crunching! Well, it turns out we can easily do it.

Assume for a moment that we have some calculation (process) that takes a considerable amount of CPU time to process:

(defn process
  "Do some really hard work with our tweets"
  [tweets]
  (map #(Thread/sleep 100) tweets))

If we know we will be consuming a substantial amount of the lazy sequence, we could encourage the sequence to go ahead and start realizing the next chunk of our sequence.

This would mean that instead of processing 10 tweets, waiting, processing 10 tweets, waiting, etc.:

... we would be able to process tweets continuously back-to-back:

Wouldn't also be great if we didn't have to think about the parallelism at all? To this end, I present motivate:

(defn motivate
  "Motivate a lazy sequence to seek slightly ahead of the sequence consumer's position."
  ([coll] (motivate coll 1))
  ([coll motivation]
     (lazy-seq
      (future (nth coll motivation))
      (cons (first coll) (motivate (rest coll) motivation)))))

Let’s compare:

(time (process (take 100 (tweets-for "riblah"))))
;=> “Elapsed time: 11545.011 msecs"
(time (process (take 100 (motivate (tweets-for "riblah") 5))))
;=> "Elapsed time: 10394.769 msecs"

The speed difference is noticeable even when processing only a 100 tweets. If we were doing more than 100 milliseconds/tweet of processing, fetching a lot more data, or dealing with a slow upstream dependency, the speed improvements would be even clearer.

The last (optional) parameter to motivate is the “motivation factor”. If your CPU-bound work is long-running, this number can be smaller without a noticeable difference. The ideal number depends on how long each IO operation takes and much processing you do with each chunk.

Essentially, the motivation you give to the lazy sequence is a trade-off between waiting for IO and wasting IO; that is, the lower the number, the more likely you are to wait on IO but the higher the number, the more IO you’ll perform unnecessarily (at least, if you aren’t guaranteed to consume the whole sequence.

Hopefully this is handy to someone else out there. I wouldn’t at all be surprised if something like this already existed (UPDATE Yup: seque) or if this completely obvious to seasoned Clojurian, but it was a pleasant moment discovering this possibility on my own.

Factories breed complexity

Sun, 26 Feb 2012 19:22:56 -0000

Having maintainable code is great. Maintainable code allows you to deliver improvements faster, happier, and more reliably.

Furthermore, the measures that developers need to take and the strategies that we have to employ to achieve maintainable code have been understood for years, if not decades. Especially in the realm of object-oriented programming, but certainly not exclusively, most of these principles boil down to reducing coupling and system complexity. A system whose parts are coupled as loosely as possible is a modular system; the parts know little of each other and a lot about themselves and they have thin and specific interfaces between each other.

Test-driven development is one of the many tools at a developer's disposal to achieve code quality. Unfortunately, there is a lot of naïveté around the benefits of TDD. A lot of developers see TDD as primarily a tool for verifying system correctness. While TDD does of course offer this benefit, and arguably better than retroactive automated testing, the real benefit of TDD is that it offers short feedback loops that guide the design/architecture of the system.

Since it is accepted that a loosely-coupled modular system is a simpler system, it stands that tools, such as TDD, which guide a design towards modularity and simplicity are good tools. A module that is tightly coupled to another is not easily tested in isolation. However, if the isolated tests are written first, it is difficult to write a passing implementation for that module that maintains such a low degree of coupling. Therefore, good TDD should guide you towards a simpler design (though it is certainly not the only way).

Unfortunately, factories work against this goal. Factories debilitate TDD's ability to give you feedback into the complexity of your design.

To be clear, I am not talking about the Factory Method Pattern or the Abstract Factory Pattern — both of which can be described as ways to decouple a particular implementation of an object from code for the creation of such an object (Wikipedia). Instead, I am talking about the "factories" for replacing fixtures in tests — something which has seemingly obsessed the Ruby (Rails, especially) community. The two primary Ruby libraries for factory-based fixture replacement are factory_girl and Machinist.

A tool such as Machinist or factory_girl generates data for the attributes you don't care about, and constructs any necessary associated objects, leaving you to specify only the fields you care about in your test (from Machinist's own README file). This sounds nice at first, because it makes your tests more readable and relevant. However, behind the scenes, these tools are still creating other objects and entities and introducing them into your test environment. By having data and objects in tests that are irrelevant to the functionality that is being tested (in isolation, remember), a developer creates an environment that permits, if not invites, silent dependencies to creep into an implementation.

Furthermore, and perhaps more significantly, by creating objects (and usually entire hierarchies of objects) with such ease and opacity, you are outright masking the dependencies (*cough* complexity *cough* coupling) between your implementation and those entities. If forced to stub out all those intricacies, the system complexity would be screamingly obvious and a developer would quickly avail herself of a rewrite to reduce complexity or thin out the interface.

Instead of having feedback that guides a developer to simplicity, fixture factories seem to guide developers to complexity by masking dependencies as one-line simplicity. In fact, that one (or five, whatever) line setup is a shotgun blast of environmental dependencies that are hidden from the architect. That complexity will come back for revenge after being ignored for so long.

It seems that factory_girl and Machinist exist to make testing components more convenient. This is, at face value, an admirable and desirable goal. However, in unit tests, the cost is too high for any system of considerable size.

Please, do the right thing and avoid the convenience and "fun" of the factory_girl temptress. You will trend towards a simpler system and as a bonus (in fact, an incredible one) your test suite will likely be exponentially faster which, in turn, will breed simplicity by letting you have more feedback more often.

P.S. It shouldn't go without mention that factories can be absolutely awesome for integration tests. Integration tests aren't used for guiding system design nor testing in isolation so the drawbacks of these tools drop away. However, both factory_girl and Machinist use RSpec as some of their very first usage examples and this troubles me deeply.

P.P.S. A lot of these arguments can be applied to fixtures too. However, they usually don't create hordes of objects invisibly and litter your environment with them. Also, they aren't as slow. But yes, the fewer factories and fixtures in a test, the better.

Update for clarity: Firstly, I am absolutely just talking about unit tests. If you are testing code that integrates with ActiveRecord or number of levels of your stack, then factories and fixtures are certainly defensible (though I still prefer to steer clear). Secondly, I've tried to be careful about where I use the words simple, easy, complex, and difficult. For the definitions that I intend, please watch (at least the first 10 minutes of) Simple Made Easy.

New design

Wed, 07 Dec 2011 05:56:59 -0000

Just a quick post to point out the new design for this blog. It's nothing special — I just needed a change.

Hopefully, the change of aesthetics will motivate me to get back into blogging.

Let me know if there is any weird display issues on your browser. I only tested the latest versions of non-IE browsers, but if there are any stupid obvious fixes to any browser, I'll implement them.

Tell me what you think, keeping in mind I'm a developer not a designer :P.