Search for Balance

Some Redis ZADD performance testing.

Fri, 01 Jun 2012 14:57:58 -0700

Redis is the bees’ knees. I use it quite a bit, and recently was working on something where I’d be creating (or inserting) a lot of items in a ZSET. I decided to do a little performance testing on ZADD. (Reposted from my Disqus comment on the http://redis.io/commands/zadd page)

Did some playing around with using ZADD to create a new ZSET of 10000 elements (using 2.6.0_rc3):

I created the data like so:
perl -le ‘for (1..10000) { my $r = rand() * 1000; print “zadd testzset $r "$_"” }’ > single

Then created three copies and modified them in vim:
“multi” (added multi to the top and exec to the bottom)
“combined” (block deleted the 'zadd testzset ’ from each line, joined all lines and added 'zadd testzset ’ back to the first line

(All time commands were done multiple times and timing was pretty stable per run)

Single ZADD statements:
time cat single | redis-cli
(integer) 1
(integer) 1
(integer) 1
…
real 0m0.497s
user 0m0.100s
sys 0m0.176s

reids-cli del testzset
(integer) 1

time cat multi | redis-cli
…
9998) (integer) 1
9999) (integer) 1
10000) (integer) 1

real 0m0.826s
user 0m0.112s
sys 0m0.192s

reids-cli del testzset
(integer) 1

time cat combined | redis-cli
(error) ERR unknown command '184’
(error) ERR unknown command '184’
(error) ERR unknown command '184’

Looks like there is a (length? buffer?) limit on a single line at about 4096 characters (for me) that impacts how many items you can add per ZADD.

After splitting “combined” into 59 zadds of length < 4096:
…
(integer) 170
(integer) 170
(integer) 87

real 0m0.120s
user 0m0.048s
sys 0m0.012s

I’d interpret this as the combined ZADDs (broken into the right-sized pieces) being the fastest, but not so far ahead of single ZADD statements, with the MULTI/EXEC being the slowest. In practice the speed/complexity of breaking bootstrapping a large zset into multiple combined ZADD statements <= 4096 characters versus single ZADD statements may be a wash. If you need it in a single transaction the MULTI/EXEC block is capable if slightly slower.

One more data point: I converted something that was creating a zset of about ~17000 entries from individual ZADD statements to the combined ZADD statements (based on my values I did 60 items per ZADD) and it cut the runtime roughly in half (~0.4s to 0.15s).

Unpause

Fri, 01 Jun 2012 14:53:33 -0700

Unpause

Why every entrepreneur should ____ more.

Thu, 30 Sep 2010 21:26:03 -0700

All you need to do to succeed as an entrepreneur is do exactly ____, because that’s what worked for me! If that worked so well for you, maybe I’ll just also create a business that does exactly what yours does? Would that also guarantee my success?

There’s been a fairly constant stream of posts such as these the past few years. A simple Google search shows titles that I’m paraphrasing here:

You must read these 10 books!
You must write every day!
You must follow these 10 rules!
You need a mentor with skills X Y and Z!
You need to study the history of ___!
You must use these 5 apps!
You must dress like this!
You must dress exactly the opposite of what that last thing said!
You need to go follow these 5 blogs RIGHT NOW!
You need to be on twitter!

How about every entrepreneur should just get their shit done and do whatever works for them. If it doesn’t create value, you shouldn’t be doing it.

Oracle sues Google over Android and Java

Thu, 12 Aug 2010 23:01:27 -0700

Oracle sues Google over Android and Java:

Isn’t using Java punishment enough?

iPad = Television

Sat, 30 Jan 2010 09:55:22 -0800

I must admit I am not at all enticed by the iPad, but thinking about it led me to an interesting line of thought: the iPad must be a TV.

Advances in technology are pretty much universally about making things smaller. Computers, cell phones, batteries – really everything. Except televisions. TVs the goal is always to make them larger.

If the iPad is a larger iTouch, and they expect it to succeed, this means to me it must be a TV, otherwise why would anyone want it?

This should be obvious to anyone who has ever had a boss.

Fri, 03 Jul 2009 23:27:01 -0700

This should be obvious to anyone who has ever had a boss.

Planning and Process

Tue, 09 Jun 2009 22:05:26 -0700

All projects requires some planning and process. The purpose of both is to mitigate risk – as they say, ‘a failure to plan is to plan for failure’. The problem is when people start hiding behind planning and process as a way to mitigate personal risk.

Recently I’ve been seeing a lot of situations where people are propping up planning and process as giant pillars from which to hide behind. In terms of process, this may come as someone who requires every action they take to be signed off on by another person who they can hold responsible if it was the incorrect, or by creating a process so rigid that you spend vastly more effort upholding the process than what was designed to protect against. The extreme example of planning gone wrong is the person who only plans and never does anything– and is thus never wrong. Too often has never being wrong been mistaken for always being right. In the case of perpetual planning you’re both never wrong and never right, you’re just useless.

Planning by its nature is speculative, and the more you do without a reality check (i.e. actually DOING something) the further from reality you become. Even worse, as you are planning rather than doing, reality is constantly changing. Actually doing something has a very convenient feature with which you can validate what you’re doing – either it works or it doesn’t. There is no analog with planning, eventually you have to just do something. The goal is to get to where you have a good plan, and acknowledge your plan cannot be perfect. Once you think you know where are going, start going there and see where you get. Then, reevaluate and try again. In programming, Dave Thomas describes this as 'Tracer Bullets’ – a great analogy.

Process is supposed to give you a framework for execution. Customer sign-off on a feature request isn’t meant to be lorded over them if what you built turns out to be wrong, but a means for you to try and make sure you are both on the same page in terms of what you are doing. Spending two weeks plugging your prototype it took you one week to create into an enterprise build system is asinine. If you can’t immediately extract benefit from process, it is dead weight.

One of my favorite quotes that sums this up more eloquently than I could ever hope to is by George Patton: “A good plan violently executed now is better than a perfect plan next week.”

For the love of crap people, it is ok to be wrong! Take some damn risks! And while you’re at it, do me a favor and kick the next smug armchair quarterback you see where his testicles should be.

Just another day out with my two older girls.

Sat, 18 Apr 2009 14:26:30 -0700

Just another day out with my two older girls.

Caught in a Lie

Thu, 05 Mar 2009 20:01:03 -0800

This evening my oldest (kindergarten) daughter said something offhand that struck me as odd. It was something like “My friends at school think it is really cool you’re a geologist." To which my immediate reply was, "I’m not a geologist, have you been telling the kids at school I am?”

I’m pretty sure this is her first experience getting caught in a deep lie. I got her to elaborate: she has been telling her teacher and classmates for a few days now that I’m a geologist, thus so she knows a lot about rocks. This has caused the lie to balloon and grow far beyond her control. She felt obligated to show her expertise, so she made stuff up about rocks to tell the class, told of trips that we would take to the beach to collect rocks for my job, etc.

It has been only a few weeks since we transferred her to a new school. We’re very happy with the school and the decision, but the transition has been hard on her. Apparently her way to try and impress her classmates and make friends was via what surely started as just a little lie, and now she knows what it feels like when a lie gets a life of its own.

She is devastatingly afraid of telling them the truth now – which I’ve insisted she do. The plan is she will take in one of the more impressive rocks from my childhood rock collection (and her teddy bear for comfort) in with her and fess up. I told her to explain that while I do know a lot about rocks, I am not a geologist, and to apologize. Her fear is that her teacher will be mad and all her new friends will not like her anymore.

My advice to her was as follows: Your friends won’t really care about the lie, they will respect you more for telling the truth. Your classmates don’t like you because you told them I’m a geologist, they like you because of you. You will feel much better because it will be over.

There are many milestones in our lives, but until you witness your own child going through it, you don’t think about things like getting caught in your first big lie as one of them. It also gives a new perspective to help coach someone through the situation, and hopefully in the future if something like this happens to me I’ll remember my own advice to her – and have a teddy bear to hold for comfort.

Kindle 2: Reference Library

Thu, 26 Feb 2009 22:24:51 -0800

I want a Kindle 2, but I want it for all the “wrong” reasons.

I have no interest in reading fiction on it. I imagine that (and the $359 price tag) is what keeps most people from buying it. I like reading actual books, and they happen to be very well designed for their purpose as it is. I also typically borrow books, or buy them used, at a significant discount. This re-use benefits all of the various readers of that physical copy, but both the original and used bookstores that it passes through. Amazon currently has over 240000 books available on the Kindle, the vast majority being fiction. None of them can be resold to someone else, or loaned to a friend for a month, or passed to your child so they can experience the same feelings you did when you were their age reading those exact same pages…

I have no interest in reading magazines or newspapers on it. I hardly consider pushing your print content to the Kindle “innovation”. There is no customization, interaction, filtering, aggregation… no hint of any of the things that would actually break from the status quo. Paper content rendered on screen is just paper with fewer trees sacrificed for the cause.

However…

I *do* have a 50lb stack of reference books next to me, and a couple hundred more pounds on my bookshelf that I would love to have at my fingertips all the time. Not only that, I’d love to just have them automatically updated as new versions come out (this a wish, not an actual feature). The idea of having to repurchase all of the books I’d want on it is a bit daunting, though.

It would make the perfect RSS reader. With free 3G wireless internet and a great large format screen it would easily beat out my T-Mobile G1 for that purpose. They claim to have about 1100 blogs – which isn’t many – but I can see them adding more. Unfortunately, my guess is relatively few of the blogs I read are carried at this point.

I have a very large collection of reference PDFs – mostly white papers and research articles – that I consider invaluable references. Like my textbooks, I’d like to have them at my fingertips without having to fire up my laptop to read them.

Though my T-Mobile G1 lessens the need for it, the fact that it has 3G internet and allows some limited web browsing would be nice as well. If the Kindle became my resource for reference material, it would make sense that Wikipedia was on it as well.

The Kindle replacing textbooks is the thing that excites me going forward. Textbooks are unwieldy, heavy, quickly outdated, and hard to search through. Carrying around a single Kindle with all their textbooks would not only save the backs of many students, but suddenly they can run text search through their books instead of having to leaf through, distribution is streamlined and uses less materials, so the books should come down in price. I really hope that technologies like the Kindle can eventually redefine the textbook industry.

Sentiment Accuracy vs. Sentiment Accuracy

Sun, 22 Feb 2009 20:27:08 -0800

** Posted as a comment to this post at lexalytics **

Great post, we’ve also found it pretty easy to build a system that gets 70-80% accuracy in almost no time at all.

Sentiment accuracy is also interesting and measuring it becomes all the more difficult when you consider it as a spectrum rather than simple agree/disagree.

For example, when dealing with financial sentiment measurements as we do, you can be a little wrong, or you can be REALLY wrong.

Getting the right polarity 80% of the time is great, but you also need to consider what 20% you missed. Humans who disagree will usually have agreement on the highly polarized articles. We expect people to disagree on the more intricate cases.

In our experience, even if you are getting the same % agreement overall human-human or human-computer, computers are much more likely to throw articles humans would all agree on into the wrong bucket.

LoveRX

Thu, 12 Feb 2009 21:57:35 -0800

I saw this article in the paper this morning, which made me wonder just how soon we’ll start seeing love pills. If they have found the centers of the brain and are narrowing down the chemistry, it can’t be very long. Here are my proposals:

Accumbenz: When the marriage counselor thinks it shouldn’t be saved. (Nucleus accumbens)

PallidumRX: Why tell your boyfriend to ‘shit or get off the pot’ when you can drug him into proposing to you. (Ventral pallidum)

Tegmental VTA: When your mistresses all start seeming the same, a little Tegmental VTA will bring back that puppy love feeling all over again. (Ventral tegmental area)

Rapheipnol: For those spinsters who are stressed out about losing their virginity in their 40s. (Raphe nucleus)

Suggested mixtures:

Accumbenz + Tegmental VTA: Rebound relationship in a bottle.

Tegmental VTA + Rapheipnol: Gives you the sense that you’re cheating on your husband even though you’re unmarried and alone.

Accumbenz + PallidumRX + Viagra: Really enhances the “Breakup-Sex” role-play.

Two Cows

Sun, 08 Feb 2009 12:05:53 -0800

Two Cows

Great term: the "Great Repression"

Sat, 07 Feb 2009 10:46:53 -0800

Great term: the "Great Repression"

John Carmack gives some hate to software patents

Mon, 26 Jan 2009 22:55:39 -0800

John Carmack gives some hate to software patents

Toys For Adults

Thu, 22 Jan 2009 21:14:09 -0800

No, not that kind of toy…

Although if Apple got into that business they’d probably do pretty well there, too.

What I’m talking about is why I think Apple is doing so well in the electronics game while companies like Circuit City are dying off en masse, and Microsoft is losing market share at an accelerating pace.

While some argue they have a superior product, and at least it is more appealing to the eye than most others, I’m going to suggest an additional reason I think they are doing well – especially when it comes to expanding their market into the mainstream.

Apple stores are like a toy store for an adult. You can come in, play around with everything, all their stuff is out in the open and you can tinker with it. Because of the openness, you can see other people playing, too, making you want to try it out. Even their products are shiny and rounded like little toys.

The feel however, isn’t like a toy store. The stores are refined and sophisticated, taking great lengths to avoid any suggestion of childishness. This is very appealing to the mainstream, mall-going crowd. At some level I think they surreptitiously plan it this way – draw them in to play with the toys but tell them they aren’t playing with toys, they are still very grown up, sophisticated people.

Microsoft is losing the mainstream because they aren’t engaging them, or more accurately, their resellers aren’t. Their resellers are places like Circuit City, Best Buy, Office Depot – which aren’t in the mall, so they aren’t in the mainstream. They make you come to them rather than putting themselves in front of the populous.

I wonder what kind of success a Windows-based (or even Linux-based) reseller would have if they spent some time on industrial design and emulated the Apple store’s strategy of the mall adult toy store. It certainly couldn’t hurt their market share.

The Important Matter of Position Sizing

Wed, 21 Jan 2009 23:06:53 -0800

The Important Matter of Position Sizing

A sneak peek at the downturn & its effect on Google

Sun, 11 Jan 2009 21:24:30 -0800

This evening my wife and I were driving around Issaquah looking for a place to have dinner. Neither of us know the area very well, so I was searching for restaurants on my G1.

Our results were very interesting: 30-40% of the places we found via Google maps had closed their doors.

You can’t really blame Google, it appeared these were all fairly recent closures, and it isn’t as if Mom from Mom & Pop’s Bar & Grill phones Google to let them know they are out of business.

It does have a very negative impact on the Google experience, however. If I have this much trouble now, what will it be like in 6 months to a year when things are really bad?

Google’s approach at curation is to let the relevant percolate to the top. This works for active areas of interest, but once your focus narrows enough, you end up with derelict pages, old information, or closed restaurants.

Maybe some other company will come along with an innovative solution to what I see as a growing curation problem by culling no longer accurate results, or maybe Google will make progress in this area on its own.

Either way, I’m guessing this experience is only going to get more commonplace as stores and restaurants close, dotcom companies go dark, and people no longer have time to work on their blogs, open source or other pet projects. Who’s going to clean up the mess?

Customization and Discovery

Sat, 10 Jan 2009 14:44:00 -0800

Customization and Discovery

The Seattle P-I Up For Sale

Fri, 09 Jan 2009 22:03:59 -0800

The Seattle P-I Up For Sale:

Another newspaper is biting the dust. It will be interesting to see how this major transition from old media to new media plays out. Will news coverage worsen? Will online news be able to grow into a similar behemoth that print news used to be? Or will some innovative paper out there figure out a model that will allow printed news to flourish again?