We’re already running Hbase and storing lots of data in there and I’d like the application to scale as easily as possible. The idea of running another framework or service just to store user data seems overkill and seems like one more system to worry about. So I’m going to run a little experiment in storing users in Hbase.

The downside to Hbase is that it’s not easy to search for things when you only want one of them by something other than the id. It’s easy to pick a row by it’s id and even to scan the table in order from there or to start and the beginning and go all the way through, but it’s not very easy to quickly pick out a random row by the value of one of it’s other fields. You’d have to start a map reduce task and start crunching the data until you found what you were looking for.

Given the simple example of a user with a long id an email address, a username and a password it would be easy to get the user by it’s id, but not very easy to get the user by the email address or username. So I’m toying with how to get it to work, probably by creating some additional tables to store keys for columns I want to search that link back to the correct user. Sort of like making my own indices.

Once I get some code written and tested I’ll probably throw up another post with some more details on whether or not it worked.

Any thoughts?

Update: Check out this Vitamin article

This is something we came across when building Sproozi. If we wanted to store petabytes of data and run hundreds or thousands of concurrent processors there was no way we could ever afford to do it on machines running windows we were paying for by the box. But it’s not because we’d have to pay for software, per se, it’s how we’d have to pay for it.

Software has traditionally been licensed by machine, when machines got bigger vendors wanted to cash in so the licences got a little bigger. They had to cover their losses when you threw a few new processors in the machine rather than getting a new one to put alongside after all. It has always been in their best interest though for you to get a bigger box than to get more cheap ones - scaling out is very hard and the software doesn’t do it well. Most RDBMS just can’t do it well and they certainly can’t get anywhere near the the scale of something like Hadoop. If you want to scale out, forget SQL servers, you need software that’s going to scale out.

But let’s forget the specific software for the time being and just assume that the big boys (MS, Oracle, IBM) will have a scaling out solution soon - don’t worry this isn’t going to kill them, but it will change them. They will still want to licence an operating system and a data storage and retrieval system to you.

What I’m almost positive you’re going to see is these companies introduce new pricing schemes to meet the needs of the cloud, they have to or they’re going to lose all that revenue to the open source projects that have a head start on them. Just look at EC2, you can already provision MS and other software and I think that’s a trend that’s just going to continue.

So while Jeff is right that if I want to buy as many cheap boxes as I could for the hardware cost of a big iron server and put windows and SQL on them and it would all cost a small fortune. It’s not really a fair argument, you’re taking an old big iron way of thinking and trying to apply it to the cloud. What it fails to take into account is how much more powerful your new cloud cluster is than the big iron box, let the software vendors figure out the economics of making their software an attractive ROI when compared to OSS because if they want to compete in the cloud they’re going to have to.

Related articles by Zemanta

• Hadoop Summit: We Have 10 Tickets to Give Away (gigaom.com)
• Watch out, Oracle: Google tests cloud-based database (computerworld.com)
• Yahoo Releases Internal Hadoop Source Code (techcrunchit.com)

Things between us dear reader, have become awkward, it’s my fault and I’m sorry.  I’ve just been quite busy, I know it’s not an excuse, but there it is anyway.

So let’s try to get back on track in the form of an update.

Sproozi is coming along, I’ve had a few setbacks with the code and we’ve changed the scope some. We had to make a decision about content for a real launch we’re selling to investors and actual users as opposed to the few that happen upon our non functional demo.

The problem was that although the crawler worked pretty well, better than I’d hoped, the results were a bit lacklustre.  We weren’t confident that the data we brought back was going to set the world on fire. So it then became a question of what is worse no data or crap data?

I’ve also been working on getting a coworking space and some geeky events going going in Hebden Bridge. Lots of work getting emails sent to see if there would be funding available, trying to gauge interest and all sorts of other aspects about it.  I’ll have a lot more to say about this in the next few days once I’ve had a chance to digest where I am.

In the meantime, go take the survey- it’s here.

Lastly I’ve actually been working on this site in the background. It’s on github already over here. I like the simplicity of the look but I want to bring a few more things on to the front page. I really want to integrate my tumblr feed and start using that a lot more for posting links, pictures and music. Also now that I’m back into posting a little more regularly (the irony I’m sure isn’t lost on you) I also want to put the blog posts on the front page. Which is where I start to get into things I don’t like about the hemingway theme.

I got some feed back on my post the other day about setting rates. It’s an interesting topic to a lot of people. One of the questions it did bring up was how to deal with negotiating on rates, or specifically dealing with a customer that doesn’t want to pay what you’re asking.

I for one set out my rate at the start in a quote, explain that it’s competitive and that’s what it costs.

I’m open to negotiation if for example you’re offering me a couple of days a week for a month, few months or a year. It costs less to develop and maintain one relationship like that than the equivalent in smaller customers - even if it is putting more eggs than I may like into one basket.

If on the other hand it’s a couple of days then I’m going to be much more firm about my rates.

One strategy for getting paid what you should, and giving a customer a price they want is to determine the budget from the outset then structure a proposal around that. Once you’ve got a number, make sure you can deliver what they want, or negotiate on a cut down version, which you can deliver within their budget. If it’s design work, cut some of the revisions, base the design on a template of their choosing. For coding work cut some of the scope, remove one or two of the lest important requirements.

A lot of times you may just find “you’re too expensive” really means, “we can’t afford it” and there are a lot of great companies out there that aren’t just being cheap, even if that’s the way it may seem on the surface and they’d be more than willing to compromise on the scope to get it done properly within their budget. Then who knows, revisit the rest in a few months time.

Other times you just can’t meet in the middle, you can’t do the work in the time allowed by the budget but don’t give up - there might still be a deal you can both cut. Maybe they offer a service you’re currently paying for, or have staff that aren’t being fully utilised you can borrow. Trading your services for theirs isn’t a bad thing, you still get a client, word of mouth referrals, case studies, testimonials and add to your portfolio and you still get something in return for it. I’ve done this in the past and it’s always worked out well for me, in fact it’s probably lead to more work from and through that client than we could otherwise expect.

If all else fails, my advice is to walk away. It’s hard but the brutal truth is that you’re risking getting stuck in a working relationship where the other side doesn’t value your skills. In my experience that attitude extends right through into how they value your opinions and the result of your labours. To them what you produce is a commodity and they’re probably going for the cheapest quote they can find. Everyone has worked with people like this, it’s not fun and it’s almost never worth the money. In fact, my advice is run - don’t walk.

Related articles by Zemanta

• Happy Customers Equal Better Business (underthegrapetree.blogspot.com)
• 5 Ways to Please Your Clients and 5 Reasons Why You Should (sitepoint.com)

So they’re not cutting jobs, except for contractors. Which is too bad for the contractors but brings up an important point if you are a contractor and you’re setting your rates.

You are responsible for your own job security and you have to charge for it. You don’t get holiday pay, you need to spend some of your time on the non billables that need to be done to operate, you won’t get a redundancy payment if through no fault of your own the company cuts your role in a cost savings exercise. You just don’t have a job.

There are various ways to mitigate the disaster and the best is probably to have as many customers as possible. The more you have the less the loss of one matters.

But you also need to be aware of the reality that you may lose customers when you set your rates. You need to bill for your desired salary + operating costs + expenses + some headroom for the time you aren’t doing billable work.

Here’s an example. Let’s say you want a £30,000 salary. You have operating costs for equipment, software, travel, expenses, taxes, accountants, rent, phones, internet etc; say £15,000/ year. Given 45 working weeks in a year, because you deserve holidays like everyone else and you’ll no doubt catch a cold or two at some point.

30,000 + 15,000 = £45,000/year MINIMUM turnover.
45,000/45 working weeks / 5 working days = £200/day

So if you want to earn a £30,000/year salary and you have enough work to keep you occupied every single day - you can afford to do it for about £200/day.

Let’s for a moment say you spend just one day a week on things you need to do, but you can’t bill a customer for directly - your accounts, taxes, your website, finding new customers, writing proposals, travelling to and from meetings, meetings, professional development.

That’s a rate £250/day and you still have absolutely no cushion. Lose a customer and even if it’s one day a month you’re not at capacity and you lose £3000 from your salary. If you lose a few or a big one and you’re idle a day a week - you’re out over £11,000!

Now if you want to build yourself in a cushion so that if you lose some work you’ve got money to eat while you spend time trying to get more customers, you need to up the rate. For the sake of argument let’s say you want to be able to eat working at 75% capacity if the times get lean and stay that way for a while. That means you need to be able to survive billing for 3 working days every week, so with the numbers above we get.

£45,000 / 45 working weeks / 3 working days = £333.33/day

Now in theory, working at capacity that would give you enough to pay yourself almost £45,000/year - but if you were smart and like the C&G contractors a major client suddenly pulled the rug out from under your feet, you’ve saved that and now have a good cushion going forward to find new customers. If they were 100% of your work, it could take sometime to build up that client base from scratch so the £15,000 really isn’t that much when you start to eat through it - literally.

Came up with a good distraction tonight and thought briefly about being an iPhone developer. Was chatting with @Simon_Chapman (not sure I’d bother clicking there - nothing but tumble weed) about the various services out there and was trying to come up with a unique way to use some of the new features offered by the iPhone 3.0 software.

Specifically we were talking about how to use the new support for in app purchases and location to build a compelling service. The first thing that came to mind was pretty obvious and no doubt you’d just end up a small fish in a big pond with some monsters, create a service to search for and buy tickets for events near you.

The next idea that came to mind is to create a map based podcasting application. Allow any geotagged podcasts to be places on a map. Browse the map and get some audio or video about things around you. Revenue could be either generated through advertising or through access to premium content.

There you go, that one’s free, unless I find the spare time to develop it myself.

Related articles by Zemanta

• iPhone 3GS (manolith.com)

I’ve been working beyond the bleeding edge, using a version of the Nutch code that’s not even made it into the Apache SVN for the project yet. To celebrate the fact that my contributions will make it in I figure its a good time to get into open source and business.

To put it briefly, and as you can probably guess, I’m pro open source. I use it extensively and I push back as much as I can. When it comes to the most of the code I write there really isn’t much commercial benefit in keeping it hidden so it just makes sense to give back.

There are two types of business on the web, one where you provide a software service and that is the product and others where you provide access to data. It’s pretty easy to tell which camp you’re in.

37 Signals for example, they’re in the first and their software probably isn’t something they just want to let people download - unless they’re incredibly brave. Doing that would mean that they’d be competing on margins for the cheapest hosting, users would flock to the cheaper services, have a poor experience and blame the software.

Sproozi on the other hand is the second type, our data is what users mostly care about and we’re not planning to be precious about our code. I’ve already been pushing some of the changes I’ve made to Nutch back into the project and we’re planning open source projects of our own in the coming months.

One of our plans we have is to build iPhone, Andriod and other phone based applications for our service and release them as open source projects. We’re planning to write them (or have them written for us) and release ‘official’ versions. Then release that code as open source project to provide a framework for developers so that they can build great things from it and on our API.

If there are any experienced iPhone, Android, Blackberry, Symbian or Pre developers out there that want to get involved, drop me a line were a ways off yet but would love to chat about it and get some very early feedback.

Related articles by Zemanta

• Android Application Development (oreilly.com)
• Palm Pre will debut with only a few apps available (computerworld.com)
• Yahoo! breeds Pig that talks elephant (theregister.co.uk)

We’re not web-designers, we’re not an advertising agency - We develop and research social strategies and products to improve engagement.

It sounds like a lot of marketing and buzz word bullshit but it boils down to the fact that people, companies and brands are realising that the web is so much more than a multimedia brochure. The simplistic metaphor you hear all the time is that it’s a conversation, it’s true. It’s just like a conversation. You have to give and take and you have to speak in a way that doesn’t make someone just want to walk away and avoid you in the future. Talking at someone doesn’t work in real life and it’s becoming less and less effective on the Internet.

I saw an interesting presentation by @mattseward from Kilo75 last weekend about conversations with customers and more generally how they try to develop relationships between brands and their customers. One of the questions that came out of the audience was whether people wanted to have conversations with all these brands. It’s a good question, but one that I think is slightly wide of the mark.

The thing is what most brands have to say isn’t terribly interesting to me, so I’m just not listening. Some though have a message which I find engaging and I do listen. 37 Signals is a good example - I use their products, I read their blog and I agree with their message and even pass it on.

How do you use the Internet to improve engagement? What tools work best and how do you use them? Get in touch, I’ll have more to say on this soon as we get closer and actually relaunch.

I did it again! I wrote this on the day I wanted to publish is, then just left it as a draft. I need to try harder, it’s only Day 7!

Was it perfect? No. Was it good enough? Yes, I think so. It might not get us funding, but we can take it and build on it and send it to anyone else that will listen.

In the plan we said that we’re going to have a functional tech demo live for 1st July - so 1st of July check it out, it should work.

A couple of things I wanted out of the API, the most important was to be able to do were switch from one implementation to another without loosing my markers or having to reload them. I really like that foot paths show up on the OSM maps.

Then I came across mapstraction - which looks very good. I’m checking it out to see if it meets my needs. I’ll let you know.

Whoops, for some reason this didn’t get posted yesterday. Must try harder.