When we started with Inkzee we didn’t think too much about other alternatives… we started with Apache and it’s not that we had big complaints about it, but having all the backend in one machine made us have performance/low resource consumption requirements and after a few months of our service running the all the different backend subsystems were almost killing the server and we started to research alternatives.

We were going to change Apache for Lighttpd but then we stepped into Cherokee statistics and after reading some reviews we decided to give it a try, the numbers spoke for themselves. We had been checking on Che

rokee for some years now, the project was mature enough for our needs so we went for it.

After a year and a half we can write our own lessons-learned from our experience:

• The Cherokee team have been so responsive to every problem or tweak we wanted to do with our server.
• The admin interface is pretty rad, if you are not used to messing with config files you’ll find no challenge.
• We didn’t miss anything: FastCGI, PHP, TLS and SSL, Load Balancer, Reverse Proxy, statics cache, …
• And the most important for us: our servers stand the load so much better now… indeed now we are still able to have our service in just one server

So, dare to try it now that Cherokee is celebrating their version 1.0 release, it’s a really nice product worth trying.

This is specially true now that it’s not just me coding, but a couple more people, so we are migrating part of the code to the Zend Framework or more precisely, to a tuned version of it we want to build. The idea is to only use what’s necessary to be able to create interface tests in a very reliable way. Right now this is not entirely possible without Selenium (our testing UI server), and what we want is to don’t need a webserver to run the tests.

We’ll be migrating parts of the UI and will be living with both environments until we’ve migrated all the code.

That’s the reason why since the beginning of the year we’ve been working on an enterprise product. It’s a simplified version of the reader, with new features, but very oriented to smart and seamless integration within a corporate web.

Check out this simple demo we recorded some time ago:

So basically it’s a way to create groups of feeds that generate a single url that can be used to subscribe to from a web widget or similar. Right now we are working on adding new features to this very simple version. We are also working with our first corporate customers to roll out the first pilots inside companies.

If you might be interested in testing it, let us know and we’ll get in contact with you!

Sorry for the lack of communication this past months. I’ll try to write about everything that’s been going on recently with Inkzee.

One of the most important news is that the team is growing. We’ve incorporated new members to the Inkzee family. We are very happy because we have so many things to do at once that we definitely need more hands to help. It’s very hard to keep pushing on your own so I’m grateful of having people helping Inkzee apart from myself.

The interesting part is that things have change considerably. When on your own, you don’t need to explain processes, code or features. Now we need to document everything, from code structure to deploying methods and that takes time. We setup an internal wiki, moved the local trac to a server accessible to the other members and did a couple of training sessions with the new members to get them up to speed.

But we are finally there and starting to work as a team!!

During this Christmas, we gave ourselves a little break and I have to say, it felt really good! We entrepreneurs love to do things, we are generally over excited by everything and it’s rare if we aren’t handling several projects at the same time. The problem is that we don’t know when to take a break. Not resting, not knowing when to stop and watch movies with popcorn affects the performance of whatever we do.

Exactly that was what was happening me. My performance in the last weeks of December was terrible and I couldn’t concentrate in my coding. So, having been here before I immediately recognized it as what it was, lack of resting. So, I took 10 days of Christmas holidays.

Now I’m back, with a lot of energy for the new year a full of surprises for the next months! So, stay tuned for some news and remember, take a break from time to time, it’s good for your health

From day one we didn’t want to use complex and cryptic protocols so we chose xmlrpc for our communication channel. It was easy to implement, had wide support in php and python and was very easy to debug. We knew that at some point we would need to switch to a more efficient protocol, but we didn’t know it was going to be so soon.

After doing some extensive benchmarking we realize that the through output of the protocol was very low, not only that, if too many xmlrpc connections were spawned it would eventually consume all resources of the process (file descriptors, sockets and memory). This was a painful lesson to learn, but we did. So we switched to the most efficient protocol we could found, that is binary. To be more precise we employ python’s cPickle binary protocol. Saying that the use of this is orders of magnitude more efficient is not even close

So after switching each subsystem to the new protocol we saw the load of the machines going down. As with all big changes in the backend of any system, it took a while to stabilize it. To avoid any havoc we actually put it into production subsystem by subsystem so during some time we had both protocols running at the same time.

And so, always remember that the choices you make will come to hunt you if not done correctly

We just wanted to wish you all a Merry Christmas and a happy New Year!! We hope that this incoming new year is even better than the past one and we wish you all the best of lucks!

Turkey cheers from the Inkzee Team

We meditated the decision, and we got to the conclusion it was worth the effort and that we would, not only learn a lot during the process, but that we would get greater productivity on the long run. So off we went to implement the 5 steps Eric proposes:

1. Continuous integration server: We created our own home brewed mini system that allows us to add new tests that get started every time we need them. For now it’s a rather rudimentary system but does the job handsomely.
2. Source control commit check: We already had a source control system, so we just added all the commit checks. At first we thought this was a waste of time, but with some time we realized how many bugs we had/introduced without this. We added 3 very basic checks, a python syntax check (with pylint) , a very basic php syntax check and the automatic triggering of all the unit tests we have. These checks also enforced a common style rules for all the code under SVN, a somehow gruesome task at first (all the old code was triggering the syntax and style checks), but very rewarding in the end.
3. Simple deployment script: We had a very small deployment script but we fine tuned it so that it work flawlessly with the new AWS infrastructure. As for now, we make all the deployments in the same fashion, something that reduces the number of flaws you can introduce in this step. The script also creates a backup copy of the previous running code, in case you need to revert to the old version because of some critical error. As to date we’ve never needed to revert anything
4. Real-time alerting: We introduced a lot of Munin plugins to monitor all type of parameters on our servers. We even developed some very simple Munin plugins for Tokyo Cabinet. We are still missing some alerting here. We have all the graphs but we still need to setup some alerting framework to detect weird situations.
5. Root cause analysis (5 whys): This is probably the most critical part of the process. We’ve realized that this 5th step is what makes all the previous ones work. This process is an iterative one, you start with a small thing but with time, it will grow into an amazing process that’s is able to detect the slightness problem way before it makes it to the live servers. We’ve become used to always ask the 5 whys and it’s helping in improving the quality of the software we code.

So all in all, it’s an incredible experience we are still figuring out but that to date, has been impressive. We’ve managed to do 14 code uploads in a single day with no bugs whatsoever, plus stopped introducing small potentially critical bugs in the code base and all the way into production. If you want to give it a try, please do read the original posts by Eric Ries, you wont regret it.

Sorry we’ve been so silent lately. It’s been a mad fall. Last post we wrote about a migration to Amazon Web Services (AWS). To be fair, the migration was a little painful. We had to set new machines with new software versions that made the system a little shaky for a while. We also run into some really nasty performance issues with the new AWS instances.

First of all, if you have a Debian image as we had, be sure to use the correct fine tuned C libraries for Xen Virtualization (that’s what AWS runs on). We started having a huge load on the machines and after investigating we discovered we were using the wrong C library and it was really killing the server.

So after changing the libraries and setting up images that would autoconfigure themselves with the latest config files and code from the latest release, we setup the monitoring software. That’s when we discovered the magic of the AWS instances. When you’re the only one running stuff in your datacenter, everything runs smoothly, but when the US East Coast woke up, the performance of the machine would just drop alarmingly. The reason is that, when no one else is using the resources of the machine your VM is running, you’ll get extra resources, but when all the resources are being used, you’ll get capped to what is configured for that AWS instance. In our case, that was killing us.

After figuring out this, we started benchmarking the backend and looking for ways to decrease the stress. When we finally did, everything went back to normal. We had to fine tune some parameters but the worse part was done. That was our great AWS adventure that didn’t end up there. In the next posts we’ll write a little bit about different issues we had and things we’ve been doing lately.

We are finally there! We’ve devoted this past month to migrate all of the Inkzee system to the new database we are deploying, Tokyo Cabinet. It’s taken us forever to finish this migration but it’s here to stay. We are still missing some key things, but the bulk of it is already there so we’re starting to begin the migration to Amazon Web Services (AWS).

Right now we’ve managed to build a custom image for our servers with all the tools we need to run the Inkzee backend. Tomorrow and next week we’ll start moving data from the old server to the one in Amazon. This wont be painless and we know it. The architecture is still not 100% stable so we expect some minor glitches but nothing too serious.

So, here we go for step 3!!

Happy summer to all!

The Inkzee Team

We needed a php class that implemented the protocol so we took a look at two of them, Net_TokyoTyrant with Pete Warden’s patch and Tyrant by Bertrand Mansion. The first one supports http and binary protocols, while Tyrant only supports the raw binary protocol.

During the first tests, Net_TokyoTyrant went crazy when inserting over 28000 records over http, so I guess there’s something wrong with that. When we switched to the binary protocol it worked as expected.

Here are some quick numbers:

Net_TokyoTyrant (100000 keys)

Time inserted: 50.3662779331 secs
Time retrieved: 57.7555668354 secs
Time deleted: 34.1996 secs

Tyrant (100000 keys)

Time inserted: 39.330272913 secs
Time retrieved: 44.3433589935 secs
Time deleted: 26.9360201359 secs

The former is slightly faster so I guess we’ll go for it. Specially important is that the author keeps it up to date, which is also a plus!

The Inkzee Team

After setting up Tokyo Cabinet, it’s python binding and Tokyo Tyrant (db server) with it’s python bindings too we did some fast tests. We drafted a new schema-less design for the new database and dumped part of some old data to Tokyo Cabinet.

For those not familiar with the term schema-less, it’s basically a database that has no table structure, that is, everything is stored as a tuple of (key, value). On one side, a key-value database is much faster to read/write but it’s much harder to maintain and keep in sync.

So, we did some queries (read only operations) in both databases an this is what we saw:

Test 1:

• All data from a feed (MySQL):  0.01699 s
• Partial data from a feed (TC): 0.00174 s

This first test wasn’t really fair, as MySQL had to retrieve all fields per record, while TC just had to access a bunch of buckets with fewer fields. We did this first test as it’s going to be the real scenario, currently we retrieve many more fields from a Feed than we should and so, the new query under TC is, not only faster because of the database, but because it’s much more lightweighted.

Anyway, we modified the test so that both queries retrieved both fields per row:

Test 2:

• Partial data from a feed (MySQL): 0.00346 s
• Partial data from a feed (TC): 0.00151 s

Here we can see that both are slightly similar. Again, this isn’t really fair, as MySQL is executing just one query against several that we do with TC. So, we changed the TC query into a multiget request (request several keys at the same time):

Test 3:

• Partial data from a feed (MySQL): 0.003533 s
• Partial data from a feed (TC with Multiget): 0.000845 s

Under exact circunstances it’s clear which one is faster. So, I think we’ll continue experimenting with Tokyo Cabinet and some more real data and see how it performs.

After altering a lot of our current code so that it worked with the new database design we’ve been experiencing problems with MySQL. It seems that, even though the solution makes the overall system much faster (like 3 to 4 times faster), some operations don’t play too well with MySQL and add an unaccepted latency to the system.

We’ve been resisting the urge to migrate to a schema-less database but it seems we have no other option but to transition to it. So, even though we thought we could have the new design working by the end of the week, we are afraid we’ll have to postpone it until further notice. We’ll keep you guys updated though!

The Inkzee Team

That means that if we ought to open up Inkzee we need to redesign the database so it can sustain a higher load of blogs and posts. We are currently working on it and we’ve done great advances. We have a prototype working with the new design but there are still some bugs and problems to resolve.

We hope the new design is finished sometime during this week. We’ll then fire up our test cases and check nothing is broken and once we’re sure the new design is as flawless as we can get it, we’ll release it to you guys! Hopefully you’ll experience a much faster site, not only on a subscription by subscription basis but specially when you request all posts from all blogs.

We’ll keep you posted!

The Inkzee Team

1. Redesign the UI and make it more user-friendly
2. Redesign the database architecture and alleviate the current scalability bottlenecks we are experiencing
3. Move all infrastructure to Amazon Web Services for easy horizontal scaling.

To date we’ve accomplished the first milestone. We did a complete overhaul of the UI for Inkzee. It wasn’t just the design but we needed to add a lot of help features. One thing we’ve seen is that most users just use a small fraction of the features the site has (and there aren’t too many). Because the current version was on alpha status, there where no help pages whatsoever. That meant that only the most advanced users were able to navigate the site. Our goal is to make it plain simple for everyone and so we realized we needed a complete help system.

So, off we went and added a lot of help features into the site. For starters, we added a help page where you a a complete (or so we think) guide to Inkzee. In there you’ll, hopefully, find the answers to the most common questions you might have.

Secondly, we changed most of our cryptic icons and turned them into fully blown buttons with their textual meaning apart from the icon.

We also added a bunch of widgets to the main page so you now will have the latest news of this blog on the main page. We also added a much more interactive “get started” widget than the old plain text layer we had before.

And finally but not least, we added help bubbles all over the interface with tips about the feature at hand.

The first reviews from our alpha users has been very positive but we will love to get some feedback from everyone. Found a spelling mistake? Couldn’t find help about a specific topic? Do you still don’t undeerstand what that button or feature does? Feel free to contact us with your suggestions and ideas!

The Inkzee Team