feedraider the blog

Search

2009-07-19T23:21:00.005+03:00

I spent this weekend on adding search to feedraider. Things progressed with such pace after a saturday lull (great weather -- not enough of that in Finland!), that I've managed to play with a working setup on my home box for most of Sunday. The result? I'm impressed how much ooomph a simple search box can bring to a site and I've only scratched the surface. Good start for sprint #2!

Search should be up and live on feedraider monday evening.

Monday: Search up and running!

Plans for 2.02

2009-07-18T00:16:00.002+03:00

I want a big search box placed onto the mainpage.

That itch will be scratched as the second sprint gets underway with an ETA of July 31st.

Also time will go into addressing the following: spammers singing up to promote NSFW/shady content, feed rotation, metrics, "related articles", ...

feedraider turns 2.01

2009-07-17T21:17:00.004+03:00

Two and a half weeks after launch and a 100 fixes later there's still no proper release notes for 2.0! I'll aim to blog about the functionality more now in future posts. No good developing in the dark.

Visible fixes/upgrades in 2.01:
- Pages are even more lighter and thus faster due adding minifying to javascripts
- Emails had a small glitch at launch time with those mails never reaching their destinations. Manual intervention got those mails to their destinations. Still, some never made it as feedback from a registrant close to 2.0 launch made clear.
- Small changes to what makes it onto the mainpage. The mainpage will be a focus in the coming month with a grand overhaul on the way. Currently there's very little interesting going on, it can get filled up with NSFW content, multiple languages are mixed together... a mess!
- Some pages gave a 404 error

This is a big project and with the recent release I'm more "in control" of what I have created. As there's less oil leaks and smoke coming out of the engine behind the service I have more focus on new functionality and features. More on this in an upcoming post.

Enjoy!

Duplicates

2009-07-12T22:13:00.003+03:00

There are some duplicate articles visible that will be fixed in a few hours time as the database is swept through a filter.

The cause? A script I'm testing ran amok. No harm done, just temporary chaos.

Roadmap for July

2009-07-09T00:15:00.004+03:00

I'm planning two sprints this month with releases on the 17th and 31st. Contents of the sprints will be announced on blog posts during those days. July will skew towards many backend changes, fending of spammers, fixes for known bugs and giving me a better idea on how fast I can move to improving the service from a user perspective.

Improved Feed Handling

2009-07-04T23:06:00.004+03:00

At launch time feed handling was a bit iffy. It all worked, but there were problematic edge cases that required manual intervention. The launch helped me figure most/all out and come up with a continuation plan that took most of the week to carry out. I looked into the "Circuit Breaker" pattern and implemented it with added feedraider specific requirements. The end result -- solid performance and less worries for me!

Favicons are offline (=default icon for all) until an upcoming fix.

Feed freshness is also being continually looked into. I noticed there are several splog submissions per day. I wonder if a certain SEO e-book is to blame. I won't pay for it to find out. These splogs will likely have some magic applied to them in near future with a disappearance spell.

Feed Update Lag

2009-06-30T22:00:00.002+03:00

As I noted to Andre earlier the 2.0 rollout came with a slight snag, that is, a lag on feed updates for a few days until things are back to normal. This time schedule calculation seems to be holding and at this pace feeds are fresh tomorrow afternoon onwards.

2.0 live!

2009-06-28T00:19:00.004+03:00

After a long, long development period that had its fair share of problems the new feedraider is live! 2.0 is mostly a much needed behind-the-scenes change that'll ensure the future of this service for years to come, with new features rolling in from 2.1 onwards.

If you run into any problems leave an email, comment here or on the getsatisfaction support forum.

Migration Check: Part 7

2009-06-19T01:47:00.008+03:00

Closing login, registration for the the final push :knock wood:

2:30PM: Data migration beginning in a moment. Preparing the lunch for MySQL took a few hours. The process will take about 24 hours.

Sunday 8:37PM: Slight bump on the way slowed progress, causing a few hours of delay. Other than that migration running on schedule.

Monday 11:21PM: Looking back I think Midsummer did eat one day while out partying. Migration of data has gone by OK. Now doing integrity checks on the data, which seem to take forever.

Wednesday 01:48AM: Whole move is on the final phases.

Wednesday 8:08PM: Down with fever. Not so good timing to get ill.

Saturday 4:57PM: I plan to turn the switch this evening to get logins+registration back up for everyone.

Migration Check: Part 6

2009-06-09T22:48:00.002+03:00

The migration on the new production server managed to find itself troubled in so many ways the whole progress stalled by over a week while troubleshooting the matter. I blame MySQL. That's been worked around. I almost feel like a DBA by now.

Current ETA being still "in June".

Migration Check: Part 5

2009-06-02T23:00:00.003+03:00

Migration humming has not stopped yet.

Meanwhile I've ordered new dedicated servers. By July feedraider will be running on shiny new Intel Core i7 920s (4x2.66GHz) phasing out the current dual-core setup. On top of two added cores the boxes will have two times more memory stacking in at 12GB's each.

Migration Check: Part 4

2009-06-01T23:42:00.001+03:00

Things are humming in the background on schedule.

Migration Check: Part 3

2009-05-30T23:27:00.002+03:00

The big picture over the migration process has been slowly cementing itself. It won't be quick due all the requirements I've imposed on it (there's about 20 steps involved).

Currently I'm planning to begin the process starting next week, while keeping the current version up. Slight downtime for 1.1 in the beginning. The migration will run for many, many days with 2.0 opening doors to public sometime in June.

Migration Check: Part 2

2009-05-29T21:22:00.002+03:00

I've glanced under the hood of the migration process and managed to turn days into hours. At the same time I must say this has been a great stress test against the database schema.

Migration Check

2009-05-28T23:14:00.002+03:00

Running the migration against production data proved that it'll take quiet a while (over 2 weeks!). Looking into optimising the migration process.

One Click Is More Than Enough

2009-05-28T00:56:00.002+03:00

Installation instructions edited (documentation always manages gets outdated). One click deployment in place. Empty site works! Now to testing migrations...

Monday Jazz

2009-05-25T23:26:00.002+03:00

No blockers left. No known release-critical bugs.

Next up I'll test the service in private, performing a test-migration of the current version and have it running for a few days in the background. This'll give me an idea on how long it'll take to migrate the current database (I'm expecting two-three days of processing).

Saturday Blues

2009-05-23T19:39:00.003+03:00

Pe-hew!

Many, many bugs fixed that have gone unnoticed till now. The code base has shrunk over time, but I suppose its the magnitude of the effort going on here that allows for things to still go wrong in certain scenarios.

Time also sinks into small, yet big hurdles like sane latin/unicode conversion to utf-8. All the encoding and converting can drain what feels like a near bottomless pit of man-hours.

Currently left on the table: private testing before going live on a staging server (as opposed to my home box), figuring out why HTTP_REDIRECT is missing as usable variable in PHP and a few others.

I can't wait to start on 2.1!

update: Moved forward to find myself taking a few steps back while at it discovering a few more minor bugs. Where do these pesky buggers come from?-) I noticed I've worked the past three days hard as today I couldn't remember if I'd had dinner.

All Over The World In Near Real-Time

2009-05-20T00:41:00.002+03:00

I've made good progress into tackling the problem of keeping all feeds in the feedraider database fresh. The code behind the system is almost done and should be enough to meet the needs for 2009 (and beyond with a bit of added work). This was one of the hardest problems on the table and I'm glad I can soon put it on the "tasks finished" pile.

The Engine Was Running The Creator Wasn't

2009-05-18T00:13:00.002+03:00

A relaxing weekend ahead with little work done on feedraider. Sometimes I do keep a breather too. ;)

Engine checkup

2009-05-14T23:45:00.003+03:00

The logs don't lie on one front: It takes five hours to rotate through all the feeds currently. I blogged about this problem prior, and will give it attention prior to 2.0 launch to have a fresh offering feeds-wise.

The Engine Purrrrs

2009-05-12T21:15:00.003+03:00

Changes have been successful and the engine isn't spilling oil all over the place anymore. That's my impression at first glance. Checked out Moa'2009 on the way home, meaning it's nine thirty and I haven't had time to grok all the logs yet!

Engine plan changes

2009-05-11T23:19:00.002+03:00

The big change in 2.0 is underneath the surface. Everything about how feedraider works has been changed. It's a big change and big changes bring a lot of uncertainty into the system. The major bug I stumbled upon few days back has to do with the self-maintenancing part of the engine. I aim to have most of the operations work without admin intervention. In this case the "AI" was a bit too heavy handed causing problems. I've now taken a more lenient approach. This means another night of testing to find out the results! The downside of testing a big system; no immediate results.

Where Was My Mind?

2009-05-10T17:33:00.004+03:00

Naturally. Naturally there had to be a major bug to delay things. After some testing I've pinned down the cause and am testing some more to confirm that the fix has no side-effects. At times like this I do wonder where my mind has been when the offending code was written.

Update: Bug squashed. Unfortunately testing the whole system takes hours of processing time.

Final Testing

2009-05-04T23:46:00.003+03:00

With MayDay celebrations behind I've been getting back to business and going over production deployment. That's how ready things are -- bar stumbling on some major bug at last minute. I've been testing everything with a comb for quite some time. After launch, will see how much "obvious" stuff I missed.