Qrowd

The Genie is out of the Bottle

Robert Yeager — Wed, 13 Aug 2008 18:48:37 +0000

A couple of nights ago I decided to post some links to Qrowd on a few web directory services, with the idea of starting to drive some traffic. I thought that by posting on some C-level directories that I could slowly build up traffic, in order to minimize swamping the server.

Well, the trickle I was expecting has turned into a flood. I’m now getting a geometric progression of increasing traffic that has exposed performance issues both with the software and with the server. What started as a few new visitors per hour is now a new visitor every few seconds. I am both pleased and terrified.

The first weakness exposed was in the software, in the way a new weptop was deep-copied from a setup account to create the populated webtop that greets new visitors. As a stopgap, I have greatly reduced the number of pages and widgets in the setup webtop until the software can be optimized.

The second weakness has been with my server. I currently use a shared hosting provider. Besides running my other product Cooqy on this server, the server is shared with a few other customers. After some troubleshooting, I am seeing the MySQL database log long-running inserts/updates, primarily at the top and bottom of the hour. It seems to me that another customer on the box may have a batch job running every thirty minutes that is tanking the server’s disk I/O performance.

My hosting provider is currently helping troubleshoot the performance issue with the server. I’ll give them a day or two to try to find the issue.

My ultimate goal with hosting is to eventually use a cloud computing service like Amazon’s. I am waiting for Amazon to provide persistent storage for their EC2 service, promised sometime “later this year”. Until then, I may have to upgrade to a dedicated server if the traffic continues to grow.

The feedback about Qrowd I have seen by other bloggers critical of the performance is entirely understandable…after all, Qrowd and Cooqy are running together on a shared server along with a few other customers. Cooqy by itself has a rather sizeable amount of traffic. People like to complain, so throwing darts at Qrowd because of performance is all too easy at this early stage. When you consider that I only pay $120/month for hosting a high-traffic site like Cooqy with its millions of hits and then start a new venture like Qrowd on it as well, then you can understand the situation.

To paraphrase Rumsfeld, you launch with the software and hardware that you have, not with the software and hardware that you want.

Update: My hosting provider found an abusive customer on my shared box, who has now been booted off the machine. So much for VPS software being able to manage system resources. Hopefully this will improve performance.

Coming out of the Coding Cave

Robert Yeager — Wed, 30 Jul 2008 04:43:22 +0000

More than a year after coding began on Qrowd, I finally have it working more or less like I first envisioned. It took over 920 subversion checkins of approximately 50,000 lines of code, most of which was done after midnight while everyone in the house was asleep.

I’ve now dumped my previous start page (Protopage…vaya con dios) and started using Qrowd’s new mashable webtop, to help find and fix issues with the new code. There’s nothing like eating your own cooking! With Qrowd’s webtop, I can configure refresh rates of my news feeds and setup news alerts. I have lots more ideas I want to build into Qrowd, but at this point it is more important to start getting the word out and see how people react to the new tool.

To illustrate why Qrowd is a better start page, today an earthquake struck LA. By adding a webtop alert into the Earthquake Map widget, my webtop will alert me whenever a major earthquake strikes the west coast. Later, I will probably add an e-mail widget to also send me an e-mail or SMS message, if the webtop alert is not cleared after a certain amount of time. The modular approach of Qrowd also means that as new software parts are added (by me or other users), other widgets can take advantage of those new capabilities.

Unlike all the other start pages out there like Netvibes, Pageflakes, Protopage, iGoogle, and Yahoo, Qrowd provides the ability to mashup native widgets on-the-fly, customizing them to do things that perhaps were never imagined when they were first built. I certainly didn’t think of adding alerts when I first built the Earthquake Map widget. The map widget was built early this year; the alert widget was built just a few weeks ago. But when the LA earthquake struck today, I wanted the Earthquake Map widget to be able to indicate alerts in the webtop. Using the visual wiring editor, I had it working the way I wanted in just a minute or two. Qrowd will become more powerful as more parts and widgets get added.

All other start pages are dumb containers for the widgets that they host. Choosing a start page came down to liking how the windows worked in laying out the widget containers onscreen. Perhaps the only real differentiator was the integrated feed reader. The Web 2.0 Ajax start pages just added a thin veneer of improvements to the start page concept, primarily just drag-drop placement of windows. Qrowd redefines the start page as a customizable information processing environment.

Admittedly, I’ve had to take some shortcuts to get Qrowd ready for launch. I would prefer to have lots more native widgets, as opposed to wrappers that wrap existing widgets. Native widgets show off the power of the visual wiring editor and the webtop, after all. But, I am only one person with so many hours in a day. I can only hope that other people will “get it” and jump on the Qrowd bandwagon.

Part Editing Enhancement and a Chuckle

Robert Yeager — Sat, 05 Apr 2008 04:14:07 +0000

Lately I’ve been spending some time enhancing the part editor with some conveniences for part developers. Now, the assembly can be executed directly from the part editor dialog, so that part implementation changes can be tested extremely fast. Coding is an iterative process, so it was critical to add this feature. Part developers will now find that editing parts with the online editor is about as fast as it would be testing them with a local IDE.

I have been very active on the OpenLaszlo developer forum the past couple of weeks, in an effort to sharpen my technical skills by helping OpenLaszlo developers with their coding issues. One person came along who was seeking to evaluate OpenLaszlo for a “big project”. If you have ever been on the receiving end of a software review process, you will probably find my answers amusing.

Latest Update: Part Properties

Robert Yeager — Sun, 23 Mar 2008 20:32:09 +0000

Significant new functionality was added to Qrowd last week, to provide for “part properties”. Beta testers who looked at the sample eBay Showcase widget before part properties existed saw that the part had over a dozen settings exposed in the wiring editor. About half of the settings were to feed the part with data, the other half defined runtime options like colors and animation speeds. Now, part properties are used to control all the runtime options.

Part properties are defined much like the part definitions, using XML metadata. Part properties can use controls like entry fields, dropdown lists, sliders, and others to let users tweak runtime settings for parts, such as colors, animation speeds, and other settings part designers want users to be able to tweak. Part properties can be grouped into tabs, so logically-related properties can be grouped together.

From within the wiring editor, when an assembly is executed a new wrench icon appears at the top of the engine output window to allow all the part properties exposed by parts within the assembly to be tweaked.

The primary benefit is that all part properties contained within an assembly get exposed in the Widget Wizard, so end users can now easily customize widgets.

Now that part properties are implemented, some effort will be devoted to adding some more sample parts and assemblies into the system. First up is a new Map part that is already available. The map part currently uses a Yahoo! Map and only supports the Flash runtime. A DHTML map will be integrated soon, as will support for geo-coding. Additional parts will be developed in the days ahead based on feedback from beta testers.

The private beta period will continue into the foreseeable future.

OpenLaszlo 4.1 DHTML: Worth the Wait?

Robert Yeager — Thu, 07 Feb 2008 03:02:54 +0000

Qrowd was built using the nightly builds of OpenLaszlo’s 4.x releases. OpenLaszlo’s 4.x releases are where the DHTML runtime was first introduced over a year ago. The 4.0.x releases until recently were pretty much disasters, in the sense that not only was DHTML not a viable runtime for serious application development, but the Flash runtime support was also torpedoed in the process. Taken together, OpenLaszlo went through a year-long period of time in which serious development could only be done using the 3.x Flash-only runtime.

After a long year of suffering the bleeding edge nightly builds while DHTML support gestated and Flash support was stitched back together, I believe OpenLaszlo 4.0.x is getting ready to blossom into 4.1 this spring, the first release in which both SWF and DHTML runtimes should be ready for development of reasonably complex applications. Based on my experience with the latest nightly builds, DHTML support is still not quite ready for prime-time large-scale applications, though.

Qrowd’s visual wiring editor was developed to run with both Flash and DHTML runtimes, for no more reason than to see how far DHTML development could be pushed. While developing on nightly builds had its ups and downs, progress was steady up through October 2007. Especially on Firefox, at one point in October the DHTML runtime was performing even better than the SWF version on Firefox. IE support has always been very problematic, however. More often than not, IE DHTML simply doesn’t work. A DHTML runtime that doesn’t work with IE isn’t of much use. Sometime after October, performance of DHTML on Firefox tanked, which was really disheartening.

As of the more recent nightly builds, Qrowd’s wiring editor launches in DHTML with Firefox and Safari, but not IE due to issues I have reported in their issue tracking system. On Firefox and Safari, the look of the app is virtually identical to what appears in Flash, but the performance is too slow to be usable. (Anyone wanting to take the DHTML wiring editor for a test drive themselves can leave a comment or e-mail me.) I keep hoping that whatever has tanked performance since October will be fixed, but so far that hasn’t happened.

On the upside, less sophisticated apps like the widgets that can be created with Qrowd run rather well in DHTML with Firefox, Safari, and IE. The differences between the Flash runtime and the DHTML runtime are rather subtle. The download footprint sizes are roughly equal. DHTML provides a better fade effect of text. With Flash, text fading requires embedded fonts, which greatly increases the download footprint. Flash offers shadow and glow effects that aren’t available in the DHTML version. The biggest downside of the DHTML runtime as far as I am concerned is that although the “major” browsers can run the widgets, notable browsers like Opera cannot. Then again, it should be possible for the Safari browser on the iPhone to run the DHTML runtime, since the iPhone doesn’t currently have Flash support.

So, was DHTML support worth the wait? I believe yes, but only if all the IE gremlins are addressed. I am happy to have a DHTML runtime that supports the major browsers (IE, Firefox, Safari) that together comprise over 95% of the browser market. One huge advantage of the DHTML runtime is that it makes for better integration with HTML websites. Qrowd uses DHTML on its website here and here. In situations like this, Flash won’t work as well, primarily due to sizing of the canvas. Since the DHTML canvas is a DIV element, I was able to grow it as the list expands, which in turn expands the CSS border. Doing this in Flash is possible, but would be harder since the Flash app would need to invoke Javascript to stretch the dimensions of its container. Having DHTML offers better solutions for certain problems…another arrow in a developer’s quiver of tools. Looking ahead, the work Laszlo Systems invested in the 4.1 release should provide many future benefits, as the technical groundwork has reportedly been laid to more quickly add future runtime options to OpenLaszlo. Silverlight may be a compelling choice, after it matures and penetrates enough of the market.

Unlike Javascript DHTML (AJAX) toolkits, OpenLaszlo provides an easy-to-use application stack that makes writing cross-browser web apps palatable. As with any higher-level abstraction, the benefits of less effort and shorter development times come at a cost of bulk in the form of the runtime library and some loss of fine-level control. For me, the benefits of an open-source, cross-platform multi-runtime web app software tool are clear…I only hope that the delay of OpenLaszlo 4.1 hasn’t caused a lost market opportunity. I have experienced many times when a superior technology loses out to an inferior technology’s marketing nimbleness…anyone recall OS/2 Warp vs. NT when it was Not There?

Step by Step

Robert Yeager — Fri, 01 Feb 2008 08:43:21 +0000

Qrowd reached an important milestone today, with the launch of the private beta. Invitations have been sent to a select few people to begin the process of finding defects and gathering feedback to help make Qrowd a better system.

I am very happy to have reached this goal. Qrowd to date has been virtually a single-handed effort that I started last June. I have worked on Qrowd maybe 50% of the time, equating to about 3-4 months of real effort.

The results so far are a system that does as much as Yahoo! Pipes and Microsoft Popfly combined, with the potential to far eclipse their capabilities thanks to the generation of both Flash and DHTML runtimes for client-side widgets via OpenLaszlo, along with the ability to process data on the server to create RSS and JSON feeds.

One of the coolest sample widgets is a MP3 player that uses SeeqPod’s API to find MP3 files on the internet that can be played and (even better) downloaded. I knocked this widget out in a few hours, so it is very crude. Still, the availability of API’s across the web is opening up a new world of possibilities for mixing and mashing up data.

I hope to eventually add WYSIWYG design capabilities into Qrowd, to make widget creation even easier (like Sprout’s widget builder.)

A long list of defects and desired new functionality awaits, but for now I can take a brief pause and enjoy this milestone. I will start to expand the circle on the number of beta invitations over the course of the following weeks and months. Feel free to send an invite request to sales at qrowd.com if you would like to take Qrowd for a test drive!

Looking back, looking ahead

Robert Yeager — Tue, 01 Jan 2008 08:53:40 +0000

Happy New Year! 2007 was a great year, so I hope that will continue into 2008. Cooqy chugged along nicely in 2007, with more and more eBay merchants using the Cooqy showcase widgets to advertise their eBay inventory. 2007 was also busy with many interesting OpenLaszlo consulting projects…so much so that I usually worked multiple projects simultaneously. Any time left over at the end of the day I would work on Qrowd, typically 12am-3am when the house was quiet.

For 2008, I am looking forward to getting Qrowd out the door initially as a private Beta hopefully sometime in January. As companies like Teqlo have demonstrated, the mashup builder space is a hard place in which to figure out what problem to solve. Yahoo! Pipes’ apparently low usage leads me to believe that RSS-only mashups are of little interest by themselves. Microsoft’s Popfly is more of a general-purpose software app builder, yet the apps I have tried are not very inspiring, at least to me. Worse, Popfly and Silverlight are buggy in my experience…when I try to use them I crash or hang the browser, even IE7.

Qrowd will have functionality similar to Yahoo! Pipes and Microsoft Popfly, plus some additional capabilities neither offer. Like Pipes, Qrowd will allow RSS feeds to be mashed up into new feeds. Like Popfly, Qrowd will also provide a general-purpose application builder. Unlike Pipes, though, technically-savvy users who know XML and Javascript will be able to create new system modules that all users can use. In the case of Popfly, Qrowd’s widgets will run in Flash or browser-native DHTML thanks to OpenLaszlo.

I see Qrowd as a foundational technology upon which other higher-level component-based applications can be built upon. One example is the Web 2.0 Start Page like Netvibes, Protopage, Pageflakes, etc. A tool like Qrowd that powered a Start Page would enable a higher level of customization, by allowing more control over the gathering and presenting of web information. Another example of a higher-level tool would be a Performance Management dashboard for enterprises, whereby the ease of data manipulation and presentation customization offered by Qrowd’s visual wiring editor would allow non-technical managers to tweak their Key Performance Indicators without waiting for their IT department to otherwise make coding changes. In my mind, Qrowd is a starting point for building tools that allow a higher level of user customization while imposing a lower hurdle of technical skills.

I’m looking forward to pulling back the curtain. Here’s to a great 2008!

It’s Alive!

Robert Yeager — Thu, 06 Sep 2007 05:10:29 +0000

Any creative endeavor starts in a vacuum of nothingness with fear, uncertainty, and doubt about whether or not what is envisioned can actually be built. Combined with the uncertainty of whether or not the creation will actually be useful and how it will be perceived by others, then you get a taste of what it is like being an inventor/entrepreneur wanting to improve the world with a new invention.

Just a few minutes ago, Qrowd took its first breaths. The main processing engine is fully integrated into the wiring editor and part libraries. Live data from the web flowed through its veins. The concept that started in my mind’s eye is turning into reality.

As exciting as it was to watch my creation come to life, it is daunting to realize how much work remains before Qrowd will be ready to be used by anyone. I call this phase the “productization phase”, where there are usually no technical pitfalls lying ahead, just a whole lot of necessary plumbing that has to be laid. Boring things like database storage, account management, error handling, etc.

Some thorny technical issues do remain, but they are entirely within OpenLaszlo’s DHTML runtime support (primarily with Internet Explorer). The pace of Qrowd’s development has been slowed to a trickle recently, to wait to see if the forthcoming OpenLaszlo 4.1 release will address these.

Still, enough of Qrowd has been built to prove the viability of my original concept. Hopefully Qrowd’s technical development will be downhill from here…

Rome wasn’t built in a day…

Robert Yeager — Thu, 09 Aug 2007 01:25:48 +0000

Having founded Cooqy in December 2005, choosing to use the letter “Q” again for my next startup seemed to be preordained. (Hopefully having the “Q” at the start of the word will prevent confusion with the letter “G”.) Cooqy was a self-imposed project to get myself familiar with Web 2.0 RIA technologies like OpenLaszlo, AJAX, and DHTML. The availability of the eBay API and a sure-fire monetization approach (Commission Junction revenue) made Cooqy a no-brainer in terms of building something new (a Web 2.0 RIA user interface for eBay) that would have an instant revenue model. Still, Cooqy was never meant to be a primary bread winner, if for no other reason than b/c an affiliate business model is precarious. It’s too easy and tempting for the Mothership to steal away the good ideas from their Affiliates. Sure enough, eBay eventually got into the widget game themselves with eBay To Go, which replicate much of the functionality of Cooqy’s widgets.

The germination for Qrowd actually began in December 2006 as a wall-less MySpace or Ning, that would use widgets as the basis for providing social networking features on any web site or web page. A little bit of code was developed, but I never could get passionate about the concept. That idea was eventually shelved as I worked on many consulting projects in the springtime.

Come late May, I was taking a break from consulting to start anew on Qrowd again. Around the same time, I had come across Yahoo! Pipes and Microsoft’s Popfly, both of which left an impression on me. I quickly wrote a “Qrowd Manifesto” document that described how to improve upon these new tools, then launched into full-scale development in June.

So what is Qrowd? Without sharing all the details, Qrowd is:

1. A user interface like Yahoo! Pipes for wiring together components, that will be geared for non-technical people.

2. An open-ended component builder, for technical people to contribute new components into the system. Components may optionally be shared with others. Components are developed with Javascript (or more correctly, ECMAScript) using a provided API. A visual WYSIWYG designer is planned for later that will emit LZX code, for building visual components.

3. Components are of three basic types: data input, data processing, and data output. Data input components suck in data from the web and convert it as may be needed into a common XML format for processing by the data processing components. Data output components may be visual or non-visual. Visual components can render themselves as either DHTML or Flash runtimes.

4. A meta-data engine that can process the wired component assemblies and execute the assembly either in the browser or on the server. Assembly output can be in many forms. On the server, output may be in the form of RSS, JSON, or e-mail. Assemblies running in the browser will be visual DHTML or Flash widgets.

[The purpose and utility of Qrowd is a topic deep enough for a separate, forthcoming post.]

Qrowd is being developed with OpenLaszlo 4.x, which provides support for DHTML and Flash runtimes. The recently released OpenLaszlo 4.03 has some problems with Internet Explorer DHTML support, so this has caused me to postpone further development on Qrowd until OpenLaszlo v4.1 gets closer to being released, which I hope will be around October. Still, OpenLaszlo’s DHTML support has come a long way since I wrote this post. I’ll share my recent experiences with nightly builds of OpenLaszlo in a forthcoming post. It’s not ready for general-purpose “prime time” just yet, but the promise of greatness is evident.

As of today, the Qrowd wiring editor UI for #1 above is more or less complete and runs in both DHTML and Flash. A simple API for #2 above has also been developed, and a rough prototype of a visual designer has been made. A few sample components for #3 above have been made…just enough to test the system. The heart of the system, the meta-data engine for #4 above, is maybe about 75% complete.

One of the hardest challenges in building Qrowd thus far was to find a way to run the meta-data engine on the server for generating RSS and other server-side output. The meta-data engine is implemented as an ECMAScript routine in a Laszlo LZX library, so I needed a way to run a Laszlo-generated DHTML program on the server. Fortunately, I stumbled upon the Crosscheck browser simulator, which is a fabulous piece of open-source software. Crosscheck emulates Mozilla and IE browsers, to facilitate cross-browser software testing. I took the Crosscheck software and augmented it with a live XMLHttpRequest implementation, making Crosscheck able to perform live network data operations for Mozilla emulation. Now, I am able to execute the Qrowd DHTML meta-data engine within Crosscheck with live network requests on the server. More information about this technique is available in this forum thread. Since OpenLaszlo uses the Dojo toolkit for its DHTML implementation, anyone needing to execute a AJAX/DHTML routine in Javascript on the server using live data requests should be able to follow the same technique.

As always, in building something new there is a “if I build it, will they come?” angst in wondering if I am expending my energies in the most productive manner. Creative people go through long periods of dark brooding, followed by short bursts of incredibly productive bliss. At least that’s how I am. While waiting for OpenLaszlo v4.1 to get farther along, I’ll be working on several consulting projects that by themselves are a lot of fun and challenging. While I would rather be working on Qrowd fulltime, it’s also nice being able to put some greenbacks in your wallet.