Raphaël Valyi's blog - tackling the distributed world

OpenERP vs Adempiere, an open letter to Adempiere experts

2009-08-23T18:59:00.000-07:00

Hi, Thanks to Smile.fr my former French employer, I had the opportunity to study and compare quite deeply the main open source ERP's two years ago. I know some of them might be biased by huge financial interests and some others were simply developed without analyzing too much at the existing alternatives, or because alternatives were less apparent at that time (Tiny had closed SVN access and no mature web client yet for instance). But I know, and had even recent confirmations that at least the Adempiere community is made of a lot of independent and very skilled folks. That's why I'm very interested in knowing why they stick to Adempiere why I would myself rather stick with OpenERP. Here is an open letter to them:

UPDATE: wow, about at the exact same hour when I got interested in Adempiere again, Red1, Adempiere Malaysian rock star who was at the origin of the Compiere fork back to 2006, just released a fantastic book about the open source ERP's story. The book focus on the Compiere, Openbravo and even Ofbiz story, largely omitting OpenERP (cited only once). Still there are really really good critics about Compiere and Openbravo VC backed models; I share them at 100% and feel sorry for not having been able to be more explicit about Openbravo back in 2008. In my opinion what made Adempiere success largely applies to OpenERP (a powerful community), I even see some architectural decisive advantages in OpenERP, that's why I'm writing that post. So for folks interested about the genese of OpenSource ERP's, aside from my own study from 2007/2008 here http://www.smile.fr/publications/livres-blancs/erp-open-source, I really recommend you take the time read the Book from Red1, it's not only about ERP's, but also about open source thermodynamics of complex systems in general:
http://red1.org/aaa/WebEconomy.zip

Hi Adempiere Community.

CONTEXT: FIRST OPEN SOURCE ERP SURVEY IN 2007:

By the end of 2007, my former employer, Smile.fr asked me to do an extensive 6 months survey about emerging Open Source ERP (to be found here, sorry French was required http://www.smile.fr/publications/livres-blancs/erp-open-source ). I quickly identified Adempiere in the top 6. But I should say its integration by Smile.fr has been quickly discarded at that time because:

Smile was not so interested in pure community driven open source. In the opinion of Smile leaders, that model hardly ever worked at Smile, probably because they wanted to scale a lot making it difficult to rely solely on individual talents rather than being backed by a strong and reliable editor. I probably see myself more weight in those talents within the oss process because I think there have been an oss speculative bubble about to burst recently and for this successful VC baked oss projects (Alfresco?) remains the exception rather than the rule. Now compare this to talented leaders that led for instance: Linux, JBoss, Compiere before 2006, Rails, Ruby, JRuby, Python, OpenERP... This is yet an other story, but that's why strong editor backed yet community oriented oss projects like OpenERP (formerly TinyERP) and """promising""" Openbravo where favored at the time while unilaterally editor driven oss like Compiere or ERP5 were dropped at the end of the list.
by 2007, the path to a strong web interface was still uncertain for Adempiere given the investment to be made. On the contrary, Smile was really web grounded already.

FEEDBACK TWO YEARS LATER: OPENBRAVO DROPPED, SUCCESS WITH OPENERP:

Now I should say I have been totally disappointed by Openbravo since then. Despite the successive $6M + $12M fund raised by Openbravo, it's still very limited in term of usable built'in features and effective refactoring since their original obsolete architecture, not to speak about the lack of effective community (the vast majority of their forums is frequented by Java/oss newbies, apparently seduced by the massively funded commercial hype, but providing no valuable code/feature/help feedback). They might still have some cache to invest, let's see if their new architecture moves (Hibernate Data Access Layer, modules, new GUI which are all really at the proof of concept stage for now) brings them more community/business momentum, but I should say I'm not really optimistic about that, especially considering the competitive OpenERP global outbreak. On the contrary, OpenERP was for sure a great surprise both for us and our customers (see for instance one tesimony here http://www.erp-infos.com/info_article/m/709/anevia-deploie-openerp-en-moins-de-trois-mois.html ), allowing a sustainable business, even if it should still be considered as maturing (quickly).

RENEWED CURIOSITY ABOUT ADEMPIERE:

2 years after that survey and more than 4 full SMB's OpenERP integrations later, I recently quit Smile.fr to start my own FOSS consulting company based in Brazil this time ( http://www.akretion.com ). While working hard on the Brazilian OpenERP localization, I see that Adempiere is historically more heavily rooted in Brazil (especially because the community divorced from Compiere and because just like in lot's of developing countries, the usual SMB's proprietary ERP's from SAP, Sage and M$ didn't really attack this market yet, giving more room to oss outsiders). This led me to re-investigate Adempiere. My findings is that Adempiere is finally much stronger than Openbravo and possibly than Compiere. Indeed, Adempiere seems what more advanced than Openbravo in the process of escaping raw PL/SQL at the profit of a higher abstraction, Java code in your case. Moreover, Adempiere definitely has a strong community and it seems that with AdempiereGWT, a viable web based interface is finally emerging.

OPENERP VS ADEMPIERE IN 2009, ROUND 2:

Now I'm sorry, I still see plenty of advantages in OpenERP over Adempiere. But some of you here will know Adempiere much better than me so that's why I ask you to defend Adempiere here so that I make a more accurate judgment. Don't take this post as an aggression, just like for OpenERP, I'm really aware of the passion behind the Adempiere project and I really hope you make the most of it, emulation is always good, and I'm simply curious about what I might have missed in it.

I think that it's just too hard to really compare the raw business features beteween OpenERP and Adempiere. I think they are roughly equivalent, at some place X will have a broader scope than Y, but Y will do the thing with less bug or in a more integrated manner, so it's really just to complex to say (while it's easy to say that both along with Compiere have a broader scope than Openbravo ERP for instance). I tend to think that OpenERP has a broader scope, especially in the service area while Adempiere might have less bugs eventually at its functional core.

So, instead of judging pure business functionality, I prefer evaluating more impartial criteria that will probably determine mid and long term business coverage. Would you agree that community dynamism and technology govern this in the long run?

About Community dynamism, I would say that both are again roughly equivalent. Both are FOSS led by very passionate people (both are farm grown, just discovered this!), and also backed by active third party integrators. I think that Adempiere is historically more grounded, but OpenERP community has been growing quicker those last 2 years.
At least Google Insight shows that, but I have other indicators showing also (forum topic counts, partners network, sucess stories...):
http://www.google.com/insights/search/#q=adempiere%2Copenerp%2Ccompiere%2Copenbravo&cmpt=q
(IMHO you can consider Openbravo really overrated here because of all the money they smoke making commercial hype, when you count real third party integrations, figures are very different)

So the next factor is technology/architecture in my opinion. I think that:
- the web is the platform. An other client that the web client might be used too, but a very good web client is required to look appealing to the market
- technology should maximize code reuse bewteen third parties (=> modules repository) in order to boost synergies (eg systemic cost savings)
- technology should boost productivity
- technology should should decrease entry barrier (please no J2EE bloat for the bloat, comes a time where the community should contribute efficiently rather than spend time in compiling standards in committees, just like J2EE unfortunately ended. Bloat that is affordable for fortune 500 ERP's is not affordable for SMB's ERP's. It's not a university where you should make the system happy, real world effciciency is required.)technology should enable very broad customizations. We live in a globalized world where the survivors are the one who maintain an original business model, the others die or get merged. This means that past time where one black box ERP could rule them all is over. Even SAP approach, "we think about lots of standard business, pre-build the thousands of tables and then you need to hire a $ expert to customize it because it's so boring and if you want to develop outside behind the tables it would just cost you $ instead", is being beaten by the open API and open module system approach). Today's ERP should be highly customizable at SMB's costs.

THE MAIN DISCRIMINANT FACTOR I SEE IS TECHNOLOGY/ARCHITECTURE, LET'S GO FOR 10 REASONS WHY I FIND OPENERP MORE VIABLE:

Concerning the technology/architecture topic, I see lots of advantages of OpenERP over Adempiere. But that might just be because I don't know Adempiere enough, so please correct me and develop why you think Adempiere is better for you, I hold no OpenERP share and I'm very open.

1) Is it possible in Adempiere that a module EXTEND any core form (eg view) at any point, replacing, adding, removing form element, rather than redefining/copying the whole view or patching the original view directly in the source code which I consider less maintainable and harder to deploy? I saw few very frameworks featuring OOP at the view level...

2) Can modules extend views and business objects in cascade? In OpenEERP it's very common a customer run 30 modules, 10 being custom, and most of the 10 being only like 40 lines at most. This sytem, many small modules and large bundles of modules really makes the most of code reuse, just like OSGI for you Java folks.

3) Can a module extends a CallOut process (or Controller/Business Object layer if you prefer) and call 'super', like for instance: do the normal thing you should do when I click here and then bring me back to my custom code which do exactly and only what I need. Might be a dumb question, but that still not possible in Openbravo. From my day to day OpenERP work, that would mean way more time, patching/duplicating large piece of codes rather than acting with a surgery scalpel.

4) Is anything you do in an Adempiere client, auto-magically natively exposed as a webservice you can programmaticaly call from any system without extra coding on the ERP side? Featuring access rule and object lifecycle validations? Because in OpenERP, this makes integration so much easier. For instance I wrote an oss Ruby on Rails connector where your Model+Controller in Rails can be seamlessly imported form OpenERP while the View layer fits in Rails, and the connector is less than 500 Ruby LOC, no extra code on OpenERP. Openbravo added that recently but they only support 50% of what OpenERP does out of the box, because they only support Cread Read Update Delete and not business methods at the moment. Where does Adempiere stands?

5) I guess that Adempiere also features a "relative ID" system where you can import any XML fixtures and update existing records with their foreign keys, without any need to think about real database ID. But is it really the case on Adempiere? (For instance is is also a definitive boost with Ruby On Rails). Speaking about this I wish everybody (including OpenERP) moves to YAML as this is mostly hand written and need to be human readable and YAML beats XML here.

6) In term of scalability. We all know that Python is an order of mangitude slower than Java. But investigation shows that OpenERP SQL is quite optimized while 80% of request time is spent in the Postgres SGBD layer, so no point in adopting a faster language for less than 10% of the time (the other being latency over a good network) at the cost of code expressiveness. Also notice that raw PL/SQL interpretation is even way slower than Python... But OpenERP optimizes SQL because it leverages both an ORM level 1 cache system and also a very smart SQL cache where results are cached in columns (so read is only one select) while the cached is flushed in a transactional manner by OOP cache invalidation listeners on business objects. All this makes it very easy to have SQL optimization. Something you would achieve polluting business code a lot if you code in PL/SQL like say with Openbravo. Moreover OpenERP has native support for MPTT (Modified Pre-order Tree Traversal, see http://dev.mysql.com/tech-resources/articles/hierarchical-data.html ) making it only one request to retrieve complex aggregates in tree hierarchies like accounts and stock locations. Does Adempiere feature any of those SQL optimisation, how?

7) I think that the OpenERP Pylons based web-client (Turbogears dependency dropped 4 months ago) is more mature than Adempiere emerging web clients (Adempiere Zk and GWT). I suspect OpenERP web client is used in production in more companies (inlcuding Tiny Saas offer) and is already fully functional and fast. Tell me if I'm wrong. That said, I'm enthusiast about AdempiereGWT but had trouble following the last updates about it.

8) OpenERP is component oriented, making it so easy to compose your screens using various sub-views of all your business object tree. Is Adempiere like this or is it the Openbravo way (page oriented) where you see a record and then linked objects are always shown in other tabs rather than in the same view?

9) Finally, I know that both OpenERP and Adempiere (and even Compiere) feature a powerfull workflow engine which still lacks in Openbravo (what old code the hell did they forked?). None are a standard (but BPEL might be over-bloated for foss), so I see no difference here.

10) I won't go in a language war, but in my Opinion the Java language is being slowly abandoned ( http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html ) at the profit of less verbose langages, some being dynamic (Ruby, Python, Groovy...), while some statically typed (Scala, Clojure, D...). Notice that on the contrary, the Java Platform and JVM (portable and JIT bytecode execution which is independent from the Java language) still have a very bright future. While having been a Java fan boy by 2003, I find Python less verbose and more effective to code the business layer. Is isn't a very big win as you might argue a statically typed language makes the platform core stronger in Adempiere, which I would agree with. Adempiere might also slowly move to JRuby/Jython/Groovy JVM backed dynamic languages for its business layer while OpenERP might have a smooth transition to Jython enabling Java/scala for its very core, so no real big issue here, exept that personally I think I'm more effective today, as an integrator, writing fewer lines Python business code in OpenERP rather than in Java.

Now I easily admit some issues with OpenERP, mainly:

adopt a large test suite to avoid bugs/regressions,
set better community management process to deal with required code refactoring, API changes, release planning,
better ensure modules play well together to minimize runtime/integration time bad surprises,
consolidate financial periods ending processes and period financial reports. But so far, I think this is being solved right now and OpenERP is currently already very usable for many companies.

WHAT IS YOUR OPINION, WHAT DO I MISS HERE?

Still I'm very curious, what you Adempiere folks will answer to those points? Are you aware of those pros of the OpenERP platform (I recognize that knowing several ERP's deeply is hard)? Or is it me that misses Adempiere key points? What are those Adempiere pros over OpenERP in your opinion?

Python runtimes for OpenERP, Jython, Unladen Swallow, where do we stand?

2009-05-24T15:24:00.000-07:00

Hi,

little time ago, I decided to showcase the world a proof of concept of what I assume the most mature/promising oss ERP so far - OpenERP - running on Jython, the Java based Python interpreter. I made a branch, adapting the OpenERP database connection use zXJDBC+JDBC Java Postgres drivers, escaped code that were not running on Jython yet (C libs wrappers). That attempt has been related here:

http://rvalyi.blogspot.com/2009/02/openerp-running-on-java-jython-round-1.html

But before all, just to make it clear: I'm posting here an update of the Python runtime situation for OpenERP. That doesn't mean I consider this is a very important issue for OpenERP, it's just that there are reasons to hope for a convergence toward the Java platform over the year, nothing more. On the contrary, my only very hot topic for OpenERP now is code quality, testing and business code refactoring/cleaning. Still, that might be for an other post, I'll now only post an update about the Java convergence and Python runtime perf status.

Now, 3 months after my first post, were do we stand? There are mostly plenty of great news as I'll expose.

But again, why Jython?

1) Speed in the long run

I prefer to make it clear: for one, OpenERP v5 is now fast enough for a lots of situations. I can't see any performance issue to deploy it in large companies and it's actually being done in several already (it might not always compete in term of native features or quality in large companies yet however). And second, yes I know that in most enterprise systems, the bottleneck rather lies in the database load and in the client/server bandwith usage/latency. Now this is especially true when the language layer is fast, for instance with Java. In Python, and OpenERP especially, the language layer might not be the bottleneck, it's still common to see 30% of CPU usage eaten by Python. So any significant improvement here will also help.

And SQL requests of OpenERP are already quite optimized in v5. For instance they try as much as possible to handle n records in an O(1) or O(log(n)) time. They also have database transactional cache thanks to their fields.function + store=True/Hash of invalidation trigger feature. They also model SQL hierarchy using MPTT. So at then end having a faster Python runtime, whatever it is will certainly help.

2) Enterprise support

Speed is one thing, but an other thing that Jython would bring to large corporations is: easy enterprise integration. In Jython, any Java class/lib can be seamlessly invoked. And Java, thanks to its design by comity mantra has at least all sort of mature features the enterprise world like to rely on: good SOAP integration (we recently had some pain with the Python Soappy implementation on the contrary), standard ESB's, JCR, JDBC, JMX, BI tools, ETL's...

Google Unladen Swallow is likely to provide the best short and mid term performance:

Still, OpenERP is probably to enjoy the best short and mid term performance thanks to Google commitment with their 'Unladen Swallow' branch of CPython. In an interview, the creators of Unladen Swallow explained that Google has a lot of "legacy" code where the high level plumbing is done in Python while the low level algorithms are done in C++ and invoked via Swig wrappers. So they said they would certainly have look in Jython for long term performance, but they also need short term performance for standard CPython and that's why they launched the Unladen Swallow project.

Unladen Swallow, wants to bring a simple JIT (Just In Time) compiler to Python and do other standard optimizations. They claim they could rich 5x perf increase by the end of 2009. Those improvements will be merged back in the standard CPython distros. So far they already have a solid 20% perf improvement. They might even try to remove the Python GIL efficiently.

In the long run, even if they relied upon the LLVM, that's however unlikely they build a Virtual Machine as sophisticated as the Java one (often considered as the best VM, especially now that they added support to dynamic languages with the InvokeDynamic JSR). So if Jython then focus on performance as they announced at PyCon 2009, Jython might win the perf war in the long run. Anyway the enterprise integration I was talking about would only be achieved on Jython.

OpenERP on Jython might be a reality by Q1 of 2010.

Yes, several very good news allow me to hope this could be done. Let's enumerate them:

most of the C libs wrappers will be avoided, Tiny, the editor of OpenERP took that commitment:

- Turbogears for the web-client layer -> done! The full blown Turbogears is no more required, on the web-client trunk branch (already very stable), Tiny completely removed the Turbogears dependency and only depends on CherryPy (pur Python) now. At the discretion of few Jython bugs, OpenERP web-client should now run unchanced on Jython.

- mx.DateTime: Tiny took the commitment to remove it in the very next months!

- the minidom XML lib: Tiny took the commitment to replace it by the Etree lib which is implemented on Jython. This as already been started in community and editor branches:

https://code.launchpad.net/~ajm-tech/openobject-server/server-50-lxml-fields-view-get

https://code.launchpad.net/~openerp-commiter/openobject-server/pap-etree-trunk

- libxml usage will be removed too.

- I investigated a bit around ModJy, the Jython standard Java servlet wrapper, and it's should be quite easy to put the OpenERP server on ModJy using a WSGI wrapper.

The best is what is being done at the database/ORM level: OpenERP is moving to SQLAlchemy!

My work on the Jython branch has been to workaround the psycopg2 native driver and replace it by zXJDBC, the Jython wrapper for JDBC + Python DPAPI2. Well the good news is that OpenERP is going away from psycopg2. They will indeed move all the SQL generation logic to the standard and awesome SQLAlchemy ORM. And SQLAlchemy is Jython compatible out of the box. That's great to see them coming back to standards (SQLAlchemy was not mature when OpenERP was already in business).

Tiny already began that work on a dedicated branch:

https://code.launchpad.net/~openerp-commiter/openobject-server/server-sa

and I encourage the community to help them do it right. Also notice that they already used SQLAlchmey for SQL generation in their emerging BI OLAP cube.

SQLAlchemy is not only a good news for Jython support. It also means that OpenERP domain (SQL filtering) logic is going to become more subtle and powerful. Currently, that part of OpenERP was not as the level of the best ORM's around such as SQLAlchemy or Hibernate. Even ActiveRecord (Rails) was doing a smarter joint job. Still, among the mature ERP's it was probably the best available, years ahead Compiere or Openbravo with their millions of pure pl/SQL legacy code (Openbravo recently supported Hibernate in their 2.50 platform, but their business code remains millions pl/SQL code lines lurking in XML CDATA statements and will still take years to be migrated to Java if ever it's to be achieved).

Oh and yes, That SQLAlchmey transition will support database independance as there are SQLAlchemy adapters for most of the market SGBD's. Especially, MySQL and Oracle will be supported. That's great because it will strengthen the OpenERP community. In some organizations Oracle is not an option (and it can arguably provide marginally superior perf), while MySQL has a much larger commuity penetration than PostgreSQL.

Unlike most large software projects, OpenERP technical quality improves slowly but keeps improving

With all that refactoring being done, OpenERP belongs to those rare kind of compex enterprise projects where the code improves over the time rather than those going desperately to chaos as businessmen add layers of crap over crap as the commercial deals are made. Also beyond the ORM transition and the webclient refactoring (removed Turbogears+changed for Mako templates), Tiny also rewrote their reporting engine an just allowed the Mako engine (a powerful standard).

My vision of software engineering is indeed pretty much a thermodynamic vision: code quality has to improve over the time much like a living system should not generate entropy in its scope. Design mistakes can very easily be done along the road and once tons of business code is build on non optimal models, then you are all screwed up: any energy you will invest in trying to correct a part of the system will actually result in a larger energy waste, pretty much like if you would like to refresh your home by letting your fridge door open: all you will get is actually get it warmer.

Why this claim? Because in business driven software, it's always easier in the short term (the only business scope as the current economic crisis teaches us), to make money supporting your existing customer base. Meaning that once mistakes are done, it's always too expensive - and thus never attempted - to re-think the core abstractions. Instead, those companies tend to make several abstractions cohabit, prefer invest in building chaotic non abstracted code instead of re-factoring the concepts that could be abstracted in the core platform. And once you have millions of lines of chaos code, you are never going to factor them back into clean/intelligible/maintainable concepts. That's pretty much how the money has been wasted in today's proprietary ERP's and even a in few claimed open ones. So they might have lot's of working features appealing to collapsing traditionnal industries, they are facing exploding maintenance costs and can't adapt further to the new emerging business as will the few viable oss ERP's do.

So overall, by early 2010, I think we should be able to see an OpenERP distributed also as a war archive you'll deploy in one click on your enterprise webserver such as Glassfish. This might shift OpenERP adoption from small SME's to large organizations which is always good because it will fuel more heavy engineering inside the project. And at the end, even the small businesses will benefit the resulting quality/usability improvement. Still I don't expect extra perf from Jython before Q2 2010 at least.

OK, enough said about Jython and Python runtimes. All that enthusiasm should also not shadow my very urgent press to Tiny to increase their quality (less bugs, less regressions). An efficient way to achieve this will be by taking the test first approach way more seriously than they have been doing so far. They just took a commitment to this too at their May Community meeting so we will see. Given what they did in the recent past (transition to international, transition to an open distributed forge, English documentation, transition to an editor business model, performance issues fixes...), I hope they will manage that challenge too.

magento-openerp-synchro module: we moved to Launchpad!

2009-03-03T15:37:00.000-08:00

Sure Google Code Hosting is great,

But we are now seeing too much interest in that connector poping all over the world to continue the development in a centralized way. Also, we have been contacted by Jordi Esteve, the primary extern OpenERP contributor (after Launchpad stats) who wanted to jump in and join efforts, so he also asked us to move the contribution process forward.

That's why we have the pleasure to announce you that we migrated our development trunk from Google Code SVN to Bazaar on Launchpad, meaning we just use the advised OpenERP community advised distributed development platform. Also notice that we took care of migrating the previous SVN commits.

That was also the occasion to split the development in two branches:

a 4.2 compatible maintenance Branch. That one is for the conservative folks who can't afford migrating to OpenERP v5 https://code.launchpad.net/~openerp-commiter/openobject-addons/4.2-extra-addons
a 5.x branch where all the new interesting stuff is expected to happen: https://code.launchpad.net/~openerp-commiter/openobject-addons/trunk-extra-addons

In both cases, the OpenERP addon module name is now called magento_openerp_synchro.
The last SVN version (revision #27) has been ported as bzr revision #3581.1.9 on v5 branch and rev #148 in 4.2 branch. Later on both branches got a fix for issue http://code.google.com/p/magento-openerp-smile-synchro/issues/detail?id=31
v5 branch got even more attention and code clean up.

So you are very much welcome to jump in and contribute to whatever branch you need. Still, before adding any serious extra feature, we would like to insist that some refactoring should be undertaken BEFORE bloating the code.

Especially:

Too much code is lying inside the OpenERP wizard layers. That was a bad design choice making it hard for additionnal third party modules to extend the connector behavior and add custom features that way OpenERP allows it (thanks to its extensive OOP design). Instead, that code should be moved INSIDE THE OBJECT LAYER. That shouldn't be to hard, it's only about extracting methods, putting them in the appropriated objects (sale order, product...) and calling those methods from the wizards instead. For instance we deployed a derived version of that connector for a customer along with the sale_supplier_direct_delivery module and we had to hardcode the connector to make it account for the direct delivery eventuallity. With the new design things should work more seamlessly. See the following tracker: http://code.google.com/p/magento-openerp-smile-synchro/issues/detail?id=32
The sale order push feature (from Magento) has not beeing integrated properly yet and is still polluting the code. It has been a wonderfull contribution by Charles Galpin, but we had not time to integrate it the way we wanted to and it has not been tested extensively. Instead, we would like to remove the push code from Magento and instead makes Magento call OpenERP and tell him to pull the given sale order reusing it's standar sale order import code. Finally that design should be able to deal with possible network or OpenERP failure and flag failing push inside Magento for later processing. Not at top priority, but something you should be aware of. See following tracker: http://code.google.com/p/magento-openerp-smile-synchro/issues/detail?id=33
Finally, we insist that we din't take care yet of exposing ALL OpenERP XML/RPC server webservice the standard Magento way. One of the consequences is that currently importing sale orders is not secure unless you block external connections by IP at say an Apache level. A much better way would be to properly expose our custom extra webservice the Magento way, see the following tracker: http://code.google.com/p/magento-openerp-smile-synchro/issues/detail?id=6

Meanwhile we welcome Jordi Esteve as a core contributor + project member. We know that things have been a bit slow with Smile and the connector recently (we are flooded by OpenERP non Magento demand), but we really hope to help moving foward as much as we can. We really hope all those inputs will streamline the installation process + natively supported features while minimizing the required development skills. Enjoy!

New planet for Magento - OpenERP synchro module

2009-03-01T17:33:00.000-08:00

Hi, I'm Raphaël Valyi, one of the main author of the synchro module between OpenERP and Magento. Some 10 months ago, after completing an extensive study about open source ERP's, I convinced Smile, my employer, to support that synchro module allowing Magento and OpenERP to work together. Some two months later, thanks to Smile and the tireless work of Sylvain Pamart, we got that connector in pretty good shape already.

Over the time, we improved it and tried to integrate community contributions as much as possible.

Since then, our OpenERP schedule at Smile got totally loaded with customer contracts letting us little time for the community even if Smile is still commited to support the effort in theory (my boss agrees on a 2 man days per month effort so far, as long as we have no higher priority, and that's the catch) . So unfortunately, it means that the connector is always lagging behind direct commercial activities.

Anyway, as I do personally see interest in pushing that connector forward, I'm taking on my spare time to give you updates about the connector in this blog, under the magento-openerp-synchro tag, so it get exported to our brand new Connector Yahoo Pipes powered planet mashup: http://pipes.yahoo.com/pipes/pipe.info?_id=7mpa3ckG3hGCuJSWPxJ3AQ

Ideally, I'll let any contributor (including Smile.fr) post in that planet when they contribute in the future. So the corporate work supported by Smile.fr will hence probably be published both here and on the corporate blog, that is http://blog.smile.fr/le-blog-des-consultants.

So I hope this planet will be a fair place to give updates about the connector while accommodating corporate, personal and community contributions. Feel also free to comment in the respective planet blogs.

Finally note that this planet is managed as a Yahoo Pipes aggregator wrapped inside a Google Gadget eventually. You can contact me to have your blog (+specific tag) included in that planet if you wish.

Raphaël Valyi

OpenERP running on Java (Jython) - ROUND 1

2009-02-01T17:19:00.000-08:00

Hi folks,

Looking for an open source ERP that doesn't suck? Well there is at least one: OpenERP. Like it or not it's Python based. For most of the things this is a perfect fit, but let's face it, Python has less enterprise acceptance and the lower layers of that ERP could also have been powered by a powerfull virtual machine. And since building a new ERP can't really take less than 3 years, the situation will hardly evolve any time soon, so we better deal with it: I bet my money OpenERP will now take the market by storm, I don't see anything stopping it to take the lead, at least for the next 5 years.

So I wanted to show the world that it was possible to back OpenERP by the Java platform, much like it's now being admitted that Java is the best bed for Ruby on Rails applications thanks to JRuby. I built a proof of concept. Now I hope some get the message and join efforts to make it happen. I'll hardly finish that journey in my spare time, but hopefully, some people start to release how important this is and how close we are and help covering the extra miles to get it in production running on Java. Of course, OpenERP doesn't need to run Java to be the best ERP already, it is just that it could be even better. And if you want to help out, I'll be there too.

what about the claimed Java based ERP's?

I've been myself a Java programmer since around 2003, even created my first open source project on Sourceforge back in 2004 (EmSim http://sourceforge.net/projects/emsim/), later on I used those skills in a variety of advanced projects when working at Amadeus and Smile.fr. But, like it or not, after deep investigation, my conclusion is that the best open source ERP was coded in Python. I've looked deep inside Java based or claimed ERP's such as Ofbiz, Neogia, JFire, Compiere, Openbravo, Adempiere but those were too limited in my opinion. Basically, the trouble is that those 'Java' based ERP don't rely on true object oriented modeling since the ground up, some of them being even more SQL based then really Java based. I could explain in deep details why, but this is beyond the scope of this post.

So my conclusion was that even if I needed to learn Python, I would get the work done faster with OpenERP than with any of those existing Java based ERP's. My experiments proved me that I was right. With OpenERP, extending the relational model, the forms and making it fit to my needs was faster than with any of the other tried ERP's, even while being a Python noob. One year later, with a few successful implementations behind me, I'm only recommending it more than ever.

But wouldn't it be great to port OpenERP on the Java platform?

The benefits of porting OpenERP on Java are beyond the scope of this post. Still, here is a short list:

Noticeable speed up. Even if the Python layer peaks up at 40% of the CPU load under high concurrent loads, getting twice as fast would be noticeable. If Jython get as optimized as JRuby which is 2 to 5 times faster than C Ruby, we will go there.
Real multithreading and less memory use under huge loads (CPython scales using process and sharing nothing accross runtimes because it has a Giant Lock in its interpreter, much like PHP or Ruby). The expected benefit there is much like what benchmarks prove for heavily loaded JRuby on Rails applications.
One click installation on lots of hardwares (imagine how easy it will be to sale to large companies when you will say it's only a .war you pu on your Tomcat server).
Penetration in large companies where Java tends to rule more than Python
Easier and solid cross database support thanks to the unified JDBC API.
Larger community for OpenERP. OpenERP is certainly the open source ERP with the largest active open source community. But just imagine how large it could get when you managed to connect with the Java community, telling them: hey, we are somewhat Java based too. If you know Java you'll be able to leverage those skills to get the job done in OpenERP. Again, JRuby proved here that connecting communities was possible.
Possibly larger fundings and exposure for OpenERP. Openbravo certainly had the large fundings it had by playing on its Java image which as a broad acceptance in the enterprise world.
And the best of all: cross language implementation: by leveraging Jython, you would be able to call the best existing Java lib (there are some good ones like JDBC, JCR, JMX, JMS, ESB's, SOAP...), but also externalize some code pieces where speed is critical to Java, meaning an easier way to reach C speed (a warmed up Java6 runs at C speed and sometimes faster) than writing C bindings.
Jython will probably ends up implementing the MOP (Meto Object Protocol) that makes it possible to share OBJECTS between JVM based languages such as JRuby, Javascript, Scala, Clojures, Groovy... Meaning that more people could adapt OpenERP to meet their needs using the language they already master. That means you could code OpenERP modules in Ruby for instance and that's not a small thing, while preserving backward compatibility. Of course, published modules would have to keep beeing closely controlled, but at least one shot veticalization modules could offer more options.
Because it's fun and we can do it.

What does it take?
Well, Jython is really alive again. Notice that Microsoft hired the orginial Jython creator to create Iron Python on the .Net platform. Next, by 2008, Sun Microsystem hired the two Jython leads, Ted Leung and Frank Wierzbicki to make it Python a first class citizen on the Java platform again. Espcially, JRuby recently proved that the JVM was able to run dynamic languages faster than their native C interpreter that don't come up with such a sophisticated virtual machine. So in theory the future is bright. A 2.5 Python compliant Jython is expected by February and the current trunk largely reflecting it already.

Still, the trouble, comes with native libraries. Since Python tends to be somewhat slow, Pythonistas tend to back lot's of Python libraries by C extensions to provide extra speed. The trouble is that the Jython interpreter, sandboxed in its virtual machine environment can't invoke the same C extension, mostly because it doesn't work the same at the lowest levels.

In some places, OpenERP uses such libraries. Some of them are psycopg2, mx.DateTime, libxml2, libxslt. Overall there aren't too many fortunately so that's why this shouldn't be too hard to have a full blown OpenERP running on the JVM. The common strategy here is to set up a wrapper over existing Java libraries in place of those C extensions. Also notice that in the near future, Jython might also support CTypes, a standard way of building C extensions for Python. This is largely because Jython is benefiting here from the fantastic JRuby work of Wayne Meissner around JFFI (Java Foreign Fuction Interface). Other synergies exist, like the Da Vinci Machine, or Invoke Dynamic bytecode instruction that would help the JVM speeding up dynamic languages while optimizing memory usage.

Current status:
After some two days of heavy hacking, I'm getting:

OpenERP server starting powered by the last Jython Java based interpreter, backed by the regular JDBC Postgres driver over a standard Java JNDI database connection pooling, instead of Pyscopg2.
It answers most of the webservices calls, be it from the clients or from the webservice API. The largest thing that isn't working here is datetime operation that are using the mx.DataTime library that isn't yet fully supported.
Also, views don't work yet (the fields_view_get method). This is due the current XML and XPath limited support. Everything need is available on the Java plateform, but it should be wrapped properly to fake the Python API OpenERP is expecting there.

Next stages:

I guess that faking the required xpath and xml API would be great. That shouldn't be that hard given that OpenERP only use a few things in those API. We should only wrapp them over Java libraries such as Xalan.
As for the mx.DateTime trouble, the OpenERP team said they are ready to move away from it. Indeed mx.DateTime has a bug with dates before 1970. That's why the Tryton fork moved away form it, proving the thing is possible (they only use it marginaly for parsing, but alternative can be found). Advised by Jim Baker from Jython, it appears that the standard python datetime, datetime.timedelta and the http://labix.org/python-dateutil library might be used instead. So aside from helping getting rid of mx.DateTime, we probably won't have much work here.
Once all that is working, the Java based OpenERP server will already be in pretty good shape to be used in production.
Of course, ideally we would package the whole thing within a standard J2EE war package and serve it with a standard servlet container such as Tomcat or Glassfish. The idea there is to write a servlet connector that will use standard Java servlet instead of the python SimpleXMLRPCServer. Then we would route HTTP request to a pool of Jython runtimes and the appropriate OpenERP server layers, much like others do with Django on Jython or the Glassfish JRuby on Rails gem supported by Sun Microsystem.
Once performance starts to proves better than CPython (Jython and JRuby guys are expecting this to happen by 2009 already) and once large companies show interrest in deploying OpenERP over their existing Java stack, may be the OpenERP team starts to support it officially and provide alternative Java based layers to increase performance even more or provide extra features (like SOAP, JMS async messaging, ESB like Mule or ServiceMix, a REST layer, JMX remote monitoring, OpenTerracotta clustering...)

Steps to follow to test the current Jython powered version:

Requirements:

you should know how to start a regular OpenERP v5 server
you should have an existing OpenERP v5 database to test again
you should have Java installed (1.6 advised)

First you have to grab a recent Jython interpreter. Make sure you have a recent version of Java installed (1.6 advised) and then grab Jython:
get a fresh version of Jython, and build it:


$ svn co https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython/  jython-dev
$ cd jython-dev
$ ant

now, put the jython-dev/dist/bin/jython command in your path.
now if you type


$ jython

that should bring you a Python 2.5, java based commande line interpreter.

Now grab my public branch of the OpenERP server:


$ bzr branch lp:~rvalyi/openobject-server/trunk-java-jython

As for the addons, you can use your regular OpenERP v5 addons, or grab them from here:


$ bzr branch lp:openobject-addons

Now open the trunk-java-jython/bin/tools/config.py and make sure you properly set up the database connection params and you addons path location.
May be it's better to leave the db_name to False so it will load all the features lazily when requested.

you can already see what happens if you:


$ cd trunk-java-jython/bin
$ jython openerp-server.py

You'll probably have errors because we didn't properly set up some required libraries.
So it's time to copy paste a few libraries from our regular Python path to our Jython path:
you can copy/paste the following directories:
pychart, pytz, reportlab
from
/user/lib/python2.5/site-packages/ (or whatever you Python sys.path)
to [...]/jython-dev/dist/Lib

There is one more catch though. Currently, there is bug in Jython preventing the reportlab lib to load fully. This is because font listings in the pdfbase/_fondata.py are just too large for the JVM spec to fit in a single Java method. The Jython folks are aware of that bug but it's low priority: see http://bugs.jython.org/issue527524
Meanwhile, helped by the fellow Charles Headius Nutter from the JRuby Sun team who faced similar issues implementing JRuby, I patched the file to make reportlab works in Jython. It's only a wrapping of large list instantiations and doesn't remove any feature. I submitted it to the reportlab developers and hope to see it included waiting for a Jython fix.
So in any case, for now, you should just move the _fontdata.py file provided in the trunk-java-jython branch and copy it in place of the reportlab/pdfbase/_fontdata.py file of your Jython reportlab installation.

Now you should be able to start you OpenERP server on Jython. Look at the following logs on my machine. Then I connected using XML/RPC with my GTK client (in that development version I'm printing all the requests, you could remove those print of course):


rvalyi@rvalyi-laptop:~/DEV/openobject_trunk/openobject-server-jython/bin$ jython openerp-server.py
[lun. févr. 02 2009 02:41:22] INFO:server:version - 5.0.0
[lun. févr. 02 2009 02:41:22] INFO:server:addons_path - /home/rvalyi/DEV/openobject_trunk/openobject-addons
[lun. févr. 02 2009 02:41:22] INFO:server:database hostname - localhost
[lun. févr. 02 2009 02:41:22] INFO:server:database port - 5432
[lun. févr. 02 2009 02:41:22] INFO:server:database user - openerp
[lun. févr. 02 2009 02:41:22] INFO:objects:initialising distributed objects services
WARNING; Python Imaging not installed, you can use only .JPG pictures !
[lun. févr. 02 2009 02:41:32] INFO:web-services:starting XML-RPC services, port 8069
[lun. févr. 02 2009 02:41:32] INFO:web-services:starting NET-RPC service, port 8070
[lun. févr. 02 2009 02:41:32] INFO:web-services:the server is running, waiting for connections...
list
()
-------------
[lun. févr. 02 2009 02:41:40] INFO:dbpool:Connecting to template1
executing the following DB query:
select datname from pg_database where datdba=(select usesysid from pg_user where usename='openerp') and datname not in ('template0', 'template1', 'postgres') order by datname
query passed!
[lun. févr. 02 2009 02:41:40] INFO:dbpool:Closing all connections to template1
['jython', 'openerp']
server_version
()
-------------
5.0.0



So the GTK client can connect to the Jython powered server and list the available databases for instance.
But if I try to really connect to the database and go further here I get an error unfortunately:

[...]
SELECT ir_act_window_group_rel.gid,ir_act_window_group_rel.act_id                 FROM ir_act_window_group_rel , res_groups                 WHERE ir_act_window_group_rel.act_id in (1)                     AND ir_act_window_group_rel.gid = res_groups.id  order by res_groups.name offset '0'
query passed!
{'view_id': (1, u'ir.ui.menu.tree'), 'src_model': False, 'view_ids': [], 'context': u'{}', 'view_type': u'tree', 'auto_refresh': 0, 'usage': u'menu', 'res_model': u'ir.ui.menu', 'domain': u"[('parent_id', '=', False)]", 'name': u'Menu', 'id': 1, 'view_mode': u'tree,form', 'target': u'current', 'views': [(1, u'tree'), (False, u'form')], 'limit': 80, 'type': u'ir.actions.act_window', 'groups_id': []}
execute
('jython', 1, 'admin', 'ir.ui.view', 'read', [1], ['model', 'type'], {'tz': False, 'active_ids': [], 'lang': 'en_US', 'active_id': False})
-------------
executing the following DB query:
SELECT "model","type",id FROM "ir_ui_view" WHERE id IN (1) ORDER BY priority
query passed!
[{'id': 1, 'model': u'ir.ui.menu', 'type': u'tree'}]
execute
('jython', 1, 'admin', 'ir.ui.menu', 'fields_view_get', 1, 'tree', {'tz': False, 'active_ids': [], 'lang': 'en_US', 'active_id': False})
-------------
executing the following DB query:
SELECT arch,name,field_parent,id,type,inherit_id FROM ir_ui_view WHERE id='1' and model='ir.ui.menu'
query passed!
executing the following DB query:
select arch,id from ir_ui_view where inherit_id='1' and model='ir.ui.menu' order by priority
query passed!
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[01]: Traceback (most recent call last):
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[02]:   File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 60, in wrapper
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[03]:     except orm.except_orm, inst:
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[04]:   File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 120, in execute
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[05]:     cr.commit()
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[06]:   File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 112, in execute_cr
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[07]:   File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/orm.py", line 1082, in fields_view_get
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[08]:     doc = dom.minidom.parseString(encode(result['arch']))
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[09]: NameError: global name 'dom' is not defined

So yeah, not everything is working yet, and especially the view layer don't work yet because we didn't provide appropriate wrappers for the xml/dom and xml/xpath Python modules used by OpenERP.

The same goes withe the web-client (eTiny), it can list the databases but won't show any view properly yet.

Still, more something interesting is that webservices, which only care about the model and controller layers are already working quite well. See yourself, in an standard python or jython console, I can for instance read all the info related to a given product (here with id 1):


>>> sock = xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/object')
>>> sock.execute("my_test_base", 1, "admin", 'product.product', 'read', [1])
[{'warranty': False, 'property_stock_procurement': [5, 'Procurements'], 'supply_method': 'buy', 'code': False, 'list_price': 38.25, 'expected_margin_rate': 0.0, 'pricelist_purchase': 'Default Purchase Pricelist (0.00) : 25.50\n', 'incoming_qty': 0.0, 'weight_net': False, 'standard_price': 25.5, 'cost_method': 'standard', 'active': True, 'price_extra': 0.0, 'mes_type': 'fixed', 'uom_id': [1, 'PCE'], 'uos_id': False, 'ean13': False, 'default_code': False, 'type': 'service', 'property_account_income': False, 'qty_available': 0.0, 'sales_gap': 0.0, 'id': 1, 'expected_margin': 0.0, 'uos_coeff': 1.0, 'virtual_available': 0.0, 'seller_delay': 1, 'total_cost': 0.0, 'purchase_ok': True, 'date_from': '2009-01-01', 'property_stock_account_output': False, 'track_outgoing': False, 'company_id': [1, 'Espace Loggia'], 'product_tmpl_id': [1, 'Onsite Senior Intervention'], 'state': False, 'loc_rack': False, 'pricelist_sale': 'Public Pricelist (0.00) : 38.25\n', 'uom_po_id': [1, 'PCE'], 'price_margin': 1.0, 'price': 0.0, 'property_stock_inventory': [4, 'Inventory loss'], 'loc_case': False, 'sale_avg_price': 0.0, 'description': False, 'track_incoming': False, 'property_stock_production': [6, 'Production'], 'purchase_avg_price': 0.0, 'weight': False, 'supplier_taxes_id': [], 'volume': False, 'normal_cost': 0.0, 'outgoing_qty': 0.0, 'dimension_type_ids': False, 'date_to': '2009-12-31', 'procure_method': 'make_to_stock', 'sale_num_invoiced': 0.0, 'variants': '', 'partner_ref': 'Onsite Senior Intervention', 'loc_row': False, 'purchase_num_invoiced': 0.0, 'sale_ok': True, 'rental': False, 'packaging': [], 'sale_delay': 7.0, 'name': 'Onsite Senior Intervention', 'total_margin_rate': 0.0, 'description_sale': False, 'property_account_expense': False, 'categ_id': [8, 'All products / Sellable / Services / Onsite Intervention'], 'invoice_state': 'open_paid', 'property_stock_account_input': False, 'track_production': False, 'sale_expected': 0.0, 'lst_price': 38.25, 'taxes_id': [], 'dimension_value_ids': [], 'produce_delay': 1.0, 'seller_ids': [], 'description_purchase': False, 'turnover': 0.0, 'purchase_gap': 0.0, 'product_manager': False, 'total_margin': 0.0}]

OK, enough said for now. I'll try to post updates on the dedicated Launchpad blueprint: https://blueprints.launchpad.net/openobject-server/+spec/jython-support-as-jython-improves
Don't hesitate to contact me if you want to push this work further.

Raphaël Valyi.

Java 7 build 44 improves jar build speed a lot indeed

2009-01-24T09:49:00.000-08:00

Hi,

from the changelog of jdk7 build 44, one can read that they improved on the speed of the jar task.

Indeed, I can confirm this for building JRuby which is quite a large project:

Before, Java 7 build 43:
>ant-jar complete
[...]
BUILD SUCCESSFUL
Total time: 1m30 seconds

Now, Java 7 build 44:
>ant-jar complete
[...]
BUILD SUCCESSFUL
Total time: 56 seconds

Quite enjoyable indeed!

OLAP component: Mondrian+JPivot or Flex OLAPDataGrid ?

2008-07-11T16:18:00.000-07:00

Recently I've been spending a few hours digging into OLAP cube components (a special table to analyse large and multi-dimmentionnal data in Business Intelligence). I wanted to produce a nice demo with an OLAP cube plugged to the OpenERP (by far the best open source ERP) database.

I actually saw a guy demoing such an OLAP Flex component along with Openbravo ERP (looking nice but the details were provided): http://opensourceerpguru.com/
Anyway I just wanted to achieve the same but using OpenERP (I feel much more comfortable with it's elegant and efficient architecture, not to tell about the features nor the business model) this time and eventually JRuby on Rails, my best Swiss knife to pull the data to the OLAP viewer.

So at first I've been stunned by the OLAP Flex component (called OLAPDataGrid) . It looks really nicer than the old fashion Mondrian JPivot. So I decided to give it a try. First time with Flex and back to the half open source crapp since long ago. So I had to follow all the Adobe crapp flow: create a fucking account, read all their commercial crapp, agree with their whatever license and finally download the stuff and start playing with it. Then I remarked that all my FlexBuilder nice OLAP samples were coming with a "Flex Data Vizualisation Trial" watermark. OK, time to remeber, FlexBuilder is not open source yet, so let's go with the Flex SDK, back to the Adobe legacy crapp, download again and try again (I heard that the SDK was open source, or sort of).

Then I tried to compile my mxml component with the following command line:


flex_sdk_3/bin$ ./mxmlc /home/rvalyi/DEV/olap_test/src/olap.mxml
Loading configuration file /home/rvalyi/Desktop/flex_sdk_3/frameworks/flex-config.xml
/home/rvalyi/DEV/olap_test/src/olap.mxml(162): Error: Could not resolve  to a component implementation.

       id="myMXMLCube"

/home/rvalyi/DEV/olap_test/src/olap.mxml(195): Error: Could not resolve  to a component implementation.

WTF ???

Googled the error message and got the official answer from an Adobe employee on a forum here: http://www.codeverge.net/item.aspx?item=101509

The OLAPDatagrid component is available only in the Flex Builder Professional
version. For the first problem, the one where you are building a Flex + LCDS
2.5.1 project, the answer is that the LCDS 2.5.1 doesn't include the
OLAPDatagrid component and that is way you are getting the errors on runtime.

You f****** b*st*rds! you got me! So that how I lost a few hours trapped by the Adobe half open source policy, go hell!

Mondrian or Flex OLAPDataGrid?
Well, back to Mondrian and JPivot...

OK, for sure Mondrian and JPivot aren't really something optimal and I feel more like it's a bloated non HTTP compliant piece of code, but hey it's free and it just works. So until Tiny.be release their awesome open source "TinyBI framework" (for October?) I'll stick with it. I'll hardly try Flex and Flash again, I promise.

Finally I should say that I really don't know anyway how that Flex component would deal with a large database as it seems it's an in memory client side OLAP solution only. Even if it were to change, I'm not sure I would feel comfortable in feeding the right data pieces as the OLAP component requires them and finally all the existing samples don't come with drill down, slicer and rotation widgets, so I'm not sure how easily one can interact with the cube.

May this post save your time.

Orkut social network community profiling with GreaseMonkey and Rails, round one.

2008-01-20T15:58:00.000-08:00

Hi,

I wanted to figure out some statistics about some Orkut (Google equivalent of Myspace or Facebook) social communities. I want to know the mean age, the gender rate and other things among given communities. And actually I'm getting a lot more as you'll see. The method I'm showing is very simple and can be reused for other purposes. It's automatic enough to grab profiles 15 by 15 and build a good sampling of the community, but not enough to grab ALL the profiles. Anyway, I guess Google would kick out crawlers wanting to grab all the profiles.

The challenge for collecting those data is that you need to be logged in to crawl Orkut communities. Then only you can grab information in the HTML pages if you manage to handle the navigation properly. I didn't really manage to automate all the process using the HPricot HTML parser.

Instead I went for a semi-automatic combination of GreaseMonkey scripts to grab the profiles and a Rails server to store the data. Now I only need to browse every members page of an Orkut community to get all its member profiles stored inside my database. Then I also have a rails action to export data in csv so I can open it in spreadsheet editor:

The first thing is to install the GreaseMonkey Firefox plugin.
Then you'll install the following user script name orkut_crawler.user.js:


// ==UserScript==
// @name          Orkut_crawler
// @namespace     http://livetribune.org/
// @description   grab orkut properties
// @include       *
// @exclude       http://diveintogreasemonkey.org/*
// @exclude       http://www.diveintogreasemonkey.org/*
// ==/UserScript==

var scripts = [
'http://localhost:3000/javascripts/orkut_crawler.js'
];
for (i in scripts) {
var script = document.createElement('script');
script.src = scripts[i];
document.getElementsByTagName('head')[0].appendChild(script);
}

What we are doing here is including the Prototype Javascript library to make it easier to grab the information using CSS selectors. I tried to paste the prototype lib directly inside the browser script but it didn't worked for some unknown reason. So anyway, this last solution works.

You can now make sure the script gets activated when you visit an Orkut page:

OK, now we need to set up our actual orkut_crawler script as well as the Rails backend.
build a simple Rails app using the ">rails orkut_crawler" command line.
You can use any database. I used the default Sqlite DB in my case.

Then create the database with that command ">rake db:create" inside the orkut crawler directory.

Now, let's make a simple persistent model to store our data:
"> ruby script/generate model item"
Now edit the db/migrate/001_create_items.rb migration file and write:


class CreateItems <>

Let's create the table: ">rails db:migrate"

Now we need to create the orkut_crawler Javascript file that'll be called by our GreaseMonkey script. make a new file called public/javascripts/orkut_crawler.js

Inside that file, you first need to copy the Prototype javascript library you'll find in public/javascripts. Then write this code at the end of the file:

/* Crawler */

function appendNewScript(src_url, id) {
var headID = document.getElementsByTagName("head")[0];     
var newScript = document.createElement('script');
            newScript.id = 'snap';
newScript.type = 'text/javascript';
newScript.src = src_url;
headID.appendChild(newScript);
}

function addTuple(line) {
    try {
    var key = line.childNodes[1].innerHTML;
    var value = line.childNodes[3].innerHTML;
    key = key.replace(" ", "_");
    key = key.replace("/", "_");
    key = key.replace(":", "");
    value = value.replace("\\", "");
    value = value.replace("\"", "");
    value = value.replace(";", " ");
    value = value.replace(":", " ");
    params += "&" + key + "=" + value;
    console.log(key, value);
    } catch(e) {}
}

var tab;

if (document.location.href.indexOf('CommMembers.aspx') > 0) {//it's a community page
        tab = $$('.listitem');
        for (var i =0; i<>
                                setTimeout(tab[i].innerHTML="<" + "iframe src='"+ tab[i].childNodes[1].href + "'/>", 100 * i );
        }
} else if (document.location.href.indexOf('Profile.aspx') > 0) {//it's a profile
    var params="";
    tab = $$('.listlight');
    for (var i =0; i< tab.length;i++) { addTuple(tab[i]); }
    tab = $$('.listdark');
    for (var i =0; i< tab.length;i++) {addTuple(tab[i]); }

    params = document.location.search + params;

    appendNewScript("http://localhost:3000/data/new" + params);//send the params back to our Rails app!



This code will grab the profile when you browse an Orkut page. If you are rather browsing a community page, then it will open all the profiles of the listed members of this page and thus grab those profiles.

Finally, we need to write a Rails controller that will persist the data (the 'new' action), render a global csv file ('index' action) and even tell how many profiles we have ('size' action). So edit app/controllers/data_controller.rb this way:

require 'cgi'

class DataController < ApplicationController
  def new
    params.delete 'action'
    params.delete 'controller'
    params.each_key {|key| params[key] = CGI.escape(params[key])}
    puts params.inspect
    if existing =Item.find_by_uid(params[:uid])
      item = existing
    else
      item = Item.new
      item.uid = params[:uid]
    end
    item.properties = params.inspect
    if item.save
      render :text => 'UPDATED' and return if existing
      render :text => 'SAVED!'
    else
      render :text => 'ERROR!'
    end
  end

  def size
    render :text => Item.find(:all).size.to_s
  end

  def index
    all_props = []
    @items = Item.find(:all)
    res = ""
    @items.each do |item|
      map = eval item.properties
      map.each_key do |key|
        all_props << key unless all_props.index key
      end
    end

    all_props.each do |key|
      res +=  key + "; "
    end
    res = res[0..-3] + "\n"

    @items.each do |item|
      all_props.each do |key|
        map = eval item.properties
        res +=  CGI.unescape(map[key].to_s).gsub("\n", " ").gsub("\r", " ").gsub("\r\n", "").gsub(";", " - ") + "; "
      end
      res = res[0..-3] + "\n"
    end
    render :text => res
  end
end



OK, we are done.
Now make sure our Rails server is working:
"> ruby script/server"

You can also track what is going on:
"tail -f log/development.log"

Now visit some Orkuts community pages with this URL pattern:
http://www.orkut.com/CommMembers.aspx?cmm=10087467&tab=0&na=3&nst=151&nid=0


As you'll see, instead of the normal member's icons, we are now loading the members profiles inside iframes. At the same time, you can ensure in your terminal our Rails server is storing all the profiles.

Finally; look at the profile you collected:
see how many profiles you have collected: http://localhost:3000/data/size
Now let's export the data as csv (this can take a while depending how much data you stored):
in your terminal, execute:
">wget http://localhost:3000/data"
Now rename the data file into data.csv and browse it in OpenOffice for instance:



Ok, you are done. Now you can start making some statistics, but that's for round two!

Getting ruby-openid OpenID gem to work with JRuby on Rails

2007-10-18T17:37:00.000-07:00

As you may know a few native methods aren't yet implemented in JRuby and a few ones aren't even possible to implement due to the POSIX native calls.

I really don't know where odes FileTest#chardev? method fits in, but JRuby doesn't have it right now.

So if ever you want to use the 'ruby-openid' gem in your JRuby on Rails using for instance the official wrapper, there is a workaround.
Indeed if you do nothing, either at the rake setup, either at startup, you'll get something like:


=> Booting WEBrick...
/home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27: FileTest#chardev? not yet implemented (NotImplementedError)
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
      from /home/rvalyi/DEV/livetribune/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
      from /home/rvalyi/DEV/jruby/lib/ruby/gems/1.8/gems/ruby-openid-1.1.4/lib/openid.rb:1:in `new_constants_in'
      from /home/rvalyi/DEV/livetribune/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
      from /home/rvalyi/DEV/jruby/lib/ruby/gems/1.8/gems/ruby-openid-1.1.4/lib/openid.rb:1:in `require'
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:32:in `require'
      from /home/rvalyi/DEV/jruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:32:in `require'
      from /home/rvalyi/DEV/livetribune/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
      from /home/rvalyi/DEV/livetribune/vendor/plugins/open_id_authentication/init.rb:120:in `new_constants_in'
      from /home/rvalyi/DEV/livetribune/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
      from /home/rvalyi/DEV/livetribune/vendor/plugins/open_id_authentication/init.rb:120:in `binding'

The workaround is to override the FileTest module and define a stub value for the chardev? method. That method is to return true or false. Returning always true for that useage seems to work well.

So at the VERY BEGINNING of your environment.rb, just write:


module FileTest
def self.chardev?(file_name); return true; end
def chardev?(file_name); return true; end
end

Then Open Id is going to play nice with your JRuby on Rails app. At the time I'm speaking, I'm on JRuby SVN head trunk and the String#split method is having a little bug that'll ouput a stack trace but fortunately this has no consequence on the functionality.

JRuby rocks!

migrating Livetribune.com on JRuby on Rails2.0

2007-10-06T17:26:00.000-07:00

update: I'm here talking about problems I encountered migrating from Rails 1.2.3/1.2.4 to Rails2.0. My app has been on JRuby since almost the beginning and I had no issue with JRuby (this time at least). Being on JRuby motivated me even more to try Rails2.0 as the routing is said to be optimized by caching as I'll explain there.

The current state of JRuby (trunk) is that low level tests tend to be faster than C-Ruby but Rails is still a bit slower. Some people, included me, tend to think this might be due to the request routing of Rails being slow in JRuby. Indeed Rails routing extensively uses regexps and regexps in JRuby are still a lot slower than in C-Ruby. In fact this is because Java regex engines work with char[] whereas Ruby (including JRuby) represents strings as byte[]. So in JRuby we have to translate byte into char[] and back all the time for regexps untill somebody ports on Java a byte based regexp engine (namely the oniguruma lib).

So for a JRuby dev, it's always nice the hear Rails2.0 made some caching optimization on routing. So I wanted to try it. I didn't finish yet to port the Livetribune application (not yet in prod BTW) but I made great progress. In order to help other people in their migration I'll relate here a few tricks I had to do to make it working.

* url_for don't accept anymore symbols to point controllers. You have to write strings instead. So for instance:
map.connect ':locale/topics/:year/:month/:day/:id', :controller => :topics, :action => 'show'
became
map.connect ':locale/topics/:year/:month/:day/:id', :controller => 'topics', :action => 'show'
else rails will complain symbol :topics have no method [].

* the usefull routing_navigator plugin doesn't work anymore out of the box in Rails2.0 . This seems to be due to the self.template_root method call in the RoutingNavigatorController that doesn't exist anymore in Rails2.0. I didn't fin the work arround so I simply dropped the plugin.

* replace @base_path by ActionController::Base.view_paths whenever its used (in the AuthGenerator plugin for instance) as suggested here:
http://dev.rubyonrails.org/ticket/9689
(still I didn't get AuthGenerator fully working yet. Rails still miss its view templates)

* Since by default Rails2.0 sessions are based on a cookie store, either set it back to file store, either explicitly define a secrete that'll be share between the client browser and your server, like:
config.action_controller.session = { :session_key => "_myapp_session", :secret => '449fe2e7daee471bffae2fd8dc02313a' }
in your environment.rb

After this I had almost everything working again. Still some trouble with AuthGenerator, please provide feedback if you get this one fixed.

Also I installed that Rails 1.2.4 version before hoping it would shout for all the potential issues. It never shouted for those I just wrote about, so may be that could have done it better (or may be it's not too late to include those pointers).

Rails is very cool and Ruby is too. But it's in those kind of migration situations you'll think Java was just safer. I mean, I think half those problems would have been statically highlighted by the compiler. For the other half, let's face it you would have used some IOC injection like Spring so you wouldn't have detected it either (that the big hypocrisy of the today's J2EE world).

I still think nothing is approaching Rails in its niche, but that was just to highlight Ruby will never replace Java fully, even as a language. Rails is OK because the compiler safety is being replaced by the community checking quite effectively, but that wouldn't be true in an enterprise world for instance.

Finally I think JRuby really is the expected silver bullet, because it will allow you to closely choose where things get dynamic but unsafe (Ruby) and where thing get static and safe (Java). Moreover, the new optional Ruby type signature that's being imagined for the new JRuby compiler ( http://headius.blogspot.com/2007/09/compiler-is-complete.html )will just allow you to compile your great Ruby lib in Java and then use it from Java (as a jar bytecode) calling it through normal Java typed methods. It means you'll use safely Ruby libs whenever their author would have decided some part of its API deserves more security (and yes, a bit less agility). I mean JRuby will let you have the choice and that's great.

Also I forgot: I had no issue due to the use of JRuby (trunk) with Rails2.0.
Have fun with JRuby and Rails2.0.

Raphaël Valyi.

JRuby IT survey using Yahoo pipes

2007-10-04T16:27:00.000-07:00

Suppose you are all convinced JRuby is the next big thing that will bring an incredible productivity to the enterprise world. What matters from now one is to track the new JRuby experiments popping all arround the world. Every day more and more people are throwing specialized Java libs into brand new web2.0 Rails applications. And there are also of course the JRuby implementation to track. Indeed great expectations, such as speed boost, optional typing, Java/Ruby tight integration, can be put into the recent compiler advances.

So what I did is a simple Yahoo pipe that will crawl the web and gather very recent stories about JRuby usage and progress. You would think you could simply use Google blog search and see the same. Well, that would be a little bit less pertinent. Indeed, contrary to Google, this pipe filters out SVN commits as well as bug reports which tend to pollute Google blog searches. By aggregating lots of sources including the JRuby core developer blogs, you are also sure you won't miss anything important.

So the pipe is there:
http://pipes.yahoo.com/pipes/pipe.info?_id=oCCxOv9w3BGq_aM_1vC6Jw

And here is the output of that pipe (truncated to the 4 last entries only) thanks to Google AJAX Feed API:

An important thing you could notice is that Technorati and Delicious aren't used. Well actually the problem with those is that they provide feeds ordered by tagging history, not by publication history. So an old story could seem recent in Delicious just because somebody just tagged it.

Anyway, Yahoo pipes looks like a terrific tool for IT survey in general. Analyzing how people build pipes might also provide good semantic information to Yahoo, just like Delicious.

Have fun and feel free to provide feedback or build better clones.

Raphaël Valyi.

Hacking the moo.fx Accordion into the ultimate DHTML Accordion widget

2007-08-22T16:35:00.000-07:00

For long, I've been looking for a good Accordion widget. First I began with the Yahoo one but unfortunately I ran into serious interference troubles when using it on the same page I was using the Google Ajax search widget.

Then I came to the Moo.fx accordion and that was the good one.
Still, this Accordion has some limitations:

If your page is large, your browser won't scroll properly to the selected accordion tab.
It would be better if the user could click on an expanded tab to collapse it back/
You might want to have a collapse/expand button on each tab.
You might want your accordion to degrade gracefully for non JavaScript enabled browsers (accessibility + SEO optimization) and have links to separate pages instead of tabs in this case.
You want to disable the Mootools Accordion from following those accessibility links because you just want to expand the tab instead. The default Accordion implementation really sucks to this respect.
You might want a tab CSS rollover effect.

I addressed all those issues for the web site I'm building now: Livetribune.com.

Issue #1:
Use the prototype scrolling feature to scroll to the expanded tab (I'm using Moo for Prototype):


     myAccordion = new fx.Accordion(myLinks, myDivs, {onActive: function(toggler, element){
         setTimeout("Element.scrollTo('" + toggler.id + "');window.scrollBy(0,-150)", 400);
    });

Issues #2 and #3:
hack Accordion a bit to allow collapsing back any tab. And also add/remove a class name to the toggle DOM element in order to then use different CSS styles, for instance to display a different button backgound-image when to tab is collapsed or expanded.
Here is my modified Accordion class, I simply changed the showThisHideOpen function :


   showThisHideOpen: function(iToShow){
       var forceHide = iToShow == this.previousClick;
       if (!this.options.alwaysHide){
           if (forceHide)
               this.previousClick = null;
           else
               this.previousClick = iToShow;
           var objObjs = {};
           var err = false;
           var madeInactive = false;
           this.elements.each(function(el, i){
               this.now[i] = this.now[i] || {};
               if (i != iToShow || forceHide){
                   this.hideThis(i);
                   $(this.togglers[i]).className = $(this.togglers[i]).className.replace(' expanded', '');
               } else if (this.options.alwaysHide){
                   if (el.offsetHeight == el.scrollHeight){
                       this.hideThis(i);
                       madeInactive = true;
                   } else if (el.offsetHeight == 0){
                       this.showThis(i);
                   } else {
                       err = true;
                   }
               } else if (this.options.wait && this.timer){
                   this.previousClick = 'nan';
                   err = true;
               } else if (!forceHide){
                   this.showThis(i);
                   $(this.togglers[i]).className += ' expanded';
               }
               objObjs[i+1] = Object.extend(this.h, Object.extend(this.o, this.w));
           }.bind(this));
           if (err) return;
           if (!madeInactive) this.options.onActive.call(this, this.togglers[iToShow], iToShow);
           this.togglers.each(function(tog, i){
               if (i != iToShow || madeInactive) this.options.onBackground.call(this, tog, i);
           }.bind(this));
           return this.custom(objObjs);
       }
   }

Issue #4, #5 and #6
We can consider it all together. For good accessibility and degradability, we need our accordion tooggle elements to be standards 'a' links. In that a href element, we put a link to a page that will display approximatively the same content than in the tab, possibly with a restful URL so that's good for SEO optimization.

Also, now that we have those a with their href, we are able to make a CSS rollover using the a:hover CSS pseudo selector to translate the background image of our toggle element. Remember: IE6 only accept the :hover selector on links and moreover the href of those links have to be defined for the :hover selector to apply properly.

But the catch is that if you use links with href as Mootools Accordion tooggle elements, your browser will follow the links and refresh the window instead of collapsing/expanding your accordion tabs. Ending your onActive custom function doesn't fix this nasty behavior. The only way I found was to patch the Accordion widget and add the return false; statement in the initialize function in the onclick trigger of Accordion.js:


initialize: function(togglers, elements, options){
        this.now = {};
        this.elements = $A(elements);
        this.togglers = $A(togglers);
        this.setOptions(options);
        this.extendOptions(options);
        this.previousClick = 'nan';
        this.togglers.each(function(tog, i){
            if (tog.onclick) tog.prevClick = tog.onclick;
            else tog.prevClick = function(){};
            $(tog).onclick = function(){
                tog.prevClick();
                this.showThisHideOpen(i);
                return false;
            }.bind(this);
        }.bind(this));
        this.h = {}; this.w = {}; this.o = {};
        this.elements.each(function(el, i){
            this.now[i+1] = {};
            //el.style.height = '0';
            el.style.overflow = 'hidden';
        }.bind(this));
        switch(this.options.start){
            case 'first-open': this.elements[0].style.height = this.elements[0].scrollHeight+'px'; break;
            case 'open-first': this.showThisHideOpen(0); break;
        }
    },

for details: have a look to Livetribune. I'm still working on this site, so please be indulgent if you are coming in a bad time. Livetribune will be ready in October.

Hope you can now build your ultimate accordion widget!

Raphaël Valyi.

Very Good Javascript lessons from Yahoo

2007-08-22T16:19:00.000-07:00

So that's it, the web2.0 is popping everywhere all over the place and you need to make your sites sexy using Javascript and AJAX.

Then you probably feel JavaScript is a crappy language you never wanted to learn. So you keep learning bits of it as you need to. But doing this you'll probably end up reading a lot of crappy examples too and that will keep you in the vicious circle of thinking JavaScript is only a toy language.

But what if you take some 2 hours and really learn the basic correctly from Yahoo people who really know what they are doing? What if you finally learn what is that damned prototype stuff and how you can prevent libraries to collide their namespaces? I did it yesterday and I'll never regret it. I still prefer Ruby (JRuby) or Java many times over Javascript, but know at least I'll know how to code JavaScript and will be much more interested and effective in making the most of existing frameworks.

So if you can, really watch those screencasts:

The JavaScript Programming Language
Advanced JavaScript

Again, I've read lot's of Javascript tutorials or even books before. But nothing approaching the clarity and effectiveness of those lessons (except may be the AJAX in Action book).

Have fun,

Raphaël

import your Google bookmarks into del.ico.us!

2007-07-03T15:59:00.000-07:00

There are plenty sites explaining how you can export your del.ico.us tags, for example to Google Bookmarks, for instance: http://blog.persistent.info/2006/10/import-your-delicious-bookmarks-into.html
(or export your Del.ico.us tags using the Del.ico.us tool in 'settings' and then import them in Firefox or Google).

But what if you want the contrary Google Bookmarks -> Del.ico.us ?
First of all, why would you do that?
Well, despite of Google offering a richer indexing of your bookmarked entries, Del.ico.us has a number of advantages:
* Del.ico.us is really oriented toward massive tag sharing, so it really helps other people to find out good pages semantically.
* Del.ico.us really uses tags as a plebiscite system to rate the page quality of a page
* Del.ico.us gives nice JSON and GET API's allowing developpers to build semantic mashups based on social tagging.

so now you know why, here is the how to import from Google Bookmarks to Del.ico.us
1) login to your Google bookmark account.
2) go to that link: http://www.google.com/bookmarks/bookmarks.html
and save that html file somwhere on your disk.
3) now login to your Del.ico.us account
4) go to 'settings' on the top right of the page. Now select: 'import / upload' in the middle of the page.
5) now just select the file you saved from Google as the file to upload. Also you probably want to removed the 'imported' tag that's going to be added by default but that's up to you. Uploading might take some time; just come bac to your bookmarks after a minute...
6) finally, sadly Del.ico.us won't share those uploaded tags with other people. That's probably because they want to ensure the quality of the tagging before sharing any tag. So unfortunately, you would now have to share your tags one by one using the buttons in the tag list. Any javascript Firebug hack to automate this would be very welcome.

That's it folks! Keeping in sync Delicious and Google Bookmark in some way might be great.

Rails 1.2 and rb-gsl ruby math lib don't play nice together!

2007-02-13T15:42:00.000-08:00

Hi,

so Rails 1.2 might be nice, but I got some HUGE headaches when migrating to Rails 1.2. This time, all my ActiveRecord objects were raising a "private method `equal?' called for #" exception when I was using require 'gsl' no matter were in my code. That sucks a lot because ruby is a slow language (yet powerful) so I really needed to use the C native maths (matrix stuff) functions from rb-gsl.

With Rails 1.1.6 everything was OK, but with Rails 1.2, I started to get errors like:


private method `equal?' called for #
/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/core_ext/class/inheritable_attributes.rb:127:in `inherited_without_layout'
/usr/lib/ruby/gems/1.8/gems/actionpack- 1.13.2/lib/action_controller/layout.rb:185:in `inherited_without_helper'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/helpers.rb:120:in `inherited_without_api'
/usr/lib/ruby/gems/1.8/gems/actionwebservice- 1.2.2/lib/action_web_service/container/action_controller_container.rb:84:in `inherited_without_action_controller'
/usr/lib/ruby/gems/1.8/gems/actionwebservice-1.2.2/lib/action_web_service/dispatcher/action_controller_dispatcher.rb:32:in `inherited'
./script/../config/../app/controllers/application.rb:6

The problem with ruby is that each framework attempts discretely to do its own magic by redefining existing methods. But frameworks end up crunching themselves this way. That's pretty much the same knd of trouble I had with gettext (see previous post)

After looking quite a while at the code, I had no idea what object was this Hash:0xb74e20fc object neither why ActiveSupport would care about it. Adding some debug logs into ActiveSupport, I saw that all GSL objects are being handled by ActiveSupport but again, I found no way to prevent such a nasty exception to occur.

At the end, I found a dirty compromise to continue using rb-gsl in my Rails app:

First it implies you patch ActiveSupport (1.4.1 in my case) to look if the method 'equal?' isn't private as it would just occur with rb-gsl:
add this method visibility check line 127 of

/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/core_ext/class/inheritable_attributes.rb

      if (!inheritable_attributes.private_methods.index('equal?')) && inheritable_attributes.equal?(EMPTY_INHERITABLE_ATTRIBUTES)

(hint: you can use that check to log the problematic 'child' object if you want to debug Rails or rb-gsl)

Second, we can't require 'rb-gsl' inside an ActiveRecord object (neither a module included in an ActiveRecord object) anymore because even if we avoid the first illegal access to the private method, a there is a second catch that would break out later: 'can't change a frozen Hash', go understand...

So you'll have to encapsulate your rb-gsl lib inside some wrapping ruby object that will contains require 'gsl' and provide ruby methods to the gsl methods you are interrested in. And eventually your controllers or ActiveRecord object will play with that wrapper to call the gsl math functions.

Yes, overall, doing that really sucks, does anyone have a better idea? Who is the culprit? Rails or rb-gsl? If rb-gsl, how modify it to make it Rails compliant again?

NB: Paul King also noticed this bug with rb-gsl:
http://rubyforge.org/pipermail/mongrel-users/2007-January/002659.html
but he could eventually add a stub empty equal? method to its problematic object. This didn't work at all in my case though.

Rails 1.2 doesn't play nice with gettex

2007-02-09T19:18:00.000-08:00

Hi,

After migrating to Rails 1.2, under certain conditions, I had suddenly lots of trouble to save any trivial ActiveRecord object. All was working like a charm before and with Rails 1.2, I started to get that exception:
In the console, I play with a dummy ActiveRecode model with only one sample string field:


>>foo=Foo.new
>>foo.save
>>foo.save SystemStackError?: stack level too deep
from
...
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction' from /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/transactions.rb:95:in `transaction' from /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/transactions.rb:121:in `transaction' from /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/transactions.rb:129:in `save_without_validation' from /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/validations.rb:752:in `save'

By under certain conditions, I mean that the bug was hardly consistent, it happened only after I my application gained some complexity (Foo excepted) and also using the auth_generator plugin.

Still, I think the problem is coming from gettext (1.9.0) that seems to not behave correctly with Rails 1.2. In any situation, removing gettext always removed that bug.

So in my case I hope gettext deveoppers will react while I'm polishing my app business logic, else I'll be forced to migrate to Globalize.

Hope this helps.

Raphaël.

Prototype 1.5 bug?, at least not backward compatible with auth_generator...

2007-02-09T18:35:00.000-08:00

Hi,

I recently upgraded my web app to Rails 1.2 coming along with Prototype 1.5. Also, In that app, I'm using the auth_generator plugin, a wonderfull plugin allowing to create a user base + login system + right policy in a snap. But what makes it very special is that this Rails plugin can display users info in a javascript generated DOM fragment that will behave accordingly to the user specific cookie allowing you to cache all your pages statically, and that's really a big win considering how slow ruby is compared to hitting only Apache for static pages.

Ok, but this is not the point. The point is that auth_generator has got some simple Ajax requests and they do not work anymore with Prototype 1.5 at least with Firefox (2.0.0.1 on Ubuntu Linux Edgy)!
Here is an example of such a request:

new Ajax.Updater('accountinfo', 'http://localhost:3000/auth/logout', {asynchronous:true, evalScripts:true, onLoading:function(request){Element.show('ident_spinner')}}); return false;")

Nothing especiall as you can see (Or please tell me if I'm wrong).
But this freezes on a Prototype exception arround the line 916:


[Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIXMLHttpRequest.setRequestHeader]" nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)" location: "JS frame :: http://localhost:3000/javascripts/prototype.js?1170853340 :: anonymous :: line 916" data: no]

I have no idea at all what should be the correct behavior, if that's a Prototype bug, a Firefox bug or an auth_generator bug.

I have no idea of what side effect this could have, but a dummy fix for me consisted in rewritting the lines arround line 916 this way:


    for (var name in headers) {
   try {
        this.transport.setRequestHeader(name, headers[name]);
   } catch (e) {
        //do nothing
      }
    }

Hope this helps, if you have a better fix, please go on and comment.

Raphaël

Update: OK, Prototype 1.5 really bombs out with Firefox, others railers discovered the bug and they found no other fix, see here:
http://dev.rubyonrails.org/ticket/6918

Ubuntu trouble: uninitialized constant Mongrel::HttpHandler (NameError)

2006-12-22T16:45:00.000-08:00

Hi,

Ok so Rails is the must have framework, and Mongrel the must have server. But may be you just had an error when trying to launch Mongrel on Debian and you are googling desperately for a solution.

So, if after typing:
mongrel_rails

you get:
uninitialized constant Mongrel::HttpHandler (NameError)

The reason on Debian might be your Gem package is outdated, so try to type:
sudo gem update --system

For me this upgraded Gem from 0.8 to 0.9, than I reinstalled Mongrel with:

sudo gem install mongrel --include-dependencies
sudo gem install mongrel_cluster --include-dependencie

and voilà, Mongrel is now working!

Hope this help.

PS: Viva Lula! viva Chavez! fudeu a direita! Com mas de 60% dos votos contra direita, foi uma boa goleada contra burguesia, valeu. (translation: no putsh in Brazil neither Venezuela, South America is going fine).

RIGHT WING PUTSCH ATTEMPT IN BRAZIL

2006-09-26T14:29:00.001-07:00

I'll try here to inform people that may not know the situation one week before the presidential elections in Brazil. After having been evinced from the power in 2002, those who always ruled Brazil since its colonisation are trying everything to prevent Lula from being elected again.

Opinions pools were pretty clear since several months ago: Lula was summing about 55% of the vote intentions against less than 30% for the right wing candidate. But the oligarchy -still very powerful in Brazil- tried to biased the public opinion. Journals such as Veja are lying more than Bush, while others like the "Folha de Sao Paulo" are obviously biasing the facts against Lula. The problem is that there aren't yet any mass media to balance those partial views from journals owned by the oligarchy.

Among the most scandalous stuff, are: the national lawyers association (OAB) is trying to impeach Lula; while the government was top blame, because they bought vote from opposition politicians to gain governability, the biggest problem is that the press only persecuted the government without even persecuting those corrupted politicians from the opposition who used to be corrupted for decades. Finally one week ago, a tape proving the corruption of Jose Serra -a former rival of Lula- has been discovered. Instead of investigating this case, the press is only focused on linking Lula with irregularities when obtaining that tape, again, they are trying to impeach Lula despite the total lack of proof and its huge popularity. They are thus trying to shunt the democratic process in Brazil because they fear to loose their privileges inherited from the colonnial situation.

Lula is very popular in Brazil, it's the first president who helped the poorest of the poor of Brazil. He is hopefully going to be elected any way. But still, the putsch attempt should be denounced anyway. It's our duty to blame the PSDB party and the mass media they manipulate wherever they go (that's why I'm telling it in english).

The issue is nothing funny. While the brazilian mass media (mainly Globo) are hiding the facts, they are more people killed in 3 months in Sao Paulo than during the whole second war in Lebanon! Corrupted politicians such as Jose Serra (PSDB) are widely responsible for letting that violence run out of control!

I'm only giving my view here. Others may defend the cause better (PT web site, , blogdareeleicao) , and the good results of Lula and popularity can be verified against numbers from impartial organisms. Here are such indicators:
numbers,
details of numbers
popularity pool

Sun accepted the annoying BeanInfo lookup in XMLEncoder as a bug, vote for it!

2006-09-22T07:30:00.000-07:00

Hi,

As I described it here, XMLEncoder is giving too much pain when marshalling javabeans in unsigned applets. I've built up a workaround framework, but still, a few BeanInfo meta classes gets looked up on the server (while they don't even exist, so it's only freezing the network and loading the server). Also I can't tell if my framework really scale to any application and if there is no side effect. If there are they would be hard to debug in such a marshalling context.

We should just be able to disable that codebase lookup. Lots of problems with applets indeed come from the code base lookup, among them: problem with ResourceBundle, Services lookup, XMLEncoder, XMLDecoder (less annoying as the workarround is fully efficient). So I reported that to Sun Microsystems and the bug has been accepted. Here is the link.

So if you find it annoying and want a fast and easy marshalling in applets, if you want a faster and smaller JGraphpad CE, please vote for that bug here:
here
(you should create an account if you don't have one).

Raph.

Applet revival: JGraphpad CE BLACKJAX applet size cut from 450K down to 110K thanks to Pack200!

2006-09-21T07:20:00.000-07:00

Pack200 is an underestimated widely deployed java tool to speed up dramatically the download time of applets or webstart applications.

I was already aware of an other 'BLACKJAX' like applet deploying Apache Derby on the client. Then I heard its creator claimed he cut Derby from 2Mb down to 600kb using Pack200..

So Pack200 is a class compressing/deflating tool deployed along with the JRE since version 1.5 (Tiger). How it works: in short applets and webstart apps are still requesting a jar archive on the server. But on the server side, you catch the fact the a different encoding (pack200-gzip) is requested by java 1.5+ clients. When requested, you'll then return the "packed" jar version, else (JRE prior to 1.5) you'll return the normal jar archive. JRE 1.5+ webstart and applet clients will deflate the packed archive on the fly saving a huge amount of server load and bandwidth.

Of course, this transfer win is only worth the CPU overhead on the client side if your archive was large enough. Make your benchmark.

Results are here:
So what about JGraphpad Community Edition? Without plugins (but with JGraph) - that's configured as the minimal online diagram editor - JGraphpad CE is weighting about 450K. Pack200 cut down the archive to 110K which is a very big win! Frankly I didn't benchmarked the extra overhead against the transfer win, but overall the win is HUGE! Overall, the applet startups 3 times faster!

So, yes it means that JGraphpad CE which is by far a more complex app than TinyMCE or FCKEditor which are 'only' online word processors, is still something 2 times smaller to download!!! (And with our "Javascript sugar", it starts up just as fast). Really amazing.

Now consider that JGraphpad CE is architectured so that you can download some plugins in the background (until you really need them) while diagramming with the core JGraph application, it turns JGraphpad CE really credible for diagram online edition.

Limits: JGraphpad CE require java 1.4.2 at least to run. But with java 1.4.2 you'll have to download the classic jar (it's totally transparent however). Only happy java 1.5 and 1.6 users will get their first download time boosted thanks to the Pack200 compression.

Server Side details:

JGraphpad CE has been designed to be server side agnostic. It only two use standard GET and POST HTTP methods to download/upload a diagram from the server. So you can use what you want to handle your uploads. I personally use Python within the MoinMoin wiki (normal file upload feature), but you could use Java, PHP or Ruby, no problem.

Still getting the Pack200 trick properly configured is server side dependent. It's easy to find out how to do with a Java server.

For an Apache server, the trick is really easy, it's explained in that excellent article.

Finally, if you can't o don't want to bother with Pack200, just put the jar in the root of the applet codebase directory as usual and this will work.

I'll soon provide a bunch of official documentation about that new JGraphpad CE release. Don't hesitate to send your feedback.

AJAX + Applet = ? ... => BLACKJAX !

2006-09-15T07:48:00.000-07:00

OK, apparently I'm not alone thinking Ajax apps and applets can interact together to provide richer Internet applications, see this other blog also. Indeed:

Ajax is better for fast responsive GUI and document centric applications.
Applets on the other side provide better graphics, DB connectivity, XML processing or any heavy and maintainable business logic.

Okay, but now we need to name this new way of bridging those technologies. I propose to coin:
"BLACKJAX" standing for:
Background Liveconnect Applet Code Kicking Javascript and XML.

That's indeed a mix between AJAX and Applet. 'Asynchronous' is better replaced by 'background' which means the same thing but makes a funnier acronyme.

A new tech is born, let's hype arround it!
I'm waiting for your comments. Cheers,

Raph.

degraded AJAX app + pre-loaded hidden java swing applet, the best deployment option for complex rich clients?

2006-08-15T08:40:00.000-07:00

1) When AJAX isn't enough to address complex visualization or complex UI (or: there is no Java2D in javascript)

The trend is definitely to port traditional desktop applications to the web using AJAX. To overcome browser incompatibilities and javascript language flaws, we are using lots of frameworks including the Google Web Toolkit for instance. Those are fine but will only work well within a limited range of use cases as I will explain.

There is a use case indeed where AJAX+frameworks aren't enough: the case where you need superior visualization features like diagram editing or diagram driven modelling for instance (think about the equivalent of the Visio desktop app to model UML online for instance).

Ultimately, the point is that javascript in browsers doesn't allow you to dynamically draw a bunch of lines without consuming too much memory. Yes there are some attempts out there:

but as you can see by yourself, those trick doesn't scale to dozens of animated lines, no way.

2) SVG applications fall into the same deployment trap as java: not everybody has got the required runtime platform

Then what? Oh yes, there are those guys that will tell you to use SVG. SVG is great indeed, but you know what, MS Internet explorer requires an external plugin (Adobe) to display SVG, so basically, your shiny SVG/AJAX app falls into exactly the same deployment trap than java: not everybody can run it because not everybody already has got the required runtime platform.

3) SVG+VML turns into a maintenance nightmare

Then what? There are those who will tell you: OK let's writte an AJAX app the will use SVG on Moz browsers and VML on Internet Explorer. Fisrt of all, unlike SVG, VML is an end of life product that's not that much powerful. And moreover, your AJAX/SVG/VML app has already turned one of those maintenance nightmares!

4) Then a java client is the best, even in 2006. But which one, webstart or applet?

So taking all this into account, the best target runtime for large deployment of complex visualization application is... JAVA. A good old one jre 1.4.2+ is already well deployed indeed and programming java is easy and best of all, java is a language able to handle the complexity.

Java? OK, but do you mean applets or webstart? Well actually I mean applets. While webstart is younger and got a better security sandbox, webstart won't allow you to integrate you application in the browser just like applets. And using all the DHTML browser facilities is very handy, especially in document centric applications like CMS or publishing tools.

Yes but aren't applet slow at startup? Well, yes they are. And you know what, that's almost normal considering all you can do with the java runtime (much more than Flash for instance). Also don't forget that applets are cached after the first use, so even if your huge applet first takes time to download just like any other web app, then only startup time matters.

5) Don't even wait for the applet startup, instantly popup a degraded AJAX app to handle the first user interactions!

OK, Javascript apps start a way faster than applets. So why not instantly popup a simple degarded javascript app to deal with the first user interactions while the heavy business logic is being loaded in a powerful hidden applet? After the applet is loaded, we swap the two and the user now plays in a full featured desktop Swing application...

6) Hidden Applet and degraded AJAX howto

I've read that some people advice to set the applet size to 1*1 pixel to hide it and then makes it larger to display it. However, unfortunately, it seems that this hack makes the resize quite slow in Firefox, giving the feeling that the app is unresponsive.

I've found a better way, style your Ajax Div container and Applet div container this way (using CSS) :
div.visibleAjaxDiv {height: none; overflow:hidden}
div.hiddenAjaxDiv {height: 1px; overflow:hidden}
div.visibleAppletDiv {height: none; overflow:hidden; visibility:visible}
div.hiddenAppletDiv {height:1px; overflow:hidden; visibility:hidden}

then switch the Ajax and Applet div class when appropriate (for instance when the user request a complex interaction in your Ajax app, after you checked that the applet has been fully loaded indeed).

7) Liveconnect as the glue between the AJAX app and the applet

What you want is that the applet will remember what the user first told you in the AJAX app (for instance the name of the diagram to be created) and you also want the AJAX app to remember what the applet told (for instance you hide again the applet after editing your diagram, but then you want the AJAX app to display the picture of the diagram when it has been uploaded...

8) Your applet needs to communicate with the server. XMLEncoder and XMLDecoder are a free lightweight persistence framework for your applet, but it requires some tuning to be used in an unsigned applet.

Hardcoding business object encoders and decoders is the best way to write unmaintainable and unnecessary code. Fortunately, java allows you to persist and read javabeans at no cost.

However, the DefaultPersistenceDelegate and Metadata java code suffer some flaws making them hard to use inside an unsigned applet (security exceptions and heavy network use), but fortunately, I found a workaround for both XMLEncoder and XMLDecoder, see those posts:

9) Conclusion

Yes this way of associating technologies seems a bit tricky and too novel for production use. However, I think this is the best available technology for complex visualization rich clients. Also, I don't consider it's more hacking than traditional AJAX frameworks. No it's not, most of this technology is traditional java programming, one CSS style sheet, two java classes to reuse and only a bit of javascript but less than a full blown AJAX app. Also, if several people start to deploy this way, we could build a framework and spread best practices. I think this is better than waiting forever that a better client runtime is widely deployed... All this is available now. Any comments will be very much appreciated.

10) An example?

Yes, I've got an example. Unfortunately, the degraded AJAX app is reduced to its simpler: only a clickable image launching the app, with a tooltip, image reloading and file save checking, but the Swing app is shiny and based on JGraph (the GPL version of JGraphpad). And it's not hard to follow that proof of concept and build a larger app following those guidelines.
see the example here: visualmodeller

hacking XMLDecoder for unsigned and network friendly applets

2006-07-17T01:33:00.000-07:00

Want to avoid that at applet startup?

network: Connecting http://yourHost.com/META-INF/services/javax.xml.parsers.SAXParserFactory with proxy=DIRECT
network: Connecting http://yourHost.com//META-INF/services/javax.xml.parsers.SAXParserFactory
network: Connecting http://yourHost.com/META-INF/services/com.sun.org.apache.xerces.internal.xni.parser.XMLParserConfiguration with proxy=DIRECT
network: Connecting http://yourHost.com//META-INF/services/com.sun.org.apache.xerces.internal.xni.parser.XMLParserConfiguration

When used in an usigned applet (yes there are complex hacks if you are willing to pay for a certificate), XMLDecoder also use to open unecessary sockets to lookup the applet code base on the server. The trouble isn't so important as only something like 4 sockets will be opened. But depending on your server load, this can easily result in a 1 second latence when opening the applet. Indeed, it appears that code base lookup when loading classes is a process that completely prevent others threads from running, be it in a simple other Thread or even in the SwingWorker thread.

Also, since this latence occurs at the applet startup is your applet is to open a server side XML file, then the latence is finally really annoying and make applets look sluggish compared fast and responsive ajax applications.

Fortunately, I found a way to avoid those 4 sockets. Instead of using XMLDecoder, use the following derivative:

(Also, to decode encode side XML files in an applet using an usigned applet, we use AppletFriendlyXmlEncoder, an applet friendly derivative of XMLEncoder. )

The full source of AppletFriendlyDecoder which extends XMLDecoder is to be found on the Visualmodeller wiki here:
http://www.visualmodeller.org/wiki.fcgi/AppletFriendlyXmlDecoder

A demo of it can be found in JGraphpad Community Edition demowed online at Visualmodeller here:
http://www.visualmodeller.org

hacking XMLEncoder for unsigned and network friendly applets

2006-06-15T09:23:00.000-07:00

XMLEncoder is very handy to serialize javabeans without the burden of hard coding encoders and decoders. Unfortunately, the current Sun implementation doesn't work properly with unsigned applet (there are hacks if you are willing to pay for an applet certificate).

Especially using XMLEncoder in an unsigned applet is very likely to result in security exceptions and will run slow and without any background threading solution.

Indeed, the DefaultPersistenceDelegate used by XMLEncoder for common javabeans uses to kill the network by opening dozens of sockets because it looks up for beanInfo classes on the server code base for every type inside your bean.

Fortunately, I worked hard and dicovered a hack: use the following encoder instead of XMLEncoder (you may adapt it a bit to your own beans in order to avoid some more specific beanInfo lookups). It's currently used in the last JGraphpad Community Edition online diagram editor.

Also, to decode server side XML files in an applet using an usigned applet, we use AppletFriendlyXmlDecoder, an applet friendly derivative of XMLDecoder.

The full source of AppletFriendlyEncoder which extends XMLEncoder is to be found on the Visualmodeller wiki here:
http://wiki.visualmodeller.org/AppletFriendlyXmlEncoder

A demo of it can be found in JGraphpad Community Edition demowed online at Visualmodeller here:
http://wiki.visualmodeller.org