Turn Geek

Lucene! O' Lucene!

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sat, 25 Oct 2008 04:05:00 +0000

Well, I recently got a chance to evaluate search platforms and it turned out to be a very interesting experience. First, the obvious choice was lets use Google - but it turned out that Google's search engine while doing a great job on the internet is one of the least equipped when it comes to enterprise search - which is more a data mining job rather than what happens on the internet - cross linking, user clicks etc. For example Google lacks faceted search - which is by far the most touted and seeming useful feature when it comes to enterprise data getting exposed on an intranet or even customers. Even the Lucene/Solr combination supports faceted search (though I have yet to try that out!)

Search Engine integration and customization still proves to be a specialist job though there is a lot of open source which exists in the area. If you are trying to do anything out of the ordinary (like the corner search box which does simple keyword search) or you have a large amount of data then you are in for some heavy duty $$ spending. The major players in the search space are still Autonomy, Endeca, FAST, Recommind, Exalead, Dieselpoint, Vivisimo.

I did find a lot of open source which can be integrated with Lucene though: Nutch, Lingpipe, Gate and Carrot. You could do lots of nifty things with this. Also CNet and Factiva seem to use Lucene.

Choosing between Drupal, Joomla and RoR

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 12 Aug 2008 15:43:00 +0000

I recently did a quote update for myself where the three strong contenders are when building community sites today. Ruby on Rails has been in news a lot as the coolest app around, in hot demand. Rails has become almost the tool of choice for startups today trying to build something quickly without fear of it getting outdated or hard to manage when they grow. A lot of rails hosting companies have also sprung up.

It may also seem that the debate between Drupal and Joomla needs to be resolved once for all. I have evaluated the two myself a few times without any conclusive results as to which one is the clear winner. I think a lot depends on what you really want to do and your goals.

For example, Joomla may be the ideal choice when it comes to quickly building up a site with nice features which come integrated or are easy to find. However if you need flexibility or are not sure how the site will evolve as time goes by (in other words if you think your website may be your business next year or so), then you need to look at Drupal and RoR. Joomla may not be a wise choice, at least based on what I have read or experimented with. So if you have a bounded task at hand, Joomla is quick, simple, elegant and beautiful.

If you are building a content site and have ruled out Joomla based on the above, then Drupal may be something you want to seriously look at. Though it is based on PhP, a not so hot technology, it has the bones to last you for a long time.

Finally, if you have a custom application you are building from scratch (so there is no previous baggage of Java or PhP) targeted towards consumers or a community, Ruby on Rails should be your choice.

Enterprise or B2B applications may be best suited for Java and you can pick the Spring/Hibernate framework there which is probably a much more safer approach considering it is much more mature and feature rich.

Finally, here is a great series of articles on Drupal from IBM, who picked it as their platform of choice between Joomla and RoR (though that was back in 2006)

Blogged with the Flock Browser

Rails v/s Spring v/s J2EE

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Thu, 07 Aug 2008 04:08:00 +0000

Well, after having researched and played around with Rails quite a bit, I have a few conclusions I have formed. For example, if you are developing a web Software as a Service model, the best framework or language to use may be Java/Spring since it is clean and lightweight compared to J2EE for doing something much simpler than a full enterprise class application which may be best done using the latter. It is also much more mature than Ruby/Rails which may be more suitable for consumer facing or retail applications like a video sharing or social networking site. If you are starting fresh, I'd highly recommend going with Ruby/Rails to reduce the development cycles required. Other good choices would be Python/Django or just PHP if that works.

While this lays down a few things regarding application development, some important decisions need to be made regarding hosting. CentOS is clearly a winner when it comes to deciding which operating system to use (most stable and free for server hosting). However I am still not clear which "cloud" provider is the best to go with. Amazon's EC2 presents some very flexible options regarding that, but it doesn't seem to come with persistent data. Amazon S3 doesn't seem like a good candidate to fill that hole since it doesn't do a good job of handling a database. There seem to be a large number of good alternatives like GoGrid, Joyent etc in this space which may give a flexible hosting option (which looks more traditional than Amazon EC2) which includes a database. Check these out and do comment if you have a recommendation...

Blogged with the Flock Browser

The Back-end Wars

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Fri, 18 Jul 2008 03:17:00 +0000

While there is a war brewing up on the web front end frameworks, viz AIR, Gears, Sproutcore and Silveright, many don't realize there is a similar war brewing on the web back-end technologies as well. The developer communities are gearing up to take each other on on different aspects like ease of use, frameworks, performance, size of community etc. Here are the major contenders:

1. PHP
2. Python
3. Ruby
4. Groovy
5. Perl
6. Java

All of these technologies have evolved in the past few years to come up with a lot of MVC frameworks which help clean up the code and reduce fresh development cycles needed to launch new sites. For example Symfony is the most popular framework for PHP. Django is the winner for Python. Rails is leading the path for Ruby and Grails is leading the path for Groovy (which seems to be the path Java developers may pick going forward). In Java, the Springs framework is quite popular now. Perl developers pick Catalyst a lot of times.

These are too many options. So which combination is going to be the winner? My viewpoint is that this is not going to be a decision based on performance of the software as many think, but more driven by adoption. For example Python and Django have been blessed by Google App Engine. That means its a clear winner. Ruby and RoR have had tremendous success in the last few years as new developers move to Ruby in large numbers because of the language features. Groovy provides a migration path for Java developers and I think it will have its share of success as long as the bridges to existing Java code work. I really don't see much future for PHP or Perl based frameworks going forward. Among Python, Ruby and Groovy, I think Python and Ruby will continue to grow at a much larger pace than Groovy community.

Blogged with the Flock Browser

Web 2.0 challenges for Ribbit

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Mon, 07 Jul 2008 04:52:00 +0000

Ribbit is an excellent example of what kind of challenges a telecom outfit can face in the Web 2.0 world. First, Ribbit is an innovator of kinds - it is perhaps the only flash/flex based phone I have seen which claims to provide connectivity to the PSTN. I don't fully understand what architecture they currently utilize, but its quite possible that they connect RTMP protocol to their traditional network on the proprietary softswitch they have on the backend.

There are several game changing plays still left when it comes to VoIP and flash, so lets summarize those before we come back to ribbit. Real time streaming using Flash today requires the Flash MX media server in the network and currently my assumption is that it doesn't support SIP (though Adobe has included support for Speex - an open source free codec in the latest version 10 of Flash). When Flash does support SIP, a lot of new players doing VoIP and Flash integration will jump in. Flash is a proprietary technology and there are a lot of open source versions of software which attempt to reverse engineer Flash (yes, this is completely legal it seems). Red5 is the most prominent of all which intends to act as a substitute for MX server. GNU flash is a client side substitute for Flash runtime.

I also didn't fully understand Ribbit's business model when I was on their site, but that is perhaps due to the fact that they are very different from a traditional telecom company from this perspective. A traditional telecom company sells minutes when connecting to PSTN or monthly charges for telephone numbers you can buy. Ribbit's model is a little different here and they seem to be mostly trying to sell their VoIP and web capabilities to other companies with web based products which will benefit with Voice features. Salesforce.COM is one integration they have done. One suggestion from me would be the more traditional customer support market where companies would benefit with Voice support to improve post sales communications with customers. On the consumer side, Ribbit's model is not clear and I am quite surprised they don't have the valley approved "Ad Model" worked out yet for that ;)

The Zimbra for Mobile: Funambol

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Mon, 23 Jun 2008 20:15:00 +0000

Funambol has an open source approach to Mobile Email and Mobile Sync. Some say that Funambol is out to take out RIM. They have a model similar to what I saw last with SugarCRM - free for Enterprise IT users or personal sync and you can buy the enterprise edition. I think that is a good model and the space needs a competitor as well. The software appears to be licensed under the AGPL, again I have seen that after a while. For those not familiar with AGPL (Affero GPL), it is a stricter form of traditional GPL license in that you cannot even host it w/o disclosing source code.

Blogged with the Flock Browser

Wix and Sprout: Online Flash Editors

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Mon, 23 Jun 2008 19:14:00 +0000

If the thought of having to learn flash to make the delightful widgets and apps ever occurred to you, then here are two tools for you: Wix and Sprout. I tried Sprout myself and it was extremely easy to use (I am still waiting for a Wix Invite). Sprout's focus seems to be the creation of widgets - small pockets of content with rich media you can put on your blogs or websites to promote anything you want or tell people about yourself (the new jazzed up way against the old boring image of yourself). Wix on the other hand is more focused on building a full website.

Other differences include the fact that Sprout is on the West Coast (and Hawaii) and Wix is in NY and the former is kind of ahead in that it recently got its Series A with Polaris ($5M).

Blogged with the Flock Browser

Linux on everything

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Wed, 14 May 2008 13:39:00 +0000

Here is a company I came across who provide linux porting on various devices and what I found interesting was the home page which was on wordpress with some default layout. That's geeky. But it was nice and warm. I quickly bounced from their site to another interesting company: Jumpbox which is again into some interesting things. Simple, but interesting. A Jumpbox is basically software which runs on virtual servers (and they restrict it only to virtual servers) and it provides a kind of starter toolkit for typical applications like joomla, crm (mostly open source apps) etc. Its great for enterprises, but I am not sure how they manage to keep the IT guy away. Eventually to manage even a website - to configure any opensource app to run properly you need a geek or you better be one. Most open source apps are easy to install at a certain level of geekness ;) I mean they are not hard. I have found most of the systems they are talking about there relatively easy to install without knowing any details about the systems. Maybe I need to learn more. I think one of the more interesting things Jumpbox does is the subscription. Even with that I still don't know why I don't need the IT guy. And if I have one, why do I need Jumpbox? It seems like a small delta. Need to understand it better...

Evaluating Hosting Providers

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sat, 03 May 2008 04:05:00 +0000

Our next phase of network evolution is finally here which means I am evaluating a bunch of hosting providers for the services they provide and hope to share some information soon on different parameters like: (a) Where are the NOCs and why that is important for you (b) Availability of High Availability options (c) Cost etc. The providers I am looking at are: hostway, rackspace, servepath, theplanet, logicworks and godaddy.

Replicating MySQL over the WAN

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sat, 03 May 2008 03:57:00 +0000

I am researching how practical it is to do real-time replication of a MySQL database over a WAN SSH tunnel. What are the constraints caused by bandwidth, MySQL *blocking* bugs (apparently there are a few out there in older versions of MySQL such as 4.x) and how real-time is the real-time replication. How well does a Master/Slave or a Master/Master MySQL system work in this case?

Challenges in peering a P2P SIP Network with a traditional SIP Network

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 22 Apr 2008 05:00:00 +0000

There are considerable challenges in peering a P2P SIP Network with a traditional SIP Network and this goes back to why SBCs exist really for peering. For example, authentication, policy enforcement, billing, accounting and media path steering are few of the functions performed by the SBCs and most of these only work when SBCs are used for peering with other SBCs!

For example the SBC needs to authenticate calls coming from the P2P SIP Network at both the signaling and media level. Several methods may be used here: (i) A certificate mechanism may be employed between the P2P SIP endpoints and the SBC. However this may soon become unmanageable since the SBC will need to authenticate a lot of certificates on a lot of connections, each belonging to a unique P2P node. However it is within the realm of possibilities (ii) A SIP proxy may be used as a fixed peering point between the P2P SIP network and the SBC. The SBC can indirectly trust all media associated with calls exchanged with the trusted SIP proxy. This avoids having to aggregate media which can introduce unnecessary relaying.

Hung calls may also result in the system when the P2P SIP nodes dies or suddenly becomes unreachable. SIP Session timers must be used to detect these calls and the SBC must implement and enforce these timers.

All calls from the P2P SIP Network passed on to the SBC must also be authenticated and accounted since the SBC lies in a different administrative domain. What if the node is authentic (has the right certificate for example), but is not allowed to make calls since the bill has not been paid? It seems that the aggregate SIP Proxy is the best foot forward. This proxy will need to make sure the call is accounted for and the endpoint is authenticated in the P2P SIP domain itself.

Peering a P2P SIP Network and a traditional SIP Network

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 22 Apr 2008 04:55:00 +0000

Peering between SIP Networks has been the subject of discussion in the SPEERMINT group of the IETF. Most of the work done has been focused on peering traditional SIP Networks using fixed signaling and media peering points (SBCs) which act as policy enforcement points for signaling and media traffic respectively.

However, when you start looking at the problem of peering a P2P SIP network with a network having an SBC, a fresh set of problems surface. This kind of peering may be desired for connecting subscribers together or to provide PSTN access.

Also, both Infrastructure P2P SIP and Application P2P SIP are in scope when looking at this form of peering. See my earlier posts on this blog for the difference between the two.

Skinning P2P SIP

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 22 Apr 2008 04:43:00 +0000

Peer-to-peer telephony has always meant different things to different people. For example, the designers of SIP designed it to be peer-to-peer application level protocol where servers like the SIP Registrar and Proxy essentially provide the minimal roles of location and routing in a large SIP network. SIP Applications were designed to work end-to-end between two endpoints or between an endpoint and media server providing functions like conferencing and/or voicemail etc. On a different note, the P2P SIP working group in the IETF is focused entirely on making the location and routing functions in large SIP networks independent of centralized servers. P2P SIP can this be divided into “Infrastructure P2P SIP” which deals with location and routing issues and “Application P2P SIP” which deals with the complete P2P Implementation of SIP applications. In both cases the use of centralized and/or stateful servers is avoided. One of the main differences between infrastructure and application P2P SIP is the scale at which they are designed to operate. Application P2P SIP can operate at a much smaller scale of a few SIP nodes in the network while infrastructure SIP typically makes sense if there are at least a few thousand nodes in the network. Infrastructure SIP also tends to break down functions like NAT Traversal to make them more peer-to-peer while application P2P SIP does not really get into these aspects.

Performance comparison of LOTS with an Asterisk

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sat, 19 Apr 2008 03:21:00 +0000

Transnexus provides a lot of interesting white papers on their website on OpenSER and Asterisk performance and these are really helpful when it comes to evaluating LOTS. Consider the following piece of data:

Each GHz of Intel CPU running at 60% CPU utilization is worth 60 calls per second with an OpenSER and only 6 calls per second when it comes to Asterisk. This means a direct 10x hardware cost of Asterisk based system compared to LOTS!

For the actual white papers, refer to

The first perception of quality

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sat, 19 Apr 2008 03:04:00 +0000

Last month we spent a considerable time in debugging our VoIP network and proving out the feasibility of telephony grids and SIP trunks using the LOTS architecture I am working on. After hooking up a SIP trunk from several VoIP Providers and making sure call quality was great, several of our customers started complaining of bad service. The culprit turned out to be high PDD! Here is what was going on:

LOTS uses OpenSER at the core of the network. By default this proxy has a timeout of 200s between an initial INVITE and when the next hop responds with a 18X. This was killing our service quality. Most of the SIP Providers are not able to figure out a good number from a bad one, so they end up routing our calls to such numbers to several termination networks and the PDD becomes huge. Here is how we ended up resolving this:

1) Tune the OpenSER to have a much smaller timeout between the INVITE and 18x
2) Introduce a local progress indication in our applications which run on the PC which indicate to users connection is in progress. Once the 18X is received, an actual ring tone is available.

This really helps!

Getting Asterisk to work behind a firewall

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sun, 24 Feb 2008 00:34:00 +0000

One of the main things you will need to get to work is Asterisk behind a firewall for the LOTS architecture. Asterisk doesn't support STUN and instead relies on pinholes and firewall policies to be tweaked. Here is what you need to do:

1) Set the externip in sip.conf to the firewall's extenal IP address. This address is used by Asterisk in all its signaling messages which are directed outside the firewall. E.g externip=123.123.123.123

2) Set the localnet variable to indicate. This is perhaps the most important thing to do since it indicates to Asterisk what is private - which means it knows what is public from a NAT perspective. If you don't specify this, Asterisk will not apply the externip and will assume every address is private. Things may not work as you expect. E.g localnet=192.168.0.0/255.255.255.0

3) In the firewall, open pinholes which redirect all traffic on the firewall IP address for the rtpstart and rtpend ports defined in rtp.conf to Asterisk.

For more general info, you can look at this link but it may have more than what you are looking for, so stay focused :).

Asterisk and LOTS

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Sun, 24 Feb 2008 00:30:00 +0000

If you are wondering about Asterisk and LOTS, don't get confused. Asterisk has a place in LOTS - at the edge of the network. A starting application is Voicemail - which kicks in when a user (or a user group) is not reachable. LOTS doesn't allow endpoints to register with Asterisk and uses Asterisk as a gateway. We set up a SIP Peering connection between the OpenSER and Asterisk. What else is on the edge of the LOTS network? The answer is SBCs, Media Gateways, High Density conferencing servers etc.

What's the equivalent of the browser in LOTS?

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Fri, 01 Feb 2008 04:27:00 +0000

No one talks about the browser when saying LAMP is the building block of Web 2.0. Browsers have gone through limitless standardization starting from transmission protocols like HTTP/XML to others which determine rendering of the digital content like CSS, Javascript etc. Browser world is dominated by few very sophisticated players like Firefox, IE etc which differ mainly in how fast they run, what management features they provide etc. This is completely different from LOTS. In LOTS, the equivalent of the browser is a smart VoIP client. The only standard for VoIP which exists out there is SIP. Like HTTP, SIP can run between LOTS server and the VoIP client. However, SIP can also run between clients themselves since its a P2P protocol. The VoIP clients can be visualized as implementing a distributed Application layer between themselves which is very unlike the web world. The clients can also coordinate between themselves to implement other sophisticated VoIP functions as a GRID. This can get very interesting compared to what today's browsers can do.

The key thing to realize is that the LOTS server itself has none of the common VoIP Server functions and these if required are implemented on the VoIP clients.

NAT Traversal Servers for SIP

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 21:36:00 +0000

After a long time, I Googled out for open source and commercial standards based NAT Traversal servers which can be used to put together a solution for SIP based networks. The results are appalling... The industry is still in paralysis with there being more whitepapers on the topic than something substantial.

I only found the following solutions: Eyeball Networks, Estacadoand Paradial. I tried contacting Estacado and didnt hear back from them at all - which is always a bad sign. That leaves two.

Apart from these the following two options are available as well.

OpenSER Mediaproxy
SBC and firewall options

Other than these, I already posted two open source STUN servers in past posts on this blog.

Just got to know that Radvision also makes a STUN server, though you have to ask for it. Not sure what the intentions or focus is, but it seems to support the latest STUN bis specs.

Two Retail VoIP Carriers I would recommend

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 19:49:00 +0000

If you need to use your SIP softphone for some manual dialing and get good rates and good quality, try out FreeCall and VoipBuster. What do they offer?

1. Good Clean Standards based VoIP Connectivity using SIP
2. G.711, G.729 and iLBC codecs
3. RFC 2833 DTMF
4. Full NAT Traversal capabilities (no STUN required)
5. Good quality
6. Cheap rates with lots of free calling within U.S

How to publish an image link in your blogger template

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 19:34:00 +0000

Blogger provides some easy ways to post images inside the posts. However, when it comes to posting one as part of your template so its visible all the time on the left or right or top center bar, then you need to search around a bit. I did. To keep it short, just follow the following two links and you are done...

http://betabloggerfordummies.blogspot.com/2007/01/making-image-link.html
http://www.bloggerforum.com/modules/xfsection/article.php?articleid=11

How to use a third party SIP softphone with your Vonage Account

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 18:48:00 +0000

This is real easy. The Vonage server uses SIP in a very standards compliant way and makes use of SIP forking etc which allows multiple phones to register as well.

Here are some pointers to help:

1) The vonage SIP Proxy is basically 216.115.20.41 or sphone.vopr.vonage.net
2) You will need to get a userid and password for your softphone account from Vonage.

That's pretty much it. After that you can register any standards compliant SIP Softphone like SJPhone or a more sophisticated solution like WebAstra to work with your Vonage account!

mediaproxy and natproxy options for OpenSER

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 15:01:00 +0000

Checkout:

http://www.voip-info.org/wiki/view/OpenSER+And+RTPProxy
http://www.voip-info.org/wiki/view/OpenSER+And+Mediaproxy

Make sure to check out the documentation of these modules on OpenSER website.

Turning on Signaling NAT Traversal in OpenSER

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 14:56:00 +0000

loadmodule "nathelper.so"
modparam("nathelper", "rtpproxy_disable", 1)

if (uri==myself) {
force_rport();

if (method=="INVITE") {

if (nat_uac_test("7")) {
fix_nated_contact();
};
}

route(1);
};
route[1] {

t_on_reply("1");

# send it out now; use stateful forwarding as it works reliably
# even for UDP2TCP
if (!t_relay()) {
sl_reply_error();
};
exit;
}

onreply_route[1] {

if (nat_uac_test("1")) {
fix_nated_contact();
};
}

NAT Traversal choices for LOTS

medhavib@gmail.com (Medhavi Bhatia (DC Metro)) — Tue, 29 Jan 2008 14:44:00 +0000

YOu will face this problem when deploying LOTS since there are lots of viable alternatives for doing this. When thinking about getting SIP signaling seamlessly through all the involved NAT boxes, you need to break up the problem into signaling NAT Traversal and Media NAT traversal since both may be solved at different points of the network. Most people think there are multiple alternatives of doing NAT Traversal, but in my opinion the best architecture is the following:

1) Do Signaling NAT Traversal always at the first hop proxy. This would be the OpenSER.
2) Split the media NAT Traversal problem into two. If you have UA to UA traffic, then use a combination of STUN and ICE. If your UAs are behind the same NAT, then you definitely need ICE. Otherwise you may not need ICE. There are other complications with ICE and reasons to avoid it as I will explain in another entry. If you don't control the NAT box and are worried that symmetric NATs will exist, then you need ICE with a TURN server to solve the problem.
3) For PSTN Calls, do the media NAT Traversal at the termination gateway or SBC. Do not overload the STUN server with traversing pinholes for PSTN calls. There is a very good reason for this.

If 2-3 sound too complicated, then there are two alternatives in my order of preference:

I) Use the OpenSER box with media/rtpproxy modules. This will solve your media nat traversal problem
II) Use an SBC. An SBC can be quite expensive and may end up costing you more in bandwidth costs as well, but it can solve 1-3 at a single point in the network.