The Mobile World Congress was full of ideas about apps, lifestyle, mobile workflows and processes that can be enhanced by mobility.  I was stunned and amazed at how little promotion there was about Ethernet and Mobile Backhaul.

In 2009 the lack of adequate Mobile Backhaul implementations were all over the press in the US and UK.  One would think that the participants at the MWC would be promoting their latest mobile broadband solutions with fanfare.  It seems that the Elephant in the room will remain uncomfortably ignored.

This should be (is) the paramount issue in Mobile Networking today and equipment vendors should be investing resources to come up with solutions.  All of the entertainment, applications and business solutions promoting data usage are useless if the bandwidth end-to-end is not there to support them.

Fortunately I was in a position to engage multiple Mobile Carriers and Telecommunications Vendors at the MWC.  Obviously the paramount concern around bandwidth has been answered with Ethernet in the backhaul but the devil is in the details.  Throwing bandwidth at the problem has proven time and time again to be a only a band-aid.

From the discussions I had with operators and vendors there were a few common themes and/or concerns.  Here they are in summary.

The single most important point Mobile Carriers are concerned about today is ensuring visibility into the service layers of their network.  In other words, they want to measure, monitor, trend  and manage their networks as accurately as possible.  Currently TDM/ATM services are used for backhauling and there are tools inherent to these services which provide a level of visibility and comfort to Carriers.  With the move to Ethernet in the backhaul, mobile carriers are loathe to give up these capabilities.

In many cases these carriers do not own their own wireline backhaul infrastructure.  These carriers could just trust their leased infrastructure to carry the traffic according to the SLAs they have contracted, however we know in this day of “elastic bandwidth” via IP/MPLS that SLA’s are easily compromised.

I can summarize this with a quote from one carrier I spoke with at the show who said,

I ordered a 20 Mbit ethernet service with a maximum latency of 10 milliseconds in October 2009 all under SLA.  I measured it last week and I was getting 7Mbit with a latency of over 40 milliseconds.  What I thought was a dedicated service was actually an MPLS based service with no guarantees, so we invoked the SLA penalties.

Active Service Monitoring

Standards based Ethernet OAM is the answer most vendors and operators have for ensuring active continuous monitoring of the network.  These technologies can be implemented in a way that the operator would know of any issues in the network, most likely, before a customer notices any problems.  Such an implementation is necessary when actively monitoring a mobile network.

What I thought was a dedicated service was actually an MPLS based service with no guarantees, so we invoked the SLA penalties

However, the devil is in the details.  Ethernet based OAM requires a lot of processing horsepower when trying to actively monitor thousands of base-stations simultaneously.  Existing routers and switches in the network aren’t (yet) architected for such a function.  A number of carriers have proven, with catastrophic results, that their routers and switches aren’t able to handle such a load.

Furthermore and probably the most important point, Ethernet OAM only measures Ethernet.  All of the services running over these phones are either IP based today or will be IP based in the future, even voice.  Carriers I have spoken with have expressed a wish to monitor the IP and IP service layers as much, if not more, than the transport layer.

Performance Management

One-way measurements is at the top of the list of Carriers wishes for service monitoring over Ethernet based networks.  While measurements on latency, litter and delay are all necessary Ethernet currently only measures round-trip performance.  This limitation is one of the main reasons why mobile backhaul has taken so long to accept Ethernet as a transport technology.

In order to provide one-way measurements there needs to be some way of handling Ethernet in a synchronous-like fashion.  Since Ethernet, by definition, is an asynchronous technology, this is no easy feat.

Standards bodies are reviewing all of the potential methods to make synchronous-like behaviors possible in Ethernet (1588v2, SyncEth) but we’re at least 1 year away from a real solution.  That doesn’t address the fact that the millions of installed Ethernet ports in carriers networks worldwide lack the hardware to take advantage of any new technological advancements.  I am not so sure that 1588 v2 or Synchronous Ethernet, in their current designs, will ever see mass adoption.

A number of carriers have proven, with catastrophic results, that their routers and switches aren’t able to handle such a load

Another equally important aspect of performance management is to use the measurements for trending purposes.  Many issues in a network aren’t visible right away but rather creep-up over time.  The ability to save measurements over-time and then apply policies and rulesets for interpretation can yield another perspective of the health of the network.  This is especially important when the transport of the backhaul network is heterogeneous to highlight any long-term degradations that can be masked by digital services.

Traffic Condiditoning

When a service traverses multiple transport/physical technologies a number of issues can (and will) arise.  These issues can manifest themselves as increased latency, jitter or re-transmission in the case of IP/data services.  These issues cause degradation of service, inefficient bandwidth usage and frame-loss.  One should also remember that there are differences between Fast Ethernet and Gigabit Ethernet that will give headaches if not allowed for.

These issues aren’t  exclusive to media or protocol conversion but can also be traced to inadequately engineered hardware.  Some base station vendors (who shall remain nameless) might be good at building radios but provide less than acceptable ethernet interfaces for backhaul.  Then we must take into account that many mobile carriers around the world have many types and brands of base stations in operation.

Carriers are now asking for backhaul solutions that can condition the traffic between the wireless and wireline networks (i.e. backhaul) to ensure that none of the aforementioned artifacts (jitter, delay, disruption) are minimized, if not eliminated altogether.  Conditioning of traffic cannot be limited to the transport or Ethernet layer.  The IP and IP service layers must also be conditioned to transit the network as efficiently as possible.

In Conclusion

The MWC was a fantastic opportunity to talk with a number of carriers and vendors who all know that there is an issue but I have yet to see anyone who has a complete end-to-end solution.  Some vendors have great radios, some vendors monitor excellently, and some vendors offer a cost efficient way of backhauling.  As has happened many times before a very clever network engineer somewhere around the world will come up with a way to solve these problems elegantly without going way over budget and ensuring a consistently good revenue stream.

Are you that engineer ?

I’m currently in Barcelona at the Mobile World Congress.  I didn’t expect this show to be very well attended this year and on Monday I was beginning to think it might be a waste of time to be here. Tuesday and Wednesday proved me wrong and I’ve been pleasantly surprised.  The amount of foot traffic, even in the smaller halls, is significant.

The buzz this year has been around mobile applications.  The amount of people in this space is mind boggling.  How applications will affect our lifestyle was apparent on every 2nd stand.  One theme I saw repeated over and over again was “How you can be a journalist armed only with an iPhone”.

This explains the dearth of updates recently as preparing and working a show such as this takes a lot of time.

It’s now time to break down the Accedian stand and try to find a beer so I’m cutting this post short.  I hope to have a larger post covering other aspects of this year’s show done by this weekend.

The first step in overcoming a problem is to admit it. Deutsche Telekom (DT) did just that earlier this week. DT is taking strategic steps to stay relevant when faced with the erosion of their core business. A Tier 1 Carrier in Europe finally woke up and smelled the coffee.

In an internal meeting, Deutsche Telekom CEO Rene Obermann outlined a new long-term strategy for the telecoms giant. The message, that went out to over 1,000 of DT’s top and middle managers, said that the DT revenue breakdown must reflect a focus on data services.

Telecommunications technology trends have all but relegated voice to being “just another data service” for almost a decade. Skype, in particular, has shown that people are willing to give up a little bit of quality if the price is right. DT is now responding to the impact of free voice calls.

A Tier 1 Carrier in Europe finally woke up and smelled the coffee

Telecommunications companies now face a choice; Become a wholesaler of large bandwidths to large enterprises and retail carriers, or move up the value chain with software solutions that strengthen relationships to the Enterprise and Consumer markets.

Mr. Obermann’s five-year strategy for DT focuses on developing applications that leverage their core telecommunications assets. Mobile applications in the consumer market and SaaS solutions to enterprise verticals are the new focus. Specifically the Electricity and Healthcare industries were mentioned.

This 5 year strategy will no doubt mean a re-alignment of resources. A cost cutting program will be put in place although no specifics were discussed. Encouraging however, was a pledge to invest over €1 Billion into infrastructure and development.

It’s been described as the Nirvana for IP based communications. The ability to harmonize the IP and the analog transport layers has been a wish for Carriers’ IP departments for quite awhile. Cisco Marketing even pushed the phrase “IP over DWDM” for so long that we’re sick of hearing about it. The hurdle is that IP over DWDM stories from router vendors leave out one important detail; The DWDM vendors must also agree to the methods and technologies to realize this goal. The problem is, IP router vendors and DWDM vendors do not speak to each other.

I had dinner with executives from a company last night that claims to have bridged this gap. Taseon is the name of the company and their technology marks the sharpest paradigm shift in Telecomms in a decade. Taseon has developed a system that intelligently integrates lambda transport between routers and switches with digital wavelength management visibility. This is achieved by accepting and carrying wavelengths originated from routers and switches and ensuring OTN management between the network elements.  So far Taseon has successfully interoperated with various Cisco network elements to ensure the solution is valid.  I hope Juniper and Alcatel will also work with them to ensure interoperability.

The picture below, taken from their website outlines the technology better than my words.

The idea behind the Taseon technology is to migrate the transponder to the switch or router. This eliminates the need for “grey interfaces” which are the discrete interfaces between DWDM systems and switches/routers. The wavelengths and the control plane are extended to the switch/router to provide a single-optical-hop between datacomms gear.  Eliminating the need for “grey interfaces” and the ability to have full digital wavelength management end-to-end is a very compelling story from both a CAPEX and OPEX perspective.

IP router vendors and DWDM vendors do not speak to each other

When I was building IP networks (sooooo many years ago) a number of IP engineers conceived an idea of aligning the IP topology with the wavelength topology. It would save us operational headaches, capacity uncertanties and perhaps assist in fast re-convergence of the network. In the meantime the router and DWDM vendors never spoke with each other to find a solution.

Having the technology is only half of the battle. A few years ago the IP and the ATM wars ended. Todays battle is between the IP and the transport departments there is no clear winner (yet). Now Taseon has the challenge to position their solution to not appear threatening to either the IP or the Transport departments in the Carriers they hope to help.

From what I saw last night I think they’re on the right track and I hope to see more of them in the future.

Classically the Germans have been masters at engineering.  You can see it in the products from audio technology (Bose, Klipsch, Blaupunkt) to automobiles (Porsche, BMW, Mercedes) to Electronics (Siemens, Bosch, Infineon).  Recently a number of issues have caused me to doubt the Teutonic “Engineering Edge”.  The coldest winter in memory could be exacerbating the problem.

Internet Connectivity

Starting about 3 weeks ago my DSL connection intermittently disconnects, for no reason.  Sometimes it comes back in a few minutes, sometimes it takes significantly longer and sometimes it will work uninterrupted for 15-18 hours.  Real-time communications like Skype are almost impossible to use.  This is the reason I haven’t been blogging much recently.

At first, I thought it was an issue with my equipment.  Being a techie at heart I have a complex home network.  Naturally I thought it was an issue with my gear.  After a week of testing, rebooting and reconfiguring I decided to call Deutsche Telekom and report sub-par service.  DT decided that a DSL line measurement was necessary and was done later that day.  Unfortunately a DT line test lasts about 30 seconds and the line didn’t drop during that time.  Their response: “Everything looks good on our end, check your home network.”

…that makes 5 weeks without adequate Internet service….

After a few more calls and cajoling DT agreed to do a long-term test.  Of course they did it at 10:30am for 30 minutes….right in the middle of the workday.  They found some errors and they now feel the problem is with them.  So we scheduled a time when they can come to my office and measure the connection both ways.  They’re coming next Friday so these Internet issues will last another 2 weeks….that makes 5 weeks without adequate Internet service.  I’m sure I’ll get a bill for the full amount.

The Rail System

For my trip I booked a train online (when my DSL was working) to Hannover.  The train was scheduled to leave Tuesday at 05:50 so I could make my 09:00 appointment with plenty of time.  When I arrived at the train station I saw that the train had been cancelled.  I had no choice but to wait for the next train in -10 degrees Celsius weather.  Then 5 minutes before the next train was to arrive that the 06:50 train had also been cancelled.  Luckily, Deutsche Bahn (German Rail) provided a substitute train.

The substitute train arrived 45 minutes late, the passengers boarded and settled in.  The train never made it out of the station, 20 minutes later we were told to disembark.  The substitute train was subsequently towed from the platform.  So much for being in Hannover at 9am.  Even Berlin reported 10’s of Millions of Euros in lost productivity due to “Snow Chaos”

The train never made it out of the station, 20 minutes later we were told to disembark

Dysfunctional train systems have been an issue all over Europe this Winter.  The Eurotunnel trains have stopped in mid-tunnel.  The French TGV has also reported trains stopping in mid-tunnel.  So far it seems that the cold temperatures are causing condensation and the water drips live circuit boards causing shorts. 

The Russians have a solution that comes from Soviet times. They dipped their circuit boards in laquer. Sometimes low-tech is the way to go.

Apple announced the iPad today and it’s a very compelling product.  With a 10 inch touch screen, 10 hours of battery life (1 month of standby!), all 140,000 apps on the App Store available and other tech specs to make the geek in you drool.  Gaming with your buddies, downloading publications through the new iBook store, watching Youtube at HD are all possible with this device.  They are all activities that use boatloads of bandwidth, exactly what the overtaxed 3G networks of today need.

Who has Apple chosen to partner up with for this device?  Well AT&T of course, haven’t they shown just how well their network handles user who use their mobile devices as intended?  To top it all off the iPad comes with an “All you can eat” data plan for $29.95.  I don’t understand how a company can, in good faith, make flat-rate offers on a network that cannot handle the traffic.

Don’t get me wrong, I think the iPad is a fantastic device.  I think it can revolutionize everything from books to being a remote control for your on-line video library.  However, this thing will live (and die) by the customer’s experience.  If the network behind isn’t able to deliver then Apple might end up with a problem they can’t solve.

Towards the end of the announcement Apple said that the device will be unlocked for use on any 3G network.  Maybe now other mobile networks will see how much pain this device can cause.  I’d like to own a vendor that makes high-bandwidth mobile backhaul equipment .

Every year for the past 11 years Lambdanet hosts the Lambdanet Forum where Carriers and Vendors exchange ideas, present forward-looking concepts, business strategies and cutting edge technologies.  In its infancy this was a casual get-together, but now the Forum is a German institution.  Today I braved the -11 Celsius temperature, hopped on a train and took part in the seminar.

The unintended overall theme of the Forum was the hard decisions facing Carriers today.  Bandwidth usage is exploding.  That bandwidth forces carriers to invest but the business models are broken.  The content driving the bandwidth usage is high-value to everyone but the Carriers.  The Carriers are  expected to shoulder the network investment and lose out on monetizing the high-margin services.  This forces an optimization in the network, technically and commercially.  But, where to start?

One thing is clear, Carriers cannot continue with the business models refined in the last decade.  There are a number of paths to choose and they all necessitate a radical change in strategy.  We have known this for years.  One of today’s presentations given by a carrier discussed the situation in detail and discussed possible models for success.  More on this in a later post

A discussion about the “Open Access Network Model” generated a lot of buzz and murmur in the audience.  Advocates of the model such as the Board Member of Lambdanet on one side and detractors such as the Vice President of the German regulatory authority on the other.  There was spirited debate ignited by pointed questions out of the audience.

Each of these topics will be discussed, in depth, in further posts this and next week.

Technological advancement breeds patent infringement cases, it seems.  Every week seems to bring a new case of companies suing each other over the use of patented technology.  In some cases, I’m wondering if it’s worth the battle. 

I’m no legal expert but it seems to take years to bring a patent infringement suit to court.  That’s not counting how long a judge or jury might take to rule on the case.  Some of these companies are so large they can just throw lawyers at the problem and prolongate the battle.  I could imagine a case like the current Apple vs. Nokia battles taking many many years to resolve.

Most of the patent infringement cases I’ve seen lately are software related.  They usually cover architecture or core components of software. It is the nature of software to be updated, relatively frequently.  Even such fundamental parts like architecture can and are updated albeit less frequently.

When a suit comes to court for software patent infringement has the software updated itself to no longer be covered by the patent?  Has the patent had it’s “time in the sun” and is now obsolete?  I wonder if the process by which patent holders can secure their royalties by someone else monetizing their patent is being obviated by the speed of technological development.

If nothing else it keeps the lawyers and journalists busy.

The solutions and boxes you sell to service providers are the absolute best, right?.  It’s the best cost-per-port, fastest, densest, most feature-rich, extensible, non-forklift upgradable, plug-and-play communications gear on the planet, right?

Good…now prove it!

I’m not talking about demonstrating how well the equipment performs in the network or even putting it through a series of tests that aren’t realistic (but prove every corner-case and keep engineers employed).  Any solution offered must not only perform well but also prove how well it has performed by reporting, in real-time, the overall health of the network.

Carriers have been offering “Service Level Agreements” (SLA for short) for a number of years.  When SLAs were first offered they were worded in a way that made difficult any attempt to prove sub-par service by the Service Provider.  With fierce competition for enterprise business they have evolved.  SLAs now have “teeth” in the form of substancial financial penalties for under-performance.

This financial risk has forced carriers to adopt a new factor in deciding which equipment they install in their network.  That factor is the ability to report events that affect the overall health of the network.  The reports must be in a protocol that is compatible with the OSS/BSS systems the carrier has chosen.  That last bit is key; “compatible with the OSS/BSS systems”.

SLAs now have “teeth” in the form of substantial financial penalties

The trend in European Carriers is to implement a new functional group within the OSS/BSS department that is focused solely on SLA reporting.  Carriers and Service Providers not only have to provide exceptional service to their enterprise customers (bound by SLAs) but they also have to prove it.  This proof comes from the reports generated by network elements, correlated with reports from other network elements and then rationalized against metrics set by carriers’ business processes.  Reporting the performance of the network is the primary function of this new group.  Now it’s the carriers turn to prove it!

For network equipment providers this impacts how carriers are engaged and with whom they cultivate relationships.  OSS/BSS  now have more influence over which gear is chosen for the network and represent a new set of gatekeepers whose function must be respected by vendors.  As one Director of a major European Telecommunications company said to me a few days ago,

“We have become an OSS/BSS company that happens to have 1000’s of kilometers of fiber in the ground”

As I’m writing this I can imagine how vendors would argue my point earlier about OSS/BSS reporting compatibility.  One argument would center on standardization and how it obviates the need to care about compatibility.  Unfortunately standardization isn’t always adopted in complex systems.

Another argument would take the opposite tact and place all the value in an integrated solution. Integration into OSS/BSS systems, with an incompatible protocol, requires an enormous amount of man-hours at a commensurate cost.  This approach has another, more serious, impact on a product’s time to market.  If the carrier is busy integrating the product into the network, they’re not selling any services based on it.

In my experience, there are two ways around this issue;

First, become the vendor that dictates, through incumbency, how the reporting/management interfaces work with the OSS/BSS systems.  Here is where Alcatel-Lucent has done a fantastic job with their SAM Network Management Product (although it is long in the tooth and running out of steam)

Second, lessen the pain and therefore the barrier to entry of your product.  Offer to pay for the integration from a third party or install engineers at no cost to accelerate the integration.  In other words “Put up or shut up”.

It’s one week into the introduction of the Nexus One and this time Google might have bitten off more than they can chew. The hype about how the handset would challenge the iPhone for smartphone supremacy have now given way to angry customers. End-user customer service is not something the Internet giant has tried to do before, the inexperience is showing.

Google’s problem is threefold;

First, if a customer chooses to purchase the phone directly for $529 then all support and service communications are done through email and web. These communications channels are second nature to Google but not adequate for the level of service mobile customers expect these days. Google is promising a 48 hour turnaround to all support requests in relation to the Nexus One; yep, that’s right 2 Days without service in case of a critical failure.

Second, if a customer agrees to a 2 year contract with a carrier then the phone is $179 and the service is handled by the Carrier. In the mass market there just isn’t enough room in the monthly subscriber fees to cover the level of customer support that is currently needed. The introduction of a complex device like the Nexus One will (if it hasn’t already) cause more headaches.

Third, No company, not even Google, is taking ownership of communication with the customer in case of hardware failures.  Reports of customers being bounced around between the Carrier, Google and the manufacturer of the device (HTC) are all over the message boards.

Google can fix this but they have to fix it fast in order to stay in this game.