• It collects garbage at seemingly random times
• It’s a multithreaded system, and your thread could be preempted
• I/O appears nondeterministic

For these reasons, Java is rarely used for true real-time systems.

Read Consistency

Another issue is read consistency. In distributed systems only two things can be moved around: state data information and the functions that process it. When you move state data in a distributed system, whether with caching or some kind of synchronous transaction system, you have two copies of the data, at least temporarily, that must stay in sync.

Durability

When you create a system that must be continuously available, it requires some form of durability. For example, you must be able to recover servers, safeguard data from loss, and so on. Durability presents a couple of challenges. First, there is a latency cost to keeping data on a disk network. Second, you have to coordinate between readers and writers.

Architectural Limits to Scalability

High-latency operations that must occur in sequence limit scalability. Globally ordered operations limit scalability. Data hotspots limit scalability. In short, if there is some point in your system where data has to line up for processing, you will have a scalability bottleneck.

In stateful systems, the challenge is to achieve availability, reliability, performance, and predictability -- including predictable performance -- in systems that can still be easily managed and serviced.

There are costs to meeting these challenges. The goal is a system that scales linearly as you add resources and capacity. In other words, each additional server you add will support at least the same number of users as an existing server.

The Solution

The collective solution is composed from some or all of these approaches: routing, partitioning, replication for availability, coordination across multiple partitions, and messaging for asynchronous updates across partitions.

Routing

In a stateful system, the purpose of routing is to make sure you bring together processing and data at the right time, in the right place. Routing moves information to servers where there is locality of state. Locality of state lets processing occur without coordination across the network.

Routing lets you take advantage of additional servers by spreading the incoming load. Routing supports partitioning, the ability to break down processing across multiple servers.

Partitioning

Partitioning designates ownership of state and responsibilities across a distributed system. Because tasks are partitioned, a single-point failure will only affect a portion of the system. In a best-case scenario, through replication, you can eliminate the loss of any information. So, partitioning makes replication possible.

Replication for Availability

But once you have partitioning, you have a big problem, because it poses a limit to scale-out. Once you start pulling information from multiple servers and pushing information to them, you have issues of transaction that involve possible data loss and inaccuracy. Once you start coordinating across partitions and servers, things become much more complicated. Enter the issue of coordination.

Coordination Across Multiple Partitions

Coordination is what makes scalability difficult for stateful systems. But good coordination simplifies the programming and supports more synchronous behavior. The alternative to using coordination is to use messaging. So, instead of coordinating synchronous operations across multiple partitions, you can use messaging to make the processing stay synchronous.

Messaging for Asynchronous Updates Across Partitions

Messaging queues let you reliably hand off responsibilities between partitions. If one partition needs to initiate processing in another partition, it can send a high priority message. However, actually waiting for confirmation that the processing has been completed would be coordination.

Message queues are durable because they make message information survivable. If the sending partition dies, the message will not be lost, and the receiving partition will process as requested.

Message queues support ordering of operations. Sometimes it’s important to perform operations in sequence -- a challenge in multithreaded systems.

Queuing has one more important aspect that supports scalability. It lets you batch operations together.

Messaging Technologies

There are many messaging options, with varying reliability, latency, and ordering guarantees.

• UDP/IP: A low level protocol, but with low reliability -- there is no guarantee that messages will actually arrive, since they can be dropped without raising an error condition. Good for predictable latency, however, since messages are not stored.

• TCP/IP: More reliable than UDP, since the protocol establishes a two-way stream-oriented conversation. TCP/IP also has an easier programming model than UDP. One disadvantage is that latency depends on network load, and is unpredictable for practical purposes.

• HTTP: Has the advantage of simplicity because it is a slightly higher level protocol. Its request-response model makes it fairly reliable.

Conclusion

In stateful systems, the challenge is to achieve availability, reliability, performance, predictability, including predictable performance, in systems that can still provide manageability and serviceability.

The collective solution is composed from some or all of these: routing, partitioning, replication for availability, coordination across multiple partitions, and messaging for asynchronous updates across partitions.

And, keep it simple. If you can’t show how your deployment works on a white board, it probably won’t work in production.

Learn more about scale-out deployments on developers.sun.com. 

2008 JavaOne
 
TS-5606
Session Title:  Flooring the Accelerator: How Open Source Is Reshaping an Industry
 
Janice J. Heiss

2008 JavaOne

Stephen Green's blog

Paul Lamere's blog
 
Janice J. Heiss

I explained the sessions that I went to:

• Java University
• Applets Reloaded
• Mars Mapping

I want to share a few URLs for the topics I covered, mainly the new Java Plug-in, and the Mapping Mars project article that I wrote:

• Mapping Mars
  
• New Java Plug-in

I also want to thank everyone in-world for their continued interest and support, and for being so understanding about my having to leave early due to my illness. I am on the mend now, and will pop in-world again soon.

Dana Nourie, Sun Microsystems

Sun's Simon Phipps, chief open source officer, and Patrick Finch, open-source community coordinator, led an almost packed JavaOne session yesterday.  They delved into the history, current status, and future challenges of open source, punctuated by many beautiful photos from Simon to, often humorously, convey the points.  Great job!

History
Open source started in the early 1980s with pioneers like Richard Stallman, who believed that software should come with freedom to revise the code and who launched the GNU Project.  Another notable figure was Bill Joy, a cofounder of Sun, who "wanted to freely collaborate with people" on software.  In fact, the philosophy behind the BSD license for UNIX is easy collaboration.

Over the years, Sun has become involved in numerous open-source communities: OpenOffice.org, Project GlassFish, OpenSPARC, NetBeans, Java technology.  "There have been many opportunities to make mistakes," said Simon.  "Nonetheless, we've done a lot of excellent work."

Toward the end of the 1990s, FOSS [free and open-source software] continued to evolve into an important trend, leading to the publication of open-source license standards in 1998 for a total of 60 licenses. Inevitably, picking the right one involves legal experts.

The Present
Now that the Internet has become pervasive, examples of open source abound, but also controversies, because "open source means different things to different people."  Still, the movement is spreading globally and changing the way of doing business, with development centers in China, India, and South America.  Traditionally conservative Gartner predicted a year ago that by 2012, 70 percent of all software will be based on open source.  That percentage now reads 90!

Who are the open-source developers?  They include those who work on kernels; who build plug-ins and device drivers; who create enterprise applications; and who integrate systems.  Be sure to count the user community, too.  Simon cited Apache as a shining example of "a democratic, highly organized community of diverse interests with well-stated governance models."

The "Adoption-Led Market"
The traditional "procurement-driven" market, whereby software projects go through the select-evaluate-build-deploy-support cycle, gated by a licensing barrier, is switching more and more to a different model. That is, project teams select from what's available in open source, evaluate, and then prototype and build a solution fast, leveraging self-support from communities with little or no vendor contact.  Once the solution stabilizes, deployment follows, leading to demands for support.

"Enterprises want insurance for software; software companies sell insurance in the form of support and bug fixes.  Users then become customers," explained Simon.  See his posting for details.

Consequently, IT managers must decide whether to hire support personnel; purchase subscriptions for support, training, warranty, and indemnity; or just risk it out.  No matter what the choice, IT now has more control over on what and when to spend its budget.

The Challenges of Licensing, Patents, and Trademarks
In quoting Eben Moglen, "A license is the constitution for the community," Simon advocated only a limited number of licenses because "license proliferation limits the ability of software projects to be  embraced by communities."  Read his white paper (PDF) on this topic.

Simon believes that as much as that software patents are a fact of life and that companies will continue to encourage employees to file patents, the entire subject is "very broken."  Currently, enterprises tackle patents through grants, covenants, and licenses, with licenses being the best approach, according to Simon, who proposes that an independent body look into the ramifications of the options.

Trademark violations are all over, for example, the Mozilla and Firefox logos.  Again, might an independent group be the answer?

What Next?
With corporations "getting up close and personal," Simon believes that management should continue to send developers to conferences like JavaOne and other collaborator and community events.  As for open source, "Jump in!" "The future is ours to shape.  The pivotal moment in IT is open source, which is definitely heading for mainstream," concluded Simon.

The Mink Connection, "Simon Phipps's view of the Web," is chock full of gems on open source.  Amazing photographs, too!  Have a look.

If you use one of the scripting languages JRuby or Groovy, you’re unlikely to use the other, and are probably curious about their relative advantages. Most of us don’t have broad expertise in both languages, so the session presented by Neal Ford, "Comparing JRuby and Groovy," TS-6050 provided valuable insights.

Ford characterized both languages as dynamic and strongly typed, but noted that they were created for different purposes. Groovy has been designed to provide a modern dynamic syntax for generating Java bytecode.  JRuby is a Java platform port of the Ruby language. Ruby already runs on all of the popular operating systems, and JRuby tries to be no more than Ruby running on the JVM.

Because of their different geneses, the two languages have different priorities. Groovy was designed to bring modern semantics to the Java bytecode, and JRuby was designed to bring the richness of the Ruby language to the Java platform.

Besides differences in syntax, Ford described many other differences between the two languages that affect the way programmers approach solving problems in each. For example, each language handles the concept of null differently. In the Groovy world, null is always null. Groovy adds a "?." operator to help avoid null pointer exceptions. For example, the expression

person?.address?.street

tests person and address for nullness, and will only execute the line of code if both person and address are not null.

In JRuby, nil is an instance of NilClass. Nil is an object because, in Ruby, everything is an object. And all objects have consistent semantics. No special operator is required to handle null or nil. Becuase nil is an object, you can call methods on it. Ford then presented the following lines of JRuby code, which define a blank? method and add it to the String class.

class String
  def blank?
     empty? || strip.empty?
  end
end

JRuby supports open classes, and when it looks on its class path, it sees that a String class is already there. The code reopens that String class and adds the blank? method to it -- in Ruby, the convention is to append the question mark character to the name of a method that returns true or false. With the following JRuby code, he then used the blank? method to create a test_blank method:

def test_blank
  assert "".blank?
  assert " ".blank?
  assert nil.to_s.blank?
  assert ! "x:.blank?
end

Here, the test_blank method operates on the String object itself, and if it is empty --  or if you strip all the spaces away and it is empty -- then the String instance must be blank.

Interestingly, if you test this code you can also assert that nil.to_s.blank? is blank. Nil is an object in Ruby, so you can call methods on it, including to_s (convert to String). And, as it turns out, nil.to_s is empty -- which is exactly what you would expect.

Ford presented many more intriguing examples of differences between the two languages that affect the way a programmer approaches the solution to a problem. It is worth a look at the transcript in Java online when it becomes available. Meanwhile, Neal Ford has made his presentation slides available on his website at nealford.com.

For more information on scripting languages, see the technical articles and tips on java.sun.com.

One of the powerful features added to the NetBeans IDE 6 was support for Ruby, JRuby, and Ruby on Rails. The Ruby programming language has become popular with a growing number of developers because of its simplicity and its productivity features. As is the case for the Java programming language, Ruby is object-oriented, although in Ruby everything is an object -- even what are called primitives in the Java language. Ruby is also open-source and has a large and active community. Ruby's creator, Yukihiro Matsumoto, known to the Ruby world as "Matz", intended the language not only to be easy to use and highly productive, but also fun.

JRuby is an open-source pure Java implementation of the Ruby programming language. Because of its tight integration with the Java platform, JRuby allows Java applications to call into Ruby code. It also allows Ruby applications to call into Java code. This Ruby-Java integration is a very handy feature for developers.

Ruby on Rails, often shortened to Rails, is a web application framework written in the Ruby programming language. The framework is designed to make web application development faster, simpler, and more efficient. Web applications built with Rails are based on the model-view-controller (MVC) architecture. Devotees claim (rightfully so) that using Rails enables them to create a working web application in minutes. Like Ruby, Rails has become very popular. It's also open source and has a large community of contributors.

With this added support for Ruby, JRuby, and Rails, developers can take advantage of the NetBeans IDE's sophisticated project management, editing, and debugging features in building their Rails applications.

In the technical session titled "The NetBeans Ruby IDE: You Thought Rails Development Was Fun Before," Sun Principal Engineer Tor Norbye and Sun senior engineer Brian Leonard, both NetBeans experts, used the NetBeans IDE to quickly build a Ruby on Rails application -- a simple web application that tallies votes for presidential candidates. In the process, they demonstrated many of the features in the IDE that make it an attractive tool for Rails development.

Norbye and Leonard presented far too many features during the session for me to cover in this short article, so I'll highlight just a few of them.

• Project management. NetBeans allows developers to create a Rails project from existing files or to create a project "from scratch." In addition, developers can specify a specific Ruby interprter when they create a project, for example, the native Ruby interpreter or the JRuby interpreter. The interpreter can also be changed during the development process.


• Code generators. One of the productivity features in Rails is the use of code generators, saving developers from doing a lot of boilerplate coding. NetBeans provides all the standard code generators that come with Rails such as scaffolds, which construct most of the models and views for a basic MVC-based Rails application, models, controllers, database migrations, and integration tests. Additional generators can be easily installed into NetBeans.


• Code Completion. Code completion is available in the NetBeans editor for all Rails APIs. It's also available for database parameter completion in ActiveRecord calls, as well as for hash keys and hash values.


• Editing in Erb Files. Erb files are the Rails version of JavaServer Pages (JSP) technology files, allowing developers to embed executable Ruby code within the HTML code on the page. An embedded block of Ruby code is called a scriptlet. All the NetBeans editing features available to Ruby are also available to scriptlets. In addition, the editing features extend to other code in an Erb file such as HTML, JavaScript, or CSS.


• Enhanced Documentation. RDoc is the Ruby equivalent to JavaDoc, that is, formal documentation of the language. The NetBeans editor does custom RDoc rendering. A nice aspect of this is that the editor preformats code fragments and highlights syntax in examples within RDoc entries. The editor also does this for Erb files.


• Quickfixes. The editor checks for many common programming mistakes, providing hints for fixes. It also identifies deprecated Rails constructs and suggests replacements.


• Source navigation. NetBeans has some nice shortcuts for Rails application editing. For example, press Ctrl-click on a class, method, field, or variable to jump to its declaration. Press Ctrl-Shift-T to jump between a file and its test. Press Ctrl-Shift-A to jump between a controller and its view.


• Context-sensitive execution. Selecting Run File (Shift-F6) performs a context-sensitive action for the current file. For example, if the context is a controller, view, or helper, selecting Run File opens the browser and shows or reloads the URL corresponding to the controller or view (starting the Rails server if necessary). By comparison, if the context is a test, selecting Run File runs the test.


• Advanced debugging. NetBeans supports a comprehensive set of debugging tasks for Rails developers such as stepping through code, setting breakpoints, inspecting local variables, navigating call stacks, and running tests. It also enables debugging of both template files as well as Ruby files.


• Java integration. NetBeans provides code templates that makes it easy to include Java modules into a Rails application. In addition, Java classes can be used just like Ruby classes. Leonard mentioned that the JRuby engineers in Sun went to great lengths to make working with Java code in Ruby more Ruby like.


• Flexible deployment. NetBeans supports deploying a Rails application to any Java servlet container such as Tomcat or GlassFish. Leonard pointed out that most deployment environments do not include Ruby containers. So this feature enables developers to deploy their Rails applications in their existing infrastructure.

Unfortunately, Leonard could not get the voting application to run. It's likely that during the talk, Norbye or Leonard added some errant code to the working code that was originally folded into the project. However, this gave Leonard a chance to show more debugging features.

I was impressed with the comprehensive set of Rails development features already in NetBeans, but enhancements are coming. Norbye said that future plans include more extensive type inference, enhanced test support, better console integration, more refactorings and quickfixes, improved Java integration, and support for additional Ruby frameworks.

Despite the application execution problems, this was an impressive display of Rails support in NetBeans.

Late afternoon this past Monday, Sun identity architect Pat Patterson, senior product line manager Daniel Raskin, and product manager Nick Wooler took the podium at CommunityOne on OpenSSO.  In particular, they described the runtime scenario; explained Fedlets, a shining capability in the upcoming Sun Federated Access Manager, OpenSSO's twin; and demo'd the process of deploying applications and configuring OpenSSO for federation.

It was fascinating to learn about the latest model of federation adopted by many enterprises: outsourcing of employee email and calendars to Google, for example, and the attendant needs for authentication across company borders.  The savvy speakers had the details down pat and took many Q&As at and after the session.  I wondered if Pat ran out of business cards?

Here's a recap of the session.  AMIS Technology also posted a review, complete with a screen shot.

In kicking off this year's session titled "Java-Centricity: Java Technology at the Hub of Your Digital Life," Sun Distinguished Engineer and chief technology officer for software, Bob Brewin, focused on rich internet application (RIAs). He noted that RIAs are evolving. "RIAs are typically associated with things that are in a browser only. And if you take a look at where we're going, the future [of RIAs] is about every device connected to the network." Brewin went on to say that not only are RIAs changing, but the rich Internet -- the expanding collection of connected devices, such as Playstation 3s, TVs, MP3 players, mobile devices, and in-dash navigation systems -- are changing too. All of these devices are starting to use the same protocols to talk to the network as well as to each other.

Noting the central role Java technology has played in RIAs, Brewin said that Java has been there all along. However, there's been a gap. "While Java is the ideal deployment platform, it hasn't been ideal for content authors and designers." Enter JavaFX technology, a development environment, runtime, and tool set based on Java technology, targeted to meet the needs of content creators and designers.

To illustrate the power of JavaFX, this session showed off a number of cool JavaFX-powered demos. Of course, the foundation of JavaFX is Java itself, so Brewin first handed the microphone over to Roberto Chinnici, the platform lead for Java EE, to highlight what's new in that platform. Following Chinnici, Danny Coward, chief architect of Sun's Client Software, came onstage to highlight what's new in the Java SE platform. The last segment of the session highlighted several cool JavaFX-based demos.

What's New in Java EE?

Chinnici started his tour of the Java EE platform by looking at the major objective of the platform's current release, Java EE 5: ease of development. Features such as annotations dramatically simplified things for developers by reducing (and often eliminating) the need to code deployment descriptors. Java EE 5 has been very successful. Chinnici said that the Java EE 5 SDK has been downloaded over 3 million times either standalone or in conjunction with the NetBeans IDE.

The good news is that the simplification theme will continue in the next release of the platform, Java EE 6, with features such as support for RESTful web services and a more extensive use of annotations across all the web APIs. Java EE 6 will also focus on rightsizing and flexibility.

Rightsizing

By rightsizing, Chinnici meant reducing the ever-increasing size of the Java EE platform to something more manageable. This involves the introduction of profiles, that is, various types of Java EE technology packages, each one designed for a specific use. For example, a web profile would include Java EE services targeted to web developers. A TELCO profile would include services designed to meet the needs of the telecommunications industry. In addition, rightsizing will involve pruning of some technologies, such as JAX-RPC or EJB 2.1, that have been supplanted by newer technologies.

Flexibility

Java EE 6 will also be a lot more flexible by adding extensibility points, making it easier to incorporate the large number of available open-source frameworks and libraries. This will make it easier for people to add technologies or other facilities on top of the Java EE platform. Chinnici pointed out that extensibility in the platform also means treating scripting languages such as Ruby as first class citizens.

Modularity

Chinnici said that rightsizing and flexibility ultimately lead to modularity. "Runtimes, applications servers, and even Java SE are becoming more modular." A good example of that is GlassFish Application Server v3, a fully modular and extensible application server that is based on the OSGI module framework. GlassFish v3 is also open-source and conforms to the Java EE platform. A technology preview release of GlassFish v3 is available. In fact, a second version of the technology preview was just released. To demonstrate the modularity of GlassFish v3, Chinnici called to the stage GlassFish Architect, Jerome Dochez. Dochez demonstrated how modularity makes GlassFish v3 much smaller than previous versions -- as small as 98K bytes -- and gives it a startup time of only one second. He also showed how easy it is to add an extension to the application server, one that immediately participates in GlassFish services such as monitoring.

Multi-Language Support

In another demo, Sun technology evangelist for web services, Arun Gupta (the "GlassFish guy"), and Sun Principal Engineer, Tor Norbye (the "NetBeans guy"), demonstrated GlassFish v3's support for multiple dynamic languages and frameworks such as Ruby, Groovy, Rails, and Grails. They showed a multi-player tic-tac-toe game developed in the NetBeans IDE and deployed to GlassFish v3. First they ran a Ruby on Rails version of the game. Then they ran a Groovy on Grails version. Both ran identically.

What's New in Java SE?

Next, it was Coward's turn to tell the Java SE platform story. Coward started by focusing on the strength of the current version, the Java SE 6 platform. He noted that there are currently over 7 million Java developers. Each month, on average, there are 750,000 downloads of the Java Development Kit (JDK) and 48 million downloads of the Java Runtime Environment (JRE).

Now Part of the Ubuntu Package

In a significant announcement for Linux developers, Coward announced that the JDK is now a core package in the Ubuntu distribution of Linux. With this packaging, Linux developers no longer need to manually download anything from Sun. Coward said "it's just there."

Modularity and Multiple Language Support Here Too

The Java SE 7 platform release is scheduled for mid-2009. Like Java EE 6, Java SE 7 will be highly modular. Coward announced a new JSR that will specify the interoperability between the module system in Java SE 7 and OSGi bundling system. This will allow developers who create applications that use OSGi bundles to run them unmodified in JDK 7.

Java SE 7 will also support a raft of dynamic language such as JRuby, JavaScript, and JavaFX Script. Coward said that an active community of Java SE 6 users has already brought over 200 languages to run on the virtual machine (VM). In Java SE 7, work is ongoing to make it easier to write interpreters and language compilers and to speed up the execution of other languages.

Applets Are Back!

For a long time, applets have taken a back seat to web technologies such as JavaServer Pages (JSP) technology. However, Java SE has given new life to applets. The runtime for the latest update to Java SE, Java SE 6 Update 10 (also known as the consumer JRE), currently available as a Beta release, enables applets to run without waiting for the whole JRE to download. Also, Update 10's JRE is enabled for JavaFX. To underscore the Java FX tie-in, Sun engineers Ken Russell and Jasper Potts demonstrated some attractive applets coded in JavaFX Script and running on top of the Java SE 6 Update 10 JRE. What was particularly impressive is that these applets can be decoupled from the browser and seamlessly moved to the desktop.

Cool Java FX Demos

Demos in the last part of the session underscored the strengths and potential of JavaFX technology. But before the demos, Brewin took the stage with Bill Joll, CEO of On2 Technologies to announce that Sun and On2 have signed an agreement to incorporate On2's video codec into the Java runtime.

Here's a brief summary of the JavaFX demos:

• Parleys.com. This web-based Rich Internet Application created by the Belgian Java Users Group (BeJUG), is an excellent learning platform. Among other things, it presents videos of sessions recorded at the BeJUG-sponsored conference JavaPolis. Steffen Janssen, founder of the BeJUG and JavaPolis, demonstrated a beta version of a new JavaFX-based Parleys.com, showing off an animated user interface.


• LiveConnect. Sun engineer Josh Marinacci wrote this social mashup application. Marinacci demonstrated how simple it was to code LiveConnect, an application that brings together social feeds, media, and messages in a single rich graphical interface. And because it links into Java libraries, JavaFX makes it easy to add services to an application like this.


• JavaFX Collaboration. JavaFX technology promises to bring the world of the designer and developer closer together. Sun engineers Martin Brehofsky and Soraya Younossi demonstrated an implementation of this collaborative vision. Taking the designer role, Younossi used tools such as Adobe Photoshop and Illustrator to design the application's user interface. Then she was able to easily pass the components of the design to Brehofsky, who incorporated those assets into a JavaFX-based development tool. It looked like a very smooth productive arrangement, with no need for the designer to learn the development tool or the developer to learn design tools.


• JavaFX Moon Tanks. Running a multi-player game called Moon Tanks, Sun engineers Chris Oliver -- the originator of F3, which morphed into JavaFX technology -- and Anthony Rogers demonstrated some of the advanced animation, audio, and video effects that developers can produce relatively easily using JavaFX. Each player has a set of tanks that can fire projectiles at an opponent's tanks. What makes this a "moon" tanks game is that the tanks move around a geodesic dome-like moon.

  

Final Thoughts

I left this session feeling good about where Java technology is and where it's headed. I'm also looking forward to some of the new tools and features that will be unfolding under the JavaFX brand. Indeed, Java technology is and will continue to be the hub of everyone's digital life.

The Conflict

The conflict that is producing the move to open source is between the "walled garden" approach, which uses proprietary software and services to produce scarcity and product demand, and the open source and standards approach, which makes software a commodity and makes it tough to add value.

According to Barr, there really is no choice.  Where consumers have an open source alternative, they invariably take it, and the proprietary approach can never recover the market.  As an example, he cited AOL, which tried to contain the user experience in a subspace of the internet.  Users demanded full access to the broader internet, however, and AOL was required to break down the garden walls and set its users free.

As another example, Barr described the attempts to limit recorded music copying with digital rights managment (DRM).  "Lead users" -- aggressive consumers who blur the line with developers -- demanded the freedom to copy the files they owned, and the MP3 format eventually conquered proprietary encryptions.  Consumers are the ones who lead the open source revolution, abandoning proprietary suppliers when they do not fill consumer needs.

But these examples are from the desktop software world, and Barr's expertise is in mobile communications.  This industry is in a great deal of flux at the moment, and the adoption of open source software is complicated by several factors.

Open Source and Profitability

The technology stack supporting mobile devices is more complex than the one for desktop computing, and the capital investment in infrastructure is high.  Moreover, the demands for reliability and security are higher than in desktop computing.  The open source development model of continuous beta release doesn't produce the bullet-proof quality required for mobile communications.  And then there are the myriad devices, each with their own special features and peculiarities, as well as the sheer volume of devices deployed -- billions and counting.

Barr touched on monetization in a mobile industry that is trending towards open software.  Proprietary products produce high profits at low volumes. Open-source products quickly become high-volume commodities.  Barr noted that the largest companies typically sell commodities at low profit per item, but still are very profitable overall.

To differentiate themselves and profit, open source companies must innovate, sell intangibles, build side channels for content add-ons, and tune the overall user experience to gratify consumers.  The challenge for software vendors in the mobile space is to make the user experience as comfortably similar as possible on many dissimilar devices.

Open Source: Problem or Solution?

So, is the move to open source/open standards the solution to the problem of disruption in mobile communications, or did it cause the problem? Neither, actually, if Barr is correct -- it is merely a symptom of the changes being demanded by consumers who want more options, and developers who want to write code to open standards.

With the shift to open source and open standards, people's expectations have changed. There is no going back to the old proprietary model. Even naive consumers know that an open source company offers them more choices, and that its products integrate better with others. It is still not clear how the mobile market will fit into the overall IT market, but clearly old models of doing business are coming undone. For now, open standards are more important to mobile communication than open source, since only open standards can simplify the technology stack.

For more ideas from Terrence Barr, see his blog.

by Dana Nourie, Sun Microsystems, Staff Writer

If you were frustrated with applets in the past, or completely given up on them, it's time to take another look at the new Java Plug-In technology. Now you can reap the following advantages from the rewritten support:

• Improved reliability
• Improved user experience
• Applets launch in the background
• Built-in JNLP support
• Per-applet command line arguments
• Heap size, Java 2D™ API acceleration options...
• Improved Java/JavaScript programming language integration
• Improved Windows Vista support
• Signed applets now work correctly in Protected Mode Internet Explorer

The next-generation Java Plug-In, new in Java SE 6 Update 10, provides powerful new capabilities to applets in the web browser, while improving the overall reliability and functionality of applets in a backward-compatible manner.

The most significant new feature of the next-generation Java Plug-In is built-in support for launching applets from JNLP files. Using the JNLP file format as the applet descriptor allows applets to instantly reuse JNLP extensions previously written for Java Web Start applications, and significantly expands the capabilities of applets in many other ways.

Applets no longer execute in a Java Virtual Machine (JVM) inside the web browser. Instead, a separate JVM machine process is launched to execute the applets. Applets launch directly from JNLP files, make use of the
same descriptor used by Java Web Start software, and allow more powerful parameters than classic “archive”, “code”, “cache_archive.”

The new plug-in provides:

• access to advanced JNLP extensions previously available only to Java Web Start software applications. A small set was previously available, with restrictions, and these restrictions have now been removed.
• access to the JNLP APIs from applets
• PersistenceService, DownloadService
• control over heap size, command-line arguments, JRE version selection, and auto-download. You have the same capabilities as Java Web Start software for applications

Now, you use something like the following on the web page:

<applet width=”500” height=”500”>
  <param name=”jnlp_href” value=”my_applet.jnlp”>
</applet>

By default, only one JVM machine is launched, but you have the opportunity to launch more than one JVM machine, and you get support per-applet command-line arguments, so you can affect heap size or other requests.

The applet behaves exactly like an application started with Java Web Start software. jnlp_href parameter bridges between the web page and the JNLP description of the applet. Applet tag and JNLP file have overlapping mechanisms for specifying things like the width and height.

Historically, the maximum heap size that could be specified for applets via the Java Control Panel has been limited. This limitation is fixed in the new Java Plug-In; applets can now utilize as much heap space as command-line applications. 

An applet may force itself into its own JVM machine instance separate from all other applets if you like:

    <param name=”separate_jvm” value=”true” />

This is useful when migrating certain kinds of desktop applications to work in the web browser.

You can also run a particular applet on a particular version of the JRE, as shown below:

<j2se version=”1.4+” ...>
<j2se version=”1.5*” ...>

This is useful for enterprises that want to QA an applet against a particular JRE version, or it supersedes earlier version selection mechanisms like CLSID in Internet Explorer. If a very old JRE version is requested, restrictions are enforced the user will be prompted if applet attempts to load unsigned code.

Note that since JNLP support is first available in Java Plug-In in Java Platform, Standard Edition 6 (Java SE 6) Update 10, version specifications like “1.4+” are basically meaningless. This will have more meaning when “1.7+” is needed.

In addition, you can use the <update> tag in the JNLP file to drastically reduce startup time for second and subsequent launches:

       <update check=”background”> 

In this case, it'll use the version of the applet that is already in the cache, and will download updated versions of the applet in the background. At the next launch the new version is picked up.

Lastly, it's much easier to maintain backwards compatibility, to create your own corporate logo or animated image for loading time, as well as controlling background colors and overall appearance, there is powerful Java/JavaScript integration, and support for access to Web Services.

Applets are mini applications that run anywhere, in and out of the browser.

For more information, see the Release Notes Next-Generation Plug-In

Ludovic Poitou, community lead for OpenDS, Sun's open-source project for a next-generation directory service; and Jim Yang, project lead for Penrose, a Java technology-based directory server, took the podium at CommunityOne earlier today on the topic, “Getting Started on OpenDS.” In clear, concise terms, Ludovic described the goals and status of OpenDS, its efficient installation process, and extensibility. The second part of the session, presented by Jim, covered how Penrose took advantage of OpenDS in its development.

Introducing OpenDS
OpenDS was launched in 2006 for the purpose of developing in the Java programming language an LDAP version 3-based directory service. Currently, the community boasts 38 committers (all from Sun), of whom are 15 developers and nine QA engineers, all working full time on the project; the other committers are part-time. In addition, there are 14 contributors and 220 users.

The goals are several fold:

• Robust services for accessing the data and consolidating data access.

• Ease of installation, use, and management.

• Embeddability, default implementations, and extensibility with service APIs.

• Superior scalability and high performance in read-write operations
  

OpenDS 1.0 is in the works and will be available soon. Not only does it comply with LDAP version 3 with support of standard and implementation extensions, it's also configurable and extensible, complete with comprehensive documentation. Tools are in place for deployment, monitoring, and interactions with the server, with unit tests also in open source.

Installing OpenDS
According to Ludovic, you can install OpenDS in only three minutes. Just follow the pointer at opends.org and run through the steps in an intuitive wizard to define the settings: server path, host name, port number, configuration of secure access, directory data, and so forth. When installation is complete, you can perform administration tasks, such as starting or stopping the server, from either the command line or in a GUI called the Status Panel.

Extending OpenDS
Ludovic encourages everyone to make the best of OpenDS, for example--

• Extend the LDAP schema by adding files to the schema directory or add LDAP schema attribute types or object classes for LDAP.
  

• Intercept LDAP operations and processes through the DirectoryServerPlugin API, which performs two tasks:
  

• Provide entry points in LDAP operations, such as preparsing and preoperations.

• Define business constraints or LDAP extensions, a real-life example being Penrose.
  

Building Service Interfaces for OpenDS
Examples of service interfaces abound: AccessControlHandler, ExtendedOperationHandler, AlertHandler, security interfaces. Not all of them are public yet. Feel free to create interfaces of your own or customize the existing ones.

Introducing Penrose
Penrose, an open-source virtual directory service, does not manage its own data. Rather, it's a collection of mappings through two main infrastructures:

• A Java background, which ensures interoperability between Penrose and other LDAP providers.

• Metadata management, which enables mapping between data models.
  

Penrose chose OpenDS as its default provider for the latter's thorough testing and detailed documentation. In an intuitive GUI, Penrose Studio, you can create connections to the database, map data (for example, a MySQL entry to an LDAP entry), manipulate data, and deploy to the server.

Conveniently, you can run Penrose in two modes: as an independent server or with OpenDS as a stand-alone server with Penrose as a plug-in. Many deployments also apply, according you much flexibility with your setup.

Before checking out the code in detail, have a look at the Penrose Server Developer Guide.

A succinct, informative session! From the audience came a question, “What are the applications on which you test OpenDS?” Ludovic responded: “We test with our own tools—LDAP certification tools, for example. But remember that the community is testing for us, which is one of the many advantages of open source. We count on our members for feedback and suggestions.”

See also the presentation.

Last year saw the birth of CommunityOne a free, full-day event that preceded the 2007 JavaOne Conference and set the tone for what would be a full week of presentations, demonstrations, and knowledge sharing by and for the open-source community. Celebrating its second appearance, this year's CommunityOne was even bigger and better than last year's version.

The event, sponsored by Sun, brought together a large and diverse gathering of many of the leading-edge open-source communities such as the Apache Software Foundation, Eclipse, Grails, NetBeans, ODF Alliance, OpenSUSE, OpenID, OpenOffice.org, OpenSolaris, Python, Ruby, Ubuntu, and others. This was more than triple the number of communities attending last year's CommunityOne event. "Large and diverse" also characterized the agenda. CommunityOne offered over 70 technical sessions in 12 tracks ranging from Chip Multithreading/OpenSPARC to Web and Application Servers -- all highlighting the challenges and successes of community-based, open-source projects. Running parallel to the 12 tracks were a RedMonk unconference that offered sessions conceived of and run in an open, participatory, community-driven style (these are also called barcamps in hacker-speak), and another unconference called StartUp Camp that focused on getting entrepreneurs in startups to share ideas. As many as 500 entrepreneurs participated. Over 3000 people attended the CommunityOne event, including students from 80 countries.

Keynote: Innovate, Collaborate, Integrate

Sun's vice president of developer and community marketing, Ian Murdock, started the day with a keynote session that set the theme for the day: Innovate, Collaborate, and Integrate. Clearly, the community, or rather communities, have reshaped the computer industry and will continue to do so in new and innovative ways. To underscore the power of collaboration, Sun CEO and president Jonathan Schwartz, in a brief appearance onstage, asked the audience to think of the Amazon River. "You can look at the Amazon as this enormous powerful force of nature, but in truth it's not one river. It's the composition of 10,000 smaller rivers, each of them lending their own capacity, volume, and force of energy to something much bigger."

But collaboration, especially if it involves many contributors, does have its downside. Customers want to tap into the power of the community, but don't want to deal with the complexity of technology that comes from multiple sources. This has spurred the rise of aggregators and integrators such as Dell and Red Hat. Murdock said these companies provide a valuable role, "They deliver the complex but powerful world of community technology to the market in the form that the market can digest." But this service to customers presents a challenge to contributors, Murdock framed the challenge as a series of questions. "What's the role of the small independent developer in this world? Are we going back to monolithic structure in the pursuit of simplicity? What's the right balance between integration and choice and flexibility?"

Murdock made the point that open source and the communities behind it are the core of Sun's business today. He said that Sun is attempting to strike the right balance. On the one hand, Sun is committed to fostering the community model and open-source software. On the other, it's focused on developing modular architectures, where developers can contribute to core platforms with edge technologies -- whether these technologies are packages, plug-ins, extensions, or adapters. This bilateral approach can benefit the core platform as well as the independent developer.

Panel Discussion: A Multitude of Models, How Communities Work

An engrossing sidelight to Murdock's keynote was a panel discussion titled "A Multitude of Models, How Communities Work." Led by Matt Asay, general manager and vice president of Alfresco and a prolific blogger on open source topics, the panel included community luminaries such as Jim Zemlin, executive director of the Linux Foundation; Mike Evans, vice president of community development at Red Hat; Jeremy Allison, the Samba project lead at Google; Marten Mickos, Sun senior vice president for databases and former CEO of MySQL; Ted Leung, Sun Microsystems principal engineer and Python project lead; and Stormy Peters, OpenLogic's director of community and partner programs. The panel discussed a wide range of questions such as "Who makes up various communities such as MySQL?" and "Is there a tension between satisfying the needs of code contributors and satisfying the needs of a business?" The speakers encouraged audience members to participate in the discussion by posting questions through the social blogging tool Twitter. This led to some interesting technical questions -- but also to a humorous one: "Did Jeremy [Allison] dress like Steve Jobs on purpose? "

OpenSolaris Launch

Perhaps the biggest announcement of the day came from Rich Green, Sun's executive vice president for software, in his talk titled "Be Brilliant Faster." Green announced the fully supported release of the OpenSolaris operating system. Created through worldwide community collaboration, the OpenSolaris OS is an example of what the open-source model can accomplish. The leading-edge operating system allows users to fully install and customize their open-source OS deployment quickly and easily. To demonstrate this, Sun Distinguished Engineer Stephen Hahn showed how easy and fast it is to fully install OpenSolaris from a LiveCD distro. The operating system installed in less than 15 minutes.

Some other cool demos ensued, and the highlight was a dramatic demonstration of the ZFS data recovery feature. Armed with a sledgehammer and a drill, Sun Fellow and Solaris Chief Technologist Jim Hughes and his partner in destruction, Sun distinguished engineer and chief architect of ZFS Jeff Bonwick, did fatal damage to two hard drives that were connected to a processor. Green quipped this was true "hard drive compression." Hughes and Bonwick then used ZFS to quickly recover the data from a backup onto a new pair of drives.

Download the OpenSolaris OS for free from the OpenSolaris web site.

Technical Sessions and More

It was impossible to go to all of the sessions during this event, and it's very difficult to relate everything I saw and heard at the sessions I attended, but here's a sampling of what I experienced at three sessions at CommunityOne:

• The NetBeans IDE: Welcome, New, and Cool. During this session, which kicked off the NetBeans track, NetBeans evangelists at Sun Brian Leonard, Bruno Souza, Gregg Sporar, and Roman Strobl demonstrated some of the cool new features in NetBeans 6.1 such as support for JavaScript and the Groovy language. Believe me, NetBeans 6.1 is loaded with cool features. In addition to its JavaScript support, NetBeans 6.1 supports the Spring web framework, is more tightly integrated with MySQL, and starts up a lot faster (up to 40%) than it used to. Try it out.
• jMaki: The Power of Ajax Made Easy. In this session, the two principal contributors to the jMaki project, Sun engineer Carla Mott and Sun Ajax architect Greg Murray, demonstrated the latest features in jMaki, a lightweight framework for creating Ajax-enabled applications. Among its other features, the jMaki framework makes available widgets from popular JavaScript toolkits such as Dojo, Yahoo, Script.aculo.us, and Google in a way that's familiar to Java technology, PHP, or Phobos developers. One of the more interesting demonstrations during the session underscored just how easy it is to take advantage of the jMaki event mechanism so that an event on one widget can trigger an action on other widgets.
• JavaFX Script Programming Language and GlassFish Application Server V3: Productivity and Power, Front to Back. In this session, Sun software engineers Tim Quinn and Josh Mariucci showed off some of the neat features in the JavaFX Script language that simplify developing Rich Internet Applications (RIA) and how the open source GlassFish Applications Server v3 provides an ideal environment for deploying and running JavaFX Script applications. Aside from being feature rich itself, GlassFish Application Server V3 has modular packaging. The core module is extremely small and has a fast (one second or less) startup time.

The general sessions and many of the technical sessions were video recorded and will be available for replay on the CommunityOne web site.

As content-rich as these sessions are, going to an event such as CommunityOne is more than sitting in on talks. There's also the great value you get in meeting people, learning about their experiences and their work, and the infectiousness of their enthusiasm. Bringing vibrant, creative, enthusiastic, fun people together -- that's what CommunityOne is all about.

Roman talked about the many features of the NetBeans IDE, such as code completion for various languages, language support like PHP and Ruby, and UML support. Then the audience was able to ask questions about special features.

It seems NetBeans does just about everything these days, including writing most of the code. So, for new Java developers, he recommends that you learn about design patterns. He says there are many books out there on the topic that you can read. In spite of the fact that NetBeans will write so much code, you can still potentially put together a sloppy application or one that does not scale well. He also recommends looking at the best practices for building web applications by studying tutorials on on the Java EE platform.

Some of the areas in web applications development that you need to understand if you're using Java technologies for web application development is JavaServer Pages. The web application is created in two ways in NetBeans and that is the visual and the code. You can use JavaServer Faces technology through drag and drop to create the user interface, but you can also get to CSS code for further modifications. This provides several tools to web site designers.

One of the questions directed at Roman also concerned UML support. The NetBeans IDE has great UML support that allows you to design applications using a standard modeling language. You then can generate source code from the UML model and update the model from changes made in their source code.  With NetBeans UML Modeling, you can create eight UML diagrams: Activity diagram, Class diagram, Collaboration diagram, Component diagram, Deployment diagram, Sequence diagram, State diagram, and Use Case diagram. You can align diagrams visually in the Diagram Editor.

Additionally, Roman talked about upcoming features that we can look forward to, such as additional jMaki widgets. Currently, there are many handy jMaki, Yahoo, and Dojo widgets, and now we can look forward to additional widgets in future releases. These widgets make pulling in services as easy as drag and drop.

I'm looking forward to some of the upcoming talks in Second Life this week. Be sure to check the schedule. 

All inworld events will be held at the JavaOne Theater at Sun's SIMs in Second Life.

Dana Nourie (aka, Dana Oceanlane), Sun Microsystems, staff writer java.sun.com

For more on JavaOne.

Janice J. Heiss

Java University has always been packed with helpful, thorough instruction, but this year it has more courses and includes more than just Java technology. As anyone involved in web development knows, you can't always do it all with Java technology alone. So, I'm glad to see that Sun has also included a lot of information for developers on topics such as:

• JRuby and Rails
• Groovy and Rails
• MySQL
• Ajax and the Dojo Toolkit
  

Of course, there is also a heck of a lot you can do on the web and on the desktop with Java technology and tools, and Java University has many courses to help developers in these areas as well:

•  Filthy Rich Clients (great stuff you can do with Swing)
• Java ME and Mobil Devices
• Java Persistence API
• Secure Java Web Services

And there is a whole lot more for developers to benefit from at Java University. In the past, I have found that the courses in Java University were handled well as far as getting everyone into the rooms, which are very large, presenting the technical materials, which included handout printed materials and presentations on the huge screens, and the speakers were animated. I expect the same and more this year.

For those of you who can't make the JavaOne Conference this year, there will be many writers from Sun covering the sessions and experiences. I'll be writing about Java University for the JavaOne Today daily paper, as well as blogging about it and sessions here.

No matter whether you are at the conference or not, be sure to watch this space as we share the conference experience and deal out technical information as we can. It should be a really great conference this year.

 Dana Nourie, Sun Microsystems staff writer for java.sun.com

 

Dana Nourie, staff writer for java.sun.com