ActiveBlog: Insights on Open Source and Dynamic Languages

Komodo IDE 7: The world's fiercest IDE has evolved!

toddw — Wed, 01 Feb 2012 10:02:56 +0000

Komodo IDE 7 has just been released! It offers groundbreaking new technologies and innovative enhancements. Here's an introduction to the most prominent features it includes.

Code Collaboration is one of the coolest things to use in a team development environment. You can easily share your files and collaborate in real time with your colleagues and friends. Peer reviews and pair programming become a much more intuitive and interactive experience with Komodo's collaboration tools.
For those that use multiple machines, Komodo's Synchronization feature gives you the ability to copy your Komodo settings (remote server accounts, toolbox, file templates...) between all the Komodo instances. So you can customize Komodo in one place and have the same setup appear in another!
The cloud is a new and exciting playground for developers. Komodo's Stackato interface provides the ability to quickly deploy web based applications onto cloud infrastructure - you can have your application running on the cloud within minutes! A slew of languages and frameworks are supported - such as Python, Django, Ruby, Rails, PHP, Drupal, Perl, Mojolicious, Node.js and even Java!
Node.js support - Komodo IDE brings in debugging, code intelligence, editor and syntax checking support for Node applications. As well as this, Komodo IDE 7 provides additional support for web and template languages like CoffeeScript, LESS, SCSS, EJS and Mojolicious.
With the ability to Code Profile your application - you can now find out why that PHP or Python code is taking so long to run! Komodo shows a graphical representation of the methods and calls made by your program, as well as outlining the CPU time spent executing your code. Automatic time sorting makes it easy to see the code hot-spots at a single glance.
Now, with every major Komodo IDE release we've aimed to make things work smarter and faster than they were working before. Komodo IDE 7 is no exception to this rule and you'll notice a lot snappier Komodo startup time, lower CPU utilization - particularly when idle, and lower memory usage for large projects. Additionally, tools like the fast-find bar make it even quicker to do the common things you use the most.

There are way more features in Komodo IDE 7 than I can outline in a single blog post, so check out the full release notes to see other great enhancements like multi-language syntax checking, HTML end-tag completions, moveable side panes and much more.

If you'd like to test out these features yourself, you can download a 21-day free trial of Komodo IDE 7.

It's also currently on sale, starting at $245 (reg. $295) until March 15th. Visit our store to purchase today.

A special thanks goes out to all the testing feedback provided by the Komodo 7 beta users during the Komodo 7 development process - you all rock!

We hope you enjoy Komodo IDE 7!

Trackback URL for this post:

http://www.activestate.com/trackback/3295

Smooth Scaling with Stackato and vSphere

jamiep — Wed, 25 Jan 2012 00:59:21 +0000

We have a number of Stackato deployments here at ActiveState, some larger than others, but one stands out as enticingly quick to get setup and scaling with little effort.

Our Stackato vSphere cluster serves as one of our primary testing platforms and at any given time. It’s running at least 25 Stackato nodes, one 'cloud controller' and the rest mostly 'DEA' nodes.

In a real elastic cloud you wouldn’t want to waste the resources or expense of running these DEA’s perpetually, you would want them only when there is a current or trending resource demand for them. We mainly idle these VMs for QA testing and benchmarking, so we are ready to instantly scale up or down.

Elasticity should be no stranger to those of us working on PaaS or IaaS products, and if you’re not already familiar with our Stackato Sandbox, it runs on Amazon EC2 with EBS backed storage. To meet user demand at any point in time, we can have the cloud controller automatically spin up new DEA instances taking into account projected load on resources and use the EC2 API to spin up a new DEA.

vSphere can also serve your own privately managed cloud in this exact manner, and I’ll be giving a short summary on how we did this on our own vSphere setup.

Taming the vSphere API

It should come as no surprise to more experienced vSphere users that to achieve scaling we made use of the vSphere API. As Cloud Foundry itself was conceived in Ruby, we choose to use RbVmomi, an open source Ruby interface to the vSphere API. If you’ve had previous dealings with the vSphere API, fear not - this Ruby interface actually makes the coalescence of the two a breeze.

For the connecting vCenter user, I recommend creating a new user with the permissions to create, start, stop and deploy new VMs, and access to any of resource pools it might need.

Diving into the Cloud Controller

I reused existing modifications to the health monitor component from our EC2 (Sandbox) scaling code, to incorporate the ‘events’ necessary to trigger a scaling activity.

These previous modifications to the health monitor are designed to probe all the available DEAs at runtime intervals and check their current resource capabilities and limits, including the DEA’s current RAM metrics. This, of course, can also include the condition when there are simply no DEAs available for deployment due to failure or full capacity.

Using a small analysis of past, current and projected app deployments, a ‘scale up’ event will be triggered and passed along the NATS message bus, which is the default PubSub messaging mechanism used by Cloud Foundry.

At the point of receiving the event message, the process flow follows something like this:

The Health Monitor component keeps track of the health of any deployed apps and the availability of DEAs
If under load, the ‘scale up’ event is triggered from the Health Monitor to the cloud controller via NATS
The cloud controller will load up the vCenter YAML configuration (more on this later) and connect to vCenter via the API
It will round-robin all hypervisor hosts currently available for deployment, so no single one becomes overloaded.
Upon finding an available host, the DEA is deployed from the template to the host and it’s corresponding resource pool and datastore.
The MRU index of the last deployed to host is kept in the cloud controllers database.
When the DEA starts up, it is already configured to register with your cloud controller and is instantly ready to start receiving new app deployments.

Scaling from a pre-built template

On the ‘subscribing’ end of these event messages is the cloud controller itself, which upon receiving a ‘scaleup’ operation, will proceed to connect to vCenter, and deploy a new DEA from a pre-defined template. To make the template, we take a new Stackato VM and run:

  stackato-admin become dea -m <cloud_controller_ip> -e <api_url>

This is the same step you would take when setting up a cluster manually. Once you’ve run this, you can shut that VM down, then right click ‘Save as template’. Once converted into a template, it as available for instant deployment across all your ESXi’s / vHosts.

Balancing the VM load on your ESXi’s

Checking the hosts status

A useful feature of the vSphere API is the ability to probe your hosts for their runtime metrics, so you can make an informed decision about where the next VM should be deployed to. We make use of the vSphere API’s HostSystem() object to get some information about which hosts are available, and which hosts are in good health.

Using the API makes it easy to locate a new host to deploy to. You don’t need to supply any host configurations, the cloud controller will take care of keeping tabs of hosts in the CC Database. The scaling code automatically chooses the next available vSphere host at runtime, so you don’t need to manually specify your hosts in a config file.

Wrapping it up in a configuration file

To ease maintenance and provide flexibility, the scale-up control settings are in a simple YAML configuration file that you upload to the stackato user’s $HOME directory, which is then parsed by the cloud controller. This configuration contains parameters like vCenter server, authentication details and template information. It is quite straight forward:

---
  server: vcenter.domain.com
  user: username
  password: password
  https: true
  port: 443
  insecure: true
  path: /sdk
  datacenter: DataCenter
  template: "Your-Template-Name"

We use SSL to connect to vCenter (recommended), but I’m also connecting with ‘insecure’ set to true. That simply means we are using self-signed SSL certificates, and forces RbvMomi to accept them.

Expanding the featureset

With the abundance of documentation on the vSphere API we could take the scaling process even further when working with different setups of datastores architecture.

Some items for consideration could be:

Limiting the maximum number of VMs on any single host
Allowing deployments from multiple/different templates
Allowing multiple vCenters in the configuration

If you have any thoughts or questions about how this could apply to your own vSphere setup please let us know, we’d love to hear them! If you’d like to try this Stackato auto-scaling on your own vSphere cluster, please contact us at stackato-sales@activestate.com, and we'll arrange access to a special VM of Stackato with this feature.

Trackback URL for this post:

http://www.activestate.com/trackback/3285

Latest ActiveState Stackato Release Extends PaaS support to OpenStack Ecosystem

dianem — Thu, 19 Jan 2012 18:14:38 +0000

The ever-expanding OpenStack ecosystem now has an enterprise-ready PaaS. Stackato, built on Cloud Foundry and hardened for the enterprise, now supports vSphere, Amazon/AMI, HP Cloud Services, OpenStack, Citrix XenServer, and KVM infrastructure models.

ActiveState joined the OpenStack community in 2011, and committed publicly to making PaaS a reality for OpenStack deployments. As showcased with Stackato’s successful deployment to HP’s OpenStack-based cloud, we’ve made good on this commitment, and are now pleased to offer OpenStack-ready Stackato for download with our latest release of Stackato.

Stackato—ActiveState’s commercial-ready, secure, multi-tenant platform for creating a private PaaS—is now the linchpin connecting Cloud Foundry and OpenStack. Stackato’s enterprise-ready OpenStack support ensures that these two important ecosystems can be securely deployed together.

With Stackato, customers can deploy applications to either a private internal cloud (like ones powered by vSphere, XenServer, KVM, or OpenStack) or one hosted with a third-party cloud-hosting provider (like those powered by Amazon, RackSpace, or HP Cloud Services).

Stackato also now has SSH support, so you can have a secure interactive shell in any of your application instances. (Personally, this is my favorite new feature in this latest Stackato release.) This is possible (and safe) because each application instance runs in it's own para-virtualized container using LXC, providing more secure multi-tenancy.

In addition to OpenStack deployment and ssh support, Stackato has a new Management Console that replaces the Admin Dashboard with a new, improved user interface:

The Management Console offers deep visibility into the activity and events in a private cloud to help administrators better manage usage. This view includes showing activities of developers who deployed an application, number of instances deployed, memory usage, data services deployed, and languages used.

And there’s more: Additional updates in this latest release include improved application lifecycle management, improved Perl deployment speed, and new pre-staging setup hooks.

Take a test drive with Stackato on a microcloud (VM) on your own desktop or deploy directly to the Stackato Sandbox today!

Trackback URL for this post:

http://www.activestate.com/trackback/3283

Don't Step in the FUD: The Real World Demands Private Cloud

bartc — Wed, 18 Jan 2012 19:28:59 +0000

This article originally appeared January 17, 2012 at VMBlog.com. You can read it here.

Last week, SD Times contributor Alex Handy published an interesting article titled The PaaS market accelerates. The piece was noteworthy not just for its insight into the burgeoning PaaS market space and introduction to the major players, but for some bizarrely unfounded assertions made by one of the article's subjects.

Really, Adam?

In his January 6^th SD Times article, Handy writes (emphases mine):

... Adam Wiggins, CTO of Heroku, said that PaaS is about the public cloud only. Although he used harsher language to describe the private cloud and PaaS market, he indicated that he and Heroku believe the private cloud to be entirely vapor.

"My opinion—and this is not a widely held opinion—is that private cloud is total BS. If you make it private it is no longer cloud. You destroy all the value you get from cloud by making it something you have to run yourself. Anyone pursuing that path is advocating a false trail," said Wiggins.

(At this point, the Enterprise IT Leads all do a spit take. I'm just curious to know what the unmentioned "harsher language" was.)

Really, Adam? "BS"? "Vapor"? Really? Is that the best you can do for FUD? Your comments are at best puerile, and at worst naïve. Let's focus on the naïve...

Save me, public cloud!

What Heroku's Wiggins is desperately trying to obscure is that in the real world, the enterprise demands private cloud and Private PaaS. Fortunately, he is correct in observing that his is "not a widely held opinion."

Wiggins suggests that customers derive value from the self-service nature of the public cloud, and that making the cloud "something that you have to run yourself" defeats its purpose. Hmm. Do I really want control over my own platform? My data is probably safe, I mean, it's in the cloud. And do I really want to deal with all that pesky provisioning and applications management and infrastructure administration and oh, I'm overwhelmed-Save me, simple public cloud!

If only it were that simple. As my colleague ActiveState VP of Engineering Jeff Hobbs noted in Handy's article, Public PaaS can lift the burden of cloud-management responsibility, lightening IT load. And public clouds also deliver benefits like shared-resource efficiencies, utility computing, and scalability. But let's be honest here: Every Public-PaaS developer promotes those same supposed advantages, and whee! The race towards commoditization is on. Indistinguishable public-cloud services foster desperate competition, which drives down price, which ultimately benefits the consumer...at least the one looking for the cheap solution.

Public PaaS: Slumlord of the cloud's low-rent district

In contrast with Private PaaS, the very nature of the Public PaaS' shared-services model enforces a lowest-common-denominator architectural approach. You're entering the low-rent district: You get what you pay for with public cloud ("First month free!"), and that means the same setup as the tenant sharing your virtual apartment building. ("Matching floor plans!") Does your company have unique data needs? Public PaaS may impose restrictive limitations on the way you manage your data. ("No pets!") Prefer a cookie-cutter approach, okay with limiting application portability, and don't mind sharing a multi-tenant virtual hallway? ("It's just like a hostel!") Then Public PaaS-at least as Adam Wiggins envisions it-might be right for you.

The real world demands Private PaaS

Recall those Enterprise IT leads? (I hope they've regained their composure.) They are the ones who are investing in private cloud and Private PaaS because it provides better security, better control, and-believe it or not, Adam-greater flexibility than public-cloud alternatives. Oh, and with the right technology, those Enterprise IT Leads can also realize those public-cloudy benefits like shared-resource efficiencies, utility computing, scalability, and even multi-tenancy.

But don't take my word for it. The U.S. Army recently invested a quarter billion dollars in private-cloud technology. Sure, you could argue that the military has special security needs that would never be well served by a public cloud. But so do very-much-not-like-the-military companies such as Federal Express. The package-delivery logistics giant has leveraged a private-cloud architecture in its newest data center, and even retrofitted an old data center to match. Even SD Times contributor Alex Handy noted in a November 2011 blog post that-despite Heroku's contention that private cloud is just "a myth and snake oil"-he could "entirely understand the need for an internal private cloud inside the major retailers [on Black Friday and Cyber Monday]."

Thanks Adam!

Don't get me wrong-I have long admired Heroku's accomplishments (particularly its carefully cultivated developer community), and Wiggins and team have produced some great technology. Let's face it, there will always be market segments that can settle for implementing a public-cloud solution. (Small business and municipal public sector are just two that immediately come to mind.) But ultimately, Wiggins' comments ignore (and even denigrate) the needs and priorities of the vast majority of the enterprise cloud market.

But you know what, the more I think about it, maybe that's not so bad. Who knows? Maybe Wiggins is right. Not that Heroku and Wiggins need my validation, but it's okay with me if they want to believe there's no future in private cloud and Private PaaS. Adam, you and Heroku can go play in your public-cloud-only sandbox. The rest of us will be just fine over here providing solutions for the real world.

###

About the Author

As President & CEO of ActiveState Software, Bart Copeland brings more than twenty years of management, finance, and technology business experience to his role. With a passion for technologies that help people lead more productive and enjoyable lives, Bart is currently focused on ActiveState’s private platform-as-a-service (PaaS) offering, Stackato. With his vision for PaaS as an enabler to accelerate cloud adoption and value in enterprises, Bart is actively involved in the strategy, roadmap, business development and evangelism of Stackato. Bart is also an active angel investor and serves as a director on a number of other tech companies. He holds an MBA in Technology Management from the University of Phoenix and a Mechanical Engineering degree from the University of British Columbia.

Originally published Tuesday, January 17, 2012 7:16 AM by David Marshallon VMBlog.

Trackback URL for this post:

http://www.activestate.com/trackback/3272

Stackato delivers PaaS with a punch: Secure Multi-tenancy on HP Cloud Services

dianem — Thu, 12 Jan 2012 19:28:59 +0000

ActiveState is pleased to offer Stackato's secure multi-tenancy capabilities for those of you deploying to the HP Cloud. HP Cloud Services–currently in private beta–is HP’s next generation of cloud infrastructure, platform services and cloud solutions for developers, ISVs and businesses. The HP Cloud is based on HP's Converged Infrastructure, a combination of HP hardware, software, services, and OpenStack technology. Stackato’s support for HP Cloud Services gives enterprise developers a more secure PaaS option.

Built on VMWare's Cloud Foundry Open Source project and hardened for the enterprise, ActiveState Stackato works on vSphere, Amazon EC2, and OpenStack, and will also support other infrastructure platforms.

While some PaaS offerings are content to rely on user-based Unix security, but ActiveState’s Stackato takes security a big step further, and is the first PaaS to bring lxc-based containerization to the HP Cloud, creating a more-trusted, commercial-grade level of security to private PaaS deployments.

So what’s the big deal about deploying a secure multi-tenant PaaS on HPCS, rather than on one that relies on Unix-based user level security, or one that shares resources amongst its tenants?

Why does multi-tenancy matter? (Hint: It’s more efficient.)

True multi-tenancy enables your organization to squeeze efficiencies out of a shared IaaS-level resource pool by running multiple applications on the same cloud servers. Multi-tenancy is what distinguishes PaaS offerings from Orchestration tools that do not manage multiple guest applications, but rather act as simple installers onto bare IaaS instances.

While multi-tenancy allows you to maximize usage of your allotment of cloud servers, having multiple applications running on the same server can be tricky (if not impractical) to secure.

Playing well with others in the PaaS playground

When you deploy a Platform as a Service, you provide a shared “playground” for a number of guest applications (tenants). The resource pool is presented as a single logical layer to those guest applications--The PaaS manages pools of OS resources and co-habits applications on shared instances.. Those guest apps should not have to be aware of the details of components that lie beneath the PaaS at the infrastructure level, nor should they have to be designed to play nicely with others. The PaaS layer should insulate both infrastructure and application from each other, and ensure that guest applications are not forced to be aware of other guest applications running on the same resource pool.

Some PaaS offerings manage the boundary between infrastructure and the application, But most do not, and PaaS solutions that offer only Unix user-level security leave openings for untrusted applications or users to exploit, and do not provide adequate security guarantees for the enterprise.

Securing the PaaS

Early on in the development of Stackato, ActiveState made the decision to extend Cloud Foundry, and take advantage of Linux technologies to provide better security. ActiveState's Stackato creates isolated, lightweight containers known as “lxc containers,”, Virtual Private Servers (VPS), or Jails. Conceptually, the containers are built atop several Linux technologies which provide each guest-application container with a segregated file system with web and ssh services.

The containers are protected from each other: a guest application in one container cannot read files owned by another container, or kill its tasks. With the Stackato-container approach, the outside world can still reach web servers and ssh servers on the containers and, more importantly, the hosting PaaS can protect its key files from guest applications and users in the containers.

The containers effectively allow Stackato to partition the resources into isolated groups to better balance conflicting demands on resource usage between the isolated groups. Stackato's innovative containerization approach has the dual benefits of running applications on a seemingly separate machine while still leveraging many of the underlying resources. As far as the guest application is aware, the PaaS has provided a private Playground. The architecture makes Private Cloud even more "private." For enterprises, the operational and security advantages of sharing these resources while isolating the guest applications can also lead to significantly lower overhead than true virtualization.

Implications for the HP Cloud Services platform

Adding support for HP Cloud Services to Stackato reinforces ActiveState’s commitment to providing the best-of-breed secure Private PaaS layer in the cloud that runs on HP's world-class hardware and software on OpenStack™ technology - giving your organization a clear alternative to Amazon Web Services.

The end result is a highly flexible, secure, multi-tenant PaaS layer that can be run in any environment (on-premises or in the cloud) helping your organization deliver a knock-out punch when if comes to securing your Cloud.

Stackato is a private PaaS that enables deployment, scaling, and management of Java, Python, Ruby, PHP, Perl, Node.js, Scala, and Clojure applications to any cloud. Stackato delivers the power of PaaS on-premise with the security, privacy, and control behind a corporate firewall. With Stackato, customers can deploy an application to either a private internal cloud (like one powered by VMWare vSphere™, Citrix XenServer, Linux KVM, or OpenStack™) or one hosted by a third-party cloud-hosting provider (such as Amazon or HP Cloud Services).

To see Stackato working with HP Cloud Services, we've provided this short screencast.

Developers and cloud administrators can download Stackato, currently in open beta including the free Stackato Micro Cloud, at: http://www.activestate.com/cloud.

Trackback URL for this post:

http://www.activestate.com/trackback/3269

Screencast: Stackato on HP Cloud Services

troyt — Thu, 12 Jan 2012 00:23:47 +0000

We had the opportunity to take HP Cloud Services (currently Beta) for a spin and get a Stackato cluster running there.

We were happy to take advantage of the computing horsepower HP made available to us to put Stackato through it's paces and to try out a build running on Ubuntu 11.04, and we're equally happy to report that it worked just as well as it did on EC2 and our internal VSphere testing cluster.

The video shows me deploying Wordpress, as I did the recent Komodo video, but also talks a bit about how we got Bugzilla running in the cloud.

Trackback URL for this post:

http://www.activestate.com/trackback/3267

Komodo IDE and Stackato: Deploying Apps and Accessing Data

troyt — Mon, 09 Jan 2012 22:49:38 +0000

I've been doing a lot of demos of Stackato lately and starting to incorporate the new Komodo IDE 7 client interface (currently RC). Without further ado, here's the screencast!

It was tough to decide what to show. Debugging applications running on the micro cloud is pretty cool but it's better demonstrated in a written tutorial (which I'll be writing up soon). Showing off the DB Explorer is a little more visually interesting.

DB Explorer, dbshell, and tunnel

As the screencast shows, it's extremely simple to connect to a locally running micro cloud from Komodo. That's because Komodo has integration with the 'stackato dbshell' command, which has been part of the client for a while longer than the newer 'tunnel' command. Komodo will have better hooks into 'tunnel' in future releases, but for now, it's fairly trivial to set up a connection to the database manually as shown.

What you'll need to follow along

If you'd like to try this all out for yourself, you'll need:

Stackato micro cloud VM: There's another screencast that shows how to get this set up. If your local system doesn't have the resources to run a VM, feel free to use the Stackato Sandbox instead.
Stackato client: Komodo uses this stand-alone executable in the background. You may need to do some minor client setup after download.
Komodo IDE 7: Download an RC build (or a trial of the release, depending on when you're reading this) and set the path to the 'stackato' executable in Komodo's Stackato preferences if it's not in your system $PATH.
Sample apps: We've put together a Github repo full of sample applications in a variety of languages.

Let us know what you think!

As always, we would like to hear your feedback on how this works for you, and what you would like to see in future releases. You can do that in the Stackato Feedback and Komodo Beta forums.

Trackback URL for this post:

http://www.activestate.com/trackback/3263

Webcast: Application Portability in the Cloud - PaaS Delivers Holy Grail

sheilar — Mon, 09 Jan 2012 19:35:26 +0000

Join us for this great upcoming webcast about cloud portability! Space is limited. Reserve your seat now.

Date: Wednesday, February 8, 2012
Time: 10:00 AM PST/ 1:00 PM EST

The ability to move applications--for instance, from a private to a public cloud, or from one hosting provider to another--is a holy grail of cloud computing. But with a little planning, application portability in the cloud can be attained.

Cloud-vendor solutions deliver cloud technologies, but at a price, and typically impose inflexible limitations, forcing IT leaders to commit to specific hypervisors, infrastructure, and APIs. That reliance on vendor-specific technology–coupled with application-specific code dependencies, configuration/provisioning requirements, and data-security mandates–can prevent organizations from realizing the idealized benefits of the cloud: reduced costs, greater efficiency, on-demand scaling, disaster recovery, and deployment flexibility.

So how do you ensure portability, avoid vendor-lock-in, and reap the rewards of the cloud?

The answer is clear: Implement a cloud strategy that includes a Platform-as-a-Service (PaaS) layer. PaaS provides the interoperability of application and infrastructure cloud services across cloud providers or hosting technology. And you realize the benefits of true cloud-application portability.

In this webinar, we’ll share insights and trends to help your organization avoid vendor lock-in and easily move applications from one cloud to another without major re-engineering of your application or deployment scripts.

Join us for a free and informative webinar where we will highlight:

Why Portability Matters in Private & Public Clouds
Understanding & Identifying Your Applications’ Cloud Dependencies
Emerging Trends & Issues effecting Cloud Application Portability
Creating a Cloud Strategy optimized for Portability
Choosing a Technology Stack for Portability
Implementing a Portable Cloud Strategy with PaaS

Sign up today to reserve your spot. Space is limited!

Trackback URL for this post:

http://www.activestate.com/trackback/3262

Bentframe: Developed with Komodo, deployed to Stackato

ericp — Thu, 05 Jan 2012 00:15:20 +0000

For years now I've been working on tools that help build and debug Rails applications, but other than the Candidate Buzz demo application, I haven't actually created many myself. I've started many apps as weekend projects over the years. I enjoy building tools to solve common problems I run into when working on these apps, and enjoy using those tools for future projects.

But when it came time to actually launch one, I would ask myself if it was worth the trouble of spending weekends and late nights trying to placate angry or confused users. Shouldn't I spent my off-work time doing something other than... work? So yet another project would be consigned to a remote corner of my repository.

Bentframe: Visualizing Bicycle/Car Collision Data with Google Maps

In early December word got out that the indefatigable David Eaves had managed to get ICBC, the provincial automobile insurance company in British Columbia, to release a dataset of collision reports involving a car and bicycle for the years 2006-2010. Each record in the data contained a year, month, and location as a latitude, longitude pair.

I figured I could whip up a visualization of this data relatively quickly, seeing how one of those web sites I never launched was a Google Map visualization I wrote in late 2007 that showed a database of foreclosed houses for sale in the U.S. I was struck by how large the database was: about 70,000 properties scattered over all fifty states, but didn't consider the economic implications. Had I not been focusing on the technical aspects of how to display tens of thousands of data points in a 600x300px window, and instead dipped into the financial press to learn about credit default swaps, short sales, and predicting the future, I might be writing this from a Caribbean island. But fortunately for fans of visualizations, I had focused on writing efficient clustering engines, and figured I could whip up something useful during the single-day hackathon.

Within the first hour I wrote a quick Ruby program to read in the data, fired up Komodo to create a new Rails app, copied over the relevant bits of the old "Closing Town" foreclosure code, and was back in Rails development (as opposed to deployment) mode.

Debugging the Initial App in Komodo

The 4-year-old foreclosure app didn't need much changing to handle the new data set. Change mentions of "properties" to "collisions". Change the default view from central Florida to Vancouver, BC.

By noon, I felt ready to go. I started the server, pointed Firefox at good old http://localhost:3000/spots, and found... an empty Google map of Vancouver. Komodo's DB Explorer showed the database was populated with over 5,000 locations, as expected. Log statements on the server side showed that it was reading the locations from the database, and returning a non-empty JSON packet. On the front end, Firebug was showing that my Ajax callbacks were firing, and creating markers at the correct locations. The problem was, they weren't showing up.

Meanwhile, my colleague for the day, Nathan Griffiths, had taken a different approach, pouring the raw CSV file into a new app at geocommons.com. While I was dealing with updating code from Rails 1.2 to 3, and pouring the usual combination of HTML, JavaScript, and CSS together into a coherent whole, Nathan had quickly mapped all the collisions, added city bike route data for the city of Vancouver to the mix, and quickly had built an impressive working app. The provincial minister for Labour, Citizens' Services and Open Government, Margaret MacDiarmid, had spent most of that Saturday morning at the hackathon talking to people. When she came by to see what we were working on, Nathan's app had locations marked as points, overlays, animations, and heat maps. I didn't think she'd be as impressed that I could debug Rails on the server and JavaScript on the client side at the same time, with a log window open as well. The minister was a family physician before she entered politics, not a retired Cobol programmer. I stayed quiet while Nathan demoed his app, wondering if I was reinventing yet another wheel.

It turns out that the markers weren't showing up because my code was using custom images provided by the authors of the Google Maps book I used in late 2007. Attempts to wget the images gave me silent 404s instead. I pulled out the code that was setting the custom images, reran the app, and my map filled with hundreds of the stock Google Map red pin images. Yay. Time to solve a few more problems, wondering if maybe I should take another look at using Geocommons instead.

But the rest went quickly. I found a source of images with digits so I could display the number of collisions in the marker, and used reverse geocoding to translate latitudes and longitudes back into street addresses.

I showed the app to a journalist, and she asked me what the top 10 spots were. I'm not displaying them yet, but I have the flexibility to add that. Vancouver added a couple of controversial bike lanes downtown during 2009 and 2010. Once I pour in the 2011 data, I hope to provide some charts that show how collision frequencies are dropping near those areas. That's why in 2012 we're still writing our own code -- to create something new without being constrained by existing tools.

Painless Deployment to Stackato

Unlike the foreclosure data app, word got out, and the aforementioned journalist was asking when my web site would be up. Four years ago there weren't any single-button deployment solutions. Today there are a few, including Stackato by ActiveState. Deploying Rails apps used to be a major pain point, and the subject of many conference sessions. Now it took one evening after work to deploy the app on Stackato.

New features of Rails 3 makes deploying on a system like Stackato easier. For example, I added a data_loader.rb file to the config/initializers directory that checked to see if the database was empty. If it was, the single line Rake::Task['db:seed'].invoke would run db/seeds.rb, which was nothing more than the program I used during the hackathon to populate the database from the CSV file. I added the CSV file to git, and the next time the app was restarted it rebuilt the database automatically.

Now that I was actually deploying the code, I had to make sure the code behaved differently depending on whether I was testing the code in development or test mode, or deploying it. For example, I was juggling two Google Map API keys, one for localhost, and one for the target URL. There was an easy fix for this, in config/environment.rb:

GOOGLE_MAPS_KEY = (Rails.env == "production" ?
                   [key for sandbox.activestate.com] :
                   [key for localhost]);

I pushed the app to the Stackato sandbox, tested it out manually and found it worked fine, notified the local cycling advocates, and watched the hits come in. The map helped me meet one of my goals, an increase in awareness that the designated routes were not 100% safe. In particular, one intersection, where the 10th Avenue bike route crossed Clark Drive, a 6-lane truck route through the city, was showing 16 collisions over the five-year period, easily one of the top ten intersections. Apparently people at city hall are discussing possible improvements.

Improving Performance

I was happy that I could release the app without any work at all, but was less pleased with the performance. When you zoomed out to a level that contained all of Vancouver, the server was taking up to 20 seconds to respond -- it was spending most of that time converting over 2,000 locations into about 200 locations that Google Maps could quickly render. This hadn't been as much of a factor with the foreclosure data, because there were few blocks that had more than one bank-owned house on them. But the ICBC data was revealing more than one collision along each block along designated routes and main streets. Quick profiling revealed that 10% of that time was spent converting rows from the data base into ActiveRecord objects, and most of the balance spent clustering those items.

I spent the next Saturday rewriting the clustering code in C. I had worked out the subtleties of the clustering algorithm in Ruby, so this was essentially a rewrite into C. I made sure the core code was independent of any scripting language API, and then wrote an interface layer between C and Ruby (so this library could be easily ported to any other language with a C interface). It went amazingly quickly, with benchmarking showing a drop in processing time of over 90%. The advantage of the C code, besides the speedup in executing machine code over interpreted Ruby bytecode, is that I knew the code's memory requirements in advance, allocated it all in one shot, freed it at the end, and could structure the code to minimize sloshing bits around. With Ruby that was out of my control. And writing a layer of C code to interface with Ruby is actually surprisingly fun.

I built a gem for the C code, added an entry for it in the app's Gemfile, ran bundle package (side note: use bundle install for gems registered with rubygems, bundle package if your app uses private gems), pushed the app to Stackato, and it worked much faster. As I zoomed out, the server was taking less than a second to process a large set of points. You can see the result at http://bentframe.sandbox.activestate.com.

Note that while I developed the C library on OSX, it didn't matter that I was deploying to Linux. Stackato compiled the gems for me during staging.

Handling Cars

Just in time for Christmas, I received an Open Office Calc .xmlx file from ICBC containing about 110,000 rows of data representing car collisions at intersections in the province for the same years, 2006 through 2010. It took about a day to modify the bike app to handle the slight differences in the car data, and the clustering C module was performing fine. However the numbers were larger, compared to the 5500 rows for the bike data. If I zoomed out long enough, the server would take over 20 seconds to respond. Benchmarking showed that most of that time was spent simply reading data from the database (using a pure SQL call to bypass Rails' ActiveRecord). But a call to determine how many rows matched a given query took no time, so I modified the JavaScript side to show a warning when there would be too much data to look at when you zoomed out too far, and left it at that. You can see the car crash map at http://carcrash.sandbox.activestate.com.

Trackback URL for this post:

http://www.activestate.com/trackback/3261

Private PaaS: More than Java for Implementation Success?

angieh — Tue, 03 Jan 2012 15:06:34 +0000

For the past few months, as we've been talking to large enterprises about Stackato and their private PaaS plans, some interesting IT strategies have emerged around the role of languages like Python, Ruby, and PHP play in the mix with Java.

As most people know, Java applications dominate the landscape in large enterprise. But as corporate IT departments are planning private PaaS implementations, what strategy are they taking regarding languages and data services?

Option 1: Java First

On the one hand, some are focusing first on Java to support the primary language amongst their developers. In these companies, Java is the primary language for legacy mission-critical applications. Often, other languages like Python and Perl were not used as much in legacy applications, or not viewed to be strategic languages for mission-critical languages. So in some large organizations, even if up to 25% of their applications use another language like Python or Perl, these organizations are implementing a PaaS for Java first with the idea of getting the majority at the first stage.

Option 2: Next-Gen Languages to Get Buy-In

However, some companies are taking a different strategy in order to get greater internal buy-in and adoption of their private PaaS. Even though most current applications may be written in Java, they feel that the younger generation of developers are using, or prefer to use, other languages like Python, Ruby, and PHP, and are demanding that IT support these options. Since the younger developers are also the ones that are more likely to try a new internal service like a PaaS for deploying and managing their applications, corporate IT groups are focusing on getting their adoption first. And the Java crowd will follow. Likewise, in addition to supporting traditional databases like Oracle and DB2, these corporate IT groups want to be able to offer next-generation data services like MongoDB to show the flexibility and options that an internal PaaS can provide. This puts corporate IT in a position of giving the customer (internal developers) what they want in order to satisfy their demands today, get early buy-in, and set-up for success tomorrow.

So which strategy are you taking, and why? Share your comments!

Trackback URL for this post:

http://www.activestate.com/trackback/3260

Three ways Stackato makes web development easier

tarag — Tue, 20 Dec 2011 23:44:53 +0000

I'm going to show you three ways that Stackato makes developing, testing and deploying web applications easier.

1. Stackato solves the "It worked on my box" problem

If you're running a Stackato cluster in production and staging and a Stackato Micro Cloud locally for development, you'll have a consistent environment across the board. What works on the Stackato Micro Cloud will work on a Stackato cluster.

2. Easy cross-browser and cross-device testing from your development environment

Run your local Stackato MicroCloud in Bridged Networking mode, and you'll be able to access it from any device on your network.You can even test your responsive layouts on your phone or tablet. Find a bug in your CSS? Fix it locally, push it to your development microcloud and reload the page on your device.

Sample app in Chrome:

And the same sample app in mobile Safari:

(Note: you'll need to change the hostname to something other than stackato.local if there are other folks running a microcloud on your network to avoid collisions.)

3. Share your changes without staging them

Bridged networking provides another benefit: you can send your team members a link to an app running on your local Stackato Micro Cloud instead of pushing your changes to staging. This can be helpful when you want feedback on a front-end feature that isn't ready for everyone to see.

Ready to try Stackato? Get the free Stackato Micro Cloud now.

Trackback URL for this post:

http://www.activestate.com/trackback/3259

Cloud Application Portability: The Holy Grail of Cloud Computing

dianem — Tue, 20 Dec 2011 21:09:43 +0000

Portable application deployment to any cloud hosting provider is the Holy Grail of cloud computing. When cloud portability depends upon the supplier creating the cloud service, the cloud provider, or the hosting technology, applications have provisioning and configuration dependencies that create vendor lock-in, preventing organizations from realizing cloud benefits like reduced costs and deployment flexibility.

Want to migrate some or all of your private cloud to a community or public cloud? You need cloud portability. Want cloud portability now? You need private PaaS.

Conflicting (or absent) cloud-interoperability standards complicate application portability

Cloud service providers frequently toss around terms like “service,” “application composition,” “orchestration,” and “platform.” But those terms often have differing meanings depending on the source. It should come as no surprise that re-deploying your application to a different cloud will require some reengineering, particularly to ensure configuration of required cloud services like databases or web servers. Unclear or conflicting terminology can make it difficult to switch between cloud platforms, and can obscure a cloud hosting provider’s basic capabilities (and/or limitations). When an application is pushed to a new cloud, interdependencies must be identified and reevaluated, with details and values reentered and checked at each stage.

Understanding your application`s cloud requirements

There are no formal standards describing cloud interoperability—specifically, your application’s migration requirements. There are some standard practices (like adding a “requirements.txt” file containing modules and packages lists to the app root), but they do not offer enough detail about data services and cloud-configuration dependencies that your app will require to scale securely and run on the (new) cloud. Several cloud-orchestration and scripting approaches—such as Right Scale, Enstratus, Dell Crowbar, Puppet, Chef—ease the burden of repeat deployment to the same cloud, but no equivalent models exist for porting an application to a new cloud model.

Standards are emerging slowly

In lieu of fully-defined cloud-computing migration standards, some vendors make their APIs public, and push their own implementation models. These vendors understandably act in their own self-interest, creating vendor lock-in to drive revenue and market share in the hope their implementation becomes a de facto standard.

However, cloud-portability standardization discussions are starting to happen. OASIS has recently brought together enterprise cloud vendors to form the TOSCA technical committee, an org tasked with defining application and infrastructure cloud services interoperability standards, relationships between parts of the service, and specific service operational behavior (e.g., deploy, patch, shutdown). But getting industry competitors (TOSCA participants include heavy lifters like IBM, CA, SAP, Citrix, RedHat, and ActiveState) to agree on interoperability standards is a daunting—and laborious—process, and market adoption is not guaranteed.

Want cloud portability? Only Platform-as-a-Service delivers it now

Platform-as-a-Services (PaaS) solutions like ActiveState Stackato can be deployed on multiple hypervisors or on different flavors of IaaS, and offer another way forward to cloud portability. A PaaS provides a computing platform and a solution stack as a service. In the three-tiered cloud-computing model, the PaaS layer lies between the SaaS and the IaaS layers.

A PaaS such as Stackato can be deployed on either a Public or Private Cloud running on VMWare’s vSphere, Amazon’s EC2, or OpenStack. Stackato provides a layer of abstraction on top of the IaaS layer.

Applications deployed on a Stackato-enabled cloud platform can be moved to a different cloud model easily and painlessly, with little to no reengineering. This keeps your application (and your business) from being locked into a single cloud-hosting provider, and gives you the flexibility of choosing the right provider to meet your organization’s needs.

How does Stackato’s cloud-portability approach work?

Stackato automatically reads your application’s top-level “requirements.txt” file, and installs any packages you've defined—Handling dependencies is completely transparent during the staging process.

Stackato goes one step further and encapsulates your application's cloud configuration. Your application’s cloud configuration is persisted using YAML in the top-level “stackato.yml“ file. The “stackato.yml“ file holds various installation details for your app, removing the need to enter those details each time the app is pushed, and ensuring values are the same each time.

When you use Stackato’s database services, your application’s code uses connection details provided as environment variables. These environment variables give your application the flexibility to point to database services wherever they reside.

Deploying Stackato as the PaaS layer on your choice of cloud enables portable deployment to any Stackato-enabled cloud. Stackato’s approach to application deployment and wealth of enterprise-class languages, frameworks, and services ease the migration of your existing applications to the cloud, and ensure that your organization can take advantage of lower costs as the competition among cloud hosting providers ramps up in the coming year and drives down pricing.

Platform-as-a-Service may be not the holy grail of cloud application portablity, but the PaaS's ability to abstract away the differences between cloud computing infrastructure providers and an enterprise solution stack makes it easier to port applications between cloud platforms, enables more-flexible application management, and helps you avoid vendor lock-in (today, not tomorrow).

Trackback URL for this post:

http://www.activestate.com/trackback/3253

Running Bugzilla in the Cloud

jand — Thu, 15 Dec 2011 00:26:29 +0000

Bugzilla is the most popular open source bug tracker, and the one we use at ActiveState for both public and private issue tracking. As such, and in the interest of dogfooding our own software, we wanted to see what it would take to deploy it to our Stackato cloud platform.

With a little investigation and reconfiguration, and with the use of Stackato's post-staging hooks, I was able to get a functional instance of Bugzilla running on Stackato.

Getting Bugzilla 4.0.2 talking PSGI on ActivePerl 5.14.2

I started off with the latest stable release of Bugzilla, which is version 4.0.2 right now. Testing that version locally with ActivePerl 5.14.2 showed that it had a conflict with the latest release of version.pm. Fortunately, this is a known issue and already has a patch in the development branch that can be applied to 4.0.2 as well.

Bugzilla is a bunch of CGI scripts, whereas Stackato prefers to run applications via the PSGI protocol. CGI scripts can be wrapped with the CGI::Emulate::PSGI module, but before attempting to do so on my own, I did a quick Google search and fortunately found that one of the Bugzilla core developers has already started this work in order too ease deployment of Bugzilla under FastCGI. The patch needed some manual tweaking to get it to apply to Bugzilla 4.0.2, but once that was done I had a version of Bugzilla that I could run under ActivePerl with 'plackup', the development PSGI server included in the Plack distribution.

Deploying Bugzilla to Stackato

Now it was time to try to deploy Bugzilla to Stackato. The first step is to create a requirements.txt file that lists all the prerequisite modules for Bugzilla from the documentation:

CGI
Date::Format
DateTime
DateTime::TimeZone
DBI
DBD::mysql
Digest::SHA
Email::Send
Email::MIME
Template::
URI

Most of them are already included in ActivePerl, but it doesn't hurt to list them again. For PSGI and CGI::Emulate::PSGI support I needed to add a couple more modules:

Plack::App::WrapCGI
Plack::App::URLMap
CGI::Emulate::PSGI
CGI::Compile

Configuring Bugzilla requires access to a database, so I decided to do this all with a little Perl script as a post-staging hook. The stackato.yml file that describes the application looks like this:

name: bugzilla

# Need to specify the framework explicitly because we don't have
# an app.psgi file.  It is called bugzilla/app.psgi, and it will only
# be created during post-staging.
framework:
  type: perl

processes:
  web: (cd bugzilla && $PROCESSES_WEB)

hooks:
  post-staging: perl setup.pl
  # for debugging purposes only
  pre-running: env

services:
  mysql: mysql-bugzilla

mem: 128M

Notice the 'processes: web:' line. This is used to customize the command that launches the application, in this case to run the default uwsgi command (stored in the $PROCESSES_WEB variable) in the 'bugzilla' directory rather than the application root.

The setup.pl script basically does the following steps:

Download Bugzilla 4.0.2. Bugzilla is not distributed via CPAN, so it cannot be installed by the requirements.txt mechanism but needs to be downloaded and unpacked explicitly.
Apply patches for the version.pm issue and PSGI support.
Create an automatic configuration file containing the database credentials as well as default admin information.
Run checksetup.pl multiple times to create the initial configuration files and set up the database schema.

I've factored out a myconfig.pl file that allows end-user customization of the initial setup without having to edit setup.pl itself:

# Sample Bugzilla configuration
%answer = (
    ADMIN_EMAIL    => 'admin@stackato.local',
    ADMIN_PASSWORD => 'changeme',
    ADMIN_REALNAME => 'Sample Admin',
    mailfrom       => "bugzilla-daemon\@stackato.local",
    smtpserver     => "stackato.local",
    urlbase        => 'http://bugzilla.stackato.local',
);

It is not strictly necessary to change this file at all, as the settings can also be modified through the Bugzilla web interface. Just make sure you rename the admin user and change the password before making the app accessible to the outside world!

Working around one more obstacle

Unfortunately this setup doesn't quite work yet: It is not possible to log into the app with the administrator name and password. Further debugging showed that none of the HTTP POST requests would deliver the actual query data. This turns out to be an incompatibility between the CGI::Emulate::PSGI module and the uWSGI server software that Stackato uses by default to serve up PSGI applications.

I've filed this as a bug against the CGI::Emulate::PSGI module and changed the custom application start command in stackato.yml to use plackup instead of the default uwsgi command:

processes:
  web: (cd bugzilla && plackup --port $VCAP_APP_PORT)

As with the post-staging hooks and the run command, Stackato is the only Cloud Foundry-based platform that allows this kind of customization.

Enough talking, let's just do it

All files needed to deploy Bugzilla to your Stackato cloud are available from github. The commands needed to deploy your own instance are:

$ git clone git@github.com:ActiveState/stackato-samples.git
$ cd stackato-samples/perl/bugzilla
  (update myconfig.pl with your email settings)
$ stackato push --no-prompt

What's next?

There are a number of additional prerequisites to enable optional functionality. They can be added to requirements.txt to enable the additional features. XMLRPC and JSONRPC have already been identified as being problematic under PSGI with FastCGI; I didn't check if they work with the regular setup.

Several optional features also require a background process to periodically collect stats or process inbound email. This functionality can be simulated by running the required commands externally using the `stackato run` command for now. Stackato will soon support background processes and scheduling on its own.

Trackback URL for this post:

http://www.activestate.com/trackback/3250

We're Hiring a Developer Community Manager!

dianem — Mon, 12 Dec 2011 22:41:48 +0000

ActiveState is searching for a Developer Community Manager to support our growing Stackato Community and lead our Stackato Developer Community Relations initiatives at ActiveState. The position will be based in our Vancouver office. The application deadline is January 31st. If you are excited about this possibility, or know someone who would be, please check out the job listing and feel free to spread the word. Thanks!

Trackback URL for this post:

http://www.activestate.com/trackback/3249

Application Portablity in the Cloud: Standards & OASIS

dianem — Wed, 07 Dec 2011 00:03:23 +0000

The ability to migrate applications from one cloud to another and give customers a way to avoid being locked in to a single cloud provider is key to a healthy cloud eco-system.

At ActiveState, we believe that standards can and will help, which is why we're pleased to announce our support for the OASIS (Advancing Open Standards for the Information Society) TOSCA initiatives. By increasing service and application portability in a vendor-neutral ecosystem, the OASIS TOSCA (Topology and Orchestration Specification for Cloud Applications) Technical Committee will be working to enhance the portability of cloud applications and services, ensure smoother migration of existing applications to the cloud and more consumer choice.

While we continue to grow ActiveState Stackato, every day we are seeing more and more applications and their workloads deployed to clouds, taking advantage of the promise of cloud computing's elasticty and usage-based cost structure. And this is a wonderful thing.

However, with each cloud provider choosing a specific hypervisor, cloud infrastrucure and their own set of resources, it becomes difficult to move from one cloud to another. Moving your application from one to the other means reinstalling, re-testing, and, in some cases, reengineering the application, because the underlying cloud infrastructures and their APIs aren't always compatible. Even moving between two providers using the same infrastructure provisioning layer such as OpenStack can be difficult because the application will be looking for databases, middleware and other services that are specific to each cloud.

We at ActiveState believe that Platform-as-a-Service (PaaS) will play an important role in enabling the portability of cloud applications and services. Standards such as those under discussion on the TOSCA technical committee will enable the interoperable description of application and infrastructure cloud services, the relationships between parts of the service, and the operational behavior of these services independent of the supplier creating the service, and any particular cloud provider or hosting technology.

ActiveState is please to bring our experiences with ActiveState Stackato, the private PaaS solution, to add to the conversation and work alongside vendores such as CA, IBM, EMC, Citrix, Red Hat and SAP AG to ensure portability of cloud applications and services.

Try Stackato today and get started creating your own private PaaS, or read the full press release.

Trackback URL for this post:

http://www.activestate.com/trackback/3247

New Stackato Client: Tunneling and HTTPS

jeffh — Fri, 02 Dec 2011 00:03:03 +0000

I'm very happy to provide an update to Troy's post from last week about connecting to database services in stackato. In that post, he said:

Since we are maintaining API compatibility with Cloud Foundry, we'll be adding an equivalent 'stackato tunnel' command to emulate 'vmc tunnel'.

Well, we've just released a new version of the stackato client which implements the 'tunnel' command. In doing this, we evaluated how the client communicated and otherwise operated, and found a few places we felt should be improved.

Password? What password?

For example, the vmc client requires you specify a password when pushing the tunnel initially, to authenticate future tunnel connections. However, if you subsequently type an incorrect password it doesn't deny access. It interprets that as if the tunnel hadn't been deployed, deletes any existing tunnel, and redeploys using the new password.

$ vmc tunnel
1: mysql-d05c5
Which service to tunnel to?: 1
Password: *****
Redeploying tunnel application 'caldecott'.
Uploading Application:
  Checking for available resources: OK
  Packing application: OK
  Uploading (1K): OK   
Push Status: OK
Binding Service [mysql-d05c5]: OK
...

Not a critical security hole, since the vmc user has been previously authenticated, but it's likely unintentional behavior. The 'stackato tunnel' command doesn't do this:

$ stackato tunnel
1. postgresql-gtd
Which service to tunnel to: 1
Password: *****
Getting tunnel url: OK, at https://tunnel-4r0b6.sandbox.activestate.com
Error: Bad password, authentication to tunnel failed

HTTPS by default...

Another aspect is that vmc tunnel operates by default over HTTP, and we have moved that to HTTPS. We have now made this the default protocol for targets, to ensure more secure operation. We went one step further and now require an explicit '--allow-http' to make an insecure HTTP connection.

$ stackato target api.sandbox.activestate.com
Successfully targeted to [https://api.sandbox.activestate.com]

and other conveniences...

On the server side, we generalized the application name 'caldecott' to just 'tunnel', as well as providing a '--url' option when first creating a tunnel to specify the mapped url. This is more for convenience, but is handy when you want to fix a name in cases where you have inflexible DNS handling.

An option to specify the password on the command line was added to support batch mode testing, as we have a growing code base of tests for our system that leverage the client and REST APIs directly.

So if you're already in the Stackato Beta program, please download the new client and try out the improvements. If you're not in the program yet, by all means sign up. We'd love to get your feedback.

Trackback URL for this post:

http://www.activestate.com/trackback/3246

Using YAML to Drive App Deployment

troyt — Thu, 01 Dec 2011 00:42:33 +0000

There's a very handy but sometimes overlooked feature of the stackato client that we'd like to let you know more about. I often have to remind myself to point it out when I'm doing demos for people, as it's something that I've quickly come to take for granted. It's the stackato.yml config file.

It's a small addition to base vmc functionality, but it really helps with consistent, automated application deployment. You see, the stackato and vmc command-line clients are normally interactive. You are prompted for confirmation of the URL...

$ vmc push wordpress
Would you like to deploy from the current directory? [Yn]: 
Application Deployed URL [wordpress.stackato.local]:

...confirmation of the detected runtime or framework...

Detected a PHP Application, is this correct? [Yn]:

...memory allocation...

Memory Reservation (64M, 128M, 256M, 512M) [128M]: 64M

You are asked if you want to create a new data service or bind to a new one, and you are prompted for a name...

Would you like to bind any services to 'wordpress'? [yN]: y
Would you like to use an existing provisioned service? [yN]: n
The following system services are available
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
Please select one you wish to provision: 2
Specify the name of the service [mysql-33b93]: mysql-wordpress
Creating Service: OK
...

Both clients have a --no-prompt option for the 'push' command, but to specify non-default values with vmc you have to add them as command line arguments. Wouldn't it be handy to specify these things in a config file?

Since a lot of Stackato configuration files on the server side are done in YAML, we chose that as the data serialization format. We made the stackato client check for a 'stackato.yml' file and use the options specified there. This bypasses all of the prompting during a 'push' or 'update' and gives you a place to see all of your configuration at a glance in a human readable format.

$ cat stackato.yml 
name: mywordpress
mem: 64M
instances: 1
services:
    mysql: mysql-wordpress
hooks:
  post-staging:
    - sh salt.sh

When it comes time to push or update the app, you can use the '-n' or '--no-prompt' switch:

$ stackato push -n
Application Url: mywordpress.stackato.local
Creating Application [mywordpress]: OK
Creating mysql Service [mysql-wordpress]: OK
Binding Service [mysql-wordpress]: OK
Uploading Application [mywordpress]:
  Ignoring path [.git]
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (16K): 100% OK
Push Status: OK
Staging Application [mywordpress]: OK                                             
Starting Application [mywordpress]: OK

What else can you do with stackato.yml?

In addition to driving the basic push operations, stackato.yml can be used for additional server-side setup operations, as you might do with the 'stackato run' and 'stackato env' commands.

You can specify the commands in a 'post-staging' or 'pre-running' hooks section. For example, the Wordpress demo suggests creating a salt.php file before pushing. I've wrapped those commands in a salt.sh bash script (see above) and run it as a post-staging hook instead.

You can set environment variables in an 'env' section. For example, if you want to set up New Relic monitoring for a Django app, file you can set the environment variables in stackato.yml instead of creating a separate newrelic.inifile:

env:
  DJANGO_SETTINGS_MODULE: settings
  NEW_RELIC_LICENSE_KEY: gg070c96385e870xxthisisnotarealkey5a12xx
  NEW_RELIC_APP_NAME: DJANGO GTD
  NEW_RELIC_LOG: stderr

You can even specify a different web server process to run your application, as in this example from the Starman demo:

processes:
  web: starman --listen $VCAP_APP_HOST:$VCAP_APP_PORT --workers 2 app.psgi

Trackback URL for this post:

http://www.activestate.com/trackback/3242

Packaging Perl for Windows with PDK - Patriot NG

troyt — Thu, 24 Nov 2011 22:10:46 +0000

We love fan mail, especially when it's on Twitter where everyone can see it. Yago Jesus, a PDK and Komodo user, let us know how much he enjoyed using our software, and pointed out some software he's created for intrusion detection on Windows.

Since we haven't made much noise lately about the Perl Dev Kit and all its handy tools, let's have a look at Patriot NG and how PerlApp can be used to deploy seriously useful Windows software.

But first, some back-story.

Perl on Windows

ActiveState was founded on the premise that dynamic languages like Perl are useful tools no matter what platform you're running on. Porting Perl to Windows was one of our first steps towards delivering "Programming for the People", and indeed ActivePerl has introduced Perl to millions of Windows users over the years. These users can take advantage of the wealth of freely available modules from CPAN and build their own applications.

However, when it comes time to deliver these applications to end users, they will need an interpreter installed on their system to run your code. It's a lot to ask of someone to install ActivePerl when they only want to run your 20 line script, and won't be doing any programming themselves. This is less of a problem on Linux and UNIX-based systems because Perl is often installed by default, but on Windows it's an annoying extra step.

The PDK deployment tools solve this problem by packaging Perl programs as stand-alone executables, Windows services, ActiveX components or .NET assemblies. Jago used the PDK to package a real-time system monitoring utility built in Perl.

Patriot NG

Patriot NG is an intrusion detection system for Windows which monitors registry key changes, new service installation, file changes and a number of other system events.

I asked Yago a few questions about his software:

Why did you chose Perl for this application?

Well, probably many people think that Perl is a language for parsing files or web-CGIs only, but Perl is a great language for writing Windows applications because it has many Windows-specific libraries for Windows internals. I recommend looking at the Win32-* modules in CPAN. Pretty impressive.

Which CPAN modules were most useful to you?

I'm using a lot of the Win32 modules, for example:

Win32
Win32::TieRegistry
Win32::TaskScheduler
Win32::Lanman
Win32::API::Prototype
Win32::IPConfig
Net::PcapUtils (to read and parse network packets) and
Prima for the graphic toolkit, because it has a very very good IDE that allows you to do interfaces in a few minutes.

Which of the PDK tools did you use to package Patriot NG?

I'm using PerlApp and PerlTray.

I noticed the software installs a few different executables. How do they work together?

The architecture is simple. A main .exe called 'patriot.exe' runs in the tray. It starts and stops, updates NIDS rules and launches the configurator. Also there are three programs, one for general activities, one for network and another for file monitoring. Moreover, there are two graphic programs, one for configurate settings and other for draw and process alerts.

I have come to Windows from Unix, so I made Patriot modular, the Unix way.

Try it out

The Perl Dev Kit has a 21 day trial license, so you can try out these deployment tools on your own code, or with the some of the sample Perl programs. If you find it useful, now is an excellent time to buy. Our Thanksgiving Weekend Sale is running through Nov. 28. You can save $50 on single licenses (or $250 off 5-packs) of Perl Dev Kit, Tcl Dev Kit, or Komodo IDE.

Trackback URL for this post:

http://www.activestate.com/trackback/3236

Thanksgiving Weekend Sale! Save $50 or More on Komodo IDE, Perl and Tcl Dev Tools

mikek — Tue, 22 Nov 2011 18:03:46 +0000

Like most people, we here at ActiveState like to loosen our belts around the holidays. This year, with our Sales Director away, we've decided to let loose with a good old-fashioned Thanksgiving / Black Friday / Cyber Whatever dev tools sale!

From Thursday through Monday (Nov. 24 - 28):

$50 off Komodo IDE:
5-packs $~~$1615~~ $1365 / single license from ~~$295~~ $245
$50 off Perl Dev Kit (PDK) and Tcl Dev Kit (TDK):
5-packs ~~$1615~~ $1365 / single license from ~~$295~~ $245
$100 off ActivePerl Pro Studio and ActiveTcl Pro Studio:
5-packs ~~$2390~~ $1890 / single license ~~$595~~ $495

Plus, if you buy Komodo IDE with Upgrades and Support, you'll get access to our support team and Komodo IDE 7 on release.

So, have a productive long weekend or gear up for the week ahead with the professional IDE for Python, Perl, PHP, Ruby, Tcl, Java and more, and the essential toolkits for Perl and Tcl.

Visit our store and save.

Happy long weekend!

Trackback URL for this post:

http://www.activestate.com/trackback/3235

Little Clouds for Free: ActiveState commits to a free Stackato Micro Cloud

dianem — Tue, 22 Nov 2011 14:52:20 +0000

ActiveState is proud to announce that Stackato will continue to be free of charge for you to use as a single node. Stackato will ship with a "micro cloud" license option. This means that you can always have your very own private Platform-as-a-Service (PaaS) for free.

During the beta period for Stackato, we were asked to clarify the usage policy for micro cloud usage when Stackato has its general availability (GA) release. We want to assure the developer community that Stackato will remain free for use as a single node for non-commercial or internal commercial use.

At ActiveState, our products all leverage community-driven open source projects, and Stackato is yet another example of our commitment to support the community. Stackato is ActiveState's commercial distribution of the open source Cloud Foundry PaaS project. We've extended it with additional languages, services and features, and hardened it for enterprise use with enhanced security.

We want to ensure that developers have the freedom to continue to develop, test and deploy using Stackato. You can expect a license similar to the Community Edition licenses for ActivePerl, ActivePython, and ActiveTcl for Stackato Micro Cloud.

Our Stackato VM download boots as a running Stackato server instance. It is able to function as a stand-alone server with all of the necessary components (router, cloud controller, DEA, etc.) for staging and running applications. This single Stackato server instance is considered a "single node" of Stackato and is commonly referred to as a "micro cloud".

After GA, the Stackato micro cloud will continue to be available to download, install and use for free for your personal projects and internal deployments.

Technical support will not be included with the Stackato micro cloud license, however developers can still participate in our community forums and access all documentation. If you require guaranteed production-level support, plan to deploy Stackato in a multi-node deployment, or wish to deploy Stackato for external commercial purposes, we will be offering different licensing and support packages with the GA release.

As part of our commitment to the developer community, we will continue to develop a hypervisor- and IaaS-agnostic PaaS platform, helping remove vendor lock-in and enabling true portability of your applications across the cloud.

For more information please contact us directly or get started with the Stackato micro cloud today!

Trackback URL for this post:

http://www.activestate.com/trackback/3233

Connecting to Database Services in Stackato

troyt — Mon, 21 Nov 2011 19:34:59 +0000

I was going to write a fairly straightforward post about Stackato's 'dbshell' command, demonstrating how you can use it to access the databases provisioned by Stackato. However, a few days ago Cloud Foundry blogged about Caldecott, demonstrating how to connect to database services on cloudfoundry.com, so the scope of this article has changed a bit.

We were eagerly awaiting news on Caldecott and were watching the Cloud Foundry vcap repo like hawks to see when it would land in the open source project. It turns out we were looking in the wrong place. All the magic is in the client, and I'm happy to report that Stackato is compatible with this new access method without requiring any server-side changes.

Stackato's 'dbshell'

Before I get into this new way of doing things, let's look at Stackato's 'dbshell' command.

To connect to one of your database services running on Stackato, you simply run 'stackato dbshell [appname]'. It opens a database port on the micro cloud or database service VM and returns either the connection credentials or automatically launches a local database client to connect with those credentials.

For example, I have a MySQL database connected to Spring application on my local micro cloud. To get the credentials:

$ stackato dbshell --json ttspring
{
    "args"  : ["-h","172.16.34.155","--port","3306","-pps7wxzfr8aSPc","-u","uJIBwrW0E8jbi","d8f4cd7772c8e4x90af7bc216310a253d"],
    "env"   : {},
    "shell" : "mysql"
}

This returns a JSON object with everything I need to connect using an external MySQL client, like the DB Explorer in Komodo IDE.

In fact, this is what Komodo 7 (Beta) does behind the scenes to automatically configure the database connection with "View Database in DB Explorer" in the Stackato interface:

If you want to connect directly with a command-line database client, you can do that too:

$ stackato dbshell ttspring
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 644
Server version: 5.1.41-3ubuntu12.10 (Ubuntu)

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> show tables;
+---------------------------------------------+
| Tables_in_d8f4cd7772c8e4b90af7bc216310a253d |
+---------------------------------------------+
| current_states                              |
+---------------------------------------------+
1 row in set (0.00 sec)

mysql>

This is great if you're working with the micro cloud, or if you are connected to a Stackato cluster where the database service VM allows external database connections. But if you are trying to connect to an installation which is more locked down, which doesn't allow you to route directly to the data service VM(s), you can't connect.

The Cloud Foundry team has covered this scenario in their implementation by using TCP over HTTP tunneling, so let's take a look at how they've done that.

Caldecott: VMC's tunnel

Instead of making changes in the VCAP codebase (the server-side parts of the system), Cloud Foundry has bundled a small Ruby Sinatra appplication in the vmc client which is automatically pushed to the targeted system when you run 'vmc tunnel'. The app, in conjunction with the client, creates a tunnel from the data service to the local machine using the Caldecott gem. You can see this app by running 'vmc apps' once the tunnel has been created, and you can even check out the code with 'vmc files' if you want to see how it works.

The good news for Stackato users is that this works just as well on a Stackato PaaS as it does on cloudfoundry.com, you just have to use the vmc client (vmc 0.3.14.beta.4) instead of the stackato client to take advantage of it.

Next steps for Stackato

Obviously, we don't expect Stackato users to keep switching back and forth between 'vmc' and 'stackato' to get their work done. Since we are maintaining API compatibility with Cloud Foundry, we'll be adding an equivalent 'stackato tunnel' command to emulate 'vmc tunnel'. We might implement it slightly differently (perhaps a lightweight Node.js application instead of Sinatra, perhaps offering HTTPS rather than just HTTP), but the goal is to ensure maximum portability and compatibility between Stackato or Cloud Foundry.

Trackback URL for this post:

http://www.activestate.com/trackback/3232

Security in the cloud with Stackato and LXC

daniilk — Thu, 17 Nov 2011 19:48:47 +0000

Stackato, our private PaaS platform, makes heavy use of LXC (Linux Containers) to provide a secure, isolated environment for your apps in the cloud. In the age where data security is vital and business-damaging hacks are commonplace, it's becoming increasingly more important to properly secure web applications from malicious attacks.

Each of your apps running with Stackato gets its own partitioned space on your server. What does this mean? From the perspective of your web app, all that is visible of the server is your app's files and processes, and nothing else. The apps can't access other apps running on the same server; can't tamper with your server's configuration; can't mess with hardware, install rogue software, stop other processes, or hack your mail server to send spam to your colleagues (or worse, users!). You get the picture. Stackato ensures that each app has access to everything it needs to operate, and nothing else. Naturally, this means that if someone does manage to hack your app, they won't get far.

At the same time, the LXC infrastructure ensures that all of your apps get a fair share of CPU, and that no one app can grab the entire processor for itself. Also, to prevent an app from going rogue and taking up all of your server's RAM, the Stackato client allows you to set a RAM allocation per-app, ensuring that your server (and your other apps) stay running even if one app is misconfigured. Stackato uses LXC to ensure that no app will go over the RAM limit you set.

(If you're interested in the low-level details of how LXC allows you to keep tight control over your apps, I recommend our article exploring the technical side of setting up LXC, specifically the "Setting resource limits" section.)

Deploy your apps with Stackato and rest easy knowing that your servers and apps are protected from both malicious activity as well as accidental resource-hogging bugs that would otherwise cripple your servers and potentially cause costly downtime and user frustration.

Trackback URL for this post:

http://www.activestate.com/trackback/3231

CloudCamp Vancouver Round-up!

dianem — Wed, 16 Nov 2011 23:27:31 +0000

I was very honored to be invited to present a lighting talk at this week's CloudCamp Vancouver. CloudCamp's unconference phenomena landed in Vancouver this week at the Vancouver Convention Centre as part of the Microsoft's Tech Days Roadshow who sponsored the room and the coffee.

Attendees came in all shapes and sizes, from large telcos (Telus) to mobile web app developers (Dynamic Leap) and a number of Government agencies including my hometown's District of Sechelt's IT manager !

Patrick Pushor of Cloud Chronicle and Jenny Yang of Metafor Software hosted the event and tutored us local cloud affectionados in the ways of unconferences and unpanels. If you haven't been to a Cloud Camp before it can sound a bit unstructured at first, no pre-planned tracks or sessions, just a few preliminary introductory words from the organizers, followed by 4-5 lighting talks (5 minutes long) on a range of topics meant to spur discussion and set the stage for for open discussions that follow setting the agenda as you go.

I was corralled into reprising my recent Cloud Expo 45 minute presentation on US Patriot Act & it's Implications for Cloud Computing as a 5 minute lighting talk - which made for some speedy slide transistions (I'm giving a webinar on the topic on 12/01/11 in which I will go into more detail and promise to talk slower).

The other lighting talks ranged from an overview of the Microsoft Azure Cloud by Jamie Wakeam, Microsoft Canada, Lynn Sutherland on Canarie a very Canadian cloud network, an intriguing Cloud-based de-Centralized Storage by Margaret Dawson of Symform and a thoughtful presentation by Simon Le Pine on taking a more holistic, all-encompassing approach to risk management in the cloud.

It was a very good turnout for a Monday night, and the break out discussions left me wanting more time, and looking forward to the next CloudCamp Vancouver event!

For more information: http://cloudcamp.org/vancouver

Trackback URL for this post:

http://www.activestate.com/trackback/3229

Setting Up the New Relic Python Agent on Stackato

troyt — Tue, 15 Nov 2011 23:20:28 +0000

Guest post: Graham Dumpleton, author of mod_wsgi for Apache and software developer at New Relic

With the recent official release of the Python agent for New Relic (our application monitoring partner) we're featuring this guest post by Graham Dumpleton. I had the pleasure of working with Graham as we tested the Python agent and the New Relic service with Stackato in recent weeks. --TT

Back in January, I could see that it was going to be an important year for Python web hosting. A number of new Python web hosting services had sprung up promising an easier way of pushing up Python web applications for hosting.

At PyCon US in March I got wind that ActiveState was looking into this area as well. Based on what I was hearing it was clear that they were approaching it in a different way, but I had no real idea what they were up to. I had thought they might just be another option amongst the public PaaS offerings. When they ﬁnally revealed Stackato it was actually a pleasant surprise and not what I had been expecting.

As I looked more closely at Stackato and actually worked out how it differed, the solution grew on me very quickly.

The primary reason Stackato was appealing was that when it comes to web application deployment I like to feel that I am in control of the environment. I am also very wary of shared hosting environments, my work on Apache and mod_wsgi making me appreciate the security issues with that style of hosting.

At the same time I can be quite lazy and don't like doing all the hard work of building a complete system from scratch each time one is required. The idea of a prepackaged private PaaS environment where I could decide where it got deployed was thus very enticing. Especially since I could also deploy it to my local system as well, allowing me to deploy during development to the same environment using the same mechanisms as I would when moving to a production setting.

The concept of a private PaaS also held interest due to my residing in Australia. The problem is that the legal and regulatory regime in Australia is such that if you were doing web application development for government organizations, you would in many cases be prohibited from deploying a web site you had developed for them outside of Australia. This would immediately rule out using any of the main Python web hosting services available as they are located overseas.

Even large private corporations here in Australia can be hesitant about overseas hosting, sometimes for legal and regulatory reasons, but also simply because of the preference for hosting the web site locally, be that internally to the organization or with a local web hosting service. Australia doesn't exactly have the best pipes out to the rest of the Internet and as a result you don't want to be running a high trafﬁc site on the other side of the world for a local market.

As far as ease of deployment, local Python web hosting services still have some way to go though. There are however local companies which offer to host any virtual machine image you want as long as you supply it. Stackato therefore provided the best of both worlds. An easy way of deploying your web applications and an ability to deploy it where you wanted. So Stackato had certainly got my interest personally. I have as a result obviously been keen to try out the New Relic Python agent with Stackato and ensure that it works as seamlessly as possible. The remainder of this post describes doing just that, providing a mini tutorial on how to get started with the New Relic Python agent on Stackato.

Step 1: Getting your application working.

The ﬁrst step is to get your Python WSGI application working on Stackato. ActiveState provides documentation and a set of samples on github to get you going.

The steps outlined here assume that you are relying on the default WSGI hosting mechanism provided by Stackato. This is using uWSGI as the WSGI hosting container. Further, the speciﬁc examples provided here are based on using the 'django-gtd' sample application provided by ActiveState on github. Where 'gtd' is used as the application name argument to the 'stackato' command, replace this with the name of your application.

Step 2: Installing the New Relic Python agent package.

To install the Python package for the New Relic agent you should add into the 'requirements.txt' ﬁle in the local project directory:

newrelic

Push up this change by running:

stackato update -n

This will cause the 'newrelic' package to be installed into the Python virtual environment on your Stackato instance.

Note: An alternative way to configure the New Relic Python agent using environment variables in the 'stackato.yml' file is outlined in the Stackato documentation. The procedure below shows off Stackato's 'run' command, which can be useful in some scenarios. There are no Python or New Relic dependencies on the local system - all the config generation is done on the server. --TT

Step 3: Generating the Python agent conﬁguration ﬁle.

Now generate the Python agent conﬁguration ﬁle. We will initially do this on the target Stackato instance rather than into the local project directory by running the command:

stackato run gtd newrelic-admin generate-config KEY newrelic.ini

Replace 'KEY' with the license key for your New Relic account. You can obtain the license key from the 'Support' page when logged into your New Relic account. The license key should be a 40 character alpha numeric string.

The 'newrelic-admin' command is a script which was installed along with the Python agent into the 'bin' directory of the Python virtual environment on your Stackato instance. When the command is run with the 'generate-conﬁg' option it will create the ﬁle 'newrelic.ini' in the 'app' project directory on the Stackato instance.

Step 4: Validate the agent conﬁguration and test the Python agent.

To test that the license key is correct and that an application running on your Stackato instance is able to contact the New Relic servers, run the command:

stackato run gtd newrelic-admin validate-config newrelic.ini newrelic.log

The 'validate-conﬁg' option to 'newrelic-admin' will read the agent conﬁguration ﬁle saved above as 'newrelic.ini' to obtain the license key and other settings. It will then attempt to contact the New Relic servers and will report some test metric data. If this succeeds, after a few minutes you should ﬁnd that the application 'Python Agent Test' now appears in the application list in your New Relic account. This should show the test web transaction.

If no data gets through for the agent test, retrieve the 'newrelic.log' ﬁle created by the test to look for errors using the command:

stackato files gtd app/newrelic.log

Step 5: Retrieve the agent conﬁguration ﬁle and customize it.

The agent conﬁguration ﬁle was initially created on the Stackato instance. You now need to pull back that ﬁle so that you have a persistent copy in your local project directory which will be pushed up with each update. To pull back a copy of the 'newrelic.ini' ﬁle run:

stackato files gtd app/newrelic.ini > newrelic.ini

Now edit the local copy of the 'newrelic.ini' ﬁle.

The ﬁrst change you need to make to the ﬁle is to uncomment and modify the value of the 'log_ﬁle' setting. You should set this to:

log_file = stderr

By setting this to 'stderr', any logging output from the Python agent will be directed to standard error output for the web process and recorded in the web process log ﬁle. You can later retrieve the log ﬁle for the web process using the command:

stackato logs gtd

While editing the agent conﬁguration ﬁle, you can also optionally change the 'app_name' setting to be an alternate name under which the Python agent should report data. By default this name is 'Python Application' and that will be the name used for the application in the New Relic UI.

Step 6: Specifying the location of the agent conﬁguration ﬁle.

When the agent test was run the name of the agent conﬁguration ﬁle was listed explicitly as a command line option. When your web process is run, we instead need to use an environment variable to tell it where it is located. To set additional environment variables for the web process, you need to add them to the 'env' section of your 'stackato.yml' ﬁle.

The name of the environment variable to add is 'NEW_RELIC_CONFIG_FILE'.

env:
  NEW_RELIC_CONFIG_FILE: newrelic.ini

Step 7: Running the web processes with the Python agent.

So long as you provide a 'wsgi.py' ﬁle containing a WSGI application entry point called 'application', the Stackato instance will automatically pick that up and use it. To perform the actual hosting of your WSGI application it will default to using uWSGI.

To have the resulting web process use the Python agent you will need to add a new 'processes' section to your 'stackato.yml' ﬁle. This section should deﬁne the 'web' command, which is how your web process will be started.

It is when the 'web' command is not provided that uWSGI will be automatically used. We want to override that command in order to wrap it, instead starting it up from the 'newrelic-admin' command with the 'run-program' option.

In overriding the command we don't want to have to replicate the original command that Stackato used by default. So that we don't need to, Stackato keeps a copy of the original command in the 'PROCESSES_WEB' environment variable and we can use that. The overridden 'web' command should therefore read:

processes:
  web: newrelic-admin run-program $PROCESSES_WEB

We are now ready to push all these changes up to the Stackato instance and try it out. To do that, again run:

stackato update -n

Access your web application and click around in the application for a while. Like with the agent test, the resulting web transactions should start showing up in the New Relic UI after a few minutes. This time the data will show up under whatever application you speciﬁed in the 'app_name' setting in the agent conﬁguration ﬁle.

If you don't see any data after ﬁve minutes, try clicking around in your application some more. The Python agent will only attempt to register with the New Relic servers on the ﬁrst request and depending on the network that may take a moment. It is only once it has been able to successfully register that it will start collecting data.

If you still have issues, then check the web process log output for any errors indicating that there is a problem.

Getting more information.

For further information on the Python agent and how to set it up for different Python WSGI applications, see the New Relic documentation on the Python agent and speciﬁcally that for using the Python agent with Stackato.

Trackback URL for this post:

http://www.activestate.com/trackback/3224

Cloud Expo Wrap-Up: Building Private Clouds, Private PaaS

angieh — Sat, 12 Nov 2011 00:52:51 +0000

Fresh back from Cloud Expo West, it was a great event to further educate, evangelize, and promote private Platform-as-a-Service (PaaS) and Stackato. Being that Stackato (our private PaaS cloud platform) is new for us in 2011, it was the first Cloud Expo we've participated in. And participate we did! We were busy the entire week giving talks, sitting on panels, attending sessions, meeting with partners, and talking to lots of people at our booth. Phew!

For the most part, exhibitors at the conference generally fell into two camps: those providing private cloud hosting services, and vendors providing various technologies to build, enable, or manage clouds. Attendees at the conference were keen to learn more about ways to build or use a private cloud to get all the benefits of "cloud" (public cloud), but internally and securely within their companies. Overall, private clouds was a big theme. So the "private PaaS" topic fit into this theme quite well as the layer on top of "private IaaS" that enables IT/Devs to deploy and manage their applications in a private cloud.

If you missed the event, here are some pictorial highlights and links to webinars so you can hear the talks that you missed!

Here, Diane Mueller is giving a talk on the US Patriot Act (and other jurisdictional regulations) and its implications or drivers for going with an enterprise private cloud. She definitely had some great interactions and questions after this session! If you missed it, register for the webinar she's giving on this exact same topic on Thurs. Dec. 1 >>

On Wed. night, The Armada Group's Brad Vaughan hosted a panel on Platform-as-a-Service (PaaS) where Diane Mueller participated alongside James Watters from VMware/Cloud Foundry, Tobias Kunze from Red Hat, Will Shulman from Mongo Lab, and Dave Nielsen from Cloud Camp. Unfortunately, this one won't be hosted by a webinar, but if you want to learn more about why you'd want a private PaaS, register for our webinar on Nov. 16 featuring Cloud Foundry and ActiveState.

And here's Troy Topnik, giving a demonstration of Stackato in the Demo Theater. If you missed this session, watch the YouTube video (screencast) Troy made, showing how Stackato automatically configures and deploys everything an application needs to run in the cloud. Watch it here >>

If you were there, what were the highlights for you?

Trackback URL for this post:

http://www.activestate.com/trackback/3223

Using Eclipse STS with Stackato

grahams — Tue, 08 Nov 2011 20:30:16 +0000

Last week, we decided to take SpringSource Tool Suite (STS) for a spin with Stackato. The point was to try a tool engineered for a standard version of Cloud Foundry with our variant, and make sure our version hadn't lost the ability to work with STS.

As an added wrinkle, our tester (yours truly) is not a regular Eclipse user. Presumably, if I could make STS Integration work, then the process wasn't excessively complex.

So how compatible is VMWare's STS tool with Stackato, and how was the experience?

Getting it installed

I started with Ubuntu 10.04 LTS. The first order of business was to install and configure the latest Stackato client to test basic operations with the Stackato server.

http://docs.stackato.com/quick-start.html#command-line-client

I ran “stackato target” to point the client to a running v5.2 Server on our network and logged in. Then I pulled down the code for some of our demos from the git repository:

http://community.activestate.com/stackato/demos#java
https://github.com/ActiveState/stackato-samples/tree/master/java/hello-spring-mysql

To test I had everything configured, I pushed the node.js ENV app from the Stackato client. Once I had Stackato working, I installed STS 2.8.0. This went very smoothly.

Getting the integration working

I followed a springsource blog post on the setup here:

http://blog.springsource.com/2011/04/13/using-cloud-foundry-from-sts/

As that post describes, there's a choice of ways to get the Cloud Foundry plugin. I went to the Eclipse Dashboard Extensions tab. It took a few moments to complete the listing on my system. When that completed, I scrolled down to the Server and Clouds section, and selected “Cloud Foundry Integration”, then clicked the Install button.

(Click image below for larger version).

Then I found the Servers view, and added New|Server|VMware|Cloud Foundry. When STS gave me the URL drop-down box, I gave it the same URL which I gave to the “stackato target” command earlier, and when STS prompted for Account Information, I gave it the same email and password I used when I logged into Stackato.

STS added the Server, and in the view, I could right away see the node.js ENV app I had pushed live earlier.

Some quick experimentation lead me to the application panel, and I found I could activate and deactivate the app from there, just as I could stop and start it from the Stackato command line.

Push a new application

I already had a known good spring demo for Stackato, so I set out to convert the Stackato demo to an Eclipse project. This turned out to be the most challenging part of the whole process, mostly due to rookie mistakes. I converted the demo into the wrong kind of project several times, before I found this video clip:

http://www.youtube.com/watch?v=XcI-2V-mmmw&noredirect=1

which tipped me that the demo had to be converted into a Spring MVC project. I had to open the STS Template project menu, and selected the Spring MVC template. Since I didn't have that template by default, STS installed it for me.

I was then able to create a new project with the Spring MVC template, and imported the files from the existing demo to supply the code. After that, I was able to drag the project onto the icon for the Cloud Foundry Server, and STS took care of the push.

(Click image below for larger version).

Summary

STS works with Stackato as easily as it works with Cloud Foundry.

Try it for yourself in the Stackato Open Beta!

Trackback URL for this post:

http://www.activestate.com/trackback/3217

Stackato 0.6.0: Admin Dashboard, New Relic and More

troyt — Tue, 08 Nov 2011 06:41:22 +0000

Just in time for Cloud Computing Expo, we've released version 0.6.0 of the Stackato micro cloud. While we were at it, we updated the Stackato sandbox so you could try out some of the new features there too.

Here's a quick look at what's new and interesting:

New Relic

As I mentioned in a post last week, Stackato users will be able to take full advantage of New Relic's excellent application monitoring service. You can create your New Relic Standard account (with a two week "Pro" trial) from the Stackato Beta page, then follow the documentation for setting up the agent and using your API key. Once you push your application to Stackato, the application monitoring data will start appearing in your New Relic dashboard within minutes.

Stackato Admin Dashboard

There's a new dashboard interface (enabled with 'stackato-admin start dashboard') to give the Stackato administrator an overview of what users are up to on the system. Events like application deployments, stops, restarts, and scaling actions are shown, and you can monitor which applications are running and which accounts are running them.

Open Beta!

When we first started showing Stackato to people back in May during the developer preview, access to Stackato was restricted to a small group of people we worked quite closely with. Over time, we expanded this group and added the developer sandbox for those that didn't want to download the Stackato VM just to try it out. Now, we really want to open the floodgates and see as many people as possible try out the system and give us feedback.

Getting started with Stackato is much easier now. Just go to the Stackato Beta page and sign-up. No surveys, no waiting.

Cloud Foundry and STS Compatibility

Though there are a number of differences between Stackato and Cloud Foundry, we've kept basic API compatibility. We've been testing this compatibility recently and found that the vmc client and SprincSource Tool Suite (STS) works just as well with Stackato as they do with Cloud Foundry.

If you're already using Eclipse or STS for your Java applications, you can keep using these tools with Stackato.

Visit us at the Cloud Computing Expo

If you're attending the Cloud Computing Expo in Santa Clara this week, come visit us at booth #415 in the exhibitor hall. I'll be giving demos and talking about Stackato at the booth for most of the conference and co-presenting a lab on Cloud Foundry and Stackato on Wednesday at 5:30 with Patrick Chanezon from VMware.

Trackback URL for this post:

http://www.activestate.com/trackback/3216

Cloud Expo: Talkin' Private PaaS

angieh — Fri, 04 Nov 2011 16:31:08 +0000

Next week, several of us from ActiveState are hitting up Cloud Expo in Santa Clara! Private PaaS is still a new space, so we're excited to be giving several talks on this topic to help people understand the benefits and capabilities of a private PaaS. If you're going to be there, come check out our sessions and visit our booth (#415).

Here are the talks we'll be giving:

Why Private PaaS is Not an Oxymoron
Bart Copeland (CEO): Monday 11/07 at 2:40pm

When you hear of Platform-as-a-Service (PaaS) what typically comes to mind are names like Heroku, EngineYard, DotCloud, and CloudFoundry.com -- hosted Public PaaS providers. But there is an emerging trend where enterprises require PaaS capabilities, but are demanding it on their own terms. They want a PaaS behind their firewall to enable a secure cloud – a private PaaS. This session will cover:

The drivers for Private PaaS
How a Private PaaS differs from a Public PaaS
What to consider when implementing and deploying a Private PaaS
Case studies of enterprises using a private PaaS

Building a Checklist for Migrating to a Private Cloud
Diane Mueller (Director, Product Management) with Nate Isley from New Relic: Monday 11/07 at 4:55pm

Your research tells you that a private cloud is the best option for you. Now what? What are the practical next steps for moving forward with your cloud initiative? A migration checklist can help you put your initiative into focus by helping you answer some basic questions such as:

What are your goals for migrating to The Cloud?
How do you prioritize apps to deploy: new projects or legacy?
How will you migrate applications and move or integrate data to new cloud infrastructure?
How will you ensure your apps are delivering great service through the migration?

Join Diane Mueller and Nate Isley as they discuss the key considerations for migrating to a private cloud. Attendees will learn how building a checklist can help them with a more successful private cloud implementation.

CTO Power Panel: Has Cloud Computing Changed IT for the Better?
Featuring ActiveState's Jeff Hobbs along with Sheng Liang (Citrix), Rick Nucci (Dell Boomi), Rich Wolski (Eucalyptus) and more
Tuesday 11/08 at 12:45pm

This fast-moving industry discussion panel, moderated by Cloud Expo Conference Chair Jeremy Geelan, will liven up your lunch-hour by answering such questions as:

How fast will the last remaining barriers to enterprise-wide cloud adoption melt away?
Are truly secure public clouds feasible, for example, or only private ones?
How exactly does a company or organization go about deciding whether to migrate only specific applications to the cloud - such as storage or security - or their overall IT infrastructure?
How much of an influence on cloud adoption is the US Government's continuing support of Cloud strategies for all government agencies?
How specifically do the panelists' own company offering/s assist CIOs and organizations/companies?
Will anything ever satisfy the thirst for security, compliance, and SLA assurances at the enterprise level of Cloud computing?
Do you agree with those analysts who say that 2012/13 will be The Year of the Cloud Database? If not, then what will it be the year of?

US Patriot Act: Making the Case for Enterprise Private Cloud
Diane Mueller (Director, Product Management): Wed. 11/10 at 2:25pm

Is the US Patriot Act causing you to hesitate on leveraging the cloud in your enterprise? Do you want to leverage the power of cloud computing but unsure what the security and privacy implications are for sensitive corporate data? Corporations are thinking long and hard about the legal and regulatory implications of cloud computing. When it comes to actual corporate data, no matter what the efficiency gains are, legal departments are often directing IT departments to steer clear of any service that eliminates their ability to keep potential sensitive information out of the hands of Federal prosecutors. Despite all the hype about every application moving into the cloud, some practical patterns are starting to emerge in the types of data corporations are willing to move to the cloud. Learn how to create a secure, compliant, private platform and cloud for developing, distributing and managing enterprise applications. In this session, we will discuss:

Implications of the US Patriot Act on cloud computing
What types of applications should stay private
Classes of applications that best leverage the cloud
Moving to a private cloud model
How to deploy an enterprise private PaaS

Cloud Foundry Open Source PaaS and Stackato Private PaaS (Bootcamp/Lab Session)
Patrick Chanezon (Cloud Foundry) and Troy Topnik: Wed. 11/10 at 5:35pm

Bring your laptops to this hands-on session to learn more about Cloud Foundry and Stackato and try out the micro cloud and/or sandbox!

Will the Real Cloud Please Stand Up: PaaS steps up to deliver on the promises of cloud
Featuring Diane Mueller (Director, Product Management) along with James Watters (Cloud Foundry), Isaac Roth (RedHat), hosted by Armada Group
Wed. 11/10 at 5:30pm

Join us for food, drinks and prizes at the Tusca Courtyard at the Hyatt Hotel Santa Clara

Hope to see you at Cloud Expo! If you can't make it, stay tuned as many of these sessions will be recorded so we'll share links to them shortly after the event.

Trackback URL for this post:

http://www.activestate.com/trackback/3214

Komodo 7 Beta is Ready!

troyt — Fri, 28 Oct 2011 18:54:03 +0000

Komodo 7 Beta 1 is now available, with Edit builds as well as IDE. If you didn't get a chance to look at the Alpha, download the Beta now, give it a try, and let us know what you think. We'd love to get your feedback!

Here are some of the new features that we've put into it:

What's New

Improved integration with Stackato, our private PaaS solution.
Collaborative editing
Code profiling
Synchronization of preferences between your Komodo installations
Debugging and code intelligence for Node.js
More languages: CoffeeScript, Mojolicious, EJS, Less, SCSS, XCSS, and Twig
And all of this running much faster on the Firefox 4 (mozilla-2.0) code base!

... and more.

Stay tuned for more posts, as we'll be writing more about each of these features in the weeks to come.

But I want to buy it NOW!

We don't want to dissuade anyone from buying Komodo IDE just because a new release is on the horizon. Don't wait - buy Komodo IDE with Upgrades & Support by December 31st, and save $50 per license. Start using Komodo IDE 6 today, and get automatic access to Komodo IDE 7 when it's released.

Trackback URL for this post:

http://www.activestate.com/trackback/3213

Virtualization in the EC2 cloud using LXC

daniilk — Thu, 27 Oct 2011 17:14:50 +0000

EC2 is already a (para)virtualized environment, which means it’s nearly impossible to run your own virtualization (KVM/VirtualBox/qemu) from inside that environment. However, Linux recently introduced a new system into the kernel, called cgroups, which provides a way to isolate process groups from each other in the kernel. A project was soon formed around this new technology, which allows for very thin, fast, and secure quasi-virtualization. It’s called LXC, short for LinuX Containers. And it works in EC2 perfectly.

Here’s how.

You’ll want a recent Linux AMI (preferrably kernel 2.6.35 or higher). I use Ubuntu Server 11.04, and the following instructions are meant for that OS. Can’t vouch for other distros, though the instructions should be easily portable. Ubuntu’s good for EC2 and LXC because they already have nice pre-made AMI images, the kernel supports LXC out of the box, and they have software repositories hosted in the EC2 cloud, which makes for extremely fast system updates. Also, any instance type works, even a t1.micro (micro instance) will suffice (my weapon of choice for testing purposes).

Start by SSH-ing into your EC2 server. You’ll need to run almost all of the following instructions as root, so let’s do:

sudo -i

to become root. Otherwise, you can prepend ‘sudo’ to the beginning of every command from now on (unless specified otherwise).

Now, we need to install a few packages:

apt-get update && apt-get install lxc debootstrap bridge-utils dnsmasq

From the packages you installed, lxc is, well, lxc. debootstrap is a utility used to create a minimal Ubuntu install within a directory (which we will do shortly). bridge-utils is a suite of utilities used for creating network bridges in Linux, which we will use to provide network access to the container. dnsmasq is a DNS/DHCP server which will allow the container(s) to identify themselves on the local network.

Now run lxc-checkconfig and make sure that the tests pass (all of them should, if you’re using Ubuntu Server 11.04).

NOTE: THIS IS IMPORTANT! Keep in mind that the effects of most of the commands from here on out (specifically iptables, sysctl, mount, brctl and any edits to /etc/resolv.conf) will not persist over a reboot, even on a EBS-backed instance. These are in-memory changes which will go away as soon as you shut down the machine. If you bring the instance back up, you’ll need to run them again, otherwise things will be broken! There are several ways to get around this: iptables rules and /etc/resolv.conf can be set by an init script, sysctl can be set in sysctl.conf, mounts can be specified in /etc/fstab, and brctl can be set in /etc/network/interfaces (add the br0 interface); however, for the purposes of this guide (I don’t use EBS-backed instances, personally), we’ll assume instance storage (config is lost on reboot).

We need to create a place on the system to hold cgroup information (required for LXC to work). I use /cgroup for simplicity. Let’s mount a cgroup environment there.

mkdir /cgroup mount -t cgroup none /cgroup

(If you want to keep cgroup mounted between reboots, do: echo "none /cgroup cgroup defaults 0 0" >> /etc/fstab instead.)

Now, let’s create a network bridge for the containers to be able to connect to the network/Internet. Simply run:

brctl addbr br0 brctl setfd br0 0 ifconfig br0 192.168.3.1 up

In the second command, we use 192.168.3.* as the container network for the purposes of this tutorial. If you’d like to use another subset (192.168.x.*), you’re free to do so, but be sure to change every instance of 3.* in this article to yourchoice.*, because those IPs will be references a large amount of times in configuration. We recommend you keep the IPs we suggest here for simplicity purposes.

Now we need to set up a few system rules for the containers to be able to reach the Internet:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE sysctl -w net.ipv4.ip_forward=1

Let’s set up DHCP/DNS on our new bridge. Open up /etc/dnsmasq.conf for editing (vim/nano/ed/cat, your choice). Uncomment the necessary lines so that the conf file looks like the following:

domain-needed bogus-priv interface = br0 listen-address = 127.0.0.1 listen-address = 192.168.3.1 expand-hosts domain = containers dhcp-range = 192.168.3.50,192.168.3.200,1h

Now, you’ll need to edit /etc/dhcp3/dhclient.conf for DNS to properly resolve locally. Add the following lines to the beginning:

prepend domain-name-servers 127.0.0.1; prepend domain-search "containers.";

(Don’t forget the dot after containers, that’s not a typo!)

Now we need to renew our DHCP lease so that dhclient will regenerate /etc/resolv.conf.

dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0

Now, let’s restart dnsmasq so it’ll re-read the new configuration.

service dnsmasq restart

Next, we need to create the environment inside the container. There’s a script that comes with lxc called lxc-ubuntu, which will set up the container. However, it’ll require a bit of tweaking for the environment to work. I’ve done the tweaking for you, and put the new script up, so simply run:

wget -O lxc-ubuntu http://bit.ly/ec2ubuntulxc
chmod +x lxc-ubuntu

(If you’d like to do the tweaking yourself, you’ll need to pull the script from /usr/lib/lxc/templates/lxc-ubuntu, change the hostname, change the mirror to match EC2′s Ubuntu mirror, fix the sshd runlevel script, and change the networking config of LXC for DHCP veth (check my script to see how it’s set up)).

Now, let’s create a new container:

./lxc-ubuntu -p /mnt/vm0 -n vm0

Wait a while for the script to finish, and your container is set up in /mnt/vm0. Let’s try it out!

lxc-start -n vm0

Type in root for the username and root for the password. Try pinging Google:

ping www.google.com

If it works, your Internet is set up! Now let’s try another thing (make sure you run this from the VM, not from the host!!):

poweroff (this shuts down the VM, and puts you in the host again)
lxc-start -n vm0 -d (this runs the VM in daemon mode)

To check if a VM is running, type:

lxc-info -n vm0

(it should say RUNNING). To test network, try pinging the VM (this might not work right away, you might have to wait up to a minute):

ping vm0 ssh root@vm0

If those two work, the VM is now in your DNS and you can address it by its hostname. Cool, huh?

Creating a new VM

Creating another VM is as simple as:

./lxc-ubuntu -n vm1 -p /mnt/vm1

The packages won’t be redownloaded, and the command should complete quickly.

Clone existing VM

If you want to clone your existing VM, you’ll need to do a few things:

cp -r /mnt/vm0 /mnt/vm1

Now edit /mnt/vm1/config and replace all references of vm0 to vm1. Do the same with /mnt/vm1/fstab. Then go into /mnt/vm1/rootfs/etc/hostname and replace the hostname with vm1. Finally, run:

lxc-create -n vm1 -f /mnt/vm1/config

Upon starting the VM, you should be able to ping it/ssh to it:

ping vm1 ssh root@vm1

If not, lxc-console into the VM and check your connection. Keep in mind you only need one br0 for all your instances, but you can create many, if you so desire.

Running services inside the container

You may want web servers to be accessible from outside the VM (from the rest of EC2, or outside EC2). To do this, you’ll need to port forward from the host to the VM. Simply run:

iptables -t nat -A PREROUTING -p tcp --dport <port on host> -j DNAT --to-destination <hostname of VM>:<port on VM>

Hibernating a container

To ‘hibernate’ a container (save the current state (running processes) of the VM for instant restoring later) do:

lxc-freeze -n vm0

and later,

lxc-unfreeze -n vm0

to restore.

Installing additional packages into the container

Your container is just like any other Ubuntu system. Therefore,

apt-get update apt-get install <whatever>

works great.

Setting resource limits

One of the benefits of LXC is that you can limit resource usage per-container. Let’s delve into the various resources you can limit:

CPU

There’s two ways of limiting CPU in LXC. On a multi-core system, you can assign different CPUs to different containers, as such (add this line to your container config file, /mnt/vm0/config or similar):

lxc.cgroup.cpuset.cpus = 0 (assigns the first CPU to the container)
or
lxc.cgroup.cpuset.cpus = 0,2,3 (assigns the first, third, and fourth CPU to the container)

The alternative (this one makes more sense to me) is to use the scheduler. You can use values to say ‘I want this container to get 3 times the CPU of this container’. For example, add:

lxc.cgroup.cpu.shares = 2048

to the config to give a container double the default (1024).

RAM

To limit RAM, simply set:

lxc.cgroup.memory.limit_in_bytes = 256M

(replacing 256M with however much RAM you want to allow).

To limit swap, set:

lxc.cgroup.memory.memsw.limit_in_bytes = 1G

Hard Disk

Well, there’s no official way to do this, it’s up to you. You can use LVM (in EC2? fun), or you can create a filesystem in a file (something like dd if=/dev/zero of=somefile.img bs=4GB count=1 && mkfs.ext3 somefile.img && mount -o loop somefile.img /mnt/vm0/rootfs) and mount it to /mnt/vm0/rootfs to limit space.

Network Bandwidth

To limit network bandwidth per container, you'll want to use the tc utility. Keep in mind you’ll need to use separate bridges (br0, br1) for each container if you go this route. Don’t forget to edit the config of each VM to match your new bridge if you do so.

Wrap-up

I hope I’ve covered most basic aspects of using LXC in the EC2 environment. LXC is an amazing technology, and the possibilities of what you can do with it are endless.

Some further reading: the main LXC site, the LXC HOWTO, and IBM’s tutorial.

NOTE: When following other guides on LXC, be very careful with messing with the network in the EC2 environment (restarting networking services or altering /etc/network/interfaces on the host) as careless network reconfiguration may cause the connection will drop between you and your instance (you’ll lose SSH), and therefore lose your instance completely. The instructions I’ve provided here have been tested and will not drop your EC2 connection.

Stackato makes heavy use of LXC to provide a secure, isolated environment for your apps (more on this soon). Check out the Stackato Sandbox today and see how easy it can be to get your apps up and running in the cloud with Stackato!

Trackback URL for this post:

http://www.activestate.com/trackback/3212