Google App Engine Blog

My summer with the Google App Engine Team

2012-01-26T13:47:00.000-08:00

Today’s post is contributed by our Summer 2011 team intern, Chris Bunch. Chris did some great work on our Logs and MapReduce APIs and is also the first “App Engine Triple Crown” winner for developing the Experimental Logs Reader API in Python, Java and Go simultaneously.

Four years ago, I was a brand-new Ph.D. student at the University of California, Santa Barbara and when our research group (the RACELab) heard about Google App Engine, we were intrigued. We thought it presented a new model that enabled apps to scale the right way without severely constricting the types of programs users would write.

But we wanted to experiment with the core functionality of App Engine: the APIs, the scheduler, etc., and so we built AppScale, an open-source implementation of the Google App Engine APIs that allows users to deploy applications written in Python, Java, and Go to the infrastructure of their choice.
Wherever possible, we implement support for the App Engine APIs with alternative open-source technologies. We’ve added support for nine different databases, database-agnostic transactions, a REST interface that users of any programming language can communicate with (via an App Engine app), and the ability to run high performance computing programs over the whole thing and talk to it from your App Engine app. And here’s my favorite part - it all deploys automatically! You don’t need to tell it what block size you want for the distributed file system, or the size of the read buffers: we configure the necessary services automatically. Since AppScale is completely open source, if you don’t like the defaults, change them!

After creating our own system to run Google App Engine apps, I wanted to see how Google does it. Therefore, I decided to become an intern on the App Engine team and see if I could give them (and by extension, the App Engine community) something amazing over the summer. I started off with some work on the MapReduce API, making the sample app much easier to use and prettier all around. I also made a YouTube video showing how it all works and how easy it is to run MapReduce jobs over App Engine.

I then looked at a recurring question that App Engine users encounter: “How can I get my logging information for my application to answer data analytic questions?” It was an excellent problem to tackle, as we have users who want to be able to determine application-specific queries that Google Analytics or the Admin Console don’t answer. Currently users have to use appcfg to grab all their application’s data to a remote machine and run some analysis script over it.

To solve this problem, I created the Logs API, which gives applications programmatic access to their logs from within App Engine itself. Applications can use it to query small numbers of logs within a single request, and they can utilize the Pipeline, MapReduce, or Backends APIs if they have lots of logs they want to analyze. Logs contain both request-level information (e.g., the URL accessed, the HTTP response code returned) as well as logging info generated by the application (the logging module in Python, the Logger class in Java, and the logging methods that Go’s appengine package provides). The Logs API is available for use as of App Engine 1.6.1 by programmers using the Python, Java, or Go runtimes, in both the production environment and the local SDK.

I had a great time putting the Logs API together, and had a unique experience interning with the App Engine team. Programming in Python, Java, and Go on a daily basis was an exciting new challenge, and I loved it!

Interested in interning with the App Engine team? Check out google.com/students for more information on internships.

Google Cloud Storage: concurrency controls and deeper App Engine integration

2012-01-19T13:17:00.000-08:00

Cross posted from the Google Code Blog

Google Cloud Storage is a robust, high-performance service that enables developers and businesses to use Google’s infrastructure to store and serve their data. Today, we’re announcing a new feature that gives you greater control over concurrent writes to the same object, and the availability of an App Engine Files API that makes it easier to read and write data from Java App Engine applications.

Write concurrency control

A number of our customers have asked us for greater control over concurrent writes, in order to implement features like strongly consistent write operations and distributed locking semantics in the cloud. In response to your feedback, we’re announcing the release of version-based concurrency control. Every time you update an object, it gets assigned a 32-bit, monotonically increasing sequence number. This version number is returned as a header with every GET or HEAD request. You can then use a conditional write operation to manage concurrent updates to the object (for example, when you want read-modify-write semantics). This feature is currently experimental.

AppEngine Files API for Java applications

Last fall, we announced the ability to read and write your Google Cloud Storage data using the App Engine Files API for Python applications. Today, we’re making the Files API available to Java App Engine applications too. This feature is currently experimental, and we’ll continue to enhance it in the months to come.

As always, we welcome your feedback in our discussion group. If you haven’t tried Google Cloud Storage yet, you can sign up and get started here.

Happy New Year from the App Engine team

2012-01-17T10:36:00.000-08:00

Happy New Year! As we return from our New Year's celebrations, brush the dust off our workstations and gear up for our first release of 2012, we thought it would be fun to take a look back at improvements we have made and what developers have accomplished with App Engine in 2011.

Let’s start with the features and functionality we added last year:

Language Support: We released the initial version of Python 2.7 support and added Go as an experimental runtime.

Storage: We launched the High Replication Datastore and added support for the Files API in Python and Java. We also announced the limited preview of Google Cloud SQL, a familiar relational database in a fully managed cloud environment.

Computation: We introduced Backends for building larger, long-lived and/or memory intensive infrastructure, Pull Queues to allow developers to “pull” tasks from a queue as applications are ready to process them and two larger Frontend instance classes. We also released the GAE MapReduce framework as an experimental feature for Python.

Security: We successfully completed the audit process for the SAS70 Type II, SSAE 16 Type II, and ISAE 3402 Type II standards.

Business Readiness: We modified our SLA, billing plans, and service limits and now offer fully supported Premier Accounts.

Best of all, with your continued support we accomplished our goal of graduating from preview and became a full fledged Google product.

We’ve seen excellent growth and adoption over the past year, with businesses like Pulse, Evite and Best Buy choosing App Engine for their applications. Even St. James’s Palace chose App Engine to host the Royal Wedding site. We had so much fun collaborating with 17 of the world’s most renowned museums for the Google Art Project and with other Googlers building iGoogle gadgets and Doodles on App Engine. We’ve added more than 1 million registered applications and have more than 150,000 active developers on the App Engine platform generating more than 5 billion page hits per day.

Back in our first blog post in 2008, we asked you to “start your engines” and what a ride we’ve taken. Thank you for making 2011 our best year yet and here’s to making 2012 even better!

Posted by Peter Magnusson, Engineering Director

Happy Birthday High Replication Datastore: 1 year, 100,000 apps, 0% downtime

2012-01-05T12:43:00.000-08:00

Once upon a time, the only way to store persistent data in App Engine was to use the Master/Slave Datastore. Although it was a transactional, massively scalable, fully managed, distributed storage system running on Google’s world-class infrastructure, its availability was tied to the availability of a single datacenter, and when you’re serving hundreds of thousands of applications, relying on any single datacenter is simply not sufficient. One year ago today we unveiled a new offering that was specifically designed to address this weakness: the High Replication Datastore (HRD). Still transactional, still massively scalable, still fully managed, still running on Google’s world-class infrastructure, but with the ability to withstand multiple datacenter outages and no planned downtime!

By the time Google I/O came around last May, HRD was performing beautifully and our customers were happy, so we took the next step and made HRD the default option for all new App Engine applications.

In June we made HRD available in our SDK so that customers could easily experiment with the new consistency guarantees (paxos on your laptop!), and we launched the first version of our migration tool to make it easy to move your apps from Master/Slave to HRD.

In October we released XG Transactions, our first HRD-only feature, which allows users to transact across entity groups.

In November we brought App Engine out of preview and added a 99.95% SLA for HRD applications.

In our most recent release we launched an updated version of our HRD migration tool that ties the duration of the read-only period to your write-rate, rather than the size of your dataset. This makes your migration quick, simple, and easy to plan for regardless of how much data you have. One App Engine customer recently migrated over 500G of Datastore data with only a 10 minute read-only period!

Throughout all this, HRD has had no system-wide downtime (planned or unplanned) and has grown to serve over 3 billion requests per day. Needless to say it’s been a phenomenal year.

We realize that moving data requires planning, testing, coordination, and a strong stomach. However, we believe strongly that HRD provides a fundamentally better service than Master/Slave, and we encourage all our customers to migrate to HRD. Over the coming months you can expect to see further improvements to our migration tools (Blob migrations are on the way!), more HRD-only features like Full Text Search, and of course, more 9s than you can shake a stick at.

Posted by Max Ross, Datastore Tech Lead

Simple development of App Engine apps using Cloud SQL - Introducing Google Plugin for Eclipse 2.5

2011-12-19T09:28:00.000-08:00

Since we added SQL support to App Engine in the form of Google Cloud SQL, the Google Plugin for Eclipse (GPE) team has been working hard on improving the developer experience for developing App Engine apps that can use a Cloud SQL instance as the backing database.

We are pleased to announce the availability of Google Plugin for Eclipse 2.5. GPE 2.5 simplifies app development by eliminating the need for manual tasks like copying Cloud JDBC drivers, setting classpaths, typing in JDBC URLs or filling in JVM arguments for connecting to local/remote database instances.

GPE 2.5 provides support for:

Configuring Cloud SQL/MySQL instances
Auto-completion for JDBC URLs
Creating database connections in Eclipse database development perspective
OAuth 2.0 for authentication.

Configuring Cloud SQL/MySQL instances
App Engine provides a local development environment in which you can develop and test your application before deploying to App Engine. With GPE 2.5, you now have the ability to configure your local development server to use a local MySQL instance or a Cloud SQL instance for testing. When you choose to deploy your app, it will use the configured Cloud SQL instance for App Engine.

Auto-completion for JDBC URLs
GPE 2.5 supports auto-completion for JDBC URLs, and quick-fix suggestions for incorrect JDBC URLs.

Creating database connections in Eclipse database development perspective
The Eclipse database development perspective can be used to configure database connections, browse the schema and execute SQL statements on your database.

Using GPE 2.5, database connections are automatically configured in the Eclipse database development perspective for the Development SQL instance and the App Engine SQL instance.

You can also choose to manually create a new database connection for a Cloud SQL instance. In GPE 2.5, we have added a new connection profile for Cloud SQL.

GPE 2.5 now uses OAuth 2.0 (earlier versions were using OAuth 1.0) to securely access Google services (including Cloud SQL) from GPE. OAuth 2.0 is the latest version of the OAuth protocol focussing on simplicity of client development.

Can’t wait to get started?
Download GPE here and write your first App Engine and Cloud SQL application using GPE by following the instructions here.

We hope GPE 2.5 will make cloud application development using App Engine and Cloud SQL a breeze. We always love to hear your feedback and the GPE group is a great place to share your thoughts.

Posted on behalf of the Google Plugin for Eclipse Team

App Engine 1.6.1 Released

2011-12-13T11:06:00.000-08:00

We have one more release this year to make our developers merry, and while some members of our team enjoy the summer sunshine down under, we’ll be taking a short winter break from releases. Don’t worry, we’ll be back to our normal schedule in January, but we couldn’t resist tempting you with some new features that will keep you up tinkering well past midnight on January 1st.

Platform Changes

Frontend Instance Classes - For applications that need more CPU and/or memory to serve requests, we’ve introduced two larger frontend instance classes. Before today, all apps were allocated a fixed instance size no matter what the app was computing in its requests. Now, apps that need more computing power can upgrade the size of their instances.

High Replication Datastore (HRD) Migration Tool Has Graduated - The HRD migration tool is now a fully supported feature. The tool allows you to easily migrate your data, limits the downtime required to complete the migration, and also allows you to choose its precise time. Every app can now start the new year off right, improving their uptime and reliability by migrating to HRD!

New APIs

Conversion API (Experimental) - Converting between formats within your application can be a pain, but with the experimental Conversion API you can now easily convert between PDF, HTML, text and images. Generating PDF invoices from HTML, displaying PDF menus as HTML or extracting text from images using OCR is now as simple as an API call.

Logs Reader API (Experimental) - Want to summarize latency by handler? Summarize request statistics by user? The new logs reader API allows you to programmatically access your logs to build reports, gather statistics, and analyze requests to your heart’s content.

Read the full release notes for Java and Python to get all the details on 1.6.1. We always love to hear what you think, so keep the feedback on our groups coming. App Engine releases will resume again with our regular schedule around the end of January.

Posted by The App Engine Team

Whentotweet.com - Twitter analytics for the masses

2011-12-05T10:00:00.000-08:00

Our post today comes from Stefan and Niklas of Whentotweet.com, a nifty site that recommends the best time of day to tweet based on your followers’ habits.

Twitter handles an amazing number of Tweets - over 200 million tweets are sent per day.

We saw that many Twitter users were tweeting interesting content but much of it was lost in the constant stream of tweets.

Whentotweet.com is born

While there were many tools for corporate Twitter users that performed deep analytics and provided insight into their tweets, there were none that answered the most basic question: what time of the day are my followers actually using Twitter?

And so the idea behind Whentotweet was born. In its current form, Whentotweet analyzes when your followers tweet and gives you a personalized recommendation of the best time of day to tweet to reach as many as possible.

Given the massive amount of data we needed to analyze, we knew it would be a huge engineering challenge to build what we wanted using the tools we had used previously. We also wanted to make sure we could offer at least a basic product for free. Not only did we need to process massive amounts of data - we also needed a way to do it without a second mortgage on our houses!

The Technology Used

As we went over the alternatives we started to sketch different ways of hosting our application. We had previous experience building web sites and knew that traditional cloud hosting would be expensive and difficult to manage for the kind of computing that we needed. After some quick back-of-the-envelope calculations it seemed clear that Google App Engine would give us both the kind of pricing we needed and a way to scale. We decided to write a quick test application to test our assumptions.

The test application blew our minds. Apart from proving our initial assumptions around pricing and scale we started appreciating the quick deploys. On previous projects we were used to one deploy per month. Almost immediately we shifted our schedule to one or sometimes several deploys per day to push new code to customers.

The main APIs that Whentotweet relies on are Google App Engine's task queues and Datastore. Whenever a new user requests a report it is added as a task. A typical report requires a huge number of interactions with external sites. By breaking down each external interaction into separate tasks in different queues it became easy to make sure we kept a steady rate of API calls to external sites without risking that a huge influx of users would break our API limits.

The initial task then spawns new tasks until finally one of the tasks decides that the report is complete and tweets a summary of the result and a link to a more detailed report. Whentotweet uses a "fail fast" technique so whenever any request fails, internal or external, the task terminates and puts itself back on the queue.

The Datastore saves a finished or ongoing analysis. Sometimes a single analysis will be updated several times a second by tasks as they finish and store their results.

The Result

After a few weeks of intense coding, we were ready to test our code on a small sized Twitter account with less than three hundred followers. The results came back in just a few minutes.

After verifying that everything had actually worked as well as we thought, we decided to try another account. This time one of the largest Twitter accounts on the planet: @techcrunch. Handling a Twitter account with over a million followers took the application one week. But after the analysis started, Whentotweet would quietly work in the background without us having to lift a finger.

Whentotweet got off to a better start than we imagined. During the initial launch thousands of people tested it on their Twitter accounts.

After a while blog posts appeared, recommending Whentotweet as an invaluable Twitter tool. Each post would generate a sudden huge spike in traffic. Sometimes, a blog owner would mail us and ask if we were ready for the sudden increase in traffic this would bring. But Whentotweet was built to scale and even massive sites such as Mashable.com didn't slow it down. The most amazing thing is that we didn't need to write a single extra line of code to handle these massive variations in load. Instead, as soon as we wrapped our head around the tools in the App Engine toolbox we knew that Whentotweet would easily scale. App Engine forced us to think outside the box and avoid the fallacies of traditional hosting that create bottlenecks.

Currently, over 38,000 people have tried Whentotweet and we see from the user feedback that they love it. Give it a try at: www.whentotweet.com

- Niklas Agevik (@niklas_a) and Stefan Ålund (@stefan_alund) of Whentotweet.com

Scaling with the Kindle Fire

2011-11-29T10:02:00.000-08:00

Today’s blog post comes to us from Greg Bayer of Pulse, a popular news reading application for iPhone, iPad and Android devices. Pulse has used Google App Engine as a core part of their infrastructure for over a year and they recently celebrated a significant launch. We hope you find their experiences and tips on scaling useful.

As part of the much anticipated Kindle Fire launch, Pulse was announced as one of the only preloaded apps. When you first un-box the Fire, Pulse will be there waiting for you on the home row, next to Facebook and IMDB!

Scale
The Kindle Fire is projected to sell over five million units this quarter alone. This means that those of us who work on backend infrastructure at Pulse have had to prepare for nearly doubling our user-base in a very short period. We also need to be ready for spikes in load due to press events and the holiday season.

Architecture
As I’ve discussed previously on the Pulse Engineering Blog, Pulse’s infrastructure has been designed with scalability in mind from the beginning. We’ve built our web site and client APIs on top of Google App Engine, which has allowed us to grow steadily from 10s to many 1000s of requests per second, without needing to re-architect our systems.

While restrictive in some ways, we’ve found App Engine’s frontend serving instances (running Python in our case) to be extremely scalable, with minimal operational support from our team. We’ve also found the datastore, memcache, and task queue facilities to be equally scalable.

Pulse’s backend infrastructure provides many critical services to our native applications and web site. For example, we cache and serve optimized feed and image data for each source in our catalog. This allows us to minimize latency and data transfer and is especially important to providing an exceptional user experience on limited mobile connections. Providing this service for millions of users requires us to serve 100Ms of requests per day. As with any well designed App Engine app, the vast majority of these requests are served out of memcache and never hit the datastore. Another useful technique we use is to set public cache control headers wherever possible, to allow Google’s edge cache (shown as cached requests on the graph below) and ISP / mobile carrier caches to serve unchanged content directly to users.

Costs
Based on App Engine’s projected billing statements leading up to the recent pricing changes, we were concerned that our costs might increase significantly. To prepare for these changes and the expected additional load from Kindle Fire users, we invested some time in diagnosing and reducing these costs. In most cases, the increases turned out to be an indicator of inefficiencies in our code and/or in the App Engine scheduler. With a little optimization, we have reduced these costs dramatically.

The new tuning sliders for the scheduler make it possible to rein in overly aggressive instance allocation. In the old pricing structure, idle instance time wasn’t charged for at all, so these inefficiencies were usually ignored. Now App Engine charges for all instance time by default. However, any time App Engine runs more idle instances than you’ve allowed, those hours are free. This acts as a hint to the scheduler, helping it reduce unneeded idle instances. By doing some testing to find the optimal cost vs spike latency tolerance and setting the sliders to those levels, we were able to reduce our frontend instance costs to near original levels. Our heavy usage of memcache (which is still free!) also helps keep our instance hours down.

Since datastore operations used to be charged under the umbrella of CPU hours, it was difficult to know the cost of these operations under the old pricing structure. This meant it was easy to miss application inefficiencies, especially for write-heavy workloads where additional indexes can have a multiplicative effect on costs. In our case, the new datastore write operations metric led us to notice some inefficiencies in our design and a tendency to overuse indexes. We are now working to minimize the number of indexes our queries rely on, and this has started to reduce our write costs.

Preparing for the Kindle Fire Launch
We took a few additional steps to prepare for the expected load increase and spikes associated with the Fire’s launch. First, we contacted App Engine’s support team to warn them of the expected increase. This is recommended for any app at or near 10,000 requests per second (to make sure your application is correctly provisioned). We also signed up for a Premier account which gets us additional support and simpler billing.

Architecturally, we decided to split our load across three primary applications, each serving different use cases. While this makes it harder to access data across these applications, those same boundaries serve to isolate potential load-related problems and make tuning simpler. In our case, we were able to divide certain parts of our infrastructure, where cross application data access was less important and load would be significant. Until App Engine provides more visibility into and control of memcache eviction policies, this approach also helps prevent lower priority data from evicting critical data.

I’m hopeful that in the near future such division of services will not be required. Individually tunable load isolation zones and memcache controls would certainly make it a lot more appealing to have everything in a single application. Until then, this technique works quite well, and helps to simplify how we think about scaling.

To learn more about Pulse, check out our website! If you have comments or questions about this post or just want to reach out directly, you can find me @gregbayer.

New Datastore client library for Python ready for a test drive

2011-11-16T09:12:00.000-08:00

Last week we announced that App Engine has left preview and is now an officially supported product here at Google. And while the release (and the announcement) was chock-full of great features, one of the features that we’d like to call specific attention to is the new Datastore client library for Python (a.k.a “NDB”).

NDB has been under development for some time and this release marks its availability to a larger audience as an experimental feature. Some of the benefits of this new library include:

The StructuredProperty class, which allows entities to have nested structure
Integrated two-level caching, using both memcache and a per-request in-process cache
High-level asynchronous API using Python generators as coroutines (PEP 342)
New, cleaner implementations of Key, Model, Property and Query classes

The version of NDB contained in the 1.6.0 runtime and SDK corresponds to NDB 0.9.1, which is currently the latest NDB release.

Given that this feature is still experimental, it is subject to change, but that’s exactly why we encourage you to give it a test drive and send us any feedback that you might have. The NDB project hosted on Google Code is the best place to send this feedback. Happy coding!

Posted by Guido van Rossum, Software Engineer on the App Engine Team

Google BigQuery Service: Big data analytics at Google speed

2011-11-14T13:35:00.000-08:00

Our post today, cross-posted with the Google Enterprise Blog, comes from one of our sister projects, BigQuery. We know that many of you are interested in processing large volumes of data and we encourage you to try it out.

Rapidly crunching terabytes of big data can lead to better business decisions, but this has traditionally required tremendous IT investments. Imagine a large online retailer that wants to provide better product recommendations by analyzing website usage and purchase patterns from millions of website visits. Or consider a car manufacturer that wants to maximize its advertising impact by learning how its last global campaign performed across billions of multimedia impressions. Fortune 500 companies struggle to unlock the potential of data, so it’s no surprise that it’s been even harder for smaller businesses.

We developed Google BigQuery Service for large-scale internal data analytics. At Google I/O last year, we opened a preview of the service to a limited number of enterprises and developers. Today we're releasing some big improvements, and putting one of Google's most powerful data analysis systems into the hands of more companies of all sizes.

We’ve added a graphical user interface for analysts and developers to rapidly explore massive data through a web application.
We’ve made big improvements for customers accessing the service programmatically through the API. The new REST API lets you run multiple jobs in the background and manage tables and permissions with more granularity.
Whether you use the BigQuery web application or API, you can now write even more powerful queries with JOIN statements. This lets you run queries across multiple data tables, linked by data that tables have in common.
It’s also now easy to manage, secure, and share access to your data tables in BigQuery, and export query results to the desktop or to Google Cloud Storage.

Michael J. Franklin, Professor of Computer Science at UC Berkeley, remarked that BigQuery (internally known as Dremel) leverages “thousands of machines to process data at a scale that is simply jaw-dropping given the current state of the art.” We’re looking forward to helping businesses innovate faster by harnessing their own large data sets. BigQuery is available free of charge for now, and we’ll let customers know at least 30 days before the free period ends. We’re bringing on a new batch of pilot customers, so let us know if your business wants to test-drive BigQuery Service.

Posted by Ju-Kay Kwek, Product Manager

App Engine 1.6.0 Out of Preview Release

2011-11-07T12:40:00.000-08:00

Three and a half years after App Engine’s first Campfire One, App Engine has graduated from Preview and is now a fully supported Google product. We started out with the simple philosophy that App Engine should be ‘easy to use, easy to scale, and free to get started.’ And with 100 billion+ monthly hits, 300,000+ active apps, and 100,000+ developers using our product every month it’s clear that this philosophy resonates. Thanks to your support, Google is making a long term investment in App Engine!

When we announced our plans to leave preview earlier this year, we made a commitment to improving the service by adding support for Python 2.7, Premier Accounts and Backends as well as several changes launching today:

Pricing: The new pricing structure announced in May (and updated based on feedback from the community) will be reflected in your bill starting on Nov 7th as previously announced.
Terms of Service: We have a new business-friendly terms of service and acceptable use policy.
Service Level Agreement: All paid applications on the High Replication Datastore are covered by our 99.95% SLA.

We are also holding a series of App Engine Office hours via Google+ this week for any users who have questions about how these changes impact their applications. The list of times can be found on the Google Developers events page, with links to join the hangout while the office hours are scheduled. Also, please don’t hesitate to contact us at appengine_updated_pricing@google.com with any questions or concerns.

In addition to leaving Preview, we have several additional changes to announce today.

Production Changes

For billing enabled apps, we are offering two more scheduler controls and some additional changes:

Min Idle Instances: You can now adjust the minimum number of Idle Instances for your application, from 1 to 100. Users who had previously signed up for “Always On” can now set the number of idle instances for their applications using this setting.
Max Pending Latency: For applications that care about user facing latency, this slider allows you to set a limit to the amount of time a request spends in the pending queue before starting up a new instance.
Blobstore API: You can now use the Blobstore API without signing up for billing.

Datastore Changes

High Replication Datastore Migration Tool: We are releasing an experimental tool that allows you to easily migrate your data from Master/Slave to High Replication Datastore, and seamlessly switch your application’s serving to the new HRD application.
Query Planning Improvements: We’ve published an article that details recent improvements to our query planner that eliminate the need for exploding indexes.

Python

MapReduce: We are releasing the full MapReduce framework in experimental for Python. The framework includes the Map, Shuffle, and Reduce phases.
Python 2.7 in the SDK: The SDK now supports the Python 2.7 runtime, so you can test out your changes before uploading them to production.

Java™

Memcache API Improvements: The Memcache API for Java now supports asynchronous calls. Additionally, putIfUntouched() and getIdentifiable() now support batch operations.
Capability Testing: We’ve added the ability to simulate the capability state of local API implementations to test your application’s behavior if a service is unavailable.
Datastore Callbacks: You can now specify actions to perform before or after a put() or delete() call.

The full list of changes with this release can be found in the release notes (Python, Java). We’d love to hear your feedback about this release in the groups. And we’d like to thank you all for investing in our platform for the last three years. We’re excited for this milestone in App Engine history, and we look forward to what the future will bring.

Posted by The App Engine Team

Oracle and Java are registered trademarks of Oracle and/or its affiliates.

ProdEagle - Analyzing your App Engine apps in real-time

2011-10-27T10:33:00.000-07:00

Today’s post comes to us from Andrin von Rechenberg of MiuMeet who has developed an easy to use analysis framework, ProdEagle, for App Engine apps. ProdEagle enables you to easily count and visualize events to help better understand both performance and usage of your site.

ProdEagle allows you to monitor your system, lets you analyse in real-time what is going on and can alert you if something goes wrong.

The story

We are a small start-up that created an incredibly fast growing social dating platform called MiuMeet. We used App Engine from the start and within 6 months we had over 1 million registered users - the system scaled beautifully. One of the world’s leading online dating companies invested in our start-up and gave us a very good piece of advice: Before you build new features, you need to understand what’s actually going on in your system. Of course we thought we knew exactly what is going on in our system. We had the App Engine Dashboard and we had Google Analytics and we ran a couple of daily MapReduces to collect some statistics. But honestly, compared to today, we didn’t have any idea what was going on.

Understanding your system

From our MapReduces we knew that most users who accessed our dating platform used our Android app and only very few users used our iPhone app. We weren’t exactly sure what the reason for that was, but we wanted to improve the experience for our iPhone users. So we had to start measuring what was different for iPhone vs Android users.

But how do you do this? You start recording everything that happens in your system. And by “everything” we don’t mean the status code of an HTTP response but human understandable events, like “an Android user starts a conversation”, “the average length of a conversation”, “how often iPhone users log in” and so on. We realized that iPhone users are much less active and reply less often - not because they don’t want to, but because they don’t know immediately that they got a new message. Android users are informed by Push-Notifications, iPhone users via email. We did not expect that this would make such a big difference, but now it’s pretty obvious what the next feature we should build to improve the iPhone experience. Our conclusions are based on hard numbers and not guesses.

We used ProdEagle to collect and analyse the data that allowed us to realize this. ProdEagle can count, display and compare events in real-time and can even alert you if something is going wrong.

How to measure events

Let’s start with a simple Python example. We would like to record the devices from which messages are sent in our system. All we have to do is add the green lines to our sendMessage function:

import prodeagle

def sendMessage(from, to, text, device):
mailbox = getMailbox(to)
mailbox.addMessage(from, text)
prodeagle.counter.incr("Message." + device)

This one line of code and a few clicks in the ProdEagle dashboard allows us to create the following graphs:

*The numbers in these graphs are just examples.

Analysing your system in “real-time”

With ProdEagle you can count whatever you want. The advantage over traditional analytics systems is that you don’t have to wait for a day for the data to appear. Data appears within 1 minute in your dashboard. This means that you can monitor your system almost in real-time and set up alerts if something goes wrong.

In the following example we measure how long it takes to execute a datastore query:

def searchPeople(query):
timestamp = time.time()
query.execute()
prodeagle.counter.incr("Search.People.Count")
prodeagle.counter.incr("Search.People.Latency",
time.time() - timestamp)

In the ProdEagle Dashboard we can specify that “Search.People.Latency” should be divided by “Search.People.Count” and plot it as a graph. Additionally we can set up an email alert that fires if the latency was more than 4000ms for 5 minutes:

If the latency is above all the red dots you get an email alert.

Being able to measure latency in real-time is particularly useful when you are trying to optimize your system and want to try out different strategies to answer queries.

The magic behind ProdEagle

Whenever you call prodeagle.counter.incr(counter_name)the ProdEagle library increments a counter in memcache - this shouldn’t add any noticeable latency to your app, unless it is the first call made by a serving instance. Every few minutes, the ProdEagle service that we run collects all these memcache counters and persists them, creates the graphs and alerts you if something is wrong.

Obviously, the system isn’t robust if your memcache gets flushed. To address this, ProdEagle uses 1024 dummy counters to check when your memcache gets flushed and to approximate the inaccuracy of the data in your graphs. This is based on the assumption that elements with the same size in memcache are freed in a “least-recently-used” fashion and that the 1024 dummy counters hit all memcache shards. We have used ProdEagle for MiuMeet for a couple of months now and our approximated accuracy was never below 99.8%.

I like! How can I get it?

ProdEagle is currently completely free of charge. You can download the python client libraries and sign up for your dashboard on www.prodeagle.com.

App Engine SSL for Custom Domains in Testing

2011-10-19T10:01:00.000-07:00

The long awaited SSL for Custom Domains is entering testing and we are now looking for trusted testers. If you are interested in signing up to test this feature, please fill in this form.

We will be offering two types of SSL service, Server Name Indication (SNI) and Virtual IP (VIP). SNI will be significantly less expensive than VIP when this service is launched, however unlike VIP it does not work in all browsers that support SSL. VIP is a premium service with a dedicated IP and full browser support. Both VIP and SNI support wildcard certificates and certificates with alternate names.

We look forward to making this widely available as soon as possible and as always we welcome your feedback in the group.

Posted by The App Engine Team

App Engine 1.5.5 SDK Release

2011-10-11T10:15:00.000-07:00

2011 has seen some exciting releases for App Engine. As the days get shorter, the weather gets colder, and all that Halloween candy starts tempting everyone in the grocery store, we’ve been hard at work on our latest action packed release.

Premier Accounts

When choosing a platform for your most critical business applications, we recognize that uptime guarantees, easy management and paid support are often just as important as product features. So today we’re launching Google App Engine premier accounts. For $500 per month (not including the cost to provision internet services), you’ll receive:

Premium support (see the

Technical Support Services Guidelines

for details).

A 99.95% uptime Service Level Agreement (see the draft agreement, the final agreement will be in the signed offline agreement).
The ability to create an unlimited number of apps on your premier account domain.
No minimum monthly fees per app. Pay only for the resources you use.
Monthly billing via invoice.

To sign up for a premier account, please contact our sales team at appengine_premier_requests@google.com.

Python 2.7

PIL? NumPy? Concurrent requests? Python 2.7 has it all, and today we’re opening up Python 2.7 as an experimental release. We’ve put together a list of all the known differences between the current 2.5 runtime and the new runtime.

Overall Changes

We know that bumping up against hard limits can be frustrating, and we’ve talked all year about our continued push to lift our system limits. With this release we are raising several of these:

Request Duration: The frontend request deadline has been increased from 30 seconds to 60 seconds. We’ve increased the maximum URLFetch deadline to match from 10 seconds to 60 seconds.
File limits: We’ve increased the number of files you can upload with your application from 3,000 to 10,000 files, and the file size limit has also been increased from 10MB to 32MB.
API Limits: Post payloads for URLFetches are now capped at 5MB instead of 1MB.

We’re also announcing several limited preview features and trusted tester programs:

Cloud SQL Preview: We announced last week that we are offering a preview of SQL support in App Engine. Give it a try and let us know what you think.
Full-text Search: We are looking for early trusted testers for our long anticipated Full-Text Search API. Please fill out this form if you’re interested in trying it out.
Conversion API: Ever wanted to convert from text to PDF in your App? Then consider signing up as a trusted tester for the Conversion API.

Datastore

Cross Group (XG) Transactions: For those who need transactional writes to entities in multiple entity groups (and that's everyone, right?), XG Transactions are just the thing. This feature uses two phase commit to make cross group writes atomic just like single group writes.

Platform Improvements

Experimental Google Cloud Storage Integration: As Google Cloud Storage, formerly Google Storage for Developers, graduates from labs, we are improving our integration by adding access via the files API.
Prediction API: Another of our friends graduating from labs is the Prediction API. Check out some of the App Engine examples in their getting started guide.

Of course, these are just the high level changes. This release is packed full of features and bug fixes, and as always, we welcome your feedback in the group.

Posted by The App Engine Team

Google Cloud SQL: Your database in the cloud

2011-10-06T12:33:00.000-07:00

Cross-posted from the Google Code Blog

One of App Engine’s most requested features has been a simple way to develop traditional database-driven applications. In response to your feedback, we’re happy to announce the limited preview of Google Cloud SQL. You can now choose to power your App Engine applications with a familiar relational database in a fully-managed cloud environment. This allows you to focus on developing your applications and services, free from the chores of managing, maintaining and administering relational databases. Google Cloud SQL brings many benefits to the App Engine community:

No maintenance or administration - we manage the database for you.
High reliability and availability - your data is replicated synchronously to multiple data centers. Machine, rack and data center failures are handled automatically to minimize end-user impact.
Familiar MySQL database environment with JDBC support (for Java-based App Engine applications) and DB-API support (for Python-based App Engine applications).
Comprehensive user interface for administering databases.
Simple and powerful integration with Google App Engine.

The service includes database import and export functionality, so you can move your existing MySQL databases to the cloud and use them with App Engine. Cloud SQL is available free of charge for now, and we will publish pricing at least 30 days before charging for it. The service will continue to evolve as we work out the kinks during the preview, but let us know if you’d like to take it for a spin.

Posted by Navneet Joneja, Product Manager for Google Cloud SQL

Project WOW

2011-09-26T11:32:00.000-07:00

Today's post is contributed by Edward Hartwell Goose of PA Consulting, who is working on an App for the UK’s Met Office to report everyone's favorite bit of small talk, the weather. We hope you find the discussion of his team's experience using App Engine illuminating.

The UK’s Met Office is one of the world’s leading organisations in weather forecasting, providing reports throughout the day for the UK and the rest of the world. The weather information they provide is consumed by a variety of industries from shipping to aircraft, and powers some of the UK’s leading media organisations, such as the BBC.

Although the Met Office is the biggest provider of weather data, they aren’t the only ones collecting information. Thousands of enthusiasts worldwide collect their own weather data, from a wide variety of weather stations - either simple temperature sensors or highly sophisticated stations that rival the Met Office’s own equipment. The question of course is: how do you harness the power of this crowd?

Enter the Weather Observations Website

The Met Office and our team from PA Consulting worked together to answer this question late last year. The end result was The Weather Observations Website, or “WOW” for short. In the 3 months since launch on the 1st June 2011, WOW has recorded 5.5 million weather reports from countries throughout the world. Furthermore, we can retrieve current reports in sub second times, providing a real time map of worldwide weather. We haven’t got the whole globe covered just yet and the UK carries the most sites, but most countries in Western Europe are reporting. We also have reports from a medley of countries throughout the world, from Mauritius, Brazil and as far away as New Zealand. We even have one site reporting at regular intervals in Oman (it’s hot!).

Better yet, as a development team of 2, since launch, we’ve spent almost no time at all doing anything but casual monitoring of WOW. No one carries a pager, and the one time we did have problems (we underestimated demand, and our quota ran out), I was able to upgrade the quota in a minute or so. And I did it from my sofa. On my phone.

How good is that?

WOW - Showing Live Temperature Data Across Europe

Lessons Learnt Building for App Engine

We learnt a lot building WOW. A huge amount in fact. And we’d love to share our insights with you. We’d also love to tell you what we love about App Engine - and why you should use it too. And so you know we’re honest - we’ll tell you what we don’t like too!

Firstly - the good stuff. We think App Engine is a fantastic tool for prototyping and any team working in an agile environment. We’re big fans of SCRUM at PA Consulting, and App Engine was a dream to work with. Compared to some of our colleagues working with difficult build procedures and environments, our full release procedure never took more than 5 minutes. Better yet, App Engine’s deployment tools all hook up with ANT and CruiseControl (our continuous build system), allowing us to run unit tests and deploy new code on every check-in to our code repository. This allowed us as developers to get on with what we do best: develop!

The APIs are great too. The documentation is fantastic and is regularly updated. We use all but the Channel API at the moment (sadly, it uses Google Talk Infrastructure behind the scenes and this gets blocked by some corporate environments). If I could offer any advice to a budding App Engine developer it would be to thoroughly read the documentation, and place the following three questions and their solutions at the forefront of everything you do:

1. Can I do it in the background? - Task Queue API

2. Can I use the low-level API? - Datastore API

3. Can I cache the result? - Memcache API

These three principles have given WOW the performance it has, and will allow it to scale effectively over the coming years. We’ve got some big changes coming over the next couple of months too that should provide even higher performance as well as exciting new features.

Future Developments

So, how about improvements App Engine could make? There are definitely a few, although how important they are will depend on the problems you’re trying to solve.

To begin with, improvements to some of the non technical elements are needed before App Engine becomes truly mainstream. Needing a credit card to pay is a showstopper for some organisations. Backup and restore is also missing, unless you implement it yourself. This is perfectly possible, but can add significant man days to your development effort if your data structure is complex and fast changing.

We also struggled with how to estimate quota to begin with too. One of the brilliant features of App Engine is how easy it is to spool up (and down) new instances to deal with demand. Unfortunately, this also means it can be quite easy to accidentally spool up too many instances and burn quota quickly. Although this has never affected the actual data, it can cause an unpleasant spike in the amount of dollars spent. We also had a similar problem with a MapReduce job getting stuck overnight that caused a scare the next morning. Hopefully the new monitoring API should provide a bit more visibility of these issues as well as automatic email notifications to help catch these issues.

Aside from that, other features will probably depend on the application you’re trying to build. Built in support for geo-queries would be invaluable for WOW. Currently we use an external library, but this adds some extra overhead on our development. Another common feature request is full text search which is essential for projects dealing with large text corpa. Both of these features would allow us to provide better search facilities for our users - for example search by site name or geographic location. These queries can be implemented in App Engine as it is now, but achieving optimal performance and optimal cost are difficult problems that we struggle to complete ourselves.

Final Thoughts

Overall, we’re really impressed by App Engine. The App Engine team regularly releases new versions, and although it does have limitations it has allowed us to concentrate on what really matters to us - the weather. We know WOW will scale without any problems, and we don’t have to worry about any of the hardware configuration or system administration that can easily consume time. Our small team of developers spends all of their time understanding the business problems and improving WOW.

We’re really looking forward to taking WOW forward in the future, we hope you can join us: http://wow.metoffice.gov.uk.

Edward Hartwell Goose (@edhgoose)

Developer

PA Consulting

Fall 2011 App Engine events

2011-09-22T07:30:00.000-07:00

Buenos días desde Buenos Aires!! Wow, we are just blazing through the summer! We had an exciting Google I/O in May where we saw the launch of the Go runtime, introduction of the Backends feature, and confirmed Google's commitment to the platform with our plans to leave preview mode, not to mention the five releases since that time. We then hit the road this summer with team members appearing around the globe, including mega events like the Cloud Computing Expo, EuroPython, and OSCON.

At Google I/O, many App Engine team members gave in-depth technical sessions. Alfred Fuller, an App Engine Datastore engineer, revealed how the High Replication datastore gives greater reliability and availability for users. Michael Handler, one of App Engine’s talented Site Reliability Engineers, discussed how App Engine works in production. At OSCON, the largest open source event in the world, yours truly gave a three-hour App Engine overview and workshop to help new users get up to speed, and earlier in the year at PyCon, described how Python users can avoid “vendor lock-in” via the Django-nonrel project by porting their applications from webapp to Django, allowing them to move on to or off of App Engine as they please with only minor configuration changes. We participate in events like these because we like to communicate with users to find out what they like & don’t like about the platform, and we often take their suggestions to heart!

This fall we have a full schedule, appearing at many developer events worldwide, including Google Developer Days and Google DevFests. At the Google events, we'll recap App Engine and those announcements we made at I/O, and you’ll learn how to build exciting applications using App Engine and other Google cloud technologies as well as how to build games in the cloud. Here are the dates, events, and locations that we will be visiting:

Fall 2011
Sep 19-20 - Google Developer Day Argentina - Buenos Aires - Wesley Chun, Chris Schalk
Sep 19-23 - Strata - New York - Chris Schalk
Sep 23-24 - PyCon Argentina - Junín - Wesley Chun
Sep 26-30 - Congresso Brasileiro de Software - São Paulo - Wesley Chun
Sep 29-Oct 1 - Python Brasil[7] - São Paulo - Wesley Chun
Oct 8-9 - Silicon Valley CodeCamp - Los Altos Hills - Wesley Chun
Oct 10 - Google Developer Day Russia - Moscow - Iein Valdez, Fred Sauer
Oct 18 - Google Developer Day Czech Republic - Prague - Iein Valdez, Fred Sauer
Oct 20 - Google DevFest France - Paris - Iein Valdez, Johan “Proppy“ Euphrosine
Nov 1 - Google Developer Day Japan - Tokyo - Takashi Matsuo, Johan “Proppy“ Euphrosine
Nov 8 - Google Developer Day Australia - Sydney - Chris Schalk, Johan “Proppy“ Euphrosine
Nov 12 - Google DevFest Singapore - Singapore - Chris Schalk, Wesley Chun
Nov 13 - Google Developer Day Israel - Tel-Aviv - Michael Manoochehri
Nov 16 - Google DevFest Indonesia - Jakarta - Chris Schalk, Wesley Chun
Nov 19 - Google Developer Day Germany - Berlin - Michael Manoochehri, Wesley Chun

If these aren't close enough to you, keep an eye out on this list as we'll add new events and locations as they are scheduled. A calendar with all of Google's developer events is also available. We look forward to meeting you soon!

Posted by Wesley Chun (@wescpy), Google cloud Developer Relations team

Pictarine: pictures in the cloud

2011-09-19T11:29:00.000-07:00

Pictarine is a photo management web application, launched in 2010, that allows people to easily manage and share all of their photos from Flickr, Picasa, Facebook, Twitter and other sites. Pictarine developers Guillaume Martin and Maxime Rafalimanana have contributed the following post discussing their experiences using Google App Engine and Google Web Toolkit.

From the start, we used Google technologies in developing Pictarine and we wanted to share our experience with them so far. In this post, we will shed some light on the weaknesses and strengths we found in Google Web Toolkit (GWT) and Google App Engine. We will also discuss how we leveraged GWT to build a new technology that allows Pictarine to seamlessly display photos from the computer directly into the browser. The following diagram is an overview of how our application works.

Building a mashup in the cloud with Google App Engine

The Pictarine team is made of a web designer and two developers who previously worked mainly with Java based enterprise technologies and had a little experience with web technologies. When we started the project in early 2009, we were quite open on learning new languages like Python or Ruby, but when App Engine announced that Java would be supported, we were really excited to give Google App Engine a try.

The first few months, learning about the App Engine environment was quite easy and dare I say fun. Testing our code on Google’s servers from Eclipse IDE was only one click away. So we built our first prototype fast and we quickly decided to adopt App Engine. Then we started to build the core of our application: the engine that uses the API from Flickr, Picasa, Facebook to fetch the users’ photos. This is where we hit the first limitations of App Engine. Most users have a lot of photos on these services and retrieving them can take some time. But App Engine has strict limits on how long a request should last: an outgoing HTTP request cannot last more than 10 seconds and cannot process a request for more than 30 seconds. So while building our architecture we found ourselves writing approximately one third of our code dealing with these limitations: paginating our requests, creating background tasks to store data in small batches, etc.

In early 2010, when we launched our alpha version, everything went smoothly. We had some good press coverage and App Engine met our expectations in handling our first users. During 2010, we worked on implementing new features requested by our users, and during this period of time we were really impressed by the way App Engine evolved. Many of the limitations were lifted and great new features were added. We are now able to use Task Queues for requests that last up to 10 minutes, which we fully use to sync our users’ photos and albums. One of the features we like the most is the Channel API, a push notification system that allows us to instantly show a photo in every connected browser as soon as it is uploaded. App Engine is still not perfect but has greatly improved and when we see its roadmap, we are quite confident it will continue to improve.

Building a fresh photo experience with Google Web Toolkit

When we started Pictarine, we wanted a fast, distraction free interface that would allow our users to focus on their photos. We wanted the interface to adapt to the screen resolution, displaying a lot of photos on large screens and fewer on small ones. We wanted it to be automatically updated when new comments or new photos are added. We wanted a web application. As we eliminated Flash quite quickly (based on our user experience...) we started to look at tools to build HTML/CSS/Javascript applications. We settled quickly on GWT: while coding in Java, with all the tools we are used to (even the debugger), we could produce optimized Javacript that would run in every browser! When we started with GWT, it was already 3 years old, so we had few complaints about it. The main issue was that we had to always keep in mind that the Java code we produced was ultimately translated to Javascript. So some Java methods, such as the Reflection API, are not allowed. Another thing that was not obvious to us when we started with GWT was that a java developer needs an intimate knowledge of HTML/CSS if he/she wants to go beyond the basic user interface provided by the GWT widgets.

What we really like about GWT in our architecture is the ability to share code between client and server: we can use the same Photo or Album class on the client and the server and the GWT RPC system allows us to automatically share the same Java object on both side. We can also have the same data validation code on both sides: we can alert the user immediately on errors and still validate the data on the server just in case.

Another great feature we like about GWT is its handling of internationalisation. From the beginning we wanted to build a website available for all Internet users, so supporting English as well as our native language (French) was almost obligatory. Fortunately, GWT makes it really easy to generate centralized localization files so that we just have to translate.

Finally, to illustrate how great Javascript generation is, when IE9 came out, we waited a few weeks for GWT to support it and our application was compatible after a recompile! Of course, the IE9 team also did a good job with their HTML5/CSS3 engine.

Building an universal uploader

After the launch of our alpha in 2010, our users were able to see and share their photos from Flickr, Picasa, Facebook. But they still had to put their photos on these websites first before coming to Pictarine. This limitation quickly became the first request on our feedback system. We needed to let our users do everything from Pictarine, including uploading photos. Uploading many photos from a website is still not a trivial process. Most websites choose Flash to allow users to upload multiple files at once, but our experience with it was that it often crashed after a while. Some use Java applets, but they are never well integrated and always look odd. At Pictarine we decided to tackle this problem by using Java Applet for their stability across all platforms but without using it to render photos or folders.

We have built a technology that uses the GWT RPC mechanism to talk to a Java Applet: photos, upload progression are rendered in HTML/CSS and the applet takes care of photos resizing and uploading. Sharing a photo from a camera is now a one-step process. This technology also allows users to browse their local files directly in their browser and it is fully integrated in our design.

We believe that this new use of Java applets can help blur the line between the Desktop and the Cloud by seamlessly integrating desktop files in any web application.

In conclusion, we can say that we are really happy with the choices we made with App Engine and GWT. App Engine is a great service that perfectly handled the spike in traffic we saw right after articles on Mashable and Lifehacker were published. So we recommend it to every lean startup out there who loves developing in Java, Python or Go. Posted by The App Engine Team

App Engine 1.5.4 SDK Release

2011-09-12T12:59:00.000-07:00

It’s been a busy four weeks for us (you may have heard), but we are still on track with our new monthly release schedule. Today we have new SDKs for you, with some new features as well as some bug fixes.

Overall Changes

Blobstore API - We’ve introduced an option to specify a limit on your blob upload size. This feature will allow you to expose blob uploads to your users, while still being able to ensure that their uploads won't exceed your preferred limits.
Datastore Query Improvements - We’re also continuing our theme from the past few releases to tune our Datastore query planner to give users more flexibility. Starting with 1.5.4, queries with equality filters on multiple properties will now continue scanning up to the 30 second Datastore query deadline. Many of these queries that used to generate an error due to inefficient indexes will now succeed.
Datastore Write Ops in the SDK - In response to the feedback from our recent Side By Side Bills release, we are now displaying the number of write ops needed to store an entity in the SDK dataviewer. Write ops include both the entity write as well as the index writes that are executed when an entity is added. You can always reduce the number of write ops by turning your indexed properties into unindexed properties, just make sure you don’t reference those properties in any of your queries! (Java, Python).

Java

Prospective Search API - We’ve released the experimental Java version of our Prospective Search API. Prospective Search lets you detect and take action on datastore entities that match certain criteria when they are written.

Python

Memcache - You can now make calls asynchronously with the Memcache API. With Asynchronous Memcache, your application does not need to block on calls to the Memcache API and can continue processing a request instead of waiting for a response from Memcache. Fear not Java friends, we’ll have matching functionality for you in an upcoming release.

Full release notes can be found in the usual location (Java, Python), and we’re always listening for your feedback in the groups. Happy coding!

Posted by The App Engine Team

A few adjustments to App Engine’s upcoming pricing changes

2011-09-09T11:52:00.000-07:00

Last week we rolled out side-by-side billing to give you a more detailed preview of how you’ll be affected by the price changes that we announced in May. We received a variety of feedback and have made a few important changes based on it. Our intent is to be as open and transparent about the changes as possible and to give you enough time to prepare. In that spirit, our Engineering Director has also shared some of his personal thoughts.

We understand that the new rates surprised some of you. We’ve been listening closely to your feedback, and we wanted to share an update on the changes we’re making to help ensure you have an accurate picture of how the new pricing will affect your app. Although prices will increase, we’re confident that you’ll find App Engine still provides great value.

Based on your feedback we’re taking the following steps:

Extended review period: We’re now giving you almost eight weeks before introducing the new pricing. You now have until November 1 to configure and tune your application to manage your costs.
Increased free Instance Hours: We are increasing the number of free Instance Hours from 24 to 28. This means that people who are using a free app to try out App Engine can run a single instance all day with a few spikes and still remain below our free quota. This will be reflected in the comparison bills soon.
Extended discount: We’ll continue to offer the 50% discount on instance prices until December 1st, at which time Python 2.7 should be available. Python 2.7 will include support for concurrent requests, which could further lower your costs.
Faster Usage Reports: We appreciate the importance of quickly being able to see the effect your tuning has on your bill and starting today we’ll provide your Usage Report (and the included comparison bills) within one day instead of the previous three.
Better analysis tools: We are working on better ways for you to model the cost of your apps. We will add the “billing” line into the instances graph on the Admin Console. We’re adding datastore billing information into the dev console to making it easier for you to track how the changes you make affect your bill, which should also help lower the cost.
Premier accounts: we know a lot of our customers are eagerly awaiting Premier accounts to get operational support, offline billing, unlimited accounts, and the SLA. So we will not wait until November 1st for this, but rather launch Premier accounts as soon as possible. If you are interested in a Premier account, please contact us at appengine_premier_requests@google.com.

We also wanted to share some of the main ways for you to lower your bill and get a better sense of the true cost of App Engine:

Set Max Idle instances: Setting Max Idle Instances to a lower level will help lower your costs as we will only charge for idle instances up to the maximum you set. This could impact your performance so it’s worth reading up on the ramifications.
Always-On reflected in bills: Currently the side-by-side bills still include the cost of always-on even though it will be retired when the new pricing launches (to be replaced by min idle instances). We’re working on a fix for this. Until then you can comfortably subtract 48 instance hours per day from the estimate.
Reserved instance hours: The simplest way to lower the charge for instance hours is to consider using reserved instance hours. They are 37.5% cheaper than on-demand, but you do need to commit to a certain number of them over the course of a week.
Managing resources: Check out this article, which provides more helpful advice on how to efficiently manage your resources and lower costs.

We launched App Engine in preview three years ago to make it easier for you to build, maintain and scale web applications. Since then, we’ve added many features like Java and Go language support, created a High Replication Datastore, and added many other APIs. And there’s a lot more cool stuff to come. It’s heartening to us that so many developers- 150,000 and counting- have chosen to use App Engine to run their apps. While we can’t continue to offer App Engine at our original prices, we can commit to listening to your feedback, acting on it, and working hard to give you a great platform for your apps at the most competitive price possible. If you find that’s not the case, or have any questions about this at all, please feel free to contact us at appengine_updated_pricing@google.com.

The App Engine Team

$50 Credit for new billing signups and budget changes

2011-08-31T14:22:00.000-07:00

At Google I/O this year, we announced that App Engine will be going out of preview before the end of the year. One aspect of leaving preview is our new pricing model. As promised we have started providing the ability for any application Admin to compare their existing cost to their new costs in the Admin Console (under “Billing History”). Now that you have an idea of what changes may impact your app, we’ve also written an article to help you optimize your application. We know many of you would rather code than think about your budget and billing settings, so to help with the transition we are extending a one-time courtesy credit of $50 to all free Apps that sign up for billing and all paid Apps that modify their budgets between now and October 31, 2011. Once you have signed up or made changes to your billing settings, a $50 credit will appear in the “Billing History” area of the Admin Console for your application. For more information on how to enable billing or change your settings, please see Billing and Budgeting Resources.

This is a big step for the App Engine Team; thank you for your continued support and feedback, and we hope you find these tools useful as we get closer to our goal of leaving preview!

Posted by the App Engine Team

App Engine 1.5.3 SDK Released

2011-08-17T10:11:00.000-07:00

We’re pleased to announce another App Engine release today. You might have noticed that the rate of releases has gone up slightly in the past few months. We’ve made some changes internally so we are looking to push out a new release every month. This month includes a few Datastore updates, some changes to Blobstore API and Memcache API, and finally a new feature for Java developers.

Python and Java Changes

Blobstore API - We’ve removed the limits on the size of blob uploads. You can now upload files of any size, allowing your app to serve images, video, or anything your internet connection can handle.

Datastore Changes

Index retrieval - We’ve added the ability for you to programmatically retrieve the list of indexes you’ve currently defined in the datastore, as well as their statuses.
Datastore Admin - You can now enable the Datastore Admin function from the Admin Console. This will allow Java users to make use of this functionality, like deleting all entities of a certain kind, without having to upload a Python version of their application. And for Python developers, you no longer need to enable this in your app.yaml file.
HRD Migration Trusted Testers - We are seeking early adopters to try out an improved HRD migration tool that requires a read-only period relative to your datastore write rate (as opposed to your datastore size, which is how the current version behaves). Please see the release notes for more information.

Python Updates

Memcache API - We now support the CAS (compare-and-swap) operation in our Python Memcache API (Java already had it). This can be used to update a value in Memcache only if no other requests have updated it between when the value was retrieved and when you go to update it.

Java Updates

Download app - Using the AppCfg download_app command, you can download any files that were uploaded from your war directory when you last updated the app version.

This release also contains small updates and bugfixes for both Python and Java so be sure to check out the full release notes. Feedback, discussion, and questions can be posted in our Google Group.
Posted by The App Engine Team

Google App Engine receives SAS70 and SSAE-16 certification

2011-08-05T16:35:00.000-07:00

As we get ready for App Engine to leave preview release later this year, we want to make clear that in addition to improving our reliability, adding new features, and introducing developer support, we are also committed to demonstrating we have sufficient controls in place to safeguard the data you store with us.

Today we are happy to announce that over the past few weeks Google App Engine successfully completed the audit process for the SAS70 Type II, SSAE 16 Type II, and ISAE 3402 Type II standards.

You can read more about the Google’s certifications and commitment to protect user data on the Enterprise blog.

Posted by The App Engine Team

App Engine 1.5.2 SDK Released

2011-07-21T08:52:00.000-07:00

As the summer heat descends on the Northern Hemisphere we thought we’d release our newest App Engine version with some changes that are sure to keep you playing around in the cool, air-conditioned indoors (hey, you don’t want your computer to overheat, right?).

Production Changes

Adjustable Scheduler Parameters - As we previously discussed, we are introducing two scheduler knobs (okay, they actually look like sliders) that will allow you to control some of the parameters that influence how many Instances run your application. Today you will be able to set the minimum pending latency and maximum number of idle instances for your application.

Datastore Changes

Advanced Query Planning - We are removing the need for exploding indexes and reducing the custom index requirements for many queries. The SDK will suggest better indexes in several cases and an upcoming article will describe what further optimizations are possible.
Namespaced Datastore Stats - Now, in addition to getting overall datastore stats, we are providing a new option to query datastore stats per namespace.

Task Queue Changes

New Task Queue details page - We’ve revamped the Task Queue details page in the Administration Console to provide more information about the tasks being run. You can now see the headers included in the enqueued task, the payload, and information from previous task runs.
1MB Pull Task Size - It’s our belief that there is only one way for size limits to go - and that’s up! So with this release we’ve increased the size for pull tasks to 1MB.
Pull queue lease modification - We’ve introduced a new method for Pull Queues that allows you to extend the lease on existing tasks if the initial lease on the task was insufficient.

Lastly, we have some exciting news related to the experimental Go runtime. While it still remains experimental, starting with 1.5.2, all HRD apps will have access to the Go runtime in production.

As always, there are also some small features and bug fixes, the full list of which can be found in our release notes (Python, Java). We look forward to your feedback and questions in our forum.

Posted by The App Engine Team

Postmortem: Java App Engine outage, July 14, 2011

2011-07-20T12:24:00.000-07:00

Summary

Last week, we posted about a limited outage on July 14, 2011. Now that our internal postmortem is complete, we thought you would also like to get more detail about what went wrong and what we are going to do to ensure this doesn't happen again.

Root Cause and Analysis

The main lesson learned is to improve our live traffic testing as a relatively minor bug triggered a corner case for some of our customers. The bug was in a new release of the infrastructure in the App Engine Java execution environment. During development, testing, and qualification, this bug was essentially hidden from view because it only manifested itself under specific load patterns. During the outage, requests to affected applications would fail with errors when traffic was routed to affected instances. Application logs would have shown affected instances experienced high latency, error rates, or were not reachable from the Internet. This could have been caught by letting the live traffic testing run longer.

In order for live traffic testing to work properly, we need to improve our monitoring as well. In this case, having more points from which to do black box monitoring would have helped immensely. We are currently working on much broader monitoring for App Engine and will be integrating more extensive black box testing in upcoming quarters.

Once again, we’d like to point out that we could have done a much better job of communicating issues to all of you. While we strive to strike a balance between letting you know about major issues and not bothering you about the day-to-day operations; we clearly should have communicated this incident to you sooner. Rest assured you’ll be better informed of issues in the future.

Timeline

July 14, 2011 - 11:30 AM US/Pacific - The new Java execution environment is released to production.

July 14, 2011 - 5:00-6:00 PM US/Pacific - The previously scheduled Master/Slave read-only maintenance period occurred.

July 14, 2011 - 8:00-9:30 PM US/Pacific - Monitoring shows error rates and latency for Java applications using the Master/Slave datastore are slowly increasing across the entire system. Investigation reveals that the new Java execution environment is malfunctioning.

July 14, 2011 - 9:30 PM US/Pacific - Rollback of the Java execution environment to the previous version begins. Latency and error rates begin to fall.

July 14, 2011 - 11:30 PM US/Pacific - Rollback of the Java execution environment to the previous version completes. Java Master/Slave applications are functioning normally.

Remediation

Faster notification on our status site and downtime-notify mailing list
More live traffic stress tests for new releases
Better black box monitoring to detect small impacts more quickly

[Edit] Clarification: no HR datastore apps were affected. Overall, the outage resulted in a 1.9% error rate, affecting approximately 0.005% of all App Engine traffic at peak.

Posted by Wesley Chun, Google App Engine Team