Google App Engine Blog

The new Task Queue API on Google App Engine

2009-06-18T14:56:00.000-07:00

With release 1.2.3 of the Python SDK, we are psyched to present an exciting new feature - the Task Queue API. You can now perform offline processing on App Engine by scheduling bundles of work (tasks) for automatic execution in the background. You don't need to worry about managing threads or polling - just write the task processing code, queue up some input data, and App Engine handles the rest. If desired, you can even organize and control task execution by defining custom queues. A quick example:

   # for each user, add a task to send a custom email message
   for u in users:
       taskqueue.add(url='/work/sendmail',
          params=dict(to=u.email, subject='Hello ' + u.name, body='this is a message!'))
  
   return # finished now, emails will be sent offline when tasks execute

   ...

   # task handler at /work/sendmail, automatically called for each task created above
   class MailWorker(webapp.RequestHandler):
      def post(self):
         mail.send_mail(
            'from_me@example.com',
            self.request.get('to'),
            self.request.get('subject'),
            self.request.get('body'))

We're eager to help you learn and experiment with Task Queues. The team recently presented the feature at Google I/O and the video is now available (slides are here). We've also prepared a set of demos to help you get started. And of course, don't miss the feature documentation. The Task Queue API is Python-only for now; we'll have a Java language version available soon.

Please note that the Task Queue API is currently a Labs release - we want to get your feedback on its usability and functionality before finalizing the API. You'll notice that its Python import path currently includes the 'labs' module (google.appengine.api.labs.taskqueue). Before the feature is promoted out of Labs, we may need to:

Change the quotas and limits which apply to Task execution (definitely, we hope to raise the number of Tasks you can use per day).
Change the API itself if there are usability or functionality issues.
Change how we bill for Task Queue usage.

Once we're ready to promote the feature out of Labs, we'll give weeks of notice and provide a transition path for our developers.

Last but not least, the 1.2.3 release is full of other new stuff as well! Stay tuned to the blog for more updates or check the release notes for exciting info on:

Asynchronous urlfetch support
Django 1.0 support

Visit the Downloads page to get SDK 1.2.3 now!

The Task Queue API is the first milestone of our plan to deliver rich support for offline processing. There's more to come, but we hope the simplicity and power of this first release opens a new range of possibilities for our developers. Try it out and let us know! We'll be watching the Group for your input.

-- The App Engine Team

App Engine @ Google I/O goodness for all to enjoy

2009-06-11T17:10:00.000-07:00

Back in April when we launched Java support, we gave the first 10,000 developers who signed up access to the new runtime. In case you haven't heard, we recently announced at Google I/O that App Engine for Java signup is now open. We're excited to see more developers joining our community!

For those who missed Google I/O or want a refresher on the sessions, the videos are now posted online. From building a complex and scalable app on App Engine to offline processing, there are a lot of things to learn from these sessions. You can find a more detailed write up of the App Engine sessions in this post to the Google Code Blog, but here's the complete list:

Apart from the sessions, we also had a Developer Sandbox featuring App Engine partners and customers. We got a chance to interview and ask many of them to share their experience developing on App Engine. Check out these video interviews to get more insight to App Engine:

Last but not least, we've also made available the handy Python cheat sheet we gave out during Google I/O. You can download it here. We want to thank our partners and the App Engine developer community for a fantastic Google I/O 2009! It was great meeting those of you who were there. We look forward to the next year!

Posted by Amanda Surya, App Engine Team
Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Changing Quotas To Keep Most Apps Serving Free

2009-06-10T18:47:00.000-07:00

Since App Engine launched, our goal has been to offer substantial free hosting resources to as many developers as possible. As previously announced, we are changing our free resource quota levels, effective on June 22nd. Our target level of free support has been 5 million page views per month for a reasonably efficient web application.

When we launched App Engine, we were intentionally generous in our free quotas, both because we didn't know resources usage of a typical request, and because we didn't offer a billing feature to allow developers to buy more resources for a higher-traffic app. Since our billing feature launched in February, developers with high-traffic applications can purchase additional resources far beyond our original fixed free quotas. Having been live for more than a year, we now have good empirical data on the average resource consumption per request, so we're able to set our quotas to more accurately support our 5 million page views target.

This change in the free quotas offered to every application is intended to allow us to continue to offer substantial free application hosting to any interested developer. We have grown a lot in the last year, with over 80,000 applications created, and with these changes to our free quotas, more than 90% of these applications will continue to serve completely free. To empirically determine reasonable levels for our quotas, we measured resource usage for all applications running on App Engine over a recent 7-day period. For each of the quotas, we took the highest daily average resource usage per HTTP request out of the 7-day period:

CPU: 0.14 CPU-seconds/request
Outbound data transfer: 6149 bytes out
Inbound data transfer: 803 bytes in

Multiplied by 5 million requests spread over a 30 day month, these per-request resource statistics translate to daily resource usage of 6.4 CPU-hours and 1.02 gigabytes of outbound data transfer. We top off the numbers by offering 6.5 CPU-hours and 1.07 gigabytes of outbound transfer. Though typically inbound data transfer is a small fraction of outbound data transfer, we made inbound and outbound data transfer symmetric to ease initial data uploads.

Finally — what do we mean by reasonably efficient applications? Simply put, efficient applications avoid unnecessary computation or data transfer, and two techniques common to efficient App Engine applications are the use of caching headers and memcache. Caching headers in an HTTP response prevent a user's browser from needlessly re-downloading information that hasn't changed, both speeding up the user experience and saving bandwidth. Similarly, memcache keeps frequently accessed data in a memory cache on App Engine servers, rather than always reading from disk in the Datastore, therefore saving CPU usage and Datastore load.

Again, these changes ensure we can keep our continuing promise to make it free to get started with App Engine.

Posted by Chris Beckmann, App Engine team

10 things you (probably) didn't know about App Engine

2009-06-05T14:44:00.000-07:00

What could be better than nine nifty tips and tricks about App Engine? Why, ten of course. As we've been participating in the discussion groups, we've noticed that some features of App Engine often go unnoticed so we've come up with just under eleven fun facts which might just change the way that you develop your app. Without further ado, bring on the first tip:

1. App Versions are strings, not numbers

Although most of the examples show the 'version' field in app.yaml and appengine-web.xml as a number, that's just a matter of convention. App versions can be any string that's allowed in a URL. For example, you could call your versions "live" and "dev", and they would be accessible at "live.latest.yourapp.appspot.com" and "dev.latest.yourapp.appspot.com".

2. You can have multiple versions of your app running simultaneously

As we alluded to in point 1, App Engine permits you to deploy multiple versions of your app and have them running side-by-side. All the versions share the samedatastore and memcache, but they run in separate instances and have different URLs. Your 'live' version always serves off yourapp.appspot.com as well as any domains you have mapped, but all your app's versions are accessible at version.latest.yourapp.appspot.com. Multiple versions are particularly useful for testing a new release in a production environment, on real data, before making it available to all your users.

Something that's less known is that the different app versions don't even have to have the same runtime! It's perfectly fine to have one version of an app using the Java runtime and another version of the same app using the Python runtime.

3. The Java runtime supports any language that compiles to Java bytecode

It's called the Java runtime, but in fact there's nothing stopping you from writing your App Engine app in any other language that compiles to JVM bytecode. In fact, there are already people writing App Engine apps in JRuby, Groovy, Scala, Rhino (a JavaScript interpreter), Quercus (a PHP interpreter/compiler), and even Jython! Our community has shared notes on what they've found to work and not work on the following wiki page.

4. The 'IN' and '!=' operators generate multiple datastore queries 'under the hood'

The 'IN' and '!=' operators in the Python runtime are actually implemented in the SDK and translate to multiple queries 'under the hood'.

For example, the query "SELECT * FROM People WHERE name IN ('Bob', 'Jane')" gets translated into two queries, equivalent to running "SELECT * FROM People WHERE name = 'Bob'" and "SELECT * FROM People WHERE name = 'Jane'" and merging the results. Combining multiple disjunctions multiplies the number of queries needed, so the query "SELECT * FROM People WHERE name IN ('Bob', 'Jane') AND age != 25" generates a total of four queries, for each of the possible conditions (age less than or greater than 25, and name is 'Bob' or 'Jane'), then merges them together into a single result set.

The upshot of this is that you should avoid using excessively large disjunctions. If you're using an inequality query, for example, and you expect only a small number of records to exactly match the condition (e.g. in the above example, you know very few people will have an age of exactly 25), it may be more efficient to execute the query without the inequality filter and exclude any returned records that don't match it yourself.

5. You can batch put, get and delete operations for efficiency

Every time you make a datastore request, such as a query or a get() operation, your app has to send the request off to the datastore, which processes the request and sends back a response. This request-response cycle takes time, and if you're doing a lot of operations one after the other, this can add up to a substantial delay in how long your users have to wait to see a result.

Fortunately, there's an easy way to reduce the number of round trips: batch operations. The db.put(), db.get(), and db.delete() functions all accept lists in addition to their more usual singular invocation. When passed a list, they perform the operation on all the items in the list in a singledatastore round trip and they are executed in parallel, saving you a lot of time. For example, take a look at this common pattern:

for entity in MyModel.all().filter("color =",
    old_favorite).fetch(100):
  entity.color = new_favorite
  entity.put()

Doing the update this way requires one datastore round trip for the query, plus one additional round trip for each updated entity - for a total of up to 101 round trips! In comparison, take a look at this example:

updated = []
for entity in MyModel.all().filter("color =",
    old_favorite).fetch(100):
  entity.color = new_favorite
  updated.append(entity)
db.put(updated)

By adding two lines, we've reduced the number of round trips required from 101 to just 2!

6. Datastore performance doesn't depend on how many entities you have

Many people ask about how the datastore will perform once they've inserted 100,000, or a million, or ten million entities. One of the datastore's major strengths is that its performance is totally independent of the number of entities your app has. So much so, in fact, that every entity for every App Engine app is stored in a singleBigTable table! Further, when it comes to queries, all the queries that you can execute natively (with the notable exception of those involving 'IN' and '!=' operators - see above) have equivalent execution cost: The cost of running a query is proportional to the number of results returned by that query.

7. The time it takes to build an index isn't entirely dependent on its size

When adding a new index to your app on App Engine, it sometimes takes a significant amount of time to build. People often inquire about this, citing the amount of data they have compared to the time taken. However, requests to build new indexes are actually added to a queue of indexes that need to be built, and processed by a centralized system that builds indexes for all App Engine apps. At peak times, there may be other index building jobs ahead of yours in the queue, delaying when we can start building your index.

8. The value for 'Stored Data' is updated once a day

Once a day, we run a task to recalculate the 'Stored Data' figure for your app based on your actual datastore usage at that time. In the intervening period, we update the figure with an estimate of your usage so we can give you immediate feedback on changes in your usage. This explains why many people have observed that after deleting a large number of entities, theirdatastore usage remains at previous levels for a while. For billing purposes, only the authoritative number is used, naturally.

9. The order that handlers in app.yaml, web.xml, and appengine-web.xml are specified in matters

One of the more common and subtle mistakes people make when configuring their app is to forget that handlers in the application configuration files are processed in order, from top to bottom. For example, when installing remote_api, many people do the following:

handlers:
- url: /.*
  script: request.py

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

The above looks fine at first glance, but because handlers are processed in order, the handler for request.py is encountered first, and all requests - even those for remote_api - get handled by request.py. Since request.py doesn't know about remote_api, it returns a 404 Not Found error. The solution is simple: Make sure that the catchall handler comes after all other handlers.

The same is true for the Java runtime, with the additional constraint that all the static file handlers in appengine-web.xml are processed before any of the dynamic handlers in web.xml.

10. You don't need to construct GQL strings by hand

One anti-pattern that comes up a lot looks similar to this:

q = db.GqlQuery("SELECT * FROM People "
    "WHERE first_name = '" + first_name 
    + "' AND last_name = '" + last_name + "'")

As well as opening up your code to injection vulnerabilities, this practice introduces escaping issues (what if a user has an apostrophe in their name?) and potentially, encoding issues. Fortunately,GqlQuery has built in support for parameter substitution, a common technique for avoiding the need to substitute in strings in the first place. Using parameter substitution, the above query can be rephrased like this:

q = db.GqlQuery("SELECT * FROM People "
    "WHERE first_name = :1 "
    "AND last_name = :2", first_name, last_name)

GqlQuery also supports using named instead of numbered parameters, and passing a dictionary as an argument:

q = db.GqlQuery("SELECT * FROM People "
    "WHERE first_name = :first_name "
    "AND last_name = :last_name", 
    first_name=first_name, last_name=last_name)

Aside from cleaning up your code, this also allows for some neat optimizations. If you're going to execute the same query multiple times with different values, you can useGqlQuery .bind() to 'rebind' the values of the parameters for each query. This is faster than constructing a new query each time, because the query only has to be parsed once:

q = db.GqlQuery("SELECT * FROM People "
    "WHERE first_name = :first_name "
    "AND last_name = :last_name")
for first, last in people:
  q.bind(first, last)
  person = q.get()
  print person

Posted by Nick Johnson, App Engine Team

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Java SDK version 1.2.1 now available

2009-05-13T17:59:00.000-07:00

Version 1.2.1 of the App Engine SDK for Java is now available to download. This was the first update to our new Java language support and includes a number of bug fixes and feature additions including:

Support for unindexed datastore properties of arbitrary types
Embedded UTF-8 characters in JSPs now render correctly
Increase in response size limit from 1MB to 10MB
Support for Thread.setContextLoader()

There are a host of updates for the JDO/JPA layer as well. The project wiki has a full list of fixes and new features.

As always, we're very interested in your feedback with the SDK, especially during this early look period -- please feel free to share your experiences in the Java runtime discussion group.

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Posted by Jason Cooper, App Engine Team

web2py support, new datastore backend for apps, and more!

2009-05-11T09:47:00.000-07:00

It has been an exciting month and a half for App Engine since the last community update. In addition to two SDK updates, Cron support, new Java runtime, etc., there have been a host of noteworthy community developments as well, only a few of which are mentioned below.

web2py Python framework now supports App Engine
We are always interested in frameworks that natively support App Engine, and the web2py framework recently came to our attention. Completely written in Python, web2py is described as a "free and open source full-stack enterprise framework for agile development of fast, secure and portable database-driven web-based applications." If your framework has native App Engine support let us know and we'll include you in a future blog post.

Alternative datastore backend for App Engine applications released
bdbdatastore is an alternative datastore backend for App Engine applications. Now at version 0.2, bdbdatastore has complete feature parity with the App Engine's production datastore, meaning you can run your App Engine application on your own hardware with bdbdatastore as the storage backend without having to change any of the code written previously to access App Engine's native datastore. Intended primarily for developers who want to host their own App Engine-based apps, bdbdatastore manages to be far more robust and scalable than the datastore included with the development server while being more manageable than larger backends like HBase and HyperTable.

App Engine supports White House town hall meeting
In late March, the White House hosted an online town hall meeting, soliciting questions from concerned citizens directly through its website. To manage the large stream of questions and votes, the White House used Google Moderator, which runs on App Engine. At its peak, the application received 700 hits per second, and across the 48-hour voting window, accepted over 104,000 questions and 3,600,000 votes. Despite this traffic, App Engine continued to scale and none of the other 50,000 hosted applications were impacted.

For more on this project, including a graph of the traffic and more details on how App Engine was able to cope with the load, see the Google Code blog.

App Engine "Sprint" held in Taipei
The Taipei App Engine Sprint was a coding event at Google Taipei office in mid-March. Seventeen Python developers split into five teams and worked for 10 hours straight to build a new application from scratch. in 10 hours. The winning application, Sight History, is a Picasa search-based application which displays a time distribution chart, slide show links, and a Google Map for photos queried from Picasa Web Albums. Only one of the four developers on this team was an active web developer, which illustrates how easy it can be to turn a good idea into a working web application with App Engine and other Google APIs.

Earth Hour 2009 site hosted on App Engine
Earth Hour, an initiative of WWF, encourages individuals, businesses, and governments to turn off their lights for one hour to show their support for action on climate change. The site aims to raise awareness of climate change and to show people and companies how easy it is to take action to avoid global warming. After launching this past December on App Engine, the site was able to scale seamlessly to handle over 450 hits per second during the Earth Hour event on March 28, making it one of the top App Engine applications served that day.

Posted by Jason Cooper, Google App Engine Team

Python SDK version 1.2.2 released

2009-05-08T15:39:00.000-07:00

We've released version 1.2.2 of the App Engine SDK for Python. Some highlights from the release notes:

Urlfetch fetch now has support for user configurable deadlines.
Datastore indexes on single properties can now be disabled by setting indexed=False on the property constructor.
Datastore now supports Key-only queries, using either SELECT __key__ or db.Query(Model, keys_only=True)
Bulk Loader improvements: New appcfg download_data command. Better backoff support and debugging output for long requests.

For a full list see the SdkReleaseNotes wiki page.

Downloads for Windows, Mac, and Linux are available on the Downloads page. This SDK update was for the Python runtime, so please post your feedback in the Python runtime discussion group.

SDK version 1.2.1 released

2009-04-22T17:52:00.000-07:00

We've released version 1.2.1 of the SDK for the Python runtime. Here are some highlights from the release notes:

Stable, unique IDs for User objects. The Users service now provides a unique user_id for each user that stays the same even if a user changes her email address.
The Images API now supports compositing images and calculating a color histogram for an image.
New allowed mail attachment types: ics, vcf
Urlfetch requests can now set the User-Agent header.
An App Engine-specific version of the Python PyCrypto cryptography library is now available. Learn more at http://code.google.com/appengine/docs/python/tools/libraries.html
The bulk loader configuration format has changed to allow non-CSV input. This change is not backwards compatible, so if you've written code for the bulk loader, it will need to be updated.
An early release of the bulk downloader is also now available in bulkloader.py. Learn more about these changes at: http://code.google.com/appengine/docs/python/tools/uploadingdata.html

For a full list see the SdkReleaseNotes wiki page.

Downloads for Windows, Mac, and Linux are available on the Downloads page. This SDK update was for the Python runtime, so please post your feedback in the Python runtime discussion group.

Posted by Jeff Scudder, App Engine Team

Early Look at Java language support expanded to 25,000 developers

2009-04-15T14:04:00.000-07:00

Have you signed up to try Google App Engine for Java yet? We were so overwhelmed with the response to our early look at Java support on App Engine that we decided to let more developers in now, without waiting. Thus, we're expanding the early look signups to 25,000 developers, in order to give more of you a chance to try it out. So please sign up, give it a try, and let us know what you think. We've already seen a number of interesting apps built and deployed. Stay tuned for us to point some of the cooler ones out in the coming weeks.

On another note, if you haven't delved down into the SDK and taken a look around, you may have missed the demo directory. Inside this directory you'll find a number of projects which demonstrate various features of Google App Engine for Java. These demos are a great place to start if you want to learn more about App Engine by example, or if you just want to get hacking on something quickly. Last but not least, we hope these demos are of practical value too, as we included example Ant build scripts and organized the code with standard directory layouts and naming conventions.

Looking to create an end-to-end Java AJAX application? Try out Sticky, the sticky note app written using Google Web Toolkit and App Engine. You'll find useful patterns you can use in your own code, the sample source itself is Apache licensed so feel free to use it directly.

Are your eyes set on a mobile app? Check out the source for Task Engine . Also built with GWT, this simple task app is written with the iPhone and Android in mind.

If scaling your infrastructure is more your cup of tea, you can check out the shardedcounter example. It demonstrates how to implement distributed incrementing counters, which are useful for understanding how to scale larger applications on App Engine.

Note: If you installed the App Engine SDK with the Google Plugin for Eclipse, the SDK and accompanying demos can be found in your Eclipse installation directory, under plugins/com.google.appengine.eclipse.sdkbundle_1.2.0.vXXX.

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Many languages, and in the runtime bind them

2009-04-13T17:00:00.000-07:00

While working on App Engine's new Java runtime, we realized that we were missing something important. Something the Python runtime already had. Something sleek, yet powerful ... and approximately 80 characters wide and 25 characters tall. Yes, we're talking about the Python runtime's nifty shell demo app.

Naturally, we wanted to take the existing shell demo and extend its functionality for our new Java environment. Our idea was that we'd create one REPL to rule them all: a shell demo that supports several different JVM-based languages simultaneously. It was an easy decision to use GWT to build the user-interface, making the terminal interface a cinch. Once we had the basic shell framework, we then added some languages: Beanshell, Clojure, Groovy, JavaScript, Python, Ruby, Scala, and Scheme.

This helps us underscore two important compatibility features of Java App Engine:

Google App Engine for Java accepts Java bytecode, not Java source code. It even supports loading bytecode generated at run time. That means that our new Java runtime can support any language with a compiler that targets the JVM. In fact, many advanced scenarios work as well - such as libraries that rely on runtime bytecode generation (such as dependency injection frameworks, AOP libraries, and expression-language runtimes).
The security sandbox in Google App Engine for Java is flexible. Other techniques for sandboxing Java can be restrictive about the permissions they grant to untrusted code, limiting the types of constructs you can use. For example, many secure Java environments don't allow you to create custom ClassLoaders or use tricks for reflection like setAccessible. Java App Engine attempts to provide this type of functionality in a secure fashion, making it possible to run more types of code (and more languages).

Now, it must be said: please note that this is a demonstration app we developed to show off the power and flexibility of our new Java runtime; it's not a production service, and you shouldn't rely on it for anything important. Further, since this is a single shared instance of this application, running server code code from many users at once, you should exercise caution in entering any sensitive data like passwords into this demo. Finally, though we've worked to ensure rich functionality for as many languages as possible, this is still a proof of concept, and there is surely some language functionality missing from our environment.

But, without further ado, may we present to you, the Lord of the REPLS.

-- Google Software Engineers, Toby Reyelts and James Robinson.

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Seriously this time, the new language on App Engine: Java™

2009-04-07T17:28:00.000-07:00

Today, we're very excited to announce the availability of a new programming language for Google App Engine. Please welcome the Java runtime!

When the two of us first heard the promise of Google App Engine, we realized that the chance to bring this kind of simplicity to Java developers was too good of an opportunity to pass up. When App Engine launched publicly, we were excited to see that Java language support was both the first and the most popular request filed in the Issue Tracker. We were also thrilled to see that this enthusiasm extended beyond the Java language to all of the various programming languages that have been implemented on top of the Java virtual machine -- not to mention all of the popular web frameworks and libraries.

But we also knew that Java developers are choosy:

They live by their powerful tools (Eclipse, Intellij, NetBeans, Ant, etc.).
They try to avoid lock-in and strive for re-use. Standards-based development (defacto or otherwise) is key.
They harness sophisticated libraries to perform language feats which are nearly magical (GWT, Guice, CGLIB, AspectJ, etc...).
They even use alternate languages on the JVM, like Groovy, Scala, and JRuby.

We wanted to give developers something that they could be ecstatic about, but we knew we would have to marry the simplicity of Google App Engine with the power and flexibility of the Java platform. We also wanted to leverage the App Engine infrastructure -- and by extension Google's infrastructure -- as much as possible, without giving up compatibility with existing Java standards and tools.

And so that's what we did. App Engine now supports the standards that make Java tooling great. (We're working on the tooling too, with Google Plugin for Eclipse). It provides the current App Engine API's and wraps them with standards where relevant, like the Java Servlet API, JDO and JPA, javax.cache, and javax.mail. It also provides a secure sandbox that's powerful enough to run your code safely on Google's servers, while being flexible enough for you to break abstractions at will.

There is a vast amount of Java code out there, much of it written without consideration of sandboxing, and we can't test it all. We know that there will be some rough edges when it comes to compatibility, but we're looking forward to working with you to smooth those out. To that end, we're giving the first 10,000 interested developers an early look at Java language support, so please sign up, give it a whirl, and give us lots of feedback.

The team has also been working on many other improvements to App Engine, which we're really excited to launch to you as well:

Access to firewalled data: grant policy-controlled access to your data behind the firewall.
Cron support: schedule tasks like report generation or DB clean-up at an interval of your choosing.
Database import: move GBs of data easily into your App Engine app. Matching export capabilities are coming soon, hopefully within a month.

Last but not least, the App Engine team will be at the upcoming Google I/O developer event on May 27-28 in San Francisco, so please come meet us in person.

We look forward to seeing your applications. Get coding!

Posted by Don Schwarz and Toby Reyelts, Software Engineers, Google App Engine Team

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

A Brand New Language on Google App Engine!

2009-04-01T14:22:00.000-07:00

Posted by Alon Levi, App Engine Team

It's been almost a year since we've launched App Engine with support for Python, and what a year it's been! We've gotten fantastic feedback from developers, and we've released loads of new features! When we launched, we promised support for another runtime language, and indeed this has been among the most requested features from our developers since day one.

Well, we fed Google's new CADIE Strategic Decision Maker the App Engine issue tracker, our groups, and various blog posts around the internet to help select a new runtime language for App Engine. Today we're excited to officially announce support for FORTRAN 77!

If you're an enterprise customer and want to take advantage of Google App Engine, but have a large and cumbersome legacy system, we want to make it easy for you to port to the cloud. By providing a Fortran 77 runtime, along with a familiar, easy-to-use deployment mechanism, we hope to make this process efficient and straightforward.

Want to give it a try? Download our SDK and deploy your application by mailing punch cards to:

Google App Engine, C/O APPCFG
1600 Amphitheatre Pkwy
Mountain View, CA 94043

and we'll take care of the rest! We welcome your feedback on our newest addition to the App Engine family--you can discuss it on our Google Group!

Google App Engine on the Road

2009-03-18T20:44:00.000-07:00

More members of the App Engine team are hitting the road in the next two weeks. If you'll be at any of the following events we'd love to meet and talk shop! Here's where you can find us:

Wednesday March 18th
5:00 pm - 6:00 pm
New York City

Pete Koomen will be at CommunityOne East participating in the Closing Panel Discussion – "Enterprises and the Cloud". The panel will be moderated by David Berlind, Editor-At-Large of InformationWeek, and the panelists include Adam Gross, Vice President of Developer Marketing, Salesforce and Lew Tucker, Vice President and CTO, Cloud Computing, Sun Microsystems, Inc.

Wednesday March 25th
9:00 am - 12:20 pm
Chicago

Joe Gregorio will be teaching a half-day tutorial at PyCon 2009 entitled An Introduction to Google App Engine. From the session description: "Google App Engine allows you to build scalable web applications and host them on Google hardware. This hands on tutorial will walk you through the steps of building a basic web application, from setting up the SDK through to using the major APIs that App Engine provides."

Thursday March 26th
1:30 pm - 2:30 pm
Philadelphia

Joe Gregorio will be giving the presentation "Under the Covers of the App Engine Datastore" on March 26th at the Emerging Technologies for the Enterprise conference in Philadephia. From the session description: "This talk walks through the internals of the App Engine Datastore, how data is stored, how indexes are built, and how queries are executed. We then discuss how those architectural decisions make the App Engine datastore scalable and finally go into how those constraints affect how you design your applications schema so your application can take full advantage of that scalability."

Wednesday April 1st
Time TBD
New York City

For the second time in two weeks we return to New York City where Joe Gregorio will be giving the same "Under the Covers of the App Engine Datastore" talk at the Cloud Computing Expo in New York City.

Posted by Joe Gregorio, App Engine Team

Jaiku moves to App Engine, new article on remote_api, and more!

2009-03-16T11:26:00.000-07:00

A number of exciting App Engine-related snippets have made the rounds over the past couple of weeks. Here's a quick summary:

Jaiku moves to App Engine

Last April, Jaiku's official blog confirmed that Jaiku was being ported to App Engine. We are pleased to announce that not only is Jaiku now hosted on App Engine, but substantial progress is being made on the related open source effort to build out the Jaiku-based microblogging platform, Jaiku Engine, which allows organizations, groups, and invidivuals to deploy their own microblogging service to App Engine. For more details, check out the official Jaiku blog posting.

Trender apps launched on App Engine

Jaiku isn't the only application to launch on App Engine in recent weeks. Enter NYT Trender and Flickr Trends which allow users to view the popularity of search terms over time based on occurrence in the New York Times and Flickr respectively. Both mashups display relevant photos for the terms entered, adding to the richness of the experience.

New remote_api article available

Googler Nick Johnson recently published a handy guide introducing the remote_api module which debuted with release 1.1.9 of the App Engine SDK. The module allows remote access to an application's datastore and Nick's guide describes how to set up and use it with plenty of sample code to help you build your own interactive console and map framework. Also, in case you missed them when they were first posted, the articles on paging and avoiding datastore contention using sharded counters are great reads as well, particularly if you're interested in building a highly scalable and efficient apps.

App Engine at WeekendApps

Over the weekend of Feb. 20th, 130 developers descended on the Googleplex in order to design, build, and launch an OpenSocial-based application as part of the WeekendApps - OpenSocial event. While OpenSocial was the star of the show, App Engine garnered a lot of interest itself with BuddyPoke! creator Dave Westwood and App Engine advocate Fred Sauer in attendance. In the end, over 14 new applications were presented, at least 3 of which were built on App Engine. Of these three, the team behind Buddy Quiz! won a Google I/O pass for "best product vision" and another, Yumie Date, recently received an Editor's Pick promotion from MySpace. Neither team had ever used App Engine prior to the event and one had never even worked with Python.

Posted by Jason Cooper, Google App Engine Team

Join App Engine at SXSW

2009-03-11T00:23:00.000-07:00

Several of the App Engine team members are excited to be heading to Austin, Texas for SXSW Interactive 2009. SXSW always brings together an talented variety of web developers, entrepreneurs, and visionaries. If you'll be there too, we'd love to meet and talk shop! If you are interested in participating in an App Engine User Study while at SXSW, please fill out this form.

Sunday, March 14th
10:00 pm - 1:00 am
The App Engine team is joining forces with the Blogger and Reader teams for a bash "Read. Write. Drink." at Six Lounge (117 W 4th St @ Colorado). Come share a beer and tales of website building and hosting with us! Bring your SXSW Interactive Badge, or find an App Engine team member at the conference and ask for an invitation. Before the party, Engineers from the App Engine team, including Tech Lead, Kevin Gibbs, are having "office hours" from 8-10pm at Halcyon Coffeeshop (218 W 4th St) where we'll hang out, sip coffee and talk shop - we want to answer your tough technical questions and to help you learn if App Engine is a good fit for your application or business. We'll have some T-shirts, and copies of our new cheat sheet!

Tuesday March 17th
10:00 - 11:00 am, Room 8
Kevin Gibbs, App Engine's Tech Lead, will be participating in the panel "Cloud Computing: Defending the Undefinable" along with Yousef Khalidi, Distinguished Engineer at Microsoft and Werner Vogels, the CTO at Amazon.com. Here's the panel description: "The brave new world of cloud computing is radically changing how we build web applications. What is a platform, what is a service, and how will the future of web applications be built? More importantly, how do these various clouds compare, and what do the differences mean? Are they ready for your world-rockin' startup? In this panel, we'll get nerdy with technical details, you'll yell at us, and we'll argue why your app should already be in the cloud."

Monday March 16th
9th Annual 20x2 Event
7:00 - 9:30 PM, The Parish on 6th St in Austin
20x2 brings together 20 different participants from all walks of creative life and each person has two minutes to answer/interpret the same question before a live audience. The question for this show is "What's It Gonna Take?" App Engine team member Lindsey Simon will be speaking.

If you're in the area, stop by! If not, you can always leave feedback in our Google Group!

Posted by Lindsey Simon, Google App Engine Team

New! Grow your app beyond the free quotas!

2009-02-24T12:40:00.000-08:00

Posted by Brett Slatkin, App Engine Team

We're psyched to announce that developers can now purchase additional computing resources on App Engine, enabling apps to scale beyond our free quotas. This has been our most requested improvement to App Engine and we're thrilled to deliver it, as promised.

When we released App Engine last April, it was completely free to use, but each application was limited to a set of fixed resource usage quotas. The free quotas are still there, but now you can grow beyond them.

You can now set a daily budget for your app that represents the maximum amount you're willing to pay for computing resources each day. You allocate this budget across CPU, bandwidth, storage, and email, and you pay for only what your app consumes beyond the free thresholds -- prorated up to the nearest penny. We've put together a handy screencast to help explain the process:

App Engine remains free to get started. However, along with many performance improvements over the past ten months, we've learned that we overestimated our initial free quota values. Therefore, in 90 days we will be reducing the free quota resources. We believe these new levels will continue to support a reasonably efficient application serving around 5 million page views per month, completely free.

The pricing for resources beyond those free quotas is:

$0.10 per CPU core hour. This covers the actual CPU time an application uses to process a given request, as well as the CPU used for any Datastore usage.
$0.10 per GB bandwidth incoming, $0.12 per GB bandwidth outgoing. This covers traffic directly to/from users, traffic between the app and any external servers accessed using the URLFetch API, and data sent via the Email API.
$0.15 per GB of data stored by the application per month.
$0.0001 per email recipient for emails sent by the application

Data stored in the datastore incurs additional overhead, depending on the number of indexes, as well as the number (and size) of associated properties. This overhead can be significant in some cases, and it's something that we have been underreporting up until now. Thus, you may notice an increase in the amount of data stored by your application that is listed in the Admin Console.

To decrease the impact of this change in the way we calculate storage usage, we've doubled the free storage quota to 1GB. To learn more about how our quotas work, upcoming changes to our quota levels, how they relate to billing, and overall resource consumption limitations, see the Quotas section of the docs. The Purchasing Additional Quota section has more detail about how to purchase additional computational resources for your app, and there's also a Billing FAQ. We've also made some changes to our terms of service to include language around payments, fees, and disallowing the use of multiple applications to avoid incurring fees.

As always, we welcome feedback in our discussion group, and don't forget to check out the App Engine sessions at our upcoming developer conference, Google I/O!

Back to the Future for Data Storage

2009-02-19T11:07:00.000-08:00

Building a massive, distributed datastore which can service requests at an extremely high throughput is something that we've focused on at Google. We created something called Bigtable that underlies the datastore in App Engine. The design for Bigtable focused on scalability across a distributed system so it may operate a bit differently than databases you've worked with before, such as not supporting joins. This isn't an accident -- when you build a system that can scale to the size that Bigtable can there's no way to do a general purpose join on data sets that size and still have them be performant.

Google isn't alone in offering an non-Relational datastore to enable scaling. For example, Amazon has SimpleDB:

A traditional, clustered relational database requires a sizable upfront capital outlay, is complex to design, and often requires a DBA to maintain and administer. Amazon SimpleDB is dramatically simpler, requiring no schema, automatically indexing your data and providing a simple API for storage and access.

There are also a range of non-relational open source datastores now available such as CouchDB and Hypertable. Those are just two examples, there are many more.

While you might think this is all new, it's actually a bit of a return to the past. You see, there was a time when "RDBMS" wasn't always the answer regardless of what the question was. At the time Codd published his paper, "A Relational Model of Data for Large Shared Data Banks," there were many different approaches to datastores. It was only in the '80s that relational databases won the majority of the mindshare. Having settled on a single metaphor the industry has developed many tools and techniques to make developing on a relational database easier.

Unfortunately that majority mindshare is also a problem because while RDBMS' are useful in many situations, they are not useful in all situations. Their dominance in the mindshare means that useful alternatives aren't used, and huge amounts of time and money can be wasted trying to force non-relational problems into a relational model.

We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Michael Stonebraker pointed out in his paper, "One Size Fits All": An Idea Whose Time Has Come and Gone, that there are common datastore use cases, such as Data Warehousing and Stream Processing that are not well served by a general purpose RDBMS and that abandoning the general purpose RDBMS can give you a performance increase of one or two orders of magnitude.

It's an exciting time, and the takeaway here isn't to abandon the relational database, which is a very mature technology that works great in its domain, but instead to be willing to look outside the RDBMS box when looking for storage solutions.

Posted by Joe Gregorio, Google App Engine Team

Web App Wednesday, Mashup, Backup, and Decay

2009-02-13T16:08:00.000-08:00

Posted by Jeff Scudder, App Engine Team

Michael Bernstein is doing something I find very interesting. Not only is he creating applications on App Engine, he creates a new one every Wednesday. His site, appropriately named Web App Wednesday, is where he is creating a new web application every Wednesday, starting first with the site itself, followed by F/OSS License Info, a single-page site that displays information about the most common Open Source software licenses. Another recent app, LinQR , is a URL shortening service and QR code generator. Good luck to Michael and we look forward to seeing what he comes up with next Wednesday.

If you're doing something cool with App Engine please let us know! We'd love to highlight it here on the blog or even have you write up an article explaining what you've done, like this new article from Omar Abdelwahed on Creating a Facebook App with Google App Engine and Best Buy Remix.

Speaking of articles, Aral Balkan has written an article on his backup and restore utility and our own Joe Gregorio published an article on a design for popularity ranking using voting and time-based decay.

The sky's (almost) the limit! "High CPU" is no more.

2009-02-12T16:05:00.000-08:00

Posted by Pete Koomen, App Engine Team

We're very excited today to announce that we've raised limits on several App Engine operations:

No more "High CPU Requests"! App Engine Apps were once allowed no more than 2 CPU-intensive requests per minute. We've made some adjustments to the way we handle requests, and have eliminated this limitation altogether. To learn more about how this works and the implications for your app, see our documentation.
Response deadline raised to 30 seconds. The amount of time an App Engine app can take to respond to an incoming request has been raised from 10 to 30 seconds! There are limits on the number of simultaneous active requests an application can process at any given moment--see our docs to learn more.
Size limits on code files, static files, and requests/responses raised to 10MB! App Engine apps can now receive requests and send responses of up to 10MB in size, and users can upload 10MB code and static files as well. Note that API requests (e.g. memcache.set(), db.put()) are still limited to 1MB in size.

These changes were inspired and driven by a great deal of developer feedback, and we're not done! Please let us know what you'd like to see next on App Engine in our Google Group!

SDK version 1.1.9 Released

2009-02-09T17:24:00.000-08:00

Today we released version 1.1.9 of our SDK. Here's what's new in this release:

You can now use the Python standard libraries urllib, urllib2 or httplib to make HTTP requests. This has been a frequent request on our issue tracker.
We've been working on a set of tools that will make the process of uploading and downloading data from App Engine applications easier. Today we're excited to announce an early release of our new bulk uploading client. You can try it out here. Let us know what you think in our Google Group!
Several updates to our datastore, including the automatic generation of single property indexes and the addition of IN and != operators to db.Query. See the Datastore API docs for more details.
A bunch of additional bugfixes and enhancements, listed in our Release Notes.

You can download version 1.1.9 of the SDK for Windows, Mac, and Linux now on our Downloads page. As always, we spend a lot of time reading (and posting!) on our Google Group, and we welcome your feedback!

Posted by Pete Koomen, Google App Engine Team

A roadmap update!

2009-02-06T17:59:00.000-08:00

Posted by Joe Gregorio, App Engine Team

The App Engine team has been plugging away and we're excited about some pretty big announcements in the near future. In the meantime, we decided to refresh our App Engine roadmap for the next six months with some of the great new APIs in our pipeline:

Support for running scheduled tasks
Task queues for performing background processing
Ability to receive and process incoming email
Support for sending and receiving XMPP (Jabber) messages

As always, keep in mind that development schedules are notoriously difficult to predict, and release dates may change as work progresses. We'll do our best to update this roadmap as our engineers continue development and keep you abreast of any changes!

You'll have the opportunity to discuss this roadmap (and all things App Engine) with us and your fellow developers during Google I/O, coming up in May. Check out the App Engine sessions we've already announced and don't forget to discuss in our Google Group!

Best Buy's Giftag on App Engine

2009-02-02T12:13:00.000-08:00

Posted by Amanda Surya, Google App Engine Team

Built by Best Buy, Giftag is a gift registry add-on for FireFox and Internet Explorer that enables you to create a wish list and share it with others. The Giftag team first built a prototype of the application on another platform and went live on September 2008. About 10 weeks later, the team relaunched Giftag on App Engine just in time for Black Friday. Since then, they've been getting positive reviews from users about the product.

We're happy to share this video created by Jerry St. Sauver, Curtis Thompson, and Thomas Bombach, Jr., the development team behind Giftag. In this video, they share their thoughts on App Engine, their development experience, tips on getting started with App Engine, and finally a demo of Giftag in action.

To learn more about what the Giftag team is up to, check out the official Giftag blog.

To share your own experience using Google App Engine, tell us your story (or even your own video) in our developer forum.

Google Code Jam's Ranking Library Released

2009-01-30T10:32:00.000-08:00

Posted by Bartholomew Furrow and Sebastian Kanthak, Google Code Jam Team

At Google, we like to use our own projects for internal development. Following that philosophy, one of the first apps ever made for Google App Engine was the contest platform for Google Code Jam. The application was a good fit: a rich web interface, real-time user interaction, and a heavily parallel design. But we faced one major challenge, which was how to handle the huge scoreboard -- in the first round, we had over 11,000 contestants.

While App Engine's datastore allows you to sort entities by score, it doesn't have a built-in mechanism to answer the following two requests:

Given a user, what is his or her rank on the scoreboard? In other words, for some arbitrary row in that scoreboard, what is its position amongst the 11,000 spots? This is useful for showing your own rank, as well as the ranks of your friends.
Give me all users on the n-th page of the scoreboard. While it is possible to get all users in sorted order, datastore queries don't allow you to start at page n, and iterating over all of the users on the first n pages takes too much time.

To solve these issues, we came up with a library that maintains a data structure to efficiently support these use cases. We imagine that other applications will have similar problems (e.g., a high-score list in a game), and so we're happy to release our work as the "google-appengine-ranklist" library.

The library supports three different operations:

Setting the score of a given user.
Given a score, what's the rank of a user with this score? This is used to answer the "What's the rank of person U?" use case.
Given a rank, what's the score S for this rank? This can be used to solve the paging problem, by constructing a query that returns the first few users with a score less than or equal to S, in sorted order

If you think this library could be useful for you, take a look at the documentation and at the example application. We'd love to hear from you in our Google Group.

Getting Creative with App Engine

2008-12-22T14:13:00.000-08:00

Posted by Marzia Niccolai, App Engine Team
Gregory Pennington and James Bogosian, Advertising Platform Solutions Team

When inspiration strikes, it's nice to go from concept to reality in the shortest possible amount of time. We're happy to let you know about two new tools that will help you shorten the path on App Engine from conception to clicks.

Zoho Creator

For those used to rapidly developing apps with Zoho Creator, we are happy to announce that Zoho have made it possible to deploy these apps using Google App Engine. Zoho Creator enables you to develop data backed business applications, and once you have developed your app in Creator, simply download the Python code and upload it to Google App Engine.

App Engine Site Creator

App Engine Site Creator is a lightweight content management system with a full GUI for content creation and administration. It provides a hierarchical site structure, an authentication system, user and group level access controls, file attachments and a WYSIWYG interface for editing page HTML.

The resulting managed site is designed to be themed and branded, and the back end Site Creator code was written with readability and extensibility in mind. Check out the code to learn more or visit the project page to learn how to download and deploy your own instance. As always, we welcome your feedback in our Google Group.

System Status Dashboard, Quota Details Page, and a Billing Sneak Preview

2008-12-16T10:11:00.000-08:00

We're excited to announce a couple new features and a preview for you today:

an App Engine System Status Site that monitors the latency and uptime of various components and provides real-time visibility into their performance
a new Quota Details Dashboard, detailing all of the resource quotas that affect your application
a sneak peak at our upcoming billing feature, which will enable your app to grow beyond our free quotas!

System Status Site

The new System Status Site provides a detailed view into the performance of various App Engine components using some of the same raw monitoring data that our engineering team uses internally. This includes:

up-to-the-minute overview of our system status with real-time, unedited data
daily overall serving status for each of our APIs, including any outages or downtime
detailed historical latency and error-rate graphs for the App Engine Datastore, Images, Mail, Memcache, Serving, URL Fetch, and Users components

In addition to the Downtime Notify Google Group, we'll use this dashboard to announce scheduled downtime and explain any issues that affect App Engine applications. You'll be able to see real data behind any issues that we experience along with explanations from our team.

We'll continue to tune this dashboard to make sure we're providing useful and accurate information about App Engine's uptime. We expect this tool will complement others offered by other companies, such as Hyperic's CloudStatus.

Quota Details Dashboard

For each App Engine application, we now provide a Quota Details Dashboard. This makes it easier to track how much of the free quota your app is using up across bandwidth, CPU, etc. Use it to get detailed information about all of the resource quotas that affect your application.

To use the Quota Details Dashboard, click the "Quota Details" link on the dashboard for any app.

Sneak Preview: Purchasing Additional Resources

As mentioned in the product roadmap, soon you'll be able to buy additional capacity beyond the free quotas. (For reference, we described our expected pricing earlier.) In the mean time, we wanted to give you a brief update on this feature.

You'll be able to buy capacity based on a daily budget for your app, similar to the way AdWords spending works. You'll have fine-grained control over this daily budget so you can apply it across CPU, network bandwidth, disk storage, and email as you see fit. You'll only pay for the resources your app actually uses, not to exceed the budget you set. Here are a couple screenshots from the beta interface:

Of course, we're still working on this feature, so we'll likely make additional changes before releasing it.

We hope you like these new features--please let us know what you think in the discussion group!

Posted by Brett Slatkin, App Engine Team