Equally True

Free PaaS options for deving, experimenting and more

2012-05-12T02:49:00.000-07:00

It seems that almost all the major Platform as a Service providers offer some kind of free plan. I thought to share a few of them for others to see and benefit. They are especially useful if you are trying to learn a new technology and try out different service providers.

Careful, don't get locked in

The providers have realized that most customers don't want to change providers once they have passed the learning curve on one provider. As a customer you should however be ready to move your deployment as quickly as possible. You can do this by avoiding the proprietary technologies provided by the service providers or at least abstract their usage in a way that makes it easy to rewrite those parts if you change providers. AppEngine is good example of a provider that offers a wide range of proprietary technologies that will make moving your deployment away from there difficult if you haven't thought about it when developing it.

If you have time to learn a bit go ahead and try do deploy your app to a few of the platforms listed below, if it goes well you might get the confidence to change to a better suited provider if the time comes in the future. Some of the providers require more effort, but if you want to try an easy one take a look at Heroku, it supports easy deployment through Git.

Google App Engine

Offers: Python, Go and Java (and any other JVM language) platform, and many useful services.

Note: Appengine offers you to many Google specific technologies like Datastore, Channel API, Tasks and Cron. This makes is more difficult to migrate out of AppEngine. Also if you use 3rd party libraries or frameworks, they may not work because of the proprietary datastore.

Free plan: 1GB database, 1GB outgoing bandwidth/day and 50k DB ops/day. Check more detailed breakdown on their pricing page. These limits are enough for most private websites or blogs, Based on peoples experiences you can get 300k+ page views per month within the limits for average app (discussion here).

Heroku

Offers: Ruby, Node.js, Clojure, Java, Python, and Scala platform. In addition they offer support for many other languages through buildpacks.

Note: Heroku is a quality provider from what I have heard. They support easy deployment through git.

Free plan: 750 processing hours (enough for 100% uptime), 5MB database. So usually the database size is the limiting factor.

Engine Yard Cloud

Offers: Ruby on rails stack.

Free tryout: 500 instance hours free (must be used within 6 months), more here.

Windows Azure

Offers: .net, node.js, Java, PHP or other (language of your choice).

Note: You'll be on Microsoft stack, so don't dream of MySQL here.

Free trial: 3 month trial includes 750 hours on small instance (enough for 100% uptime), 1GB database, 20GB storage, 20GB outbound bandwidth

Cloud Foundry

Cloud foundry is an interesting one. They use their Open Source PaaS platform. You can download and install this image to your own machine if you want. This also means you can privately run their PaaS platform on any infrastructure you choose, EC2 for example.

Offers: Ruby, Node.js, Java or any other technology. MySQL, Postgres, Mongo or Redis database. Read more about their infrastructure.

Free during beta. Beta will end during 2012, but pricing after that is not announced.

Redhat Openshift

Offers: Java, Ruby, Node.js, Python, PHP and Perl. MySQL, PostgreSQL and MongoDB databases. More about their features here.

Note: The service looks very developer friendly, and already on the landing page you can see them advertising "No Lock in". They also have community section in their site following the open source traditions.

Free plan: 3x 500MB ram and 1GB HD. Their whole operations seems to be around free plan.

dotCloud

Offers: Any technology (well, maybe not every, but their supported technology list is impressively long).

Note: They show their roadmap, which is nice touch of openness. Many technologies seem to be in "upcoming" or "alpha" stage.

Free plan: Two services of your choice, for example Python and MongoDB. Databases have 10MB hd and 10MB ram in free plan.

AppHarbor

Offers: .NET platfor as a service. Many additional services available as Add-Ons.

Free plan: Free plan includes 1 web worker unit. In addition they have wide variety of Add-Ons that have free option including: Mysql, CouchDb, MongoDb and RabbitMQ (task queue).

Amazon EC2

EC2 is actually an IaaS (Infrastructure as a Service), but I included it here, because if you have some administrative skills you can quite quickly set up any platform you like.

Offers: Virtual machine (Linux or Windows) and related services.

Note: You'll have to manage the virtual machine yourself, so you'll most probably need to get familiar with linux commandline. Managing any security or other problems on the OS level yourself isn't for everybody and requires some skills and effort.

Free plan: 1 year only. 750 hours per month Micro Instance (enough for 100% uptime), 100MB storage, 30GB Elastic Block Storage and 5GB S3 storage. Detailed limitations here. Biggest limitation for many is low CPU performance of the Micro instance. Here are some benchmark results that show it's CPU is comparable to the Nokia N900 smartphone.

Cloud Database providers

You can also get Database as a service. Using database through public internet does bring some performance and security issues, but they are easy for testing out new databases.

MongoDB: MongoLab offers 240MB free database.
MySQL: Xeround offers 10MB database for free.
Apache CouchDB: Cloudant offers 250MB database for free.

Reddit discussion about this post: Link

Want to learn more about PaaS? O'Reilly has a new book called Understanding PaaS (released Jan 2012).

Follow me in Twitter: @JuhaFinn

170+ Django conference videos

2012-05-04T10:32:00.000-07:00

It's great that many software conferences nowadays put their videos online, DjangoCon has been especially good on this. Those who can't attend the real conference for budget or geographical reasons get a chance to experience some great presentations. It's great for pushing the overall web development level upward.

I collected here mainly DjangoCon presentations, but there are a few other quality talks in the Other Videos section.

In case someone is interested of scraping data from pages with JQuery and Chrome console here is script I wrote. It gets title, urls and date from blip.tv, reverses the order and prints it as a list html element. I load JQuery to the page with a JQuerify bookmarklet. I find JQuery quite handy for small tasks like this. Here is the script:

$.each( $(".EpisodeList .clearfix").toArray().reverse(), function(i,data){ 
    console.log('<li><a href="' + $(data).find("h3 a")[0] + '">' +$(data).find("h3").first().text().trim() +'</a> <span style="color:gray;font-size:0.9em;">' + $(data).find(".MetaDataPairs .ReleaseDate h6").text() + '</span></li>');
});

5 selected videos

Here are five videos to get you started dealing with some common topics: caching, forms, deployment and security.

DjangoCon 2011 Keynote - David Eaves
Cache rules everything around me
Safely deploying on the cutting edge
Advanced security topics
Advanced Django Form Usage

3 useful features from Opera Dev tools that Chrome is missing

2012-04-29T13:06:00.000-07:00

Let me just start by mentioning that I am not biased towards Opera. I still use Chrome as my main browser and I think it provides the best overall user and developer experience from the available browsers. I used Opera as my main browser a while and still think it offers good innovative things others are missing, it's just refreshingly different from the FF, IE, Chrome lineup. One of the features I still miss on Chrome is the smooth mouse gestures integration. Anyway -- I think Opera has a few useful features in their development tools that are worth knowing.

1. Easy remote debugging

Opera Developer Tools (they call it Dragonfly) offers a nice feature easily do remote debugging. All Opera browsers (Desktop, Mobile, Mini, TV) have ability to act as remote debugging client. On the client you have to go to opera:debug page and input the ip of your desktop computer. Then on the desktop you click start listening for remote debugging connections. When you click connect on the client you just open any web page and the development tool on desktop is integrated to the mobile, you can for example: browse the DOM, inspect scripts or use the javascript console.

Edit: Chrome offers a remote debugging functionality too, but using opera you don't have in install any additional applications/packages for client or host. Crome has a page showing how to get it working with Android browser. It doesn't seem to be as easy as Operas way though.

Adobe Shadow: Operas remote debugging solves very similar problem to Adobe Shadow. However, for developers like me who use Linux it's not an option so this is a very good alternative.

2. Opera Mobile & Mini simulator

You'll have to install Opera Mobile or Mini simulators separately, but they work really well. You can choose a device from long list of devices and the simulator is customized accordingly depending on which device you choose. This is a very easy way to test your website with touch interface (mouse acts as a finger instead of a pointer in a simulator).

Google alternatives: You can use Android emulator to achieve similar simulation if you have Android SDK and eclipse set up, it is just quite slow to start up and use. You can also get Ripple as a Chrome extension does the emulation in the browser.

3. Network Request crafter & header modification

These features are in the Network section of the Developer Tools. The Make Request is a tool that allows you to write and send your own HTTP requests. Every now and then you need a tool like this. This can be used when hackproofing your site. It is still very stripped down on features, but you can easily copy-paste any request from the previous network log page. It's easy to for example test if your CSRF protection works by re-sending a "submit post" request and check if the second one is registered.

Another useful feature in the Network section is to make the browser override the headers to simulate being any other browser (not just User-Agent, but all the headers). You can also define your own headers to be used.

Edit: A guy in the Hackernews comment mentioned that there is a Chrome extension called REST Console, it apparently has the ability to send crafted HTTP requests.

Conclusions

The point of this post wasn't to tell how good Opera is. I was just mentioning that Opera is still serious contender in the browser competition and it has features that developers should know about. I'll still keep using Chrome as my main browser, but I'll have Opera just one click away if I want to do something Chrome can't do.

Edit: Ironically my blogger template didn't render properly on Opera Mobile when I released this. I changed a better one that should make it readable on Opera Mobile too.

More ways to get involved in discussion about this post:

Follow me in Twitter: @JuhaFinn

Takeaways from Google Developer Day Berlin

2012-04-13T14:15:00.000-07:00

Here key points I had in my notes from last Novembers Google Developer Day 2011 in Berlin

Keynote

Keynote was the start for the day. There were over 2000 developers on the huge hall. There was impressive display with google-style animation and upbeat background music while we were waiting for the keynote to start. And lets not forget that they had two beatboxers on the stage doing the transition sound effects.

Takeaways:

Cloud, Cloud, Cloud. They were emphasizing how concept of cloud computing is still growing in importance.
Google+ Hangouts is coming and developers are encouraged to try it out. They had some demos of hangout apps.
Android has huge userbase and it is growing (8 billion app installs, 300k apps on store).
They were promoting AppEngine as backend for mobile apps, which in my opinion is a good and easy solution.
Chrome has reached stage where it can run advanced web app that really behave like desktop apps, like 3d modeling. Improvements include JS speed optimization, WebGL improvements, SVG support, new HTML5 and CSS3 features (like 3d transforms and caching).
Demos and examples of advanced web-apps:

Pixlr: In browser photo editing: http://pixlr.com/editor/ .
3DTin: In browser 3D modeling: http://www.3dtin.com/ Try it out!
D3 SVG library (check examples: http://mbostock.github.com/d3/ex/),
Using Kinect as input to draw real-time 3D model of you (http://www.chromeexperiments.com/detail/kinectwebgl/) Really cool and innovative, developer is a real hacker :)
Three.js 3D library. (source and screenshots https://github.com/mrdoob/three.js/)
HTML5 Audio demo: http://gddhtml5comp.appspot.com/
Mask Images with CSS3: http://girliemac.com/blog/2010/09/20/201/

Bring Intents to web with webintents. Some examples: After uploading image, give intent to photo editor, photo editing web app can then let user process it and save it back to your app. Seems to be just future vision at this point.
Shadow dom, isolate and encapsulate parts of dom (http://glazkov.com/2011/01/14/what-the-heck-is-shadow-dom/)
In App payments were released for Chrome webstore. 5% to Google, 95% to Developer, sounds good to me!

3DTin running in Chrome:

People on stage:

James Whittaker - I had to double check, since his website says he works for Twitter, but he was there. Resigned Google on February.
Jonathan Beri - Talked about hangouts
+ More whose names I didn't catch

Kinect+WebGL

Follow me in Twitter: @JuhaFinn

Extracting uploaded zip file with Django in memory

2012-04-12T09:09:00.001-07:00

There are many situations where some time ago I woudn't hesitated to just use filesystem. But now I have started to realize the value of minimizing the usage of filesystem. Databases are designed to keep data and offer easy solutions for backup and security for example. Keeping more and more data in databases can be a good idea in some cases. I'm not saying you should always just push your data to the database, but it should be a consideration. Also using AppEngine lately has affected my thinking since it doesn't even offer access to filesystem.

This tutorial explains how to allow users to upload zip files, open them and insert the contents into your database, all without accessing the filesystem. It is only applicable to handling text-files, but modifying it to allow binary files should not be difficult (encode the files with Base64 for example).

Caution: In real implementations at least the type of file as well as size should be validated to avoid filling the database or memory. Even if you check file size evil user might send zip file that is small but GB's when uncompressed, you can check uncompressed size with file_size method of ZipFile library (thanks RobIII for pointing this out in his comment).

Target:

Offer user form to upload zip file
Deflate the file in-memory
Insert the content of the deflated items to the database

Steps in this tutorial:

Prepare view and tamplate
Python code to open the zipfile and save contents to db

Prepare view and tamplate

I like to use the django Form class. This is how I set it up:

from django import forms
class UploadFileForm(forms.Form):
    file  = forms.FileField()

Code for the view:

  def upload(request):
    if request.method == 'POST':
      form = UploadFileForm(request.POST, request.FILES)
      if form.is_valid():
        handlezipfile(request.FILES['file'])
        return HttpResponseRedirect('/upload_successful')
    else:
      form = UploadFileForm()
    context = {'form': form}
    context.update(csrf(request))
    return render_to_response('upload.html', context)

Simply doing dictionary lookup with the name in the form for request.FILES gives us reference to the file. We actially get a UploadedFile object that is a wrapper around the file. It offers following methods: read(), multiple_chunks() and chunks(). It also offers two properties: name and size.

Template code:

This looks a lot like any other form but it's important to add the enctype attribute to allow sending the file along with the response.

Open the zipfile and save contents to db

I am using the zipreader script I wrote a while ago (Github repository) that offers a bit simpler way to open and process zip files compared to the zipfile library.

from zipreader import fileiterator
def handlezipfile(zipfile):
  for filename,content in fileiterator(zipfile):
    tf = TextFile()
 tf.content = c
 tf.filename = filename
 tf.save()

The method simply iterates through all files and folders and inserts them to the database using our own TextFile model which has two fields: name (CharField) and content (TextField). This is assuming the zip archive only contains text files. Now the text files are in database and you can use them however you want.

Here is the model I used for the file:

class TextFile(models.Model):
  name = models.CharField(max_length=256)
  content = models.TextField(blank=True)

Follow me in Twitter: @JuhaFinn

How to cache outgoing API calls in Django

2012-04-12T09:07:00.001-07:00

Most of the articles about caching in Django are about caching users requests. Here I want to show how to use it to cache outgoing requests to 3rd party services.

Useful during development phase

I was recently doing some development for a Django project that used StackOverflow API. After some testing I got a notice that I used up the API quota (it's only 300 for unregistered apps). So I thought it's best to add caching to outgoing API calls. I would probably be able to stay under typical quota limits, but it also has other benefits. Firstly I don't have to worry about flooding the 3rd party servers when I continuously refresh pages, add items, re-add items etc. Secondly after adding caching speeds up tasks that do batch updates because all API calls could be avoided. This happened quite a few times when database schemas change, or I changed how I transformed some of the dayt. You'll probably end up implementing caching at some point so implementing the caching for development help is a good choise, especially since Django makes it so easy.

Session on Caching from last years Djangocon

My requirements for caching (for development setup)

Django documentation page about caching has good examples about using different caching techniques. Here we will use the low level caching features to achieve what we want:

Persistently store cache entries to database
Set long expiration time for the data
Use url as the key for the data
Store the content of the http request (probably JSON)
Write a wrapper method to make usage easier

Setup

Add cache settings to settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'dev_cache',
        'TIMEOUT': 60*60*24*7*4,
        'VERSION': 1,
        'OPTIONS': {
            'MAX_ENTRIES': 10000
        }
    }
}

This tells django to use cache table named "dev_cache" for storing the entries. The entries will expire after 4 weeks. Maximum number of entries is set to 10000.

Create table for the cache with name defined in LOCATION (you can decide):

python manage.py createcachetable dev_cache

Write the caching code

Here I am using the requests http library to fetch json data from the API. This function returns de-serialized python object.

from django.core.cache import cache
import requests
import json

def fetchJson(url):
    cached = cache.get(url)
    content = ""
    if not cached:
            r = requests.get(url)
            if(r.ok):
                cache.set(url, r.text)
                content = r.text
            else:
                # Write some proper error handling code here
                print "Error - status code: " + r.status_code  
    else:
        # Return the cached content
        content = cached

    return json.loads(content)

Adjusting for production deployment

The settings I used are probably not what you want for production setup. You should read the Django docs about caching to understand and adjust the settings to suit your needs.

Flushing the database backed cache

It seems that some cache backends have differing ways to clear them. DatabaseCache has clear() method that you can call to remove all cached entries from the database.

Follow me in Twitter: @JuhaFinn

How to: Import non-Eclipse Github project into Eclipse

2011-11-12T15:55:00.001-08:00

This howto article will instruct how import a Github repository into your Eclipse workspace, even if it's not an eclipse project. You can skip step 3 if you just want to inspect the code without making it runnable (let it be General Project in Eclipse). I will show the example snippets with python and Java, but it's possible to use this process with other Eclipse supported runtimes too.

Requirements:

I tested this on Eclipse Helios, but it should work on older and newer releases too
You should have EGit installed and configured
If you want to make the code runnable you should have necessary language plugins installed, for example PyDev for Python (Java is usually included by default).

TL;DR

Here is short version for the advanced users:

Clone or fork the repository: (Import..>Git project>Clone...)
Create General Project from the repository: (Import>Git Project, select the repository you just cloned).
Transform the General Project to language specific project: Add nature and buildCommand elements to .project file).
(for Java projects) Add libraries and source directories from Project>Properties>Java Build Path.

Step 1: Clone or fork the repository

This step adds the repository to know GIT repositories list in Eclipse. See the list in File>Import>"Projects from Git". Skip this step if you already have it there.

If you are not planning to make any changes to the repository it is easiest to clone the read-only repository to your computer. You can find the git link to this read-only repository on the project main page (for example https://github.com/django/django ). I will use Django repository as an example on this article. The read only repository link should look like this:

https://github.com/django/django.git

If you plan to modify the code you should fork the repository instead. You can do this by clicking "Fork". This will get you link to your own personal repository. You must be logged to do this.

Now that you have the link you can clone the repository:

In Eclipse go to File>Import... and select Git>"Projects from Git". On the "Select a Git repository" page click Clone...
Now paste your link to URI field.
If everything is configured correctly you should get to "Branch Selection" page. Select the branch that you want to use and click next.
If you want the code to be under your eclipse project (I like it so), you should select folder two levels inside your workspace. The first folder is the project name and second is folder for the code. I use "C:\myworkspace\django\django". The second folder will not be visible in any way when you use the project in Eclipse. Click Finish.
Eclipse clones the repository into your harddrive and in the end you should see the new clones repository in the "Select a Git repository page"

Step 2: Create General Project from the repository

This step creates a General Project that includes the code from the repository

Create the project:

Again go to File>Import...>"Projects from Git".
Select the repository you created in step 1 and click Next.
On next page you should select: "Import as general project" and click Next and Finish. If your repository was an eclipse project you could select "Import existing projects".
You should now see your project on the "Package Explorer" with all the contents of the repository.

Step 3: Transform project into language specific project

This step will transform the General project with no ability to run the code into a language specific project (Python projectfor example) with ability to run the code.

You will need to modify the .project file to change the type of the project. You will need to add two elements into it: buildCommand and nature.

buildCommand element for Python:

<buildCommand>
<name>org.python.pydev.PyDevBuilder</name>
<arguments> </arguments>
</buildCommand>

buildCommand element for Java:

<buildCommand>
<name>org.eclipse.jdt.core.javabuilder</name>
<arguments> </arguments>
</buildCommand>

nature element for Python:

<nature>org.python.pydev.pythonNature</nature>

nature element for Java:

<nature>org.eclipse.jdt.core.javanature</nature>

If you are using some other language you can obtain these elements by creating a new empty project of that type and opening the .project file on root of that project. You should see both of these elements there.

Now that you have these two elements you can add them to .project file: buildCommand inside buildSpec and nature inside natures. You can find .project file on Navigator on the root of your project. Doubleclick it to open it. Now insert the elements. Your .project file should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>django</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.python.pydev.PyDevBuilder</name>
       <arguments>
    </arguments>
    </buildCommand>
    </buildSpec>
   <natures>
   <nature>org.python.pydev.pythonNature</nature>
   </natures>
  </projectDescription>

Now you can Right Click a code file and see the execution options in the "Run as"-menu. For python project this is all you need to run the code properly. For other languages you may need to additionally set the classpath or similar things. For example Java projects need their classpath to be set.

Adding classpath on Java projects

Java projects additionally require you to tell Eclipse where the executable java code is inside the project. It normally reads this from the .classpath file on project root folder.

Configure classpath:

You can manually configure the source folders by selecting the project and clicking Project>Properties>Java Build Path.
Open Source tab and click add folder and select the root folder for the java code (sometimes it it /src sometimes it can be the root depends on the project structure).
If project needs libraries add them on Libraries tab
Click Ok and the .classpath should be generated to tell Eclipse where the code is located.

Problems?

If you problems with eclipse projects sometimes it helps to Close and re-open the project. You can do this by selecting the project and click Project>Open/Close Project.

This process may not work for all kinds of projects but at least I have tried it with multiple Java and Python project

Follow me in Twitter: @JuhaFinn

Importance of testing with large data sets

2011-11-10T04:28:00.000-08:00

Edit: Title modified. I later realized that the word 'training' can refer to training machine learning algorithms (especially when speaking of large data sets) so I started to use word 'testing'. I was trying to use the word 'training' to refer to the training process of your development skills.

My awakening

When I was getting started on my first real programming job I got a task to build a component for a larger system. I took the challenge with enthusiasm and started designing. After a while I had good looking component with good unit test coverage and integration was tested too. Then I got the task to performance test it. I hadn't done performance testing much before so I felt a bit insecure but I thought why would there be a problem, all my unit test showed it was solid code.

Surprisingly after running the tests with millions of data points something started breaking, and the CPU went up. This was the point I started realizing the true value of those algorithm classes that seemed theoretical at the time. Additionally, I realized its really hard to catch all race conditions in multithreaded applications with unit tests. Later on I did manage to catch the concurrency bugs after some debugging and improve the performance to acceptable level, but it opened my eyes for the importance of testing and training with realistic and large data sets.

How I see it here are most important advantages for testing with large datasets:

Catch concurrency bugs
Find performance problems
Nice numeric results for the marketing purposes
Get confidence on your product

Get used to large data sets while learning

It's sometimes hard or impossible to get suitable large data set that represents real data for your product. That's why it's important that already when you are learning a new technology you train your skills with large data sets. This way you will learn the performance and concurrency challenges when running your own throwaway training programs where it's easier to find sample data rather than later building the product for a company.

When you are in school or home learning new technologies it's usually easiest to test your skills with simple proof of concept mini-applications. This works for learning the basics but it may also let you use inefficient or error-prone approaches that would not work in high-demanding use cases. Plan your first application in a way that you can run large data sets through it. Even if it's purpose is for you to just learn the technology. Nowadays there are so many easy to use libraries for accessing xml-files, data-files, databases, etc. that it will not add much effort on your application development to support importing large sets of data. And learning to import data is surely a skill that you'll need later.

In addition to being useful it's also nice feeling when you see your application processing millions of records. It's not a simple throwaway application anymore, but it has potential to provide some real value. Maybe you can create statistical analyses based on data. Maybe you can transform the data into something more useful. Anyway, it's closer to being a real application. Now let's see where to find sample data.

Web is full of open data sets for you to use

If you have good list of open data sets in your mind, you will come up with more useful ideas for apps that can actually use real world data. Even if it's just for the purpose of learning a new technology. Here is a list of some open data sets. Feel free to add your in the comments.

Few data sets to get started:

StackExchange data dumps: All stackexchange sites release their data in XML format. You can download them using torrent. Largest is Stackoverflow dump that contains 6+ million posts. Recommended!
Wikipedia dumps: Wikipedia offers it's data in various formats
Large opensource projects can offer some interesting information for example about lines of code, commits or project timeline. Check Chromium for example.
Many countries have started to release Government data in various formats. In US there is DATA.gov. Check your local organizations of open data.
ProgrammableWeb offers good list of open API's. Many may not provide large sets of data but at least they are real-world data.
GIS data: There are many websites providing open GIS data sets. For example National Elevation Dataset. (thanks to Doug for pointing this out).

After getting comfortable with one data source its much easier to use that in the future apps. I would recommend using the Stackexchange data dumps for their varying size size but simple structure.

Follow me in Twitter: @JuhaFinn

What is Google App Engine and what you should know about it

2011-01-01T13:57:00.000-08:00

This article is to provide background for App Engine for other post that I'm writing about my experiences about GAE and who I think should be using it.

So for those that don’t know Goole App Engine (or GAE) is a cloud platform that runs Python and Java based applications. Or more specifically it runs all JVM based languages on Java side (GAE/J). GAE gives an url for your application (your-app-name.appspot.com) and you can even use it with your own domain which gives you access to the Google Apps. It offers a lot of features and free plan that offers generous daily quotas (more on that on my next post).

Memcache:

There is built-in support for using memcache in GAE which allows to make your site very efficient and responsive (and it saves your precious daily CPU quota).

Only Bigtable as database:

You’ll have to use Google’s Bigtable datastore which can feel a bit limiting if you come from SQL world, but it does enforce you to write efficient data queries. It supports the GQL language which is almost like SQL but with some limitations.

GAE is a cloud service:

Google handles starting new instances and replicating the data automatically. You can concentrate on the important part, creating the application. This is great risk less opportunity to jump on the cloud movement if you feel your mind is stuck on the one-server one-database mentality.

No filesystem:
GAE doesn’t give you access to any kind of filesystem, but you can serve static files if you include them to your package. I don’t actually mind this too much, although it did seem a bit unusual at first.

Versioning:

One cool thing that GAE offers is versioning. You can define the version of your app in the conf file and it will not override the previous version. You can have multiple vesions of your app running on different url’s (http://7.latest.your-app.appspot.com where 7 is the version) and you can set which is version is served on your app’s url.

Python:

I’m more familiar with the python side and there are some pre-installed frameworks that you can immediately start using. For exaple it has limited version of Django 0.96 (which is a bit old). But it is possible for you to install Django yourself too. It isn’t straightforward since you can’t use Django’s datamodels directly, but there is a project called app-engine-patch (http://code.google.com/p/app-engine-patch/) that will make the process easier (I have only tried it but am not currently using it).

Java and JVM:

I’m not too familiar with the Java side but here are the basics. You can use JDO to interface with the datastore. This does require some xml configuration, but I’m sure most Java programmers are used to it. For rendering GAE/J supports JSP’s that many developers are probably already familiar with.

Now the interesting part is that you can run other languages as Java on top of the JVM. Check out a list from here of the supported languages. Some interesting ones:

Jython (Python),
Groovy,
JRuby (Ruby)
Rhino (JavaScript)
Scala
Quercus(PHP). Yes, you can even run PHP on GAE :).

How to listen to podcasts with increased speed on your iPod

2010-09-17T11:56:00.000-07:00

Introduction

I have iPod Nano (4th gen) and I noticed that there is setting to listen to audiobooks with increased speed. The setting can be found from Settings>Playback>Audiobooks. I noticed that changing this actually speeded up some of the mp3 podcasts I was listening, but not all. If you haven't heard of people listening podcasts or audiobooks with increased speed don't worry, it will not make them sound like bunch of squirrels talking. It just increases the tempo without affecting the pitch. My iPod nano plays them with 125% speed giving 25% increase. You can hear the difference clearly when music is playing, but there is not too big difference when you hear people talking, but it does save you 20% of the time :).

I started to investigate how I could get all the mp3 audiobooks and podcasts to play with increased speed. It seems that when you import songs to iTunes or iPod (without iTunes) they analyze the file and decide if it is of type "audiobook" or not. The speed settings are only available for tracks that iPod considers as audiobooks. Below are two different methods describing how to get your mp3 podcasts and audiobooks (or any other mp3's) to play with increased speed.

Additional benefit: If your iPod considers the track as audiobook it also means the iPod remembers the position in the track even if you listen to something else in between. This is really handy and saves you time to scan the position where you stopped listening last time.

Get mp3 podcasts or audiobooks to play with increased speed without using iTunes

I like to use Floola as an iTunes replacement, since I it's lighter and I don't have my music library in iTunes. And it's free. I have found that this process doesn't work 100% of the time, but I haven't yet found the reason. It seems that iPod uses some secret magic to detect if track is audiobook or not. It does work maybe 90% of the time. The iTunes method seems to work more reliably.

Process:

Download the mp3 podcast file. You can use one of my favourite podcasts Techzing for testing: http://techzinglive.com/?p=435 . It works perfectly for me.
Right click it on windows explorer (I'm sure Mac and Linux have similar operation too) and select "Details" page. There you see field called Genre. Change that value to be "AudioBook" (without the "").
Now open Floola (or any other iTunes alternative) and import the file to your iPod. I like to keep my podcasts on separate paylist.
Eject the iPod.
Now in iPod mainmenu go Settings>Playback>Audiobook and select "Faster". This will increase speed by 25%.
Now start playing the podcast as you would normally. You should hear it with increased speed.
Verifying success: When you are playing the podcast press the center button for a few seconds. You should see a menu with options "Slower", "Normal" and "Faster". If your iPod doesn't recognize it as audiobook you see "Add to On-The-Go", "Browse album" and so on". If this didn't work for you, I suggest to try the iTunes method (see below).

Get mp3 podcasts or audiobooks to play with increased speed with iTunes

I found that even if I set the genre as AudioBook iTunes still doens't recognize my podcasts as audiobook. The good news is that iTunes has the Media Type option which makes to make the iPod to consider a track as an audiobook.

Note: I'm using iTunes 8 for this. There might be differences with different versions.

Process:

Import mp3 to iTunes (I trust you can do this if you use iTunes) and move it to the iPod.
Right click the file (the one under iPod) and select "Get info". Select "Options" -tab. Select0 "Media Kind" to be "Audiobook". Click Ok. You can also to dhis step for multiple files at once.
Eject the iPod.
Verifying success: When you are playing the podcast press the center button for a few seconds. You should see a menu with options "Slower", "Normal" and "Faster". If your iPod doesn't recognize it as audiobook you see "Add to On-The-Go", "Browse album" and so on". If this didn't work for you, I suggest to try the iTunes method (see below).

Getting even faster playback speeds

If you want different speeds than what your iPod lets you choose you'll have to pre-process the media files before importing them to iPod. I have used the Audacity-tool to increase the tempo by 60%. I find that acceptable rate but I do have to put more attention to listening not to miss anything. Here is link to one tutorial that explains this process: http://www.tunequest.org/speeding-up-podcasts-part-2/20070219/. This takes some time to do. There might be some automated ways to do this, but so far I haven't stumbled into any good ones. I have only used this method a few times for some longer podcasts.

Let me know if you know better ways to speed up the audiobooks or podcasts.

Understand your brain

2010-09-16T06:39:00.000-07:00

I have recently been quite interested of how humans think and make decisions. Especially the balance between conscious and unconscious mind is wonderfully complex. I read a book from Malcolm Gladwell called Blink: The Power Of Thinking Without Thinking. It was really interesting book and also very entertaining to read. It explains how the unconscious mind works and what are the good and bad features of it. For example unconscious part of brain can analyze big amounts of visual information and detect the important information from it. There is also a negative side there that the marketing people like to use. Choosing a product in a shop involves many unconscious decisions which are hard to control.

Knowing the strengths and weaknesses of our brain is important when we want to make optimal decisions. Biggest thing was that now I have more trust in my unconscious decisions in certain situations. It's still though to explain to other people how we "just know" the answer, but at least I know my self that my unconscious mind can process the information in a rational way. I have studied poker and there the unconscious decisions are highly valued, it wouldn't be first time when the best players say in an interview that they "just knew" what the other person had - that's the unconscious part of brain working and processing all the information.

I attached an interesting talk by David Rock. I haven't read his book, but he seems to be entertaining talker and he explains some interesting aspects of the unconscious mind in this video. I recomment watching.

Google Tech Talk: Your Brain at work

Procrastination - how getting conscious about it helped me

2010-09-13T12:54:00.000-07:00

I only recently googled the definition of procrastination:

"procrastination refers to the counterproductive deferment of actions or tasks to a later time." (Wikipedia)

I got interested about this term. Maybe because I'm not native english speaker I had managed avoid truly understanding this word for so long. It seems that there was a lot of theories and science behind this term, and the definition seemed to reflect my feelings surprisingly well. It seems I'm not the only one feeling like that. Good for me! Don't get me wrong, I still manage my tasks, but if there was no procrastination I would get a nice efficiency-boost on my everyday life, and probably also relieve my stress too.

You can't beat what you don't know

I started to read more about this interesting term and I also listened to some related podcasts. I didn't really expect doing that would "fix" my behavior in anyway, I was just interested. To my big surprise I noticed that I had more strength to fight procrastination, it seemed that just being conscious of it helped me to fight it.

After thinking about it for a while it starts to seem logical. Procrastination comes from our inner needs and feelings. By being more conscious about it we can put more logic into our decisions. Previously it was easy to drift doing other stuff and avoid for example doing the dishes. Now when I "drift" I realize what I am doing, and I can use a moment to think: "Is this what I really want to be doing". Surely just being conscious about it won't make it go away, but I feel that it gives me a chance to improve.

My tips to learn more about procrastination

1. Search basic information about procrastination from wikipedia and google. It's good to know the basics about procrastination, I'm pretty sure that even the wikipedia page includes new information for everyone. And it'll at least increase your general knowledge.

2. Listen to procrastination podcasts. I found that the podcasts were interesting to listen to and give you tools to reduce procrastination. They also work as motivation boosters to give you some confidence. Two that I tried: Stop procrastinating now (last podcast from 2008) and iProcrastinate.

3. Learn more about how brain works. If you understand your decision making process better, you can consciously affect more easily. There is a lot of interesting information about this.

Good procrastination

No matter what you do procrastination will still be there. You might be able to do it less, but it's something that people just need to do every now and then. So why wouldn't you do something that is actually useful as your procrastination activity. We all have long to-do lists, and I'm sure it's possible to find something from there that you are willing to do when you feel like procrastinating. If you clear one of those tasks that you'll have to do anyway you actually reduce future stress. Paul Graham has written excellent essay related to this called: Good and Bad Procrastination. I highly recommend reading it.

Slightly related: Penn&Teller on self esteem

*procrastination alert* I happened to watch today an episode of Penn&Teller Bullshit about self esteem. I interestingly found that it actually issued some of the same topics as what I found when researching about procrastination. For example fearing failure is big factor in procrastination and this is tightly connected to self esteem. Well, just watch it an see make up your own mind. You can try searching youtube if you want to see it online.

Resources to keep up with trending topics

2010-07-25T02:47:00.001-07:00

Here are some of my resources and tools to follow the wonderful world of technology. Feel free to suggest new ones to me.

General tech resources:

Blog: Mashable - Although this is concentrated on social media, they have so many posts every day that everybody can find something interesting here. While writing this I found post of 10 videos exploring Google's development center (I suggest muting the first video).
Blog: Engadget - Mostly concentrated on devices. Lots of well-written posts every day. You are guaranteed to find articles about the trending devices here. Be prepared for Apple-biased opinions.
Blog: GigaOM - The first tech-blog I started following. Good mix of well written articles about entrepreneurship, mobile industry, financial sector and technology. Good way to keep up with the industry. Nowadays there seems to be more articles about consumer devices and services (like Mashable and Engadget).

Entrepreneurship resources

Note: I'm not en entrepreneur, but I have noticed that entrepreneur related resources really help you to know about the newest things happening. And who knows, I might be involved in a startup at some point.

Blog: Arctic Startup - Interesting entrepreneurship blog mainly concentrating on startups around scandinavia.
Podcast: TWiST (This Week in Startups) - Entertaining video-podcast by Jason Calacanis about entrepreneurship. I usually listen to audio only. Well produced and they often have good quests.
User generated news: Hackernews - This is quite technical site, but many entrepreneurs hang out here. And it is run by the respected Y-combinator.

Programming resources

User generated news: Hackernews - Many entrepreneur-minded developers post interesting articles here. They also have nice community.
Aggregated news: Coder i|o - Aggregated and ordered list of links about trending topics. Bit similar as Hackernews or reddit, but I somehow like the simplicity.
Blog: CODING HORROR - One of most popular programming blogs around by Jeff Atwood. He is also one of the Stackoverflow founders and main developer (see next item). Posts are well written and entertaining. This makes the blog less personal and more like a magazine.
Q&A: Stackoverflow - Questions and answers for developers. You certainly don't have to have something to ask to enjoy your time on this site. Just browsing questions and answering the ones you can is good and educating. The points system gives an addicting element to this site - can you get more points than your friend ?
Podcast: Stackoverflow podcast - This used to be my favourite podcast, but they stopped producing new episodes. This is made by Joel Spolsky (Author of Joel on software) and Jeff Atwood (author of Codinghorror). Their 87 existing episodes are excellent way to spend your time driving, but be prepared for a lot of talk about now irrelevant development steps of Stackoverflow.
Podcast: TechZING - Currently my no.1 podcast. Made by Justin Vincent (creator of Pluggio and Swarm SG) and Jason Roberts (creator of Appignite). Good mix of entertaining offtopic talk and deep technical topics from two charismatic personas. Their different opinions and thoughts bring a nice friendly conflict into the podcast.

Equally True

Free PaaS options for deving, experimenting and more

Careful, don't get locked in

Cloud Database providers

170+ Django conference videos

Categories:

5 selected videos

Other videos

3 useful features from Opera Dev tools that Chrome is missing

1. Easy remote debugging

2. Opera Mobile & Mini simulator

3. Network Request crafter & header modification

Conclusions

Takeaways from Google Developer Day Berlin

Keynote

Takeaways:

People on stage:

Extracting uploaded zip file with Django in memory

Prepare view and tamplate

Open the zipfile and save contents to db

How to cache outgoing API calls in Django

Most of the articles about caching in Django are about caching users requests. Here I want to show how to use it to cache outgoing requests to 3rd party services.

Useful during development phase

Session on Caching from last years Djangocon

My requirements for caching (for development setup)

Setup

Write the caching code

Adjusting for production deployment

Flushing the database backed cache

How to: Import non-Eclipse Github project into Eclipse

Importance of testing with large data sets

What is Google App Engine and what you should know about it

How to listen to podcasts with increased speed on your iPod

Introduction

Get mp3 podcasts or audiobooks to play with increased speed without using iTunes

Get mp3 podcasts or audiobooks to play with increased speed with iTunes

Getting even faster playback speeds

Understand your brain

Procrastination - how getting conscious about it helped me

Resources to keep up with trending topics