Heroku Dev Center Articles

Process Scheduler

2013-05-23T23:45:33Z

Process Scheduler is an add-on that allows for easy scheduling of Heroku processes depending on the hour of the day and day of the week.

Configuring Process Scheduler to scale your processes of any kind is done in a few clicks through a calendar web UI and will allow you to save on dyno hours with very minimal efforts.

Provisioning the add-on

Process Scheduler can be attached to a Heroku application via the CLI:

$ heroku addons:add process-scheduler
-----> Adding process-scheduler to sharp-mountain-4005... done, v18 (free)

After installing Process Scheduler, it should be configured to properly scale your processes.

Service setup

The Process Scheduler dashboard allows you to configure the schedule for each of the process types of your application.

The dashboard can be accessed via the CLI:

$ heroku addons:open process-scheduler
Opening process-scheduler for sharp-mountain-4005…

or by visiting the Heroku apps web interface and selecting the application in question. Select Process Scheduler from the Add-ons menu.

The first time you visit the Process Scheduler dashboard, you will need to fill in your Heroku API key and your time zone before configuring your processes schedules.

Removing the add-on

Process Scheduler can be removed via the CLI.

This will destroy all associated data and cannot be undone!

$ heroku addons:remove process-scheduler
-----> Removing process-scheduler from sharp-mountain-4005... done, v20 (free)

Support

All Process Scheduler support and runtime issues should be submitted via one of the Heroku Support channels. Any non-support related issues or product feedback is welcome at the Process Scheduler UserVoice.

Rcmmndr

2013-05-23T23:45:31Z

Rcmmndr (http://addons.heroku.com/rcmmndr) is a recommendation engine add-on based on Apache Mahout.

Adding a recommendation engine to an application will enable the application’s users to discover items or other people they might not have found by themselves. Providing similar functionality with hard-coded search algorithms may be difficult once the application gets bigger and complex.

Rcmmndr is a collaborative filtering solution, which essentially means that, it will take all the current preferences that have been made in your application, and try to guess new preferences for a specific user. It is not specific to item types, but it only depends on the present preferences data. You can recommend books, movies, cars to your users, or even other users to follow.

Rcmmndr is easily accessible via a REST API and has supported thin client libraries for Java, Ruby and Python.

You can refer to the Official Mahout documentation or “Mahout In Action” book if you would like to get extensive information on how the recommendations are done. But to summarize; Rcmmndr will take all the preferences that you have created and calculate a user similarity based on these preferences, which then will be used to make recommendations to the users. Sometimes there might not be enough information to make recommendations for a specific user, at this case the server will return an empty json body.

Rcmmndr is using a sensible-default recommendation algorithm,so it is ready to make recommendations as soon as you put your data in. But you can also customize the recommender and tune it better for your domain.

Provisioning the add-on

Rcmmndr can be attached to a Heroku application via the CLI:

A list of all plans available can be found

here

$ heroku addons:add rcmmndr
-----> Adding rcmmndr to sharp-mountain-4005... done, v18 (free)

Once RCMMNDR has been added a RCMMNDR_ACCESS_URL setting will be available in the app configuration and will contain the canonical URL used to access the newly provisioned RCMMNDR service instance. This can be confirmed using the heroku config:get command.

$ heroku config:get RCMMNDR_ACCESS_URL
http://api.rcmmndr.com/api_key/{your_api_key}

After installing Rcmmndr the application should be configured to fully integrate with the add-on.

API Reference

This is the general api reference that you can refer to see how the web service is implemented.

Preference API

You can do the crud operations about preferences using this api. Please refer to bulk api for batch operations.

Rcmmndr is a colloborative filtering service. The flow starts with creating preferences. A single preference contains:

user_id : the actual user on your app that made the preference
item_id : the item that the user expressed the preference for, such as a book or a movie
rating: the level of the preference if applicable.

So for instance, if you own an ecommerce website and you want to create a recommendation widget based on your sales, you might want to set a default value to the rating field which is the same for all users. But if you have a rating value as well you can use it.

Creating a preference

api_key/{apiKey}/preference/{uid}/{iid}/{rating}

curl -X POST "http://api.rcmmndr.com/api_key/{API_KEY}/preference/1/101/5"

This command will add a preference for user with id 1 to the item 101 with a weight of 5.

Deleting preferences with specific user id

To delete all the preferences that belong to a specific user id:

curl -X DELETE "http://api.rcmmndr.com/api_key/{API_KEY}/preference/{uid}"

Deleting all preferences

This will delete all the preferences that you have created on that account

curl -X DELETE "http://api.rcmmndr.com/api_key/{API_KEY}/preference/_all"

Exporting your preference data

You can get all your data in csv format like the following:

curl -X GET "http://api.rcmmndr.com/api_key/{API_KEY}/preference/_all"

Recommendation API

Get a recommendation for a user

api_key/{apiKey}/recommend/{uid}

curl -X GET "http://api.rcmmndr.com/api_key/{API_KEY}/recommend/1"

This will return the top 10 recommendations for user 1

Bulk API

Please note that currently,the process of loading a dump file will override your existing data.

To load a preferences file dump:

curl -X POST "http://api.rcmmndr.com/api_key/{API_KEY}/preference/_bulk" --data-binary @preferences.tsv -H "Content-Type: text/plain"

To load a gzip compressed preferences file dump:

curl -X POST "http://api.rcmmndr.com/api_key/{API_KEY}/preference/_bulk" --data-binary @preferences.tsv.gz -H "Content-Type: text/plain" -H "Content-Encoding: gzip"

Stats API

You can use this api to check for your account statistics. Stats API will return the results in json format.

curl -X GET "http://api.rcmmndr.com/api_key/{API_KEY}/_stats"

Settings API

You can use the settings api to finetune your recommender and check how different algorithms behave on your dataset.

Get Current Settings

curl -X GET "http://api.rcmmndr.com/api_key/{API_KEY}/_settings"

>>> {"response":"tenant has no customizations"}

If you have defined a custom recommender, you will see the definition here. Rcmmndr uses a JSON based DSL to express custom recommenders. If you do not see any customizations such as the case above, it means that you are using the default recommender. (GenericUserBasedRecommender)

Update Settings

curl -X POST "http://api.rcmmndr.com/api_key/{API_KEY}/_settings" -H "Content-Type:application/json" -d '{
         "recommender": {
              "impl":"SlopeOneRecommender"
         }
    }'

This will use the SlopeOneRecommender instead of the default GenericUserBasedRecommender.

Now if you do a get on the same Rest url, you will see that your settings has been updated:

{
	"recommender": {
    	"impl":"SlopeOneRecommender"
	}
}

Available Recommenders

Different recommender algorithms or same algorithms with different parameters will lead to different recommendations on the same dataset for the same users. You can finetune the system as you wish using the settings DSL. Please remember check the Apache Mahout Documentation for choosing the suitable Recommender.

Using SlopeOneRecommender

{
	"recommender": {
    	"impl":"SlopeOneRecommender"
	}
}

You can post the descriptor above to switch to SlopeOneRecommender. SlopeOneRecommender does not have any parameters so the DSL is simple. Please check the Apache Mahout Documentation for details about SlopeOneRecommender.

Using GenericUserBasedRecommender

{
"recommender": {
    "impl": "GenericUserBasedRecommender",
    "params": {
        "UserSimilarity": {
            "impl": "PearsonCorrelationSimilarity"
        },
        "UserNeighborhood": {
            "impl": "NearestNUserNeighborhood",
            "params": {
                "n": 2
            }
        }
    }
}
}

You can post the descriptor above to switch to GenericUserBasedRecommender. A UserBasedRecommender requires a UserSimilarity and a User Neighborhood. Please check the Apache Mahout Documentation for details.

Supported User Similarity Algorithms

PearsonCorrelationSimilarity
LogLikelihoodSimilarity
TanimotoCoefficientSimilarity
EuclideanDistanceSimilarity

are currently supported.Please refer to the Apache Mahout Documentation for details.

Supported User Neighborhood Algorithms

NearestNUserNeighborhood

is currently supported. You can change the size of the User Neighborhood with the “n” parameter. N should be between the range 2-50 at the moment.

GenericItemBasedRecommender

{
"recommender": {
    "impl": "GenericItemBasedRecommender",
    "params": {
        "ItemSimilarity": {
            "impl": "PearsonCorrelationSimilarity"
        }
    }
}
}

You can post the descriptor above to switch to GenericItemBasedRecommender. An ItemBasedRecommender requires an ItemSimilarity. Please check the Apache Mahout Documentation for details.

Supported Item Similarity Algorithms

PearsonCorrelationSimilarity
LogLikelihoodSimilarity
TanimotoCoefficientSimilarity
EuclideanDistanceSimilarity

are currently supported.Please refer to the Apache Mahout Documentation for details.

Estimation API

You can get an estimation to a user’s specific preference for an item as follows:

curl -X GET "http://api.rcmmndr.com/api_key/{API_KEY}/estimate/{user_id}/{item_id}"

Evaluation API

While experimenting with these different recommenders, you might want to check which recommender performs better with your domain by evaluating how closely the estimated preferences match the actual preferences.

For this purpose Mahout provides some built in evaluators, and some of them are currently available in the Evaluation Api.

Supported Evaluators

AverageAbsoluteDifferenceRecommenderEvaluator
RMSRecommenderEvaluator

You can check the details of these evaluators form the Mahout documentation but in short, Root-mean-square (RMSRecommenderEvaluator) more heavily penalizes the estimates that are much different from the expected ones.

Evaluation is done by spitting the dataset into two, one is used for running the original recommender and the other is used for comparing the results.

Evaluator Parameters

trainingPercentage: For instance 0.7 means that the evaluator will train with the %70 of the data and test with %30 of the data.

evaluationPercentage: 0.1 means that %10 of the data will be used in evaluation process. Note that during the beta, this propery might be limitied to a lower default until the development is finalized.

AverageAbsoluteDifferenceRecommenderEvaluator

{
	"evaluator": {
    	"impl": "AverageAbsoluteDifferenceRecommenderEvaluator",
    	"params": {
        	"trainingPercentage": 0.1,
        	"evaluationPercentage": 0.1
    	}
	}
}

RMSRecommenderEvaluator

{
	"evaluator": {
    	"impl": "RMSRecommenderEvaluator",
    	"params": {
        	"trainingPercentage": 0.7,
        	"evaluationPercentage": 0.2
    	}
	}
}

Sample Usage

curl --include -X GET http://api.rcmmndr.com/api_key/{API_KEY}/_evaluate -d  '{
"evaluator": {
    "impl": "AverageAbsoluteDifferenceRecommenderEvaluator",
    "params": {
        "trainingPercentage": 0.1,
        "evaluationPercentage": 0.1
    }
}
}' -H "Content-Type:application/json"

Assessing the Result

The evaluation result will be a double value similar to the following:

{
	"score": 0.9635416666666665
}

This is the average difference between the estimated preferences and the actual preferences. The result above was extracted from a dataset whose preferences were in the range 1 to 5. So the result means that on average our recommendation is of by 0.9 ,Depending on your requirements you can interpret this in different ways,for our test scenario its does not seem great but not too bad as well. You can try this evaluation with different evaluator and recommender settings.

A perfect recommendation’s score would be 0. Meaning that there are no differences at all. But this is probably impossible without some domain specific hints to the recommender.

Using with Ruby or Rails 3.x

You can use the RubyRcmmndr gem to start making recommendations.

gem 'RubyRcmmndr', '0.0.1', :git => 'https://github.com/Rcmmndr/RubyRcmmndr.git'

Update application dependencies with bundler.

$ bundle install

Creating a Client Instance

require 'RubyRcmmndr'
client = RubyRcmmndr::Client.new("API_KEY")

Adding Preferences

client.create_preference(1,102,2.5)

this will add user ‘1’ a preference of ‘2.5’ for the item ‘102’

Making Recommendations

recs=client.get_recommendation(1)

the Ruby Data Structure ‘recs’ contains the recommendation results as itemid,value pairs

Usage Statistics

You can check the usage statistics for your account as follows:

client.get_usage_stats()["total_preferences"]
client.get_usage_stats()["distinct_keys"]

Deleting Preferences

Deleting Preferences for specific user

client.delete_preferences_of_user(1)

this will delete all the preferences of user ‘1’

Deleting All Preferences

client.delete_all_preferences

This will delete all the data in your account.

Bulk API

Bulk API is a better way of inserting many number of preference values. It is done by posting a text file (comma or tab separated) to the server.

client.bulk_update_preferences("preferences.tsv")

where the contents of preferences.tsv are similar to the following: (userID,itemID,preference)

196,242,3
186,302,3
22,377,1
244,51,2
166,346,1

Using with Python/Django

You can use the PyRcmmndr module to start making recommendations.

Installation

pip install git+https://github.com/Rcmmndr/pyrcmmndr.git

Creating a Client Instance

import pyrcmmndr
pyrcmmndrClient=pyrcmmndr.RcmmndrClient("API_KEY")

Adding Preferences

pyrcmmndrClient.create_preference(1,101,1.0)

this will add user ‘1’ a preference of ‘1.0’ for the item ‘100’

Making Recommendations

recs=pyrcmmndrClient.get_recommendation(1)

the dictionary ‘recs’ will contain the recommendation results as itemid,value pairs

Usage Statistics

You can check the usage statistics for your account as follows:

stats=pyrcmmndrClient.get_usage_stats()
assert stats['total_preferences'] == 1
assert stats['distinct_keys'] == 1

Deleting Preferences

Deleting Preferences for specific user

pyrcmmndrClient.delete_preferences_of_user(1)

this will delete all the preferences of user with id ‘1’

Deleting All Preferences

pyrcmmndrClient.delete_all_preferences()

This will delete all the data in your account.

Bulk API

Bulk API is a better way of inserting many number of preference values. It is done by posting a text file (comma or tab separated) to the server.

pyrcmmndrClient.bulk_update_preferences('preferences.tsv')

where the contents of preferences.tsv are similar to the following: (userID,itemID,preference)

196,242,3
186,302,3
22,377,1
244,51,2
166,346,1

Using with Java

You can use the rest client wrapper for Java and other JVM based languages.

https://github.com/Rcmmndr/java-client

Creating a Client Instance

rcmmndrClient = new DefaultRcmmndrClient("API_KEY");

Creating a Client Instance with custom Http Client (Apache Commons)

rcmmndrClient = new DefaultRcmmndrClient("API_KEY",httpClientInstance);

Adding Preferences

rcmmndrClient.createPreference(1l, 100l, 1.0f);

this will add user ‘1’ a preference of ‘1.0’ for the item ‘100’

Making Recommendations

rcmmndrClient.getRecommendation(10l);

this will return the recommendations for user 10 as a map

Usage Statistics

defaultRcmmndrClient.getUsageStats().getDistictKeys()

this will return how many different users are in the system who have at least one preference

defaultRcmmndrClient.getUsageStats().getTotalNumberOfPreferences()

this will return the total number of preference objects

Deleting Preferences

Deleting Preferences for specific user

defaultRcmmndrClient.deletePreferencesOfUser(5l)

this will delete all the preferences of user 5

Deleting All Preferences

defaultRcmmndrClient.deleteAllPreferences()

This will delete all the data in your account.

Bulk API

Bulk API is a better way of inserting many number of preference values. It is done by posting a text file (comma or tab separated) to the server.

defaultRcmmndrClient.bulkUpdatePreferences(new FileInputStream(new File("…/preferences.tsv")))

where the contents of preferences.tsv are similar to the following: (userID,itemID,preference)

196,242,3
186,302,3
22,377,1
244,51,2
166,346,1

Using with Clojure

There is a ready to run clojure webapp sample located here:

https://github.com/Rcmmndr/rcmmndr-clojure-sample

This sample is currently using the Java client

Monitoring

You can access the Rcmmndr dashboard from the CLI.

$ heroku addons:open rcmmndr

Migrating between plans

Application owners should carefully manage the migration timing to ensure proper application function during the migration process.

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade rcmmndr:basic
-----> Upgrading rcmmndr:basic to sharp-mountain-4005... done, v18 ($49/mo)
       Your plan has been updated to: rcmmndr:basic

Removing the add-on

Rcmmndr can be removed via the CLI.

This will destroy all associated data and cannot be undone!

$ heroku addons:remove rcmmndr
-----> Removing rcmmndr from sharp-mountain-4005... done, v20 (free)

Support

All Rcmmndr support and runtime issues should be submitted via on of the Heroku Support channels. Any non-support related issues or product feedback is welcome at support@rcmmndr.com

Add-on Plan Creation

2013-05-22T16:07:09Z

Add-on Providers can create Add-on plans using the Add-on Provider interface.

Care should be taken when creating plans since they are immutable. Plans must be immutable since there is no support for updating the name or price of a plan once it has been installed.

Once an add-on plan has been created, it is eligible for installation by any one of our beta or alpha users.

Add-on Provider API

2013-05-21T21:54:02Z

The Add-on Provider API is used when Heroku calls your add-on service. In these examples Heroku is issuing the requests and your add-on is providing the responses.

All calls should be protected by HTTP basic auth with the add-on id and password specified in your add-on manifest. All calls are required.

Provision

Request: POST https://username:password@api.heroku.com/heroku/resources
Request Body: {
  "heroku_id": "app123@heroku.com",
  "plan": "basic",
  "region": "amazon-web-services::us-east-1",
  "callback_url": "https://api.heroku.com/vendor/apps/app123%40heroku.com",
  "logplex_token": "t.abc123",
  "options": {}
}
Response Body: {
  "id": 456,
  "config": {"MYADDON_URL": "http://myaddon.com/52e82f5d73"},
  "message": "your message here"
}

Only information which is immutable between requests is provided at the time of provisioning. If you wish to access information which can change (e.g., owner email address or application name) then it is available, after the API has returned a 200 response to Heroku, via the App Info API. It is the responsibility of the provider to periodically check this endpoint to ensure data is consistent.

The region attribute in the request specifies geographical region of the app that the add-on is being provisioned to. Use this to provision the resource in geopgraphical proximity to the app, ignore it (if your add-on is not latency sensitive) or respond with an error if your add-on does not support apps in the region specified.

The logplex_token attribute is provided so that logs can be sent to the app's log stream. For more information, see the article describing logplex input.

The options object in the request are extra command line options passed in to the heroku addons:add command.

The id value returned in the response may be a string or integer value. It should be an immutable value that can be used to address the relationship between this app and resource. This means that a plan change can not change the value of this id, you can however update the config later to point the app to a different resource.

The config object and message in the response are both optional.

Heroku can also pass command-line parameters given to the addons:add action during add-on provisioning to the add-on provider. Providers that require access to advanced features will receive additional attributes based on the features they've requested.

Plan change

Request: PUT https://username:password@api.heroku.com/heroku/resources/:id
Request Body: {"heroku_id": "app123@heroku.com", "plan": "premium"}
Response Body: {"config": { ... }, "message": "your message here"}

The config object and message in the response are both optional.

Deprovision

Request: DELETE https://username:password@api.heroku.com/heroku/resources/:id
Request Body: none
Response Status: 200

Exceptions

Request: https://username:password@api.heroku.com/
Request Body: any
Response Status: 422
Response Body: { "message": "your message here" }

For 422 and 503 responses the message property from your JSON response will be displayed to the user. For all other responses, and when no message property is available, a standard error message will displayed.

Single sign-on

Request: POST https://username:password@api.heroku.com/sso/login/
Request Body: id=<id>&token=<token>&timestamp=<ts>&nav-data=<data>&email=<user-email>&app=<app-name>&other=vals
Response Header: Set-Cookie: heroku-nav-data=<data>
Response Body: Web view for logged-in user

Optional values can be passed through by appending them to the Heroku API SSO URL. Users can be automatically logged in to and add-on dashboard via SSO by directing them to

https://api.heroku.com/myapps/<heroku_app_id>/addons/<addon-name>?other=vals

See manifest formats for more information about configuring single sign-on via GET or POST.

List Apps

This endpoint returns a list of apps that have installed your add-on.

Request: GET https://username:password@api.heroku.com/vendor/apps
Response Body: [
  { "provider_id": "1", 
    "heroku_id": "app123@heroku.com", 
    "callback_url": "https://api.heroku.com/vendor/apps/app123%40heroku.com", 
    "plan": "test" 
  },
  { "provider_id": "3", 
    "heroku_id": "app456@heroku.com", 
    "callback_url": "https://api.heroku.com/vendor/apps/app456%40heroku.com", 
    "plan": "premium" 
  }
]

Pagination

Results larger than 4000 will be paginated. Pagination information is passed via the Link HTTP header. Use those URIs to navigate the pagination rather than constructing them in your client code.

Example:

Link: <https://api.heroku.com/vendor/apps?offset=100>; rel="prev",
      <https://api.heroku.com/vendor/apps?offset=1000>; rel="next"

The possible rel values within that header are:

next: Shows the URL of the immediate next page of results.
prev: Shows the URL of the immediate previous page of results.

App Info

This endpoint returns app information for an app which has installed the Add-on. Information returned is mutable and should be updated when the most recent data is required.

Request: GET https://username:password@api.heroku.com/vendor/apps/:heroku_id
Response Body: { 
  "id": "app123@heroku.com",
  "name": "myapp", 
  "config": {"MYADDON_URL": "http://myaddon.com/52e82f5d73"}, 
  "callback_url": "https://api.heroku.com/vendor/apps/app123%40heroku.com", 
  "owner_email": "user@email.com",
  "region": "amazon-web-services::us-east-1",
  "logplex_token": "t.abc123",
  "domains": ["www.the-consumer.com", "the-consumer.com"]
}

Update config Vars

This endpoint allows the Add-on provider to update the config vars defined in the Add-on provider's manifest.

Request: PUT https://username:password@api.heroku.com/vendor/apps/:heroku_id 
Request Body : { "config": {"MYADDON_URL": "http://myaddon.com/ABC123"}}
Response: 200 OK

Postmark

2013-05-13T19:09:13Z

Postmark add-on removes the headaches of delivering and parsing transactional email for your web app, with minimal setup time and zero maintenance. We have years of experience getting email to the inbox, so you can focus on what you do best.

Use our Send API or our simple SMTP interface to start sending in minutes. Our Inbound API parses incoming email and forwards it to you via a web hook, making it easy to handle incoming email in your application.

Provisioning the add-on

Postmark can be attached to a Heroku application via the CLI:

A list of all plans available can be found on the Postmark add-on page.

$ heroku addons:add postmark:10k
-----> Adding postmark to sharp-mountain-4005... done, v18 (free)

Once Postmark has been added a POSTMARK_API_KEY setting will be available in the app configuration and will contain the api key to access Postmark API or SMTP interface. This can be confirmed using the heroku config:get command.

$ heroku config:get POSTMARK_API_KEY
22874e6c-32f8-4e30-a0ee-45e1ed144a6b

After installing the Postmark add-on you also need to create at least one sender signature and configure your application to fully integrate with the add-on.

Creating sender signatures

Sender signatures are required in order to verify that, well, you really own the mailbox, and that you are not a spammer (yes, we hate spam too). You must have a sender signature for each from address used in your application.

To create a sender signature, open the Postmark Dashboard from the terminal:

$ heroku addons:open postmark
Opening postmark for sharp-mountain-4005…

You can also proceed to the Postmark Dashboard from your application page on the Heroku web site.

Setting up a local development environment

Postmark automatically sets environment variables on your Heroku servers, but sometimes you may need to recreate the environment locally. Foreman aims to solve this for you. You can save the environment variables in your .env file and let Foreman do the rest.

  $ bash -c 'echo "POSTMARK_API_KEY=`heroku config:get POSTMARK_API_KEY`" >> .env'
  $ bash -c 'echo "POSTMARK_SMTP_SERVER=`heroku config:get POSTMARK_SMTP_SERVER`" >> .env'
  $ bash -c 'echo "POSTMARK_INBOUND_ADDRESS=`heroku config:get POSTMARK_INBOUND_ADDRESS`" >> .env'

Sending emails via the Postmark SMTP interface

You want to try Postmark, but your application uses SMTP to send email? You can instantly switch your current email delivery without having to modify the application. This feature lets you use plain SMTP to send your messages and, to use it, you need to only change your configuration to point to the Postmark SMTP server.

The SMTP listener endpoint is available at smtp.postmarkapp.com on port 25 and 2525 (to get around firewall issues). We support both plain text authentication and CRAM-MD5. We recommend the latter as it is a lot more secure. CRAM-MD5 encrypts just the authentication process, but the message content is still sent as plain text. You can encrypt the entire connection using TLS via the standard STARTTLS SMTP extension. STARTTLS is supported by the vast majority of mailers and some of them, like Ruby’s ActionMailer automatically switch to encrypted communication when they detect that the server supports it.

Here is a sample configuration for Ruby Mail library:

Mail.defaults do
  delivery_method :smtp, {
    :address => ENV['POSTMARK_SMTP_SERVER'],
    :port => '25',
    :domain => 'yourapp.heroku.com',
    :user_name => ENV['POSTMARK_API_KEY'],
    :password => ENV['POSTMARK_API_KEY'],
    :authentication => :plain,
    :enable_starttls_auto => true
  }
end

And Rails/ActionMailer:

ActionMailer::Base.smtp_settings = {
  :port           => '25',
  :address        => ENV['POSTMARK_SMTP_SERVER'],
  :user_name      => ENV['POSTMARK_API_KEY'],
  :password       => ENV['POSTMARK_API_KEY'],
  :domain         => 'yourapp.heroku.com',
  :authentication => :plain,
}
ActionMailer::Base.delivery_method = :smtp

Sending emails in Ruby

We support the official Ruby gem called postmark. It’s integrated with the Ruby Mail library and requires it to be installed. You can install both manually using the following command:

$ gem install mail postmark

If your application is using Bundler, add the following entries to your Gemfile and update application dependencies with bundle install:

gem 'mail'
gem 'postmark'

The following example will use the gem to send a text email with attachments:

message = Mail.new do
  from            'leonard@bigbangtheory.com'
  to              'Dr. Sheldon Cooper <sheldon@bigbangtheory.com>'
  subject         'Have you seen these pictures of yours?'
  body            'You look like a geek!'
  add_file        '1.jpeg'

  delivery_method Mail::Postmark, :api_key => ENV['POSTMARK_API_KEY']
end

message.attachments['sheldon.jpeg'] = File.read('2.jpeg')

message.deliver
# => #<Mail::Message:70185826686240, Multipart: true, Headers: <From: leonard@bigbangtheory.com>, <To: sheldon@bigbangtheory.com>, <Message-ID: ba644cc1-b5b1-4bcb-aaf8-2f290b5aad80>, <Subject: Have you seen these pictures of yours?>, <Content-Type: multipart/mixed; boundary=--==_mimepart_5121f9f1ec653_12c53fd569035ad817726>>

Sending emails in Ruby on Rails 3.x

Our postmark-rails Ruby gem was designed to make switching to the Postmark API as easy as possible. Ruby on Rails applications will need to add the following entry into their Gemfile specifying the Postmark client library.

gem 'postmark-rails'

Update application dependencies with bundler.

$ bundle install

Add this to your config/application.rb:

config.action_mailer.delivery_method   = :postmark
config.action_mailer.postmark_settings = { :api_key => ENV['POSTMARK_API_KEY'] }

That’s it. Now send emails as usual.

class TestMailer < ActionMailer::Base

  def tagged_message
    mail(
      :subject => 'Did you know Postmark has a Heroku add-on?',
      :to      => 'sheldon@bigbangtheory.com',
      :from    => 'leonard@bigbangtheory.com',
      :tag     => 'my-tag'
    )
  end

end

Sending emails in Ruby on Rails 2.x

Rails 2.x applications will need to add the following to config/environment.rb:

Rails::Initializer.run do |config|

  # ...

  config.gem 'postmark-rails'
  require    'postmark-rails'

  config.action_mailer.postmark_api_key = ENV['POSTMARK_API_KEY']
  config.action_mailer.delivery_method  = :postmark

  # ...

end

Usage example:

class SuperMailer < ActionMailer::Base

  def email
    from       "leonard@bigbangtheory.com"
    subject    "Hello from Postmark"
    recipients "sheldon@bigbangtheory.com"
    tag        "big-bang"
  end

end

See the postmark-rails gem documentation for more examples.

Sending emails in Python

Python users will need to install the python-postmark library. It’s possible to install the library from PyPy:

$ pip install python-postmark

The following example will use the python-postmark library to send a tagged email.

import os
from postmark import PMMail

message = PMMail(api_key = os.environ.get('POSTMARK_API_KEY'),
                 subject = "Hello from Postmark",
                 sender = "leonard@bigbangtheory.com",
                 to = "sheldon@bigbangtheory.com",
                 text_body = "Hello",
                 tag = "hello")

message.send()

Check out the PMMail class’ documentation for more information on usage.

Sending emails in NodeJS

NodeJS users can use the postmark package available to install using NPM:

$ npm install postmark

The following example will use the postmark package for NodeJS to send a tagged text email:

var postmark = require("postmark")(process.env.POSTMARK_API_KEY)

postmark.send({
    "From": "leonard@bigbangtheory.com",
    "To": "sheldon@bigbangtheory.com",
    "Subject": "Hello from Postmark",
    "TextBody": "Hello!",
    "Tag": "big-bang"
}, function(error, success) {
    if(error) {
        console.error("Unable to send via postmark: " + error.message);
       return;
    }
    console.info("Sent to postmark for delivery")
});

Sending emails in Clojure

Clojure users can use clojure-postmark to interact with the API. To get started add the following entry to your project dependencies (we assume you’re using Leiningen):

[postmark "1.1.0"]

The following example will use the clojure-postmark library to send a tagged text email:

(use '[postmark.core :only (postmark)])

(def api-key (get (System/getenv) "POSTMARK_API_KEY"))

(def send-message (postmark api-key "leonard@bigbangtheory.com"))

(send-message {:to "sheldon@bigbangtheory.com"
               :subject "Hello from Postmark"
               :text "Hello!"
               :tag "big-bang"})

Using Postmark Inbound

Postmark makes email parsing easy, too. We provide you with a unique inbound email address, which can be used for receiving emails. Postmark then parses the incoming emails and sends you the data in a nicely formatted JSON document via web hook. To see it, run:

$ heroku config:get POSTMARK_INBOUND_ADDRESS
3e5317xxxx48ce3bxxxx1a7be74f665f@inbound.postmarkapp.com

Depending on your use, you may not want your users to see the @inbound.postmarkapp.com email. There are two options: MX Record Forwarding (beta) and custom forwarding with Gmail/Google Apps.

Our MX Record Support is in beta until we’ve released a web-based configuration UI. It’s currently API-only, and you can read the documentation on configuring MX Records.

If you’d like to wait until the feature is included in Postmark’s UI, or if you do not have access to edit your DNS records, you can read our help article to learn how to configure a custom forwarding email address from your Gmail/Google Apps account.

In order for your application receive the emails that we parse, you will need to tell Postmark where to send the JSON data for each inbound email it processes, which is done via an HTTP POST to a URL of your choice. You can set this URL in the Settings page for your Postmark server on the Postmark Dashboard.

To test your inbound server, we recommend using RequestBin first.

Here is a simple Ruby/Sinatra application that does basic inbound processing:

require 'sinatra'
require 'json'
require 'logger'

logger = Logger.new(STDOUT)

class Comment
  attr_accessor :attributes

  def self.create_from_inbound_hook(message)
    self.new(:text => message["TextBody"],
             :user_email => message["From"],
             :discussion_id => message["MailboxHash"])
  end

  def initialize(attributes={})
    @attributes = attributes
  end
end

post '/inbound' do
  request.body.rewind
  comment = Comment.create_from_inbound_hook(JSON.parse(request.body.read))
  logger.info comment.inspect
end

Let’s assume the inbound hook is set to http://myapp.heroku.com/inbound. Then, if a user sends an email to 3e5317xxxx48ce3bxxxx1a7be74f665f+discussion7864@inbound.postmarkapp.com (the part after plus sign is called “mailbox hash” and can be used to transfer additional information), it will be processed by Postmark and forwarded to the app’s inbound hook as a JSON object. The example app creates a new instance of Comment object using JSON object provided by Postmark.

:::ruby    
I, [2013-02-28T05:18:56.407910 #2]  INFO -- : #<Comment:0x00000002979260 @attributes={:text=>"That's what you said about the Green Lantern movie. You were 114 minutes of wrong. ", :user_email=>"sheldon@bigbangtheory.com", :discussion_id=>"discussion7864"}>

Migrating between plans

Application owners should carefully manage the migration timing to ensure proper application function during the migration process.

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade postmark:gold
-----> Upgrading postmark:newplan to sharp-mountain-4005... done, v18 ($49/mo)
       Your plan has been updated to: postmark:gold

Removing the add-on

Postmark can be removed via the CLI.

This will destroy all associated data and cannot be undone!

$ heroku addons:remove postmark
-----> Removing postmark from sharp-mountain-4005... done, v20 (free)

Support

If you need help or have questions, you can contact Postmark at any time at support@postmarkapp.com or @postmarkapp on Twitter. For system status, visit http://status.postmarkapp.com.

Direct to S3 File Uploads in Python

2013-05-07T13:44:25Z

Web applications often require the ability to allow users to upload files such as images, movies and archives. Amazon S3 is a popular and reliable storage option for these files.

This article demonstrates how to create a Python application that uploads files directly to S3 instead of via a web application, utilising S3's Cross-Origin Resource Sharing (CORS) support.

A complete example of the code discussed in this article is available for direct use in this Github repository.

Uploading directly to S3

The main advantage of direct uploading is that the load on your application's dynos would be considerably reduced. Using server-side processes for receiving files and transferring to S3 can needlessly tie up your dynos and will mean that they will not be able to respond to simultaneous web requests as efficiently.

If your application relies on some form of file processing between the client's computer and S3 (such as parsing Exif information or applying watermarks to images), then you may need to employ the use of extra dynos and consider the boto Python library for handling the S3 upload.

The application uses client-side JavaScript and Python for signing the requests. It will therefore be a suitable guide for developing applications for the Flask, Bottle and Django web frameworks. The upload is carried out asynchronously so that you can decide how to handle your application's flow after the upload has completed (for example, a page redirect upon successful upload rather than a full page refresh).

An example simple account-editing scenario is used as a guide for completing the various steps required to accomplish the direct upload and to relate the application of this to a wider range of use-cases. More information on this scenario is provided later.

Overview

S3 is comprised of a set of buckets, each with a globally unique name, in which individual files (known as objects) and directories, can be stored.

For uploading files to S3, you will need an Access Key ID and a Secret Access Key, which act as a username and password. The access key account will need to have sufficient access privileges to the target bucket in order for the upload to be successful.

Please see the S3 Article for more information on this, creating buckets and finding your Access Key ID and Secret Access Key.

The method described in this article involves the use of client-side JavaScript and server-side Python. In general, the completed image-upload process follows these steps:

A file is selected for upload by the user in their web browser;
JavaScript is then responsible for making a request to your web application on Heroku, which produces a temporary signature with which to sign the upload request;
The temporary signed request is returned to the browser in JSON format;
JavaScript then uploads the file directly to Amazon S3 using the signed request supplied by your Python application.

This guide includes information on how to implement the client-side and server-side code to form the complete system. After following the guide, you should have a working barebones system, allowing your users to upload files to S3. However, it is usually worth adding extra functionality to help improve the security of the system and to tailor it for your own particular uses. Pointers for this are mentioned in the appropriate parts of the guide.

Prerequisites

The Heroku Toolbelt has been installed;
A Heroku application has been created for the current project;
An AWS S3 bucket has been created.

Initial setup

Heroku setup

In order for your application to access the AWS credentials for signing upload requests, they will need to be added as configuration variables in Heroku:

If you are testing locally before deployment, remember to add the credentials to your local machine's environment, too.

$ heroku config:set AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy
Adding config vars and restarting app... done, v21
    AWS_ACCESS_KEY_ID     => xxx
    AWS_SECRET_ACCESS_KEY => yyy

In addition to the AWS access credentials, set your target S3 bucket's name:

$ heroku config:add S3_BUCKET = zzz
Adding config vars and restarting app... done, v21
    S3_BUCKET     => zzz

Using config vars is preferable over configuration files for security reasons. Try to avoid placing passwords and access keys directly in your application's code or in configuration files.

S3 setup

You will now need to edit some of the permissions properties of the target S3 bucket so that the final request has sufficient privileges to write to the bucket. In a web-browser, sign in to the AWS console and select the S3 section. Select the appropriate bucket and click the ‘Properties’ tab. Select the Permissions section and three options are provided (Add more permissions, Edit bucket policy and Edit CORS configuration).

CORS (Cross-Origin Resource Sharing) will allow your application to access content in the S3 bucket. Each rule should specify a set of domains from which access to the bucket is granted and also the methods and headers permitted from those domains.

For this to work in your application, click 'Add CORS Configuration' and enter the following XML:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <CORSRule>
        <AllowedOrigin>yourdomain.com</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Click 'Save' in the CORS window and then 'Save' again in the bucket's 'Properties' tab.

This tells S3 to allow any domain access to the bucket and that requests can contain any headers. For security, you can change the 'AllowedOrigin' to only accept requests from your domain.

If you wish to use S3 credentials specifically for this application, then more keys can be generated in the AWS account pages. This provides further security, since you can designate a very specific set of requests that this set of keys are able to perform. If this is preferable to you, then you will need to also set up an IAM user in the Edit bucket policy option in your S3 bucket. There are various guides on AWS's web pages detailing how this can be accomplished.

Direct uploading

The processes and steps required to accomplish a direct upload to S3 will be demonstrated through the use of a simple profile-editing scenario for the purposes of this article. This example will involve the user being permitted to select an avatar image to upload and enter some basic information to be stored as part of their account.

In this scenario, the following procedure will take place:

The user is presented with a web page, containing elements encouraging the user to choose an image to upload as their avatar and to enter a username and their own name.
An element is responsible for maintaining a preview of the chosen image by the user. By default, and if no image is chosen for upload, a default avatar image is used instead (making the image-upload effectively optional to the user in this scenario).
When a user selects an image to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described earlier in this article. The image preview is then updated with the selected image once the upload is complete and successful.
The user is then free to move on to filling in the rest of the information.
The user then clicks the “submit” button, which posts the username, name and the URL of the uploaded image to the Python application to be checked and/or stored. If no image was uploaded by the user earlier the default avatar image URL is posted instead.

Setting up the client-side code

This setup does not require any additional, non-standard Python libraries, but some scripts are necessary to complete the implementation on the client-side.

This article covers the use of the s3upload.js script. Obtain this script from the project's repo (using Git or otherwise) and store it somewhere appropriate in your application's static directory. This script currently depends on both the JQuery and Lo-Dash libraries. Inclusion of these in your application will be covered later on in this guide.

The HTML and JavaScript can now be created to handle the file selection, obtain the request and signature from your Python application, and then finally make the upload request.

Firstly, create a file called account.html in your application's templates directory and populate the head and other necessary HTML tags appropriately for your application. In the body of this HTML file, include a file input and an element that will contain status updates on the upload progress. In addition to this, create a form to allow the user to enter their username and full name and a hidden input element to hold the URL of the chosen avatar image:

To see the completed HTML file, please see the appropriate code in the companion repository.

<input type="file" id="file" onchange="s3_upload();"/>
<p id="progress">Please select a file```
<div id="preview"><img src="/static/default.png"  /></div>

<form method="POST" action="/submit_form/">
    <input type="hidden" id="avatar_url" name="avatar_url" value="/static/default.png" />
    <input type="text" name="username" placeholder="Username" /><br />
    <input type="text" name="full_name" placeholder="Full name" /><br /><br />
    <input type="submit" value="Update profile" />
</form>

The preview element initially holds a default avatar image (which would become the user’s avatar if a new image is not chosen), and the avatar_url input maintains the current URL of the user’s chosen avatar image. Both of these are updated by the JavaScript, discussed below, when the user selects a new avatar.

Thus when the user finally clicks the submit button, the URL of the avatar is submitted, along with the username and full name of the user, to your desired endpoint for server-side handling. The JavaScript method, s3_upload(), is called when a file is selected by the user. The creation and population of this method is covered below.

Next, include the three dependency scripts in your HTML file, account.html. You may need to adjust the src attribute for the file s3upload.js if you put this file in a directory other than /static:

<script type="text/javascript" src="http://code.jquery.com/jquery-1.9.1.js"></script>
<script type="text/javascript" src="https://raw.github.com/bestiejs/lodash/v1.1.1/dist/lodash.min.js"></script>
<script type="text/javascript" src="/static/s3upload.js"></script>

The ordering of the scripts is important as the dependencies need to be satisfied in this sequence.

Finally, in a <script> block, declare a JavaScript function, s3_upload(), in the same file again to process the file upload. This <script> block will need to exist below the inclusion of the three dependencies:

function s3_upload(){
    var s3upload = new S3Upload({
        file_dom_selector: '#file',
        s3_sign_put_url: '/sign_s3_upload/',

        onProgress: function(percent, message) { 
            $('#status').html('Upload progress: ' + percent + '%' + message);
        },
        onFinishS3Put: function(url) { 
            $('#status').html('Upload completed. Uploaded to: '+ url);
            $("#avatar_url").val(public_url);
            $("#preview").html('<img src="'+public_url+'" style="width:300px;" />');
        },
        onError: function(status) {
            $('#status').html('Upload error: ' + status);
        }
    });
}

This function creates a new instance of S3Upload, to which is passed the file input element, the URL from which to retrieve the signed request and three functions.

Initially, the function makes a request to the URL denoted by the s3_sign_put_url argument, passing the file name and mime type as GET parameters. The server-side code (covered in the next section) interprets the request and responds with a preview of the URL of the file to be uploaded to S3 and the signed request, which this function then uses to asynchronously upload the file to your bucket.

The function will post upload updates to the onProgress() function and , if the upload is successful, onFinishS3Put() is called and the URL returned by the Python application view is received as an argument. If, for any reason, the upload should fail, onError() will be called and the status parameter will describe the error.

If you find that the page isn't working as you intend after implementing the system, then consider using console.log() to record any errors that occur inside the onError() callback and use your browser's error console to help diagnose the problem.

If successful, the preview div will now be updated with the user's chosen avatar image, and the hidden input field will contain the URL for the image. Now, once the user has completed the rest of the form and clicked submit, all three pieces of information can be posted to the same endpoint.

It is good practice to inform the user of any prolonged activity in any form of application (web- or device-based) and to display updates on changes. Thus the status methods could be used, for example, to show a loading GIF to indicate that an upload is in progress, which can then be hidden when the upload has finished. Without this sort of information, users may suspect that the page has crashed, and could try to refresh the page or otherwise disrupt the upload process.

Setting up the server-side Python code

This section discusses the use of Python for generating a temporary signature with which the upload request can be signed. This temporary signature uses the account details (the AWS access key and secret access key) as a basis for the signature, but users will not have direct access to this information. After the signature has expired, then upload requests with the same signature will not be successful.

As mentioned previously, this article covers the production of an application for the Flask framework, although the steps for other Python frameworks will be similar.

To see the completed Python file, please see the appropriate code in the companion repository.

Start by creating your main application file, application.py, and set up your skeleton application appropriately:

from flask import Flask, render_template, request
import time, os, json, base64, hmac, sha

app = Flask(__name__)

if __name__ == '__main__':
    port = int(os.environ.get('PORT', 5000))
    app.run(host='0.0.0.0', port=port)

The currently-unused import statements will be necessary later on.

Next, in the same file, you will need to create the views responsible for returning the correct information back to the user's browser when requests are made to various URLs. First define view for requests to /account to return the page account.html, which contains the form for the user to complete:

@app.route("/account/")
def account():
    return render_template('account.html')

Please note that the views for the application will need to be placed between the app = Flask(__name__) and if __name__ == '__main__': lines in application.py.

Now create the view, in the same Python file, that is responsible for generating and returning the signature with which the client-side JavaScript can upload the image. This is the first request made by the client before attempting an upload to S3. This view responds with requests to /sign_s3/:

@app.route('/sign_s3/')
def sign_s3():
    AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY_ID')       
    AWS_SECRET_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    S3_BUCKET = os.environ.get('S3_BUCKET')

    object_name = request.args.get('s3_object_name')
    mime_type = request.args.get('s3_object_type')

    expires = int(time.time()+10)
    amz_headers = "x-amz-acl:public-read"

    put_request = "PUT\n\n%s\n%d\n%s\n/%s/%s" % (mime_type, expires, amz_headers, S3_BUCKET, object_name)

    signature = base64.encodestring(hmac.new(AWS_SECRET_KEY, put_request, sha).digest())

    url = 'https://%s.s3.amazonaws.com/%s' % (S3_BUCKET, object_name)

    return json.dumps({
        'signed_request': '%s?AWSAccessKeyId=%s&Expires=%d&Signature=%s' % (url, AWS_ACCESS_KEY, expires, signature),
         'url': url
      })

This code performs the following steps:

The request is received to /sign_s3/ and the AWS keys and S3 bucket name are loaded from the environment.
The name and mime type of the object to be uploaded are extracted from the GET parameters of the request (this stage may differ in other frameworks). The parameters are provided by the JavaScript discussed in the previous section.
The expiry time of the signature is set and forms the basis of the temporary nature of the signature. As shown, this is best used as a function relative to the current UNIX time. In this example, the signature will expire 10 seconds after Python has executed that line of code.
The headers line tells S3 what access permissions to grant. In this case, the object will be publicly available for download.
Now the PUT request can be constructed from the object information, headers and expiry time.
The signature is generated as an SHA hash of the compiled AWS secret key and the actual PUT request.
The prospective URL of the object to be uploaded is produced as a combination of the S3 bucket name and the object name.
Finally, the signed request can be returned, along with the prospective URL, to the browser in JSON format.

You may wish to assign another, customised name to the object instead of using the one that the file is already named with, which is useful for preventing accidental overwrites in the S3 bucket. This name could be related to the ID of the user’s account, for example. If not, you should provide some method for properly quoting the name in case there are spaces or other awkward characters present. In addition, this is the stage at which you could provide checks on the uploaded file in order to restrict access to certain file types. For example, a simple check could be implemented to allow only .png files to proceed beyond this point.

It is sometimes possible for S3 to respond with 403 (forbidden) errors for requests which aren't formatted correctly. If this happens then it may be necessary to quote the signature and remove any whitespace. The urllib module and string strip() methods can be used to achieve this.

Finally, in application.py, create the view responsible for receiving the account information after the user has uploaded an avatar, filled in the form, and clicked submit. Since this will be a POST request, this will also need to be defined as an ‘allowed access method’. This method will respond to requests to the URL /submit_form/:

@app.route("/submit_form/", methods=["POST"])
def submit_form():
    username = request.form["username"]
    full_name = request.form["full_name"]
    avatar_url = request.form["avatar_url"]
    update_account(username, full_name, avatar_url)
    return redirect(url_for('profile'))

In this example, an update_account() function has been called, but creation of this method is not covered in this article. In your application, you should provide some functionality, at this stage, to allow the app to store these account details in some form of database and correctly associate the information with the rest of the user’s account details.

In addition, the URL for the profile page has not been defined in this article (or companion code). Ideally, for example, after updating the account, the user would be redirected back to their own profile so that they can see the updated information.

Running the app

Everything should now be in place to perform the direct uploads to S3. To test the upload, save any changes and use foreman to start the application:

You will need a Procfile for this to be successful. See Getting Started with Python on Heroku for information on the Heroku toolbelt and using Foreman. Also remember to correctly set your environment variables on your own machine before running the application locally.

$ foreman start
15:44:36 web.1  | started with pid 12417

Press Ctl-C to return to the prompt. If your application is returning 500 errors (or other server-based issues), then start your server in debug mode and view the output in the Terminal emulator to help fix your problem. For example, in Flask:

...
app.debug = True
port = int(os.environ.get('PORT', 5000))
app.run(host='0.0.0.0', port=port)

If you are receiving 403 errors back from S3, then the most common reason is that there is an issue with your signature. As mentioned earlier, you should consider quoting the signature properly and removing any whitespace.

Summary

This article covers uploading to Amazon S3 directly from the browser using Python to temporarily sign the upload request. Although the guide and companion code focuses on the Flask framework, the idea should easily carry over to other Python applications.

Penetration Testing and Network Scanning

2013-05-07T00:39:44Z

Coordinated penetration tests and network security scans are allowed on Heroku.

Planning

Since overaggressive scans are hard to distinguish from Denial of Service attacks, please notify Heroku in advance of all automated security scans and provide the following information:

Approximate date and time window you’ll be conducting the test (e.g., “between 8am-noon US/Pacific time, 18 September 2012”). Multiple windows are fine.
Source IP address range you’ll be using.
Which specific applications, hostnames, and URLs you’ll be testing.
Contact information, including a phone number, for the individual or team conducting the test.

Scanning

We also ask the following of your penetration tests:

Provide updates at the start and end of each test. It’s OK if it slips a couple of hours.
Rate limit HTTP requests to no more than 250 requests per second, summing together across all tools and source IPs. If you need to go above that you may need to be assigned a specific testing time window.
If you find any vulnerabilities in our platform or any add-on services, please treat them as confidential and notify us at security@heroku.com at once. If it’s particularly serious, the Heroku Security Team’s PGP key is here.

Heroku Postgres Metrics Logs

2013-05-06T14:57:04Z

Overview

Heroku Postgres Production Tier database users will see database related events on their app's stream. This can be useful for recording and analyzing usage over time.

Log format

2013-05-07T17:41:06+00:00 source=HEROKU_POSTGRESQL_VIOLET measure.current_transaction=1873 measure.db_size=26219348792bytes measure.tables=13 measure.active-connections=92 measure.waiting-connections=1 measure.index-cache-hit-rate=0.99723 measure.table-cache-hit-rate=0.99118

The attributes found on the log are:

measure.db_size: The number of bytes contained in the database. This includes all table and index data on disk, including database bloat.
measure.tables: The number of tables in the database.
measure.active-connections: The number of connections established on the database.
measure.current_transaction: The current transaction ID, which can be used to track writes over time.
measure.index-cache-hit-rate: Rate of index lookups served from shared buffer cache, rounded to five decimal points.
measure.table-cache-hit-rate: Rate of table lookups served from shared buffer cache, rounded to five decimal points.
measure.waiting-connections: Number of connections waiting on a lock to be acquired.
source: The database name the measurements relate to.
The log line's timestamp is the time at which the measurements were taken.

How Heroku Works

2013-05-01T20:59:37Z

This is a high-level, technical description of how Heroku works. It ties together many of the concepts you'll encounter while writing, configuring, deploying and running applications on the Heroku platform.

Performing one of the Getting Started tutorials will make the concepts in this documentation more concrete.

Read this document sequentially: in order to tell a coherent story, it incrementally unveils and refines the concepts describing the platform.

The final section ties all the definitions together, providing a deploy-time and runtime-view of Heroku.

Defining an application

Heroku lets you deploy, run and manage applications written in Ruby, Node.js, Java, Python, Clojure and Scala.

An application is a collection of source code written in one of these languages, perhaps a framework, and some dependency description that instructs a build system as to which additional dependencies are needed in order to build and run the application.

Terminology (Preliminary): Applications consist of your source code and a description of any dependencies.

Dependency mechanisms vary across languages: in Ruby you use a Gemfile, in Java a pom.xml, in Python a requirements.txt and so on.

The source code for your application, together with the dependency file, should provide enough information for the Heroku platform to build your application, to produce something that can be executed.

Knowing what to execute

You don't need to make many changes to an application in order to run it on Heroku. One requirement is informing the platform as to which parts of your application are runnable.

If you're using some established framework, Heroku can figure it out. For example, in Ruby on Rails, it's typically rails server, in Django it's python <app>/manage.py runserver and in Node.js it's node web.js.

Terminology: Procfiles list process types - named commands that you may want executed.

For other applications, you may need to explicitly declare what can be executed. You do this in a text file that accompanies your source code - a Procfile. Each line declares a process type - a named command that can be executed against your built application. For example, your Procfile may look like this:

web: java -jar lib/foobar.jar $PORT
queuty: java -jar lib/queue-processor.jar

This file declares a web process type and provides the command that needs to be executed in order to run it (in this case, java -jar lib/foobar.jar $PORT). It also declares a queuty process type, and its corresponding command.

The earlier definition of an application can now be refined to include this single additional Procfile.

Terminology: Applications consist of your source code, a description of any dependencies, and a Procfile.

Heroku is a polyglot platform -- it lets you build, run and scale applications in a similar manner across all the languages -- utilizing the dependencies and Procfile. The Procfile exposes an architectural aspect of your application (in the above example there are two entry points to the application) and this architecture lets you, for example, scale each part independently. An excellent guide to architecture principals that work well for applications running on Heroku can be found in The Twelve-Factor App.

Deploying applications

Git is a powerful, distributed version control system that many developers use to manage and version source code. The Heroku platform uses git as the primary means for deploying applications.

When you create an application on Heroku, it associates a new git remote, typically named heroku, with the local git repository for your application.

As a result, deploying code is just the familiar git push, but to the heroku remote instead:

$ git push heroku master

Terminology: Deploying applications involves sending the application to Heroku using git.

Deployment then, is about using git as a transport mechanism - moving your application from your local system to Heroku.

Building applications

When the Heroku platform receives a git push, it initiates a build of the source application. The build mechanism is typically language specific, but follows the same pattern, typically retrieving the specified dependencies, and creating any necessary assets (whether as simple as processing style sheets or as complex as compiling code).

Advanced: Buildpacks lie behind the slug compilation process. They're open source - enabling you to extend Heroku to other languages and frameworks. Buildpacks take your application, its dependencies, and the language runtime, and produce slugs.

For example, when the build system receives a Rails application, it may fetch all the dependencies specified in the Gemfile, as well as generate files based on the asset pipeline. A Java application may fetch binary library dependencies using Maven, compile the source code together with those libraries, and produce a JAR file to execute.

The source code for your application, together with the fetched dependencies and output of the build phase such as generated assets or compiled code, as well as the language and framework, are assembled into a slug.

Terminology: A slug is a bundle of your source, fetched dependencies, the language runtime, and compiled/generated output of the build system - ready for execution.

These slugs are a fundamental aspect of what happens during application execution - they contain your compiled, assembled application - ready to run - together with the instructions (the Procfile) of what you may want to execute.

Running applications on dynos

Heroku executes applications by running a command you specified in the Procfile, on a dyno that's been preloaded with your prepared slug (in fact, with your release, which extends your slug and a few items not yet defined: config vars and add-ons).

Think of a running dyno as a lightweight, secure, virtualized Unix container that contains your application slug in its file system.

Terminology: Dynos are isolated, virtualized Unix containers, that provide the environment required to run an application.

Generally, if you deploy an application for the first time, Heroku will run 1 web dyno automatically. In other words, it will boot a dyno, load it with your slug, and execute the command you've associated with the web process type in your Procfile.

You have control over how many dynos are running at any given time. Given the Procfile example earlier, you can start 5 dynos, 3 for the web and 2 for the queuety process types, as follows:

$ heroku ps:scale web=3 queuty=2

When you deploy a new version of an application, all of the currently executing dynos are killed, and new ones (with the new release) are started to replace them - preserving the existing dyno formation.

Terminology: Your application's dyno formation is the total number of currently-executing dynos, divided between the various process types you have scaled.

To understand what's executing, you just need to know what dynos are running which process types:

$ heroku ps
== web: 'java lib/foobar.jar $PORT'
web.1: up 2013/02/07 18:59:17 (~ 13m ago)
web.1: up 2013/02/07 18:52:08 (~ 20m ago)
web.2: up 2013/02/07 18:31:14 (~ 41m ago)

== queuty: `java lib/queue-processor.jar` 
queuty.1: up 2013/02/07 18:40:48 (~ 32m ago)
queuty.2: up 2013/02/07 18:40:48 (~ 32m ago)

Dynos then, are an important means of scaling your application. In this example, the application is well architected to allow for the independent scaling of web and queue worker dynos.

Config vars

An application’s configuration is everything that is likely to vary between deploys (staging, production, developer environments, etc.). This includes resource handles to backing services such as databases, credentials, or environment variables that provide some contextual information to your application.

Heroku lets you run your application with a customizable configuration - the configuration sits outside of your application code and can be changed independently of it.

The configuration for an application is stored in config vars. For example, here's how to configure an encryption key for an application:

$ heroku config:add ENCRYPTION_KEY=my_bad_ass_long_random_key 
Adding config vars and restarting demoapp... done, v14
ENCRYPTION_KEY:     my_bad_ass_long_random_key

Terminology: Config vars contain customizable configuration data that can be changed independently of your source code. The configuration is exposed to a running application via environment variables.

At runtime, all of the config vars are exposed as environment variables - so they can be easily extracted programatically. A Ruby application deployed with the above config var, can access it by calling ENV["ENCRYPTION_KEY"].

All dynos in an application will have access to the exact same set of config vars at runtime.

Releases

Earlier, this article stated that to run your application on a dyno, the Heroku platform loaded the dyno with your most recent slug. This needs to be refined: in fact it loads it with the slug and any config variables you have assigned to the application. The combination of slug and configuration is called a release.

Terminology (Preliminary): Releases are an append-only ledger of slugs and config vars.

All releases are automatically persisted in an append-only ledger, making managing your application, and different releases, a cinch. Use the heroku releases command to see the audit trail of release deploys:

$ heroku releases
== demoapp Releases
v103 Deploy 582fc95  jon@heroku.com   2013/01/31 12:15:35
v102 Deploy 990d916  jon@heroku.com   2013/01/31 12:01:12

The number next to the deploy message, for example 582fc95, corresponds to the commit hash of the repository you deployed to Heroku.

Every time you deploy a new version of an application, a new slug is created and release is generated.

As Heroku contains a store of the previous releases of your application, it's very easy to rollback and deploy a previous release:

$ heroku releases:rollback v102
Rolling back demoapp... done, v102
$ heroku releases
== demoapp Releases
v104 Rollback to v102 jon@heroku.com   2013/01/31 14:11:33 (~15s ago)
v103 Deploy 582fc95   jon@heroku.com   2013/01/31 12:15:35
v102 Deploy 990d916   jon@heroku.com   2013/01/31 12:01:12

Making a material change to your application, whether it's changing the source or configuration, results in a new release being created.

A release then, is the mechanism behind how Heroku lets you modify the configuration of your application (the config vars) independently of the application source (stored in the slug) - the release binds them together. Whenever you change a set of config vars associated with your application, a new release will be generated.

Dyno manager

Part of the Heroku platform, the dyno manager, is responsible for keeping dynos running. For example, dynos are cycled at least once per day, or whenever the dyno manager detects a fault in the running application (such as out of memory exceptions) or problems with the underlying hardware that requires the dyno be moved to a new physical location.

Terminology: The dyno manager of the Heroku platform is responsible for managing dynos across all applications running on Heroku.

This dyno cycling happens transparently and automatically on a regular basis, and is logged.

Terminology: Applications with only a single web dyno are idled after a period of inactivity. When an idled application receives HTTP traffic, it will be unidled - causing a delay of a few seconds. Scaling the web dynos will avoid idling.

Because Heroku manages and runs applications, there's no need to manage operating systems or other internal system configuration. One-off dynos can be run with their input/output attached to your local terminal. These can also be used to carry out admin tasks that modify the state of shared resources - for example, database configuration.

Terminology: One-off Dynos are temporary dynos that can run with their input/output attached to your local terminal. They're loaded with your latest release.

Here's the simplest way to create and attach to a one-off dyno:

$ heroku run bash
Running `bash` attached to terminal... up, run.8963
~ $ ls

This will spin up a new dyno, loaded with your release, and then run the bash command - which will provide you with a unix shell (remember that dynos are effectively isolated virtualized unix containers). Once you've terminated your session, or after a period of inactivty, the dyno will be removed.

Changes to the filesystem on one dyno are not propagated to other dynos and are not persisted across deploys and dyno restarts. A better and more scalable approach is to use a shared resource such as a database or queue.

Terminology: Each dyno gets its own ephemeral filesystem - with a fresh copy of the most recent release. It can be used as temporary scratchpad, but changes to the filesystem are not reflected to other dynos.

The ephemeral nature of the file system in a dyno can be demonstrated with the above command. If you create a one-off dyno by running heroku run bash, the Unix shell on the dyno, and then create a file on that dyno, and then terminate your session - the change is lost. All dynos, even those in the same application, are isolated - and after the session is terminated the dyno will be killed. New dynos are always created from a slug, not from the state of other dynos.

Add-ons

Applications typically make use of add-ons to provide backing services such as databases, queueing & caching systems, storage, email services and more. Add-ons are provided as services by Heroku and third parties - there's a large marketplace of add-ons you can choose from.

Heroku treats these add-ons as attached resources: provisioning an add-on is a matter of choosing one from the add-on marketplace, and attaching it to your application.

For example, here is how to add a Redis backing store add-on (by RedisToGo) to an application:

$ heroku addons:add redistogo:nano

The add-on service provider is responsible for the service - and the interface to your application is often provided through a config var. In this example, a REDISTOGO_URL will be automatically added to your application when you provision the add-on. You can write code that connects to the service through the URL, for example:

uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

Terminology: Add-ons are third party, specialized, value-added cloud services that can be easily attached to an application, extending its functionality.

Add-ons are associated with an application, much like config vars - and so the earlier definition of a release needs to be refined. A release of your applications is not just your slug and config vars; it's your slug, config vars as well as the set of provisioned add-ons.

Terminology: Releases are an append-only ledger of slugs, config vars and add-ons. Heroku maintains an append-only ledger of releases you make.

Much like config vars, whenever you add, remove or change an add-on, a new release is created.

Logging and monitoring

Heroku treats logs as streams of time-ordered events, and collates the stream of logs produced from all of the processes running in all dynos, and the Heroku platform components, into the Logplex - a high-performance, real-time system for log delivery.

It's easy to examine the logs across all the platform components and dynos:

$ heroku logs
2013-02-11T15:19:10+00:00 heroku[router]: at=info method=GET path=/articles/custom-domains host=mydemoapp.heroku.com fwd=74.58.173.188 dyno=web.1 queue=0 wait=0ms connect=0ms service=1452ms status=200 bytes=5783
2013-02-11T15:19:10+00:00 app[web.2]: Started GET "/" for 1.169.38.175 at 2013-02-11 15:19:10 +0000
2013-02-11T15:19:10+00:00 app[web.1]: Started GET "/" for 2.161.132.15 at 2013-02-11 15:20:10 +0000

Here you see 3 timestamped log entries, the first from Heroku's router, the last two from two dynos running the web process type.

Terminology: Logplex automatically collates log entries from all the running dynos of your app, as well as other components such as the routers, providing a single source of activity.

You can also dive into the logs from just a single dyno, and keep the channel open, listening for further events:

$ heroku logs --ps web.1 --tail
2013-02-11T15:19:10+00:00 app[web.2]: Started GET "/" for 1.169.38.175 at 2013-02-11 15:19:10 +0000

Logplex keeps a limited buffer of log entries solely for performance reasons. To persist them, and action events such as email notification on exception, use a Logging Add-on, which ties into log drains - an API for receiving the output from Logplex.

HTTP routing

Depending on your dyno formation, some of your dynos will be running the command associated with the web process type, and some will be running other commands associated with other process types.

The dynos that run process types named web are different in one way from all other dynos - they will receive HTTP traffic. Heroku's HTTP routers distributes incoming requests for your application across your running web dynos.

So scaling an app's capacity to handle web traffic involves scaling the number of web dynos:

$ heroku ps:scale web+5

A random selection algorithm is used for HTTP request load balancing across web dynos - and this routing handles both HTTP and HTTPS traffic. It also supports multiple simultaneous connections, as well as timeout handling.

Tying it all together

The concepts explained here can be divided into two buckets: those that involve the development and deployment of an application, and those that involve the runtime operation of the Heroku platform and the application after its deployed.

The following two sections recapitulate the main components of the platform, separating them into these two buckets.

Deploy

Applications consist of your source code, a description of any dependencies, and a Procfile.
Procfiles list process types - named commands that you may want executed.
Deploying applications involves sending the application to Heroku using git.
Buildpacks lie behind the slug compilation process. Buildpacks take your application, its dependencies, and the language runtime, and produce slugs.
A slug is a bundle of your source, fetched dependencies, the language runtime, and compiled/generated output of the build system - ready for execution.
Config vars contain customizable configuration data that can be changed independently of your source code. The configuration is exposed to a running application via environment variables.
Add-ons are third party, specialized, value-added cloud services that can be easily attached to an application, extending its functionality.
A release is a combination of a slug (your application), config vars and add-ons. Heroku maintains an append-only ledger of releases you make.

Runtime

Dynos are isolated, virtualized unix containers, that provide the environment required to run an application.
Your application's dyno formation is the total number of currently-executing dynos, divided between the various process types you have scaled.
The dyno manager is responsible for managing dynos across all applications running on Heroku.
Applications with only a single web dyno are idled after a period of inactivity by the dyno manager. Scaling to multiple web dynos will avoid this.
One-off Dynos are temporary dynos that run with their input/output attached to your local terminal. They’re loaded with your latest release.
Each dyno gets its own ephemeral filesystem - with a fresh copy of the most recent release. It can be used as temporary scratchpad, but changes to the filesystem are not reflected to other dynos.
Logplex automatically collates log entries from all the running dynos of your app, as well as other components such as the routers, providing a single source of activity.
Scaling scaling an application involves varying the number of dynos of each process type.

Migrating From PostGIS 1.5 to PostGIS 2.0

2013-05-01T01:11:33Z

PostGIS 2.0 is now available in public beta to Heroku Postgres customers on production tier plans (Crane and up). It brings improved stability and features to spatial applications.

Installation

Unlike PostGIS 1.5, it is available as a Postgres extension on Postgres 9.2 databases. To install PostGIS in your database, run the following from a psql console:

CREATE EXTENSION postgis;

PostGIS 1.5 to 2.0 Migration

To migrate an old Heroku Postgres database running PostGIS 1.5 to the new architecture, please follow these steps:

Create a target database

$ heroku addons:add heroku-postgresql:ronin --version=9.2 --app your-app
Adding heroku-postgresql:ronin on hgmnz... done, v29 ($200/mo)
Attached as HEROKU_POSTGRESQL_LAVANDA_URL
The database should be available in 3-5 minutes.
 ! The database will be empty. If upgrading, you can transfer
 ! data from another database with pgbackups:restore.
Use `heroku pg:wait` to track status..
Use `heroku addons:docs heroku-postgresql:ronin` to view documentation.

Create the postgis extension on the new database

$ heroku pg:psql LAVANDA --app your-app
> CREATE EXTENSION postgis;

Transfer your data to a new Postgres database

The fastest way to do this is to use the pgbackups:transfer command available from the heroku pg-extras plugin.

Install the plugin via

$ heroku plugins:install git://github.com/heroku/heroku-pg-extras.git

This command requires the pgbackups addon. If you don't already have it installed on your app, install it now:

$ heroku addons:add pgbackups --app your-app
Adding pgbackups on hgmnz... done, v32 (free)
You can now use "pgbackups" to backup your databases or import an external backup.
Use `heroku addons:docs pgbackups` to view documentation.

Place your application in maintenance mode, and scale down any workers:

$ heroku maintenance:on --app your-app
$ heroku ps:scale worker=0 another_worker=0 --app your-app

Finally, transfer data from your old database to your new one:

$ heroku pgbackups:transfer <OLD_DATABASE_COLOR> LAVANDA --app your-app

Depending on your data size, the transfer may take a while. You can track progress of the transfer by tailing your application logs from another shell window:

$ heroku logs --tail -p pbackups --app your-app

Verify data and promote your new database

You can now gain a connection to your new database and run queries verifying that your data looks as expected. You can also point a staging application to it to verify that your it works well with Postgis 2.0.

After you're ready to use the Postgis 2.0 database as the primary for your application, simply promote it:

$ heroku pg:promote HEROKU_POSTGRESQL_LAVANDA --app your-app
Promoting HEROKU_POSTGRESQL_LAVANDA_URL to DATABASE_URL... done

Now that the new database is up and running, scale it back up:

$ heroku maintenance:off --app your-app
$ heroku ps:scale worker=4 another_worker=2 --app your-app

Remove old database

Once your new database has been migrated and promoted, you can safely remove your old one:

$ heroku addons:remove HEROKU_POSTGRESQL_<OLD_DATABASE_COLOR> --app your-app