Installing rails_xss in Rails 2 is a very good method to see how the XSS protection is going to change things after the upgrade to Rails 3. Depending what kind of content management system solutions, asset packaging tools or helper methods you are using, you can have problems in a plethora of different places. As a developer solving these issues in UserVoice, I would like to share my process for the XSS protected strings handling.

Typically problems that arise look like this. To solve them all, first step is to identify the different types of problems. Reserve some time for this, as there might be tens of different problem patterns, which all might not be easy to find.

The Process

Here I explain how I approached the problem and present some lessons learned during the process. This is not a definitive guide on the subject, but a rather a set of principles which I recommend to be applied during and after the rails_xss installation.

1. Install rails_xss

Install rails_xss and its dependency (erubis). After this, your Rails 2 application escapes all your “unsafe” strings by default. Explore your application and run tests to see on a general level how serious problems the default HTML escaping causes.

2. Identify the Problems

You should identify all the different types of problems and address each problem with a solution. While testing and reviewing the codebase of UserVoice, I identified many different problem patterns. I have described some of the most common ones:

2.1 Custom Helpers

For example, the return value of these kind of helpers can be marked html_safe by calling .html_safe. In this case it’s even better to use the Rails helper link_to, whose return value is html_safe by default. Additionally, you don’t have to worry about escaping the name of the user separately using the h method.

  def user_link(user)
    "<a href=#{user.url}>#{h(user.name)}</a>".html_safe
  end

  # better:
  def user_link(user)
    link_to(user.name, user.url)
  end

2.2 Printing Conditional Strings in Views

Often you want to print a certain string depending on some condition. In these cases the resulting string must be marked html_safe. Possibly you also want to extract a commonly occurring pattern in your codebase to its own helper.

  <span <%= 'style="display: none"'.html_safe unless content.visible? %> >
    Content shown if content.visible?
  </span>

2.3 Concatenating Strings in Views

Your view files might include concatenation of unsafe and safe strings like this.

  <%= '<hr />'.html_safe + link_to('User Information', user_path(user)) %>

By default, unsafe + safe results in an unsafe string, which means that the link above gets unexpectedly escaped. To prevent this, ‘<hr />’ can be marked html_safe or even better, just extract it out side the ERB interpolation tag.

2.4 Printing Custom User-Defined HTML

  <%= user.custom_html.html_safe %>

In these cases you need to mark the user-defined HTML html_safe. At the same time, you should also ensure that the user input is sanitized, because you do not want to allow printing arbitrary HTML in your views. A good sanitizing includes defining the set of allowed tags and denying any others. Using a good and well-tested library for sanitizing HTML is also a good idea, unless you really know what you are doing, which you most probably are not.

3. Design a Strategy

When identifying all the various cases of printing HTML, JSON and HTML entities, you might start to wonder how to cover all the cases in all your application’s views. The short is answer is that 100% coverage requires probably too much effort. However, there are many ways how you can come close to that. OWASP states about XSS vulnerability finding: “complete coverage requires a combination of manual code review and manual penetration testing, in addition to any automated approaches in use”.

3.1 Custom Automated Tool

Now that rails_xss had escaped all the output strings by default, I needed to find the places where HTML has been doubly-escaped. So I thought it wouldn’t hurt to write a utility method for matching the doubly-escaped HTML strings:

class HtmlDoubleEscapeReporter
  def self.assert_sane(str)
    if (str.match(/<[a-z]/) || str.match(/&(quot|rarr|larr|amp|#)/)) && 
        !str.match(/looks something you do not want to print/
      send_problem_report('#{str}' looks something you do not want to print")
    end
    return str
  end
end

This basically recognizes some commonly occurring strings when HTML has been accidentally escaped twice (as usually you don’t want to print this). Depending on the application, the regular expressions might be much different but the basic idea stays the same. The idea is not to have a fully-functional problem analysis tool, but to just report about any suspicious strings which the team can check and fix if necessary. The method send_problem_report represents a method which reports the problem using e.g. Airbrake or ExceptionNotifier if it happens on a deployed application instance. It could also open a debugger using ruby-debug by calling debugger. When the debugger line is reached when running tests, you are immediately able to see the backtrace to determine if there is a problem or not. Another way is of course throwing an exception, which would print nice error in your CI system and send an exception report in testing environments. This is useful during manual testing, as testers might not notice all the problems.

Notice also the !str.match(/looks something you do not want to print/ part in the if condition. This prevents situations where the error message is used in another sanity check causing an infine loop. Whether this is possible or not, depends on your codebase, however, this occurred to me a couple of times. This check prevents that from happening.

The tricky part is, when to call this method? One solution is to hook it in all HTML printing code by alias_method_chaining simple_format, content_tag_string and link_to, which are the ones rails_xss also chained.

module ActionView
  module Helpers
    module TextHelper

      def simple_format_with_double_escape_reporting(*args)
        HtmlDoubleEscapeReporter.assert_sane(simple_format_without_double_escape_reporting(*args))
      end
      alias_method_chain :simple_format, :double_escape_reporting
    end

    module TagHelper
      private
      def content_tag_string_with_double_escape_reporting(*args)
        HtmlDoubleEscapeReporter.assert_sane(content_tag_string_without_double_escape_reporting(*args))
      end
      alias_method_chain :content_tag_string, :double_escape_reporting
    end
    module UrlHelper
      def link_to_with_double_escape_reporting(*args, &block)
        HtmlDoubleEscapeReporter.assert_sane(link_to_without_double_escape_reporting(*args, &block))
      end
      alias_method_chain :link_to, :double_escape_reporting
    end
  end
end

Remember, that this kind of hooking might not work for your application, so plan and test carefully.

3.2 Manual Testing

In addition to this automated check, you most definitely want to run adequate manual regression tests on your application and involve several people in testing. Additionally, every time you find a problem, you want to grep the whole application for similar cases, as usually these problems exist in many places.

3.3 Code Review

Grepping all the code for suspicious patterns might not always be enough. Therefore you should review the code as much as possible. After seeing various problem situations, you have the best knowledge and “eye” for the problems that might still be hidden in the code. Therefore reading as much code through as possible is a highly recommendable practice in addition to automated and manual testing. Just because testing everything is often impossible.

4. Execute your Strategy

Follow your strategy and make notes on the progress. After completing all the planned steps, instruct your whole team on how to keep the application XSS safe using the new tools which you installed. This helps to keep the application both functional and secure in the future.

~ Raimo

Today I wanted to talk about another feature we've added to the iOS SDK that I didn't cover before: custom stylesheets. Since you are integrating UserVoice into your app, we thought it would be good for you to have an easy way to tweak the look and feel to match your existing UI a bit better. To do this, we extracted most of the colors that our UI works with into a UVStyleSheet class, and added a simple API that allows you to substitute your own stylesheet.

To create a custom stylesheet, add a new Objective-C class to your project and call it something like CustomUVStyleSheet. Make it a subclass of UVStyleSheet. Then you can override any of the instance methods of UVStyleSheet to provide your own colors. Here's a simple example:

#import "CustomUVStyleSheet.h"

@implementation CustomUVStyleSheet

- (UIColor *)backgroundColor {
    return [UIColor colorWithRed:0.15f green:0.15f blue:0.15f alpha:1.0f];
}

- (UIColor *)darkZebraBgColor {
    return [UIColor colorWithRed:0.1f green:0.1f blue:0.1f alpha:1.0f];
}

- (UIColor *)lightZebraBgColor {
    return [UIColor colorWithRed:0.15f green:0.15f blue:0.15f alpha:1.0f];
}

@end

Then, all you have to do is to pass your stylesheet to the SDK before launching UserVoice.

[UVStyleSheet setStyleSheet:[[[CustomUVStyleSheet alloc] init] autorelease]];

And with that, you should be able to run your app and see custom colors throughout the UserVoice UI! You can find a complete list of the colors you can override in UVStyleSheet.h, with explanations of how they are used.

-Austin Taylor
Developer, UserVoice

At the same time we are announcing that the SDK now supports the iPad! Landscape mode on the iPhone is now ready to use as well.

In addition to polishing the SDK itself, we have released an example iOS app on GitHub showing how to use it. We think this will make it a lot easier for you to get everything set up correctly in XCode.

To get started, download the example project and pull down the SDK. Replace the demo URL with your own UserVoice site. Log into your UserVoice account and add an iOS app under Settings -> Channels -> iOS Apps. Copy the API key and secret into the example codebase. That should be all you need to do to see your own feedback forum in the example app.

To set up the SDK in your existing app, you will need to do the following things.

• Run XCode 4.2 (there is an XCode 3 branch, but it is not up to date)
• Have your project inside an XCode workspace
• Download the SDK and drag it into your workspace
• Add libUserVoice.a under the Link Binary With Libraries build phase
• Set User Heading Search Paths to $(BUILD_PRODUCTS_DIR)
• Set Always Search User Paths to Yes
• Add -ObjC and -all_load to the Other Linker Flags setting
• Drag the Include group from the SDK into your project
  • Create groups for any added subfolders should be selected
  • Uncheck any targets
  • You may want to rename the group something like UVHeaders
• Drag the Resources group from the SDK into your project
  • Create groups for any added subfolders should be selected
  • Add to your target
  • You may want to rename the group something like UVResources

Once you have completed these steps, you are ready to launch the UserVoice UI from your code. Import UserVoice.h and call one of the three methods on the UserVoice class.

1. Standard Login: This is the most basic option, which will allow users to either sign in, or create a UserVoice account, from inside the UserVoice UI. This is ideal if your app does not have any information about the user.

[UserVoice presentUserVoiceModalViewControllerForParent:self
                                                andSite:@"YOUR_USERVOICE_URL"
                                                 andKey:@"YOUR_KEY"
                                              andSecret:@"YOUR_SECRET"];

2. SSO for local users: This will find or create a new user by passing a name, email, and unique id. However, it will only find users that were previously created using this method. It will not allow you to log the user in as an existing UserVoice account. This is ideal if you only want to use UserVoice with your iOS app.

[UserVoice presentUserVoiceModalViewControllerForParent:self
                                                andSite:@"YOUR_USERVOICE_URL"
                                                 andKey:@"YOUR_KEY"
                                              andSecret:@"YOUR_SECRET"
                                               andEmail:@"USER_EMAIL"
                                         andDisplayName:@"USER_DISPLAY_NAME"
                                                andGUID:@"GUID"];

3. UserVoice SSO: This is the most flexible option. It allows you to log the user in using a UserVoice SSO token. This is ideal if you are planning to use single signon with UserVoice across multiple platforms. We recommend you encrypt the token on your servers and pass it to the iOS app.

[UserVoice presentUserVoiceModalViewControllerForParent:self
                                                andSite:@"YOUR_USERVOICE_URL"
                                                 andKey:@"YOUR_KEY"
                                              andSecret:@"YOUR_SECRET",
                                            andSSOToken:@"SOME_BIG_LONG_SSO_TOKEN"];

We can't wait to see what you guys do with this! As always, we are eager to hear your feedback on the iOS SDK feedback forum.

-Austin Taylor
Developer, UserVoice

~ Scott

parameters.Add(new WebPair(p.Name, p.Value.ToString()));

to

var encodedName = OAuthTools.UrlEncodeStrict(p.Name.ToString());
parameters.Add(new WebPair(encodedName, p.Value.ToString()));

and recompiling RestSharp. The value is encoding later in the OAuth workflow in OAuthTools.cs. I suspect there is a neater solution, which I'll have a think about before submitting a pull request. In the meantime this will get you going. The part of the spec that refers to this states:

3.4.1.3.2.  Parameters Normalization

   The parameters collected in Section 3.4.1.3 are normalized into a
   single string as follows:

   1.  First, the name and value of each parameter are encoded
       (Section 3.6).

~ Scott

~ Scott

# helper method
def symbolize_hash(hash)
  symbolized_hash = {}  
  # puts hash.inspect
  
  hash.each do |key, value|
    symbolized_hash[key.to_sym] = value
  end
  return symbolized_hash
end

consumer = OAuth::Consumer.new(KEY, SECRET)

puts "Get request token"
response = consumer.request(:get, "#{SITE}/api/v1/oauth/request_token.json")
request_token_hash = JSON.parse(response.body)
puts request_token_hash

request_token = OAuth::RequestToken.from_hash(consumer, 
   symbolize_hash(request_token_hash["token"]))

# this is just a library we use to create SSO tokens, similar code 
# snippets can be found under the SSO docs here
encoder = Sso::Schemes::AesCbc128::Base64Encoder.new(SUBDOMAIN, API_KEY)

# create the SSO token setting the user to be an admin so 
# we can list the tickets back later
sso_token = encoder.process({ 
  :guid => "test@test.com",
  :expires => "2012-12-31",
  :email => "test@test.com",
  :avatar_url => "http://external.com/users/1.png",
  :display_name => "Testie McTest",
  :admin => "accept"
})

# get the access token for this new user
response = consumer.request(:post, "#{SITE}/api/v1/oauth/authorize.json", nil, {}, {
  :sso => sso_token, :request_token => request_token_hash["token"]["oauth_token"]
})
user_hash = JSON.parse(response.body)
access_token = OAuth::AccessToken.from_hash(consumer, symbolize_hash(user_hash["token"]))

# create the params for the ticket
ticket = {
  "ticket[message]" => "Everything is broken......",
  "ticket[subject]" => "Help!!!"
}

# create a ticket
response = access_token.request(:post, "#{SITE}/api/v1/tickets.json", ticket)
body = JSON.parse(response.body)

# get our ticket 
response = access_token.request(:get, "#{SITE}/api/v1/tickets.json?per_page=1")
body = JSON.parse(response.body)

puts body["tickets"][0].inspect

~ Scott

#! /usr/bin/env ruby

require 'rubygems'
require "oauth"
require 'json'

# Trusted Client
KEY = 'CLIENT KEY'
SECRET = 'CLIENT SECRET'
SITE = 'http://your_subdomain.uservoice.com'

consumer = OAuth::Consumer.new(KEY, SECRET, :http_method => :post)
access_token = OAuth::AccessToken.new consumer

forum = {
  "forum[name]" => 'Exciting New Stuff'
}

response = access_token.post("#{SITE}/api/v1/forums.json", forum)
forum_hash = JSON.parse(response.body)

puts forum_hash.inspect

And that, as they say, is that!!! Simple huh?

~ Scott

~ Scott

I’m Kevin, and I wanted to take a couple minutes to introduce myself and talk a little about the state of operations when I first came to UserVoice. I was hired on as the sole Operations Engineer. It sounds like a fancy title, but it really means I wear many different hats. On top of handling the day-to-day operations of our systems, I’m the DBA (oh no!), the network guy, the office IT guy, and, of course, the resident geek. Whether it be the WiFi interference in our loft office, our MongoDB installation, or the water cooler that was oddly plugged into a battery-backed UPS, I’ve been fixing problems since I got here.

My first real task here at UserVoice was to come up with a backup solution for our MySQL database. To implement a quick hack, I wrote a bash script that would do the necessary work which would run from ‘cron’ once a day. As we already had a master/slave setup with our database, it was simple to take a ‘mysqldump’ of the slave database. The basics of the script are:

mysqldump thisisourdbname | gzip -9 > /backups/mysql_dump-`date +%m%d%y-%H%M`.sql.gz

This ran successfully from ‘cron’ for a few days, until I noticed the disk was filling up too quickly. I needed to come up with something better and more robust. I also needed a more permanent storage solution, as well as a rotation plan, so I wouldn’t have to manually rotate the backups.

We already use Amazon S3 for different things here at UserVoice, making it easy to create a new bucket in our existing account to dump our backups in. But, before I put our entire database in the cloud, I wanted to make sure it was encrypted with a password, split into multiple files for easy upload and retrieval, and compressed as much as possible. I chose the 7z archive file format, which uses the LZMA compression algorithm, because it met all these requirements and it’s free, open source, and cross-platform. I added the following to my existing bash script to include the needed compression/splitting/encrypting:

mysqldump thisisourdbname | 7z a -si -v250m -psillypass /backups/mysql_dump-`date +%m%d%y-%H%M`.sql.7z

This further compressed our dumps, and split them into nice 250 meg volumes (except that it took about 4.5 hours to complete the dump, during which the slave wouldn’t replicate any data from the master). This was obviously not good, so I tweaked it a bit. I fixed that by running the dump and compressing it a little bit first with:

DATETIME=`date +%m%d%y-%H%M` mysqldump thisisourdbname | gzip -1 > /backups/mysql_dump-$DATETIME.sql.gz

...then uncompressing it while compressing it again into 7z format:

gzcat /backups/mysql_dump-$DATETIME.sql.gz | 7z -si -v250m -psillypass /backups/mysql_dump-$DATETIME.sql.7z && rm -f /backups/mysql_dump-$DATETIME.sql.gz

This shortened the ‘mysqldump’ portion of the process to only 30-40 minutes. I was comfortable with these figures and ready to wrap it up, all the while including weekly uploads to Amazon S3.

I polished my Ruby skills and wrote a rudimentary script that uses the ‘aws-s3’ library to read a list of files as arguments from the command line that uploads to a bucket on our Amazon account. To easily run this script once a week from ‘cron’:

find /backups -name \*.7z.\* -type f -mtime -1 | xargs s3_backup.rb

This will find all the files modified in the last day, matching the 7z pattern used when creating volumes, then send it to my S3 script as command line arguments care of ‘xargs’. The S3 script handles the upload and notifies the rest of the team via a Campfire notification.

This isn’t a perfect solution yet, but works just fine if you monitor the disk usage. I have yet to implement the automatic rotation, but plan on using 'logrotate' to rotate and manage backup files on the database server. I will also create another host with more space and daily ‘rsync’ the /backups folder for archiving and weekly backups to S3.

This is all part of the excitement that goes on here at UserVoice. I’ll continue to update you on my progress and all fun problems I get to work out. I’d love to hear your feedback about this subject, or if you have any suggestions for further discussion topics.

~ Kevin