First I tried Revel

I started off with Revel because it was popular (in terms of stars on github). It’s not for everyone, but I think Revel is a solid piece of engineering and has some really nice aspects: Good way of representing middleware, awesome code reloading in development, and you’re up and running really quickly with defaults.

One thing about Revel is is that you have “controllers” that are structs. You add methods on these controllers - these are handlers. The first thing I tried to do here is add a field to my struct, and set it in a piece of middleware:

1
2
3
4

type TicketsController struct {
  *revel.Controller
  User *User
}

Then I wanted to set this in a piece of middleware called ‘UserRequired’. This middleware would validate your session and find the appropriate user. If there was no session, it would redirect you to a login secreen. The great thing about this is that once I’m in my handler, I have the current user easily available!

1
2
3

func (c TicketsController) Index() revel.Result {
  return c.Render("Hello " + c.User.Name)
}

I actually implemented this in Revel. But to do so, I had to do ugly reflection in my middleware. Each time I wanted to have middleware interact with my structs in this way, I’d have to write more reflection. Given I wanted to do this for a bunch of stuff, I was bummed out.

Then I tried Traffic

Then I stumbled upon Traffic. Traffic aims to be a lightweight framework – akin to Sinatra. I liked the API for building up a router, and how it’s not very intrusive.

Traffic also has middlware. You can communicate from middleware to handler via SetVar and GetVar:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// Middleware:
func (c *PingMiddleware) UserRequired(w traffic.ResponseWriter, r *http.Request, next traffic.NextMiddlewareFunc) (traffic.ResponseWriter, *http.Request) {
  if nextMiddleware := next(); nextMiddleware != nil {
     if user := getUserFromRequest(r); user != nil {
       w.SetVar("user", user)
       w, r = nextMiddleware.ServeHTTP(w, r, next)
     }
  }
  return w, r
}

// Handler:
func root(w traffic.ResponseWriter, r *http.Request) {
  user := w.GetVar("user").(*User)
  fmt.Fprintf(w, "Hello, %s.\n", user.Name)
}

This is not too bad. But, I was still bummed about the type assertions required.

Web Frameworks: no thanks

Both Revel and Traffic are good web frameworks, and I’d recommend them to anyone looking for a framework. But as I get more experience, I’m finding I don’t want a framework.

I want to add web handling capabilities to my application, not shoehorn my app into a web framework.

For instance:

• Don’t tell me how to do app config. I don’t want to use your INI config file format. I want to use TOML, or I want to use environment variables ala 12-factor apps, or I want to use etcd.
• Don’t tell me where I put my controllers, models, and views. For instance, in Revel you need to put your controllers in a package called ‘controllers’, and models in a package named ‘models’. Nope!

Another fallacy of many web frameworks is they assume you only want one application per executable. They assume this by using global state. This prevents you from binding different servers to different interfaces/ports to give you a public web experience and a protected health/monitoring API. Global state sucks.

Enter gocraft/web

So then I got the urge to write my own library. I had two strong goals:

• Goal 1: Not a framework.
• Goal 2: Pass information from one layer of middleware to the next layer of middleware, and from middleware to action handlers without refection or type assertions in my app code.

Here’s an example of what you can do with gocraft/web. I have some middleware which finds a user and sets it on the context. Later middlware and handlers can read the current user:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

type Context struct {
  User *User
}

func (c *Context) FindUser(rw web.ResponseWriter, req *web.Request, next web.NextMiddlewareFunc) {
  if user := getUserFromRequest(req); user != nil {
    c.User = user
    next(rw, req)
  } else {
    rw.WriteHeader(http.StatusForbidden)
  }
}

func (c *Context) SayHello(rw web.ResponseWriter, req *web.Request) {
    fmt.Fprint(rw, "Hello, " + c.User.Name)
}

func main() {
  router := web.New(Context{}).                 // Create your router
    Middleware(web.LoggerMiddleware).           // Use some included middleware
    Middleware(web.ShowErrorsMiddleware).       // ...
    Middleware((*Context).FindUser).            // Your own middleware!
    Get("/hello", (*Context).SayHello)          // Add a route
  http.ListenAndServe("localhost:3000", router) // Start the server!
}

Contexts

When building up your router, you’ll see that many of the handlers have a signature like this: (*Context).MyFunction. If you’re new to Go, this is weird. This syntax is called a method expression. All it does is represent a function whose first parameter is, in this case, *Context. This lets us write our handlers like this:

1
2
3
4
5
6
7
8
9
10
11
12

func (c *YourContext) Root(rw web.ResponseWriter, req *web.Request) {
  if c.User != nil {
    fmt.Fprint(rw, "Hello,", c.User.Name)
  } else {
    fmt.Fprint(rw, "Hello, anonymous person")
  }
}

// Which is equivalent to this:
func Root(c *YourContext, rw web.ResponseWriter, req *web.Request) {
  // ...
}

They’re not controllers; they’re contexts. They don’t control anything. They’re simply a struct, a piece of data, that is created for you on each request, that lets you set request-specific fields on as it gets passed down your middleware.

They’re also a great place to embed your database connection and logging, since you can instrument them on a per-request basis:

1
2
3
4
5

func (c *YourContext) Root(rw web.ResponseWriter, req *web.Request) {
  c.Log("Entering request...")
  post := c.Select(&Post{}, "SELECT title, body FROM posts WHERE `id` = ?", id)
  // ...
}

Nested routers, contexts, and middleware

I believe another innovative aspect of gocraft/web is that you can have a hierarchy of routers, contexts, and middleware. Imagine:

• You want to run an AdminRequired middleware on all your admin routes, but not on API routes. Your context needs a CurrentAdmin field.
• You want to run an OAuth middleware on your API routes. Your context needs an AccessToken field.
• You want to run session handling middleware on ALL your routes. Your context needs a Session field.

Let’s see how this is represented:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

type Context struct {
  Session map[string]string
}

type AdminContext struct {
  *Context  // When you nest contexts, you need to embed the parent context in the first slot.
  CurrentAdmin *User
}

type ApiContext struct {
  *Context
  AccessToken string
}

rootRouter := web.New(Context{})
rootRouter.Middleware((*Context).LoadSession)

apiRouter := rootRouter.Subrouter(ApiContext{}, "/api")
apiRouter.Middleware((*ApiContext).OAuth)
apiRouter.Get("/tickets", (*ApiContext).TicketsIndex)

adminRouter := rootRouter.Subrouter(AdminContext{}, "/admin")
adminRouter.Middleware((*AdminContext).AdminRequired)

// Given the path namesapce for this router is "/admin", the full path of this route is "/admin/reports"
adminRouter.Get("/reports", (*AdminContext).Reports)

When a request comes in for "/admin/reports":

• We’ll allocate a context for the rootRouter
• Then we’ll run middleware on the root router, in this case LoadSession.
• Do routing. We’ll find the route for (*AdminContext).Reports on the adminRouter.
• Run middleware on the adminRouter, in this case AdminRequired.
• Finally, invoke the action.

Cargo cult of Anti-Reflection

I was able to implement this because of Go’s awesome refection capabilities. Yet when I talk to some people about this, they see it as an absolute downside. They refuse to consider using it; they get worried that their code isn’t statically verifiable by the compiler any more; they worry that it’s “too slow”.

This is utter nonesense.

I’ve started some initial benchmarking of web routers and frameworks at golang-mux-benchmark - gocraft/web shows some great performance numbers.

Second, I don’t see too much of a difference between reflection and type assertions. Strangely, type assertions have no stigma attached to them, yet result in run time panics if you type assert to the wrong type.

Finally, gocraft/web does manual type checks on your handlers. If you get it wrong, you’ll get a panic at application boot with a helpful message. This offers a significant chunk of the benefits of static type checking.

go get github.com/gocraft/web

There you have it. I believe I’ve created a compelling tool that helps you build web apps in Go.

There’s a bunch more documention and details about gocraft/web here: https://github.com/gocraft/web.

Give it a try, and tell me what you think!

Problems

We had three specific problems at UserVoice:

1. We have different user interfaces to UserVoice: the front-end where folks come to look at ideas and vote, the admin console where admins moderate their ideas and respond to tickets, the API, widgets, a Facebook app, the iOS SDK, etc. All of these interfaces have their own controllers. We have the potential for duplicate logic: there could be an ideas#create endpoint in multiple controllers.

2. In the API especially, folks would pass in all kinds of invalid input. For instance, if an ideas#create endpoint took params[:idea][:name], they’d just pass params[:idea] as the string "Here's my idea". This would pollute our exception logs.

3. Some of our models became extremely fat, with a sequence of lifecycle callbacks that feed into each other. It became hard to reason about and modify these models, especially when you’re dealing with logic that has to work in multiple contexts (create vs update vs business action A vs business action B).

Solution

We’ve done three things to solve this.

First, as is more and more common in the Prime Stack, we’re been using the Service Layer pattern more. Each service object is essentially a module of code that has one primary method and one responsibility. For instance: TwitterPoster.post(tweet, user).

We’ve called these service objects ‘Mutations’, as quite often they represent a change of state in our database from state A to state B.

Second, we took a hard look at input validation and made some big changes. In particular, in Rails apps you’re presented with a params hash from the user, and you’re supposed to write a function against it. Rails validates the params hash in a few ways:

1. ActionPack forces the hash to have a general structure by construction: the hash only has strings as keys, and values can be strings, more hashes, arrays, and files.
2. attr_accessible or strong_attributes does attribute whitelisting.
3. ActiveModel does field validation.

This trio is usually pretty effective. However, there are some problems:

1. The structure of the hash is frequently assumed to be correct. For instance, it’s taken for granted that params[:user] is a hash, and params[:user][:role_id] is a string (and not, say, an array)
2. attr_accessible suffers from context blindness: you’re frequently going to have an end user UI and an admin UI. You want admins to have access to more fields.
3. It works best if you are mapping inputs directly to database fields.

Lastly, on our more beastly AR models, we started removing lifecycle callbacks and putting them into the service objects. This makes the AR models quite anorexic. The focus is on context: each Mutation operates in a single context, so there’s very little conditional logic. Each Mutation is a clear, linear sequence of code.

The Mutant Baby

And so were born Mutations! Simply, it lets you validate and specify inputs on your service objects.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class UserSignup < Mutations::Command

  # These inputs are required
  required do
    string :email, matches: EMAIL_REGEX
    string :name
  end

  # These inputs are optional
  optional do
    boolean :newsletter_subscribe
  end

  # The execute method is called only if the inputs validate. It does your business action.
  def execute
    user = User.create!(inputs)
    NewsletterSubscriptions.create(email: email, user_id: user.id) if newsletter_subscribe
    UserMailer.async(:deliver_welcome, user.id)
    user
  end
end

and is callable like this:

1
2

outcome = UserSignup.run(params) # option A - get back an outcome that has outcome.success?, outcome.errors, and outcome.result
user = UserSignup.run!(params)   # option B - return the result, and raise an exception if there's a problem.

First, you specify the required and the optional inputs to the “function”. You also specify their type and constraints. If the inputs don’t match the constraints, and can’t be coerced to them, then the inputs are invalid and execute won’t be called. If everything checks out, then the execute method is called. Within execute, all of the inputs are safe, whitelisted values. You know all of the required values are present, and they’re the right types. This makes it a lot easier to write correct code against these inputs.

This looks pretty simple but it turns out to be pretty useful. There are a lot of synergies that arise and writing your application like this has a variety of benefits.

For complete documentation, see the README.

Reduce Code Duplication

By building an internal API (service layer) that your controllers can use, you’ll have less code duplication if you have multiple controllers that operate on the same types of data. This is straightforward and can be achieved by building any service layer, as I imagine many other folks using the Prime Stack have found.

Security & Correctness

One of the core benefits is security. There have been several Rails exploits in the past year (CVE-2013-0155 and CVE-2012-5664) that are related to an ‘incorrect datatype’ – you expected params[:id] to be a string, but in fact it was an array. These are cases where there was a bug in the code due to lack of robustness, and causing that bug also happened to reveal a security issue. There are plenty of other cases where bugs of the same type – “type checking bugs” – are simply non-exploitable bugs.

Using Mutations decreases the likelihood of issues like this popping up. If you specify that a parameter must be a string, then inside the execute method, it is guaranteed to be a string. If an ignorant or malicious user passes in an array, then an error will be raised. If the parameter can be safely coerced to a string (eg, an Integer or boolean is passed; 3 becomes “3” and false becomes “false”), then the coercion is done and you have string. This makes it much easier to write correct code, and correct code is much more likely to be secure.

Reasoning About Logic

The benefit I’m most excited about is that it’s really easy to reason about a Mutation, because it only does one thing. In contrast, it’s more difficult to reason about callback soup.

With a Mutation, you have a single business rule that you’re coding against. You’re likely to have a short, linear method.

With fat models, first of all you don’t know what the possible business operations are. But let’s say you’re thinking about creation. You have to look at the callbacks, and figure out which ones will be called in this context. In some cases, you have to look at model instance variables, which encode which context you’re in. Sometimes, information is passed from one callback to another callback. And when you’re writing callbacks, you have to make sure your piece of code works correctly in all contexts.

Callbacks are great for small apps and for simple things that are context-insensitive. For moderately complicated models, callbacks are a terrible way to encode your business rules.

Documentation

How often have you seen methods with signatures like this?

1

def do_task(user, idea, options = {}); end

user and idea are clear, but what are valid options? If you’re lucky, do_task is a short function and you can see which keys and values can be in options. If you’re not, the method is implemented like this:

1
2
3

def do_task(user, idea, options = {})
  user.foo(task_helper(idea, options))
end

And now you’re on a wild goose chase. The standard solution to this is a comment above the method. But as dogma goes, code shouldn’t need comments because code should be clear already.

Mutations solves this fairly nicely: all “options” are documented as code, not as comments. It’s extremely clear what options can be, including their types and constraints. Example:

1
2
3
4
5
6
7
8
9
10
11
12
13

class MyTask < Mutations::Command
  required do
    model :user
    model :idea
  end

  optional do
    string :name_override
    integer :count, default: 1
  end

  def execute; ...; end
end

Notice that you know what is required, what is optional, what default values are, and what the data types of these fields are. This is more verbose than the original method, of course, but could be on-par with the original method plus a comment block.

Forms

Mutations help with form development in three ways:

1. Multi-model forms are handled well. The standard solution is to use accepts_nested_attributes_for. Sometimes this works great, but other times you find yourself bending over backwards to do validations and other logic with callbacks. With Mutations, the structure of a nested hash can be well specified, and it’s easy to write logic against that instead of handing it over directly to ActiveRecord.

2. Forms where the input tags don’t map exactly to database fields are handled nicely. The standard solution is to add an attr_accessor to the model, whitelist it, and then operate on it in callbacks. This sucks. It’s much nicer to specify this parameter as an input and operate on it in a simple method.

3. Dual client-side/server-side errors are painless with Mutations. When you run a Mutation against a user’s profile, for instance, this is what you can get back:

1
2
3
4
5
6

outcome = UpdateProfile.run(params)
unless outcome.success?
  outcome.errors.symbolic # => {name: :required, address: {street: :invalid, state: :not_state}}
  outcome.errors.message # => {name: "Name is required", address: {street: "Street is invalid", state: "That's not a known state"}}
  outcome.errors.message_list # => ["Name is required", ...]
end

Conclusion

Mutations are great if you’re working on a bigger, long-lived application in a team setting. The service layer aspect gives you a scalable architecture. The input validation aspect makes it easy to pass user input directly into your service layer. You’ll have more robust and secure code.

To learn more, take a look at the Mutations home page.

Comments on Hacker News

One problem we started running into was content that included formatting we hadn’t anticipated. Early on, we didn’t allow any formatting other than line-breaks (converted to paragraphs). Later, we introduced textile support for different types of user-generated content. Once textile was added, we wrote styles for a lot of common markup, but it was easy to overlook certain elements like ordered lists or code blocks that weren’t part of our test content. It was also easy to be inconsistent in how the markup was styled from one part of an interface to another.

Later, when we prepared to launch our WYSIWYG editor and full HTML support for knowledge-base articles, we knew we would begin encountering all kinds of content that needed to be taken into account.

Enter Typeset.css

Because I’m tired of styling content over and over again, I wrote some basic CSS to solve ALL THE PROBLEMS once and for all. Here’s what Typeset.css does:

• It includes styles for HTML5 content-based semantic markup and resets layout-based styles (incorrectly) applied to user-generated content.
• It inherits font styles, allowing it to be dropped in anywhere on a page where content markup needs styles.
• It plays well with CSS resetters, normalizers bootstrappers, boilerplaters and other popular CSS frameworks.
• And it renders styles consistently across browsers.

Essentially, Typeset.css is a no-nonsense CSS typography reset for styling user-generated content like blog posts, comments, and forum content. Pop it in and BAM: content styled.

For examples of all the HTML elements styled with Typeset.css, go to http://joshuarudd.github.com/typeset.css/ or check out the Typeset.css GitHub repository.

You, too, can use it!

At UserVoice, we like to open source projects that support our core business but aren’t core to our business. Typeset.css is no exception, so please check it out and let me know what you think! I hope you find it useful!

You can also take a look at some of UserVoice’s other open source projects at http://developer.uservoice.com/opensource/.

Deploys

Whenever we deployed our code using capistrano (more great software!), it would restart Passenger on every webserver to ensure our code changes were picked up. In Passenger’s design, the first request to come in after starting actually spawns and ‘runs’ the rails app. Subsequent requests are very fast if using the smart or smart-lv2 spawn methods, but when it needed to be restarted, every request sent to the machine was backed up and waiting until the master Passenger process could fork workers to deal with them. Since our rails app is fairly large, it takes about 15-20 seconds for the app to fully load, and because of how Passenger does restarts we needed to wait for it before restarting Passenger on the next machine. This made our deploys really slow (and developers grumpy) since we needed to do rolling restarts with a large wait time between machines to account for the slow startup time, and to minimize impact to our users.

To keep the response times under control, we needed to wait 15 seconds on each machine for passenger to restart, before going on to the next webserver. With our 6 web machines, that took a minimum of 1.5 minutes just to restart the web processes, that time didn’t include all the other things that needed to happen while deploying (code checkout, asset packaging, etc). It took at least 5 minutes in total to deploy new code… Who’s keeping our deploy times under control? Passenger wasn’t. In came Unicorn to the rescue!

Passenger and Unicorn are very similar in that they both start a master process, and use fork(2) to spawn worker processes, drastically reducing time to spawn workers to handle requests. The difference between Unicorn and Passenger is that Unicorn is much simpler in design and uses a pre-fork model, it starts one instance of the rails app, and forks worker processes before accepting requests. This enables Unicorn to restart in the background while still accepting and responding to requests, the same way nginx works, allowing for zero-downtime deploys. Unicorn took us from 5 minute deploys, down to 1 minute. I’ll tell you how.

Unicorn uses accept(2) on a shared socket, which is the unix way of sharing a central queue. Now when we deploy, all we need to do is issue the master process a USR2 signal. A little ruby in before_fork, automates the killing of the old process:

1
2
3
4
5
6
7
8
9
10

before_fork do |server,worker|
  old_pid = '/var/www/current/tmp/pids/Unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

What happens:

• When code is changed, a USR2 signal is issued to the current running master Unicorn process.
• The running master Unicorn process with old code, tries to start the Unicorn binary again from scratch.
• If the re-execution of the master process is successful, before_fork takes over and kills old master process.
• If re-execution fails, the new master process won’t spawn, and will also not kill the old one running known good code.
• All connections are still actively served by old workers, until new ones spawn, and take over by issuing accept(2) on the shared unix socket.

CPU Usage

The biggest difference was in CPU usage. With Passenger I used passenger_max_pool_size 10 and passenger_spawn_method smart-lv2, and I used worker_processes 10 for Unicorn. When I increased the load to both Unicorn and Passenger to 10 concurrent requests, Unicorn became the clear winner. Data colllected from Ganglia showed our CPU usage drop with Unicorn. With Passenger, we were barely able to handle our current load without adding more machines to the cluster, when we tried Unicorn we noticed as much as a 10% decrease in CPU usage.

The CPU usage was curbed (peaks aren’t as thick), allowing us to gain a little bit of CPU headroom in following weeks before having to add more hardware.

Memory Usage

After running Unicorn and Passenger side by side in production, we found that Passenger utilized marginally less memory than Unicorn did, likely because of the dynamic spawning/killing that takes place, also somewhat explained the increased CPU usage with Passenger.

The graphs are drawn with rrdtool, and are therefore averaged out to make the graph smoother. The black line represents when we rolled out Unicorn across our entire web cluster. You can see on that the memory usage stayed about the same, but slightly beter on the Passenger side.

Response Times

Unicorn had an added bonus of slightly quicker response times, especially as # of concurrent requests increased

Response time going down during the Unicorn rollout.

We also noticed that during periods of high concurrent requests, Unicorn would be more reponsive than Passenger.

Conclusion

It took us 5 minute or more to change code on in our application… not exactly what you want when you’re trying to stay agile. We chose Unicorn because it performed more consistently across load levels, and allowed us to drastically cut down the time it takes us to deploy code across our growing infrastructure. Passenger is a wonderful ruby webserver, and because of it we were able to spend more engineering time on our application, rather than tweaking and fiddling with our webserver.

Key qualities we are looking for in a designer are:

• Craftsman. You care deeply about the quality of your work and are continually improving and challenging yourself.
• Communicator. You are able to communicate and share your ideas through written briefs, sketches and wireframes, prototypes, wild hand gestures, and high-fidelity mockups.
• Builder. You don’t just think of cool stuff. You make it. You understand that an idea is just an idea unless you bring it to life.
• Collaborator. You understand the importance of being a team member, leaning on and learning from the skills of others, and putting team before individual accomplishments.

Moreover, you are someone who:

• knows what you’re doing (we’re not here to babysit you);
• has a strong work ethic, is reliable, and delivers on time;
• can design engaging experiences with typography, color, space and imagery;
• understands that content and copy are just as much a part of design as visuals are;
• utilizes best practices when designing across desktop and touch interfaces (and knows the differences between them);
• can produce production-level semantic HTML and CSS, understands how to design and code the non-visual stuff (accessibility, etc.);
• earns bonus points with print design experience (though not required), able to hand off assets to a printer and work with them to make sure your designs are reproduced as intended;
• able to work directly with our talented engineering team to implement the hard stuff;
• and…works well with others, because we want to work well with you.

Most importantly, we’re looking for the right person, not someone who can just check off all the items on the lists we’ve made here. It’s quite possible you bring qualities we didn’t know we even needed. If so, tell us. We want to know you. Each person on our team is important to us, and we strive to create a strong company culture that everyone contributes to and is proud of.

We believe that proven experience doesn’t come from the number of college degrees you may or may not have earned. We’re not interested in how many awards or honors you may or may not have received (although congrats to you if you have). We expect that you understand and are engaged in some of the latest tech and design goodness coming out of startups these days, but we aren’t looking for someone who is ruled by them.

Still interested? Here are some things you can expect working at UserVoice:

• We’re a highly collaborative team, and we do our best to limit process so that it doesn’t get in the way (but also streamlines getting things done). You can read more about how we use Trello and Google Docs to make UserVoice better every day. And, who knows, that may change in a couple of months.
• We have two offices: our headquarters in San Francisco, CA, and Raleigh, NC. We’re looking for someone to join us in San Francisco.
• Our creative team is headed up by Joshua Rudd (Head of UX Design) and includes John Long (UX Designer), Brad Graham (Front-end Developer), Chad Bercea (Graphic Designer), Andy Parker (Copywriter), and quite possibly you (what do you want to be called?). Our team works directly with everyone else in the company to ensure that we create cohesive designs and experiences across product and marketing.
• Our engineering team does some amazing work. And while we say we have a creative team and an engineering team, the truth is that we don’t throw designs over the fence and expect them to make it happen. We work together from the very beginning because design is not just how something looks but also how it works. And how something works influences how things look.
• We work hard because we love what we do, and we play hard because we like each other. However, we don’t live at the office. We have lives. Many of us have families. Some of us are rock stars (but not the design-ninja-rock-star-shit kind many companies say they’re looking for).

If you can see yourself working with us or have any questions, we want to get to know you better. Email us at design-jobs@uservoice.com, tell us about yourself, and include links (or attachments) to some of your work you are proud of, including which roles you played. We’d also like to know your favorite childhood cartoon (our job Turing test!).

Sincerely,

Joshua Rudd
Head of UX Design
UserVoice.com

A couple of weeks ago we soft-launched an updated UserVoice developer site. We are super excited about it for a variety of reasons.

You’ll notice a couple of things right away about the new site…

• Easier Navigation
  The site is now its own mini-site and is much easier to navigate.
• Updated Documentation
  There is now a comprehensive documentation index which should makes it easy to browse through all technical documentation on a given subject (blog posts, UserVoice FAQs, and other articles).
• Open Source & Third-Party Projects
  The open source page shows off some of the amazing work of people on our development team and the third-party page links up a number of integrations created by other developers.
• We’re Hiring!
  That’s right! Our new Work With Us page highlights this much better, so if you’re a developer in the Raleigh or San Francisco area please reach out to us. We have a couple of open positions now and hopefully more in the future.

And now for some of the technical things that may not be so obvious…

Octopress

The new site is now powered by Octopress. Our marketing site runs on Expression Engine. This makes a lot of sense for our marketing team. When we originally launched the developer site we thought it made sense to keep things simple and use the same platform. But through trial and error we’ve found that our developer needs are actually quite different from our marketing needs.

Octopress bills itself as a “blogging framework for hackers” (“hacker” in the generic sense describing all programmers).

Octopress generates HTML pages from simple text files which makes it a breeze for developers to use. Because the content is stored in the file system (rather than in a database) you can use tools like Git to version and track changes to the entire web site. We are super excited about the transition because it should make it much easier for us to blog and keep this site up-to-date – which means better content for you.

Code Samples

One of great benifits of using Octopress is that it does a great job of presenting syntax highlighted source code like this JavaScript example:

Octopress includes a number of helpful ways for marking up code that should make it much easier for us to blog and share about some of the great stuff we are working on.

Responsive Design

One other feature of the new developer site that we are really happy about is that we’ve taken the time to make the whole thing responsive. This means that you will get a more optimized view of the Web site if you are looking at it on an iPhone or in a smaller window. To see what I mean, just resize your browser’s window and you will see the layout magically transform to work better on smaller devices.

All of this is accomplished through the magic of media queries. If you’re on the bleeding edge of the tech world responsive design is old news, but we like to celebrate the small things!

Cheers!

• Ruby 1.9.3 is faster (see response time difference at the end)
• It has ‘correct’ string encoding support
• We want to stay current with the Ruby ecosystem.
• First of all, let me state clearly: Rails 2.3.x does NOT SUPPORT Ruby 1.9.x out of the box, unless you want to get your hands dirty with some monkey-patching.

This guide will outline some of the things you’ll need to do in order to get things working.

Getting your script/server to run

Ruby changed the way that files are loaded with requires. So in order to get script/server and friends to work, you’ll need to tweak them from this:

1
2
3

#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'commands/server'

To this:

1
2
3

#!/usr/bin/env ruby
require_relative '../config/boot'
require 'commands/server'

We also had to change our tests so that ruby test/unit/some_test.rb would work:

1

require_relative '../test_helper'

Once you get your tests going, you’ll likely need to add lines like this to various files if they contain UTF-8 characters.

1

# encoding: utf-8

Woot! Now some of your tests might run (but they’ll still likely fail).

Upgrade your gems, use mysql2

One of the first gems you’ll need to upgrade is mysql – it doesn’t support encodings properly. You’ll want to use the mysql2 gem. Also, if your tests indicate there’s problems in any of your other gems, upgrading the gem is probably a good idea.

If you can, you should deploy all of these gem upgrades to your master branch as soon as possible, so that you can minimize the changes you experience when upgrading Ruby.

Monkey patch, ahoy!

If you weren’t properly scared yet, reading these monkey patches will surely do it. If not, you are probably insane. Read on!

Patches to environment.rb:

Patches to config/initializers/rails2314_ruby193_monkey_patches.rb:

Patches to config/initializers/utf8_monkey_patches.rb:

Migrate any serialized columns

We use some YAML-serialized ActiveRecord columns in our schema. Unfortunately, if you have any UTF-8 characters in these serializations, you’ll need to migrate them. Ruby 1.8 uses the syck YAML library, and Ruby 1.9.3 uses the psych library. So you’ll need to manually use Syck::load on each record, and re-save it with Psych::dump.

Don’t use TMail

TMail most certainly does not support Ruby 1.9.x. If you want to continue to use TMail to send your mail (this is the default in Rails 2.3.14), you’ll need to monkey patch the heck out of the gem. This was actually our first approach at UserVoice, and it was 99% successful.

However, it felt a bit risky to run our ticketing system on this monkey patched gem, so we just re-wrote a few methods in ActionMailer to use the new Mail gem (included by default in Rails 3). If you send a lot of mail with more than just ascii characters, you’ll likely want to do the same.

Deployment

First, make sure you backport anything possible into your application currently running on 1.8.7 and iron out the kinks with that. For instance, any gems that you upgraded are a great candidate for this. This ensures that when you upgrade to 1.9.3, you’ll be minimizing risk.

We deployed 1.9.3 slowly. First, take 1 server out of your HAProxy rotation, so no web requests are going to it. Then upgrade that machine with 1.9.3 and load your git feature branch on that machine. Next, make sure it works by hitting that machine yourself (don’t put your customers on it yet.) When that works, squirt a small amount of traffic from real users. If at any point you start seeing errors, take it out of rotation again and fix it.

Now, if you leave that server for a few days and it’s not producing errors, go ahead and migrate the rest of your cluster.

Final thoughts

So there you have it! We decided to upgrade Ruby before Rails because it seemed more tractable (which is probably true). But it’s certainly not as easy as I thought – I might go to Rails 3 first if I had to do it again. But, UserVoice is faster now! Here’s a graph of our performance increase for one of our slower, more ruby-intensive pages: as you can see, it’s a roughly 50% performance increase!

Installing rails_xss in Rails 2 is a very good method to see how the XSS protection is going to change things after the upgrade to Rails 3. Depending what kind of content management system solutions, asset packaging tools or helper methods you are using, you can have problems in a plethora of different places. As a developer solving these issues in UserVoice, I would like to share my process for the XSS protected strings handling.

Typically problems that arise look like this. To solve them all, first step is to identify the different types of problems. Reserve some time for this, as there might be tens of different problem patterns, which all might not be easy to find.

In this article, I’ll explain how I approached the problem and present some lessons learned during the process. This is not a definitive guide on the subject, but a rather a set of principles which I recommend to be applied during and after the rails_xss installation.

1. Install rails_xss

Install rails_xss and its dependency (erubis). After this, your Rails 2 application escapes all your “unsafe” strings by default. Explore your application and run tests to see on a general level how serious problems the default HTML escaping causes.

2. Identify the Problems

You should identify all the different types of problems and address each problem with a solution. While testing and reviewing the codebase of UserVoice, I identified many different problem patterns. I have described some of the most common ones:

Custom Helpers

For example, the return value of these kind of helpers can be marked safe by calling .html_safe. In this case it’s even better to use the Rails helper link_to, whose return value is safe by default. Additionally, you don’t have to worry about escaping the name of the user separately using the h method.

1
2
3
4
5
6
7
8

def user_link(user)
  "<a href=#{user.url}>#{h(user.name)}</a>".html_safe
end

# better:
def user_link(user)
  link_to(user.name, user.url)
end

Printing Conditional Strings in Views

Often you want to print a certain string depending on some condition. In these cases the resulting string must be marked safe. Possibly you also want to extract a commonly occurring pattern in your codebase to its own helper.

1
2
3

<span <%= 'style="display: none"'.html_safe unless content.visible? %> >
  Content shown if content.visible?
</span>

Concatenating Strings in Views

Your view files might include concatenation of unsafe and safe strings like this.

1

<%= '<hr />'.html_safe + link_to('User Information', user_path(user)) %>

By default, unsafe + safe results in an unsafe string, which means that the link above gets unexpectedly escaped. To prevent this, <hr /> can be marked safe or even better, just extract it out side the ERB interpolation tag.

Printing Custom User-Defined HTML

1

<%= user.custom_html.html_safe %>

In these cases you need to mark the user-defined HTML safe. At the same time, you should also ensure that the user input is sanitized, because you do not want to allow printing arbitrary HTML in your views. A good sanitizing includes defining the set of allowed tags and denying any others. Using a good and well-tested library for sanitizing HTML is also a good idea, unless you really know what you are doing, which you most probably are not.

3. Design a Strategy

When identifying all the various cases of printing HTML, JSON and HTML entities, you might start to wonder how to cover all the cases in all your application’s views. The short is answer is that 100% coverage requires probably too much effort. However, there are many ways how you can come close to that. OWASP states about XSS vulnerability finding: “complete coverage requires a combination of manual code review and manual penetration testing, in addition to any automated approaches in use”.

Custom Automated Tool

Now that rails_xss had escaped all the output strings by default, I needed to find the places where HTML has been doubly-escaped. So I thought it wouldn’t hurt to write a utility method for matching the doubly-escaped HTML strings:

1
2
3
4
5
6
7
8
9

class HtmlDoubleEscapeReporter
  def self.assert_sane(str)
    if (str.match(/<[a-z]/) || str.match(/&(quot|rarr|larr|amp|#)/)) &&
        !str.match(/looks something you do not want to print/)
      send_problem_report('#{str}' looks something you do not want to print")
    end
    return str
  end
end

This basically recognizes some commonly occurring strings when HTML has been accidentally escaped twice (as usually you don’t want to print this). Depending on the application, the regular expressions might be much different but the basic idea stays the same. The idea is not to have a fully-functional problem analysis tool, but to just report about any suspicious strings which the team can check and fix if necessary. The method send_problem_report represents a method which reports the problem using e.g. Airbrake or ExceptionNotifier if it happens on a deployed application instance. It could also open a debugger using ruby-debug by calling debugger. When the debugger line is reached when running tests, you are immediately able to see the backtrace to determine if there is a problem or not. Another way is of course throwing an exception, which would print nice error in your CI system and send an exception report in testing environments. This is useful during manual testing, as testers might not notice all the problems.

Notice also the !str.match(/looks something you do not want to print/) part in the if condition. This prevents situations where the error message is used in another sanity check causing an infine loop. Whether this is possible or not, depends on your codebase, however, this occurred to me a couple of times. This check prevents that from happening.

The tricky part is, when to call this method? One solution is to hook it in all HTML printing code by alias method chaining simple_format, content_tag_string and link_to, which are the ones rails_xss also chained.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

module ActionView
  module Helpers
    module TextHelper

      def simple_format_with_double_escape_reporting(*args)
        HtmlDoubleEscapeReporter.assert_sane(simple_format_without_double_escape_reporting(*args))
      end
      alias_method_chain :simple_format, :double_escape_reporting
    end

    module TagHelper
      private
      def content_tag_string_with_double_escape_reporting(*args)
        HtmlDoubleEscapeReporter.assert_sane(content_tag_string_without_double_escape_reporting(*args))
      end
      alias_method_chain :content_tag_string, :double_escape_reporting
    end
    module UrlHelper
      def link_to_with_double_escape_reporting(*args, &block)
        HtmlDoubleEscapeReporter.assert_sane(link_to_without_double_escape_reporting(*args, &block))
      end
      alias_method_chain :link_to, :double_escape_reporting
    end
  end
end

Remember, that this kind of hooking might not work for your application, so plan and test carefully.

Manual Testing

In addition to this automated check, you most definitely want to run adequate manual regression tests on your application and involve several people in testing. Additionally, every time you find a problem, you want to grep the whole application for similar cases, as usually these problems exist in many places.

Code Review

Grepping all the code for suspicious patterns might not always be enough. Therefore you should review the code as much as possible. After seeing various problem situations, you have the best knowledge and “eye” for the problems that might still be hidden in the code. Therefore reading as much code through as possible is a highly recommendable practice in addition to automated and manual testing. Just because testing everything is often impossible.

4. Execute your Strategy

Follow your strategy and make notes on the progress. After completing all the planned steps, instruct your whole team on how to keep the application XSS safe using the new tools which you installed. This helps to keep the application both functional and secure in the future.

1

parameters.Add(new WebPair(p.Name, p.Value.ToString()));

to

1
2

var encodedName = OAuthTools.UrlEncodeStrict(p.Name.ToString());
parameters.Add(new WebPair(encodedName, p.Value.ToString()));

and recompiling RestSharp. The value is encoding later in the OAuth workflow in OAuthTools.cs. I suspect there is a neater solution, which I’ll have a think about before submitting a pull request. In the meantime this will get you going. The part of the spec that refers to this states:

3.4.1.3.2. Parameters Normalization

The parameters collected in Section 3.4.1.3 are normalized into a
single string as follows:

1. First, the name and value of each parameter are encoded
(Section 3.6).

I’m Kevin, and I wanted to take a couple minutes to introduce myself and talk a little about the state of operations when I first came to UserVoice. I was hired on as the sole Operations Engineer. It sounds like a fancy title, but it really means I wear many different hats. On top of handling the day-to-day operations of our systems, I’m the DBA (oh no!), the network guy, the office IT guy, and, of course, the resident geek. Whether it be the WiFi interference in our loft office, our MongoDB installation, or the water cooler that was oddly plugged into a battery-backed UPS, I’ve been fixing problems since I got here.

My first real task here at UserVoice was to come up with a backup solution for our MySQL database. To implement a quick hack, I wrote a bash script that would do the necessary work which would run from ‘cron’ once a day. As we already had a master/slave setup with our database, it was simple to take a ‘mysqldump’ of the slave database.

The basics of the script are:

1

mysqldump thisisourdbname | gzip -9 > /backups/mysql_dump-`date +%m%d%y-%H%M`.sql.gz

This ran successfully from ‘cron’ for a few days, until I noticed the disk was filling up too quickly. I needed to come up with something better and more robust. I also needed a more permanent storage solution, as well as a rotation plan, so I wouldn’t have to manually rotate the backups.

We already use Amazon S3 for different things here at UserVoice, making it easy to create a new bucket in our existing account to dump our backups in. But, before I put our entire database in the cloud, I wanted to make sure it was encrypted with a password, split into multiple files for easy upload and retrieval, and compressed as much as possible. I chose the 7z archive file format, which uses the LZMA compression algorithm, because it met all these requirements and it’s free, open source, and cross-platform. I added the following to my existing bash script to include the needed compression/splitting/encrypting:

1

mysqldump thisisourdbname | 7z a -si -v250m -psillypass /backups/mysql_dump-`date +%m%d%y-%H%M`.sql.7z

This further compressed our dumps, and split them into nice 250 meg volumes (except that it took about 4.5 hours to complete the dump, during which the slave wouldn’t replicate any data from the master). This was obviously not good, so I tweaked it a bit. I fixed that by running the dump and compressing it a little bit first with:

1

DATETIME=`date +%m%d%y-%H%M` mysqldump thisisourdbname | gzip -1 > /backups/mysql_dump-$DATETIME.sql.gz

…then uncompressing it while compressing it again into 7z format:

1

gzcat /backups/mysql_dump-$DATETIME.sql.gz | 7z -si -v250m -psillypass /backups/mysql_dump-$DATETIME.sql.7z && rm -f /backups/mysql_dump-$DATETIME.sql.gz

This shortened the ‘mysqldump’ portion of the process to only 30-40 minutes. I was comfortable with these figures and ready to wrap it up, all the while including weekly uploads to Amazon S3.

I polished my Ruby skills and wrote a rudimentary script that uses the ‘aws-s3’ library to read a list of files as arguments from the command line that uploads to a bucket on our Amazon account. To easily run this script once a week from ‘cron’:

1

find /backups -name \*.7z.\* -type f -mtime -1 | xargs s3_backup.rb

This will find all the files modified in the last day, matching the 7z pattern used when creating volumes, then send it to my S3 script as command line arguments care of ‘xargs’. The S3 script handles the upload and notifies the rest of the team via a Campfire notification.

This isn’t a perfect solution yet, but works just fine if you monitor the disk usage. I have yet to implement the automatic rotation, but plan on using ‘logrotate’ to rotate and manage backup files on the database server. I will also create another host with more space and daily ‘rsync’ the /backups folder for archiving and weekly backups to S3.

This is all part of the excitement that goes on here at UserVoice. I’ll continue to update you on my progress and all fun problems I get to work out. I’d love to hear your feedback about this subject, or if you have any suggestions for further discussion topics.

Well, it’s spring and people are planting flowers, and so it’s something we need to get online quickly so we can not miss the opportunity. Orchard CMS is going to let us focus on the business value rather than worrying about building all the infrastructure for our site.

If you don’t have time to watch the whole video it is all transcribed lower on the page as well, pretty sure that’s the fastest I’ve seen something like that created… definitely worth a play!

Orchard is a free, open source project aimed at delivering a highly modular and extensible CMS (Content Management System) using the Razor syntax.

Orchard was originally shipped in January of this year by the OuterCurve Foundation who simultaneously released an online gallery for Orchard modules and themes. This gallery is itself open source and is used to power other .NET community websites such as NuGet and Blogengine.net

The UserVoice widget builds on the existing UserVoice WebMatrix libraries and code but adds the ability to switch between the new range of UserVoice products Feedback, Helpdesk and Full Service. The widgets module contains two widgets that let you add UserVoice functionality to your website built on Orchard CMS.

1. Feedback—the feedback widget adds the feedback tab to your site. This allows visitors to click the UserVoice feedback tab to share ideas, ask questions or contact you via the UserVoice interface.
2. Suggestions—the suggestions widget provides a list of recent/popular suggestions that you can display as content on a page.

The gallery page for the UserVoice widget can be found here

And the codeplex site can be found here

As you may have noticed, we’ve launched a new API. It’s awesome. We-built-our-whole-Facebook-app-with-it awesome. And we’re very excited to see third parties start to sink their teeth into it.

A company called Mediaslurp has done just that, and we’re delighted that thanks to their hard work the UserVoice Wordpress Plugin is updated and better than ever. MediaSlurp targets multimedia content such as real-time streams and audio and video podcasts, and brings them together in an interface for users to find content relative to them, and easily play that content. Getting customer feedback is key for them, and they decided to dive into the new API to see if they couldn’t get the Wordpress plugin working. Says Mediaslurp’s TJ Downes:

Working with the UserVoice API is really straightforward. Anyone who has worked with REST APIs should be able to quickly and easily implement the UserVoice API. We’ve had no difficulty understanding the API and picking up where Shane and Peter left off. Given that the API is so simple to use, we expect to expand on the functionality included in the UserVoice Idea List Widget in the near future. Since the API also delivers the data in JSON format, we expect that it will be very simple to implement real-time updates of the UserVoice data via AJAX and Flash applications.

Swing by the Wordpress plugin page to pick up your very own UserVoice Wordpress plugin. Interested in building your own UserVoice widget? Check out the documentation links right here on developer.uservoice.com, and drop us a line at support[at]uservoice.com if you have any questions, comments, or ideas. We want to help you do awesome stuff.

Along with the release of WebMatrix itself Microsoft have authored several helpers to even further reduce the effort involved in creating engaging websites. In their own words:

They [WebMatrix Helpers] provide you a simple and consistent way of performing common web development tasks that otherwise would require a great deal of custom coding. With a few lines of code you should be able to secure your web site using membership, store information in Windows Azure Storage, integrate your site with Facebook, among others things.

We are very excited that UserVoice was chosen as one of the helpers that Microsoft decided to create for the launch of WebMatrix. The helper makes it very easy to integrate UserVoice into your website and not just adding the widget, it also allows SSO and provides the ability to make some public api calls as well!

Check out the UserVoice WebMatrix Helper.

For a great example of both WebMatrix and the UserVoice Helper you can also check out the "Getting Started Screencast" below