Next Update

Quietly Refactoring

Thu, 18 Jun 2009 18:24:01 -0600

Customers are always requesting new features, as they should, and we'd like to spend all of our time on those requests. Unfortunately, there are other areas that need attention as well. We've recently put a hold on new things and made some serious improvements to Sifter that very few people will ever notice.

What's been happening?

The last month or so we've been refactoring our billing logic and administrative tools to as well as improving some of the back-end processes and some of the interface around managing billing information.

This has been the most ambitious and overwhelming release for Sifter since launch. Ironically, it's almost invisible to the average user. Even to account holders it will be barely noticeable. However, it has dramatically simplified and cleaned up the largest and most cumbersome module of code within Sifter.

Credit Cards & Trials

When we initially launched, we consciously made a decision not to promote the fact that you could signup for Sifter without a credit card. We knew that we'd lose some potential customers, but that was a tradeoff we were willing to make in order to feel more confident that the people who signed up were seriously interested in Sifter.

Naturally, we've received a handful of emails from people letting us know that they disagreed with our decision, but we weren't in a huge hurry to change it. We knew we wanted to change this. We just weren't sure when.

We finally decided make the update so that credit cards wouldn't even be mentioned until you were ready to provide one. Unfortunately, that change wasn't trivial, and it rippled through our billing and account management system. More importantly, it was a change that didn't directly benefit our existing customers. So it was continually put off as we focused on adding new features to help existing customers.

Entirely removing credit cards from the registration process was only a small piece of the solution. So, in the interest of openness, and making sure it doesn't seem like we're sitting around twiddling our thumbs, here's a little more insight into the latest release.

Ghost Authorizations

For subscription-based businesses with a free trial that collect credit card information days in advance of billing, you want to validate the credit card before accepting it. This creates pending authorizations, which, if not collected or voided, will disappear within a couple of days.

The credit card companies refer to these as ghost authorizations. They lock up the amount for the credit holder until the authorization disappears, and in some cases, it causes confusion for people who think they've been charged when they haven't. In the near future, some, if not all, of the credit card companies will begin penalizing merchants for ghost authorizations.

As a result, we needed to update our code so that we didn't leave any ghost authorizations lying around. This wasn't a huge change, but anytime we're talking about handling money, it's not something that you want to take lightly. Interestingly, this was one of the more enjoyable pieces of work in this release.

Old Accounts Cleanup

Sifter doesn't offer perpetually free trials, and I don't anticipate that changing anytime soon. We made this decision early on, and it's one that we believe in deeply. We're incredibly liberal about extending trials, but we believe that the majority of our attention and resources should be spent on paying customers. We also want to discourage people from squatting on good account subdomains.

In order to keep things running smoothly, we'll begin automatically purging old and unused accounts. Sixty days after a trial has expired, if the account hasn't been upgraded to a paying account or had the trial extended, we'll automatically schedule the account for deletion and notify the account holder of the impending deletion. Two weeks after the notification, we'll automatically delete the account and all of its data.

Account Holds

Paying customers, that is customers who have at least one paid invoice, will now be able to place their account on hold. Accounts that are on hold will not be billed, but will also not be usable. The basic administrative features will still be accessible, but the rest of the interface will not be available until the account is reactivated.

So, if you only need access to your Sifter account 8 months out of the year, you can put your account on hold during the slow times to save some money. Given the economy, we hope this is useful to helping some of the smaller businesses alleviate costs without having to completely cancel your account.

Quicker Cancellations

The cancellation process used to take place immediately. Unfortunately, it could be a fairly long-running process. Now, instead of processing them immediately, we delete them in the background. This way, we're not dragging out the goodbye.

Better Credit Card Information

We've significantly upgraded the experience of managing billing information with Sifter. Most of the improvements are behind the scenes, but rest assured, it should be significantly easier now. Whenever we have issues with your card when we run billing, we do a better job of capturing the precise reason and letting you know what went wrong.

Outbound Email

All of Sifter's outbound emails are processed in the background because it can take a few seconds to connect to the mail server and send. Unfortunately, on occasion during sending, the mail server is busy or times out. This is incredibly rare, but as we grow, even rare events become more frequent due to the increase in volume. Sifter is sending almost 50,000 emails per month now, so we've improved the email handling to catch these emails and make sure that they still go through.

All Around Refactoring

This release made some really nice changes to the one part of Sifter that needed some love. The code and associated tests are now exponentially more readable, and we squashed a handful of minor but irritating little bugs. I can't express enough how helpful it was to have extensive unit tests written prior to going into this. It wasn't easy by any means, but without those unit tests, I'd be incredibly scared of releasing this update.

The Moral of the Story

With software development, it's easy to get caught up in the hustle of adding new features and striving to make everyone happy with public-facing tangible improvements. It's incredibly difficult to say that you're going to put off customer-facing features that you desperately want to give them in order to refactor some of your code. However, if it doesn't happen, you're just asking for trouble.

Sifter isn't going anywhere, and this is a valuable investment that needed to be made. I hope everyone understands, and now it's time to start working on those new features that will have a more tangible impact on your experience. The first things will be enhancements to data import and export, and then we'll be moving onto the API.

Milestone Improvements

Wed, 27 May 2009 10:50:09 -0600

Long-term, there's a very big role for milestones within Sifter. However, as our ambitions became grander, the task became more daunting. As a result, the feature simply continued to be delayed. Finally, I just sat down for a weekend and knocked out the functionality. I explicitly avoided any fancy reporting or visualizations for the sake of providing immediate relief to the teams that needed milestones to organize their issues into more digestible chunks. (Ourselves included.)

The result was some very basic milestone management functionality. It got the job done, but it wasn't particularly convenient or powerful. However, by using a very basic version of the feature in the real-world, we were able to get a much better grasp of what we needed to do next.

MIlestones and the Dashboard

The most obvious pain-point with phase 1 of milestones was that it wasn't easy to get a feel for the status of the current milestone or to dive right into the subset of issues that are assigned to me for the current milestone. Almost immediately, I began sketching ideas and thinking about what needed to be done.

Working Out the Concept

With most dashboards, there are two general types of questions. The first is status questions. These questions ask for data about your current position. The second is activity. These are questions about where you've been, what's been accomplished, or what's happening right now. Recognizing this distinction become the first step in rethinking our approach to the dashboard.

Initially, I had to work out the idea of separating the dashboard into more focused and digestible pages.

With Sifter, we had previously been trying to merge activity and status into one all-powerful dashboard. This made it difficult to get an overview of all of your projects because each project took up a significant amount of vertical space.

The new dashboard separates status and activity. While the activity view is still very basic at this point, there are some plans in the works to provide advanced filtering to help you view the activity that's important to you. For now, though, that will have to wait until the next phase. So now Sifter is designed to provide a quick view of where the project and next milestone stand. Then, there's an optional page for activity that will soon see even more improvements.

Rapidly Sketching Ideas

With new features, I've always found that the best way to make progress is to just sit down and start sketching and exploring. In this case, I started by thinking about the pieces of information that were relevant, and then tried to get them all into a reasonably concise format.

When designing a module like this, it helps to jot down the bits and pieces that will need to be displayed. Then it's easy to sketch up a quick idea for a starting point.

During the process, I tried to simplify the display to the absolute minimum. However, in doing so, I felt that I took it to far for the time being. All that really mattered was that we communicate the project's completion and the milestone's completion. Next, we need to provide immediate access to the outstanding issues so that you can get in and get to work.

With each idea, I like to rapidly iterate and sketch the concepts. Below each concept, I make notes about the strengths and weakness of each one.

Once I felt comfortable with an idea, I like to rapidly iterate and create different versions making notes about my decisions along the way. At some point, I end up with a concept that I feel good about, and I move into Photoshop.

Exploring Designs

In the case of milestones, I invested a considerable amount of time mocking up detailed ideas. Unfortunately, I decided that they were great in theory, but that they would be too difficult to execute on successfully at this point. I may revisit the idea later, but it wasn't worth it yet.

Visually, this was the simplest concept, but since I wasn't completely sold on it, I decided it wasn't worth implementing yet.

The ideas took on several different forms, and at no point did I feel committed to a particular idea. I just rapidly worked through all of the little ideas and tweaks until I'd get to a point where I was comfortable.

One of the most challenging aspects was balancing milestones and overall project status as some projects may not even use milestones.

Some versions ended up having too much visual noise while others were too simple to work successfully in all of the different scenarios. Ultimately, it became more of a balancing act than an opportunity to really nail the design. Ultimately, I decided that getting a simpler workable solution out the door would give us an opportunity to make improvements over time.

The first fully implemented version was clean and simple, but there wasn't enough differentiation between projects.

The first iteration was too focused on being clean. It overlooked the value of visually separating the projects. Also, we wanted to feature the fact that projects could be sub-divided into milestones. By adding the link for administrators from the dashboard, we can hopefully draw attention to the feature so that everyone can get some benefit from it.

By making the projects stand out, the list became more scannable.

The other significant change was about preparing for a more advanced way to filter activity. Some customers who were managing many projects were frustrated by how crowded the dashboard became. It was very difficult to scan everything and get a quick overall feel. The activity summaries were a significant part of the overcrowding.

We also have plans for adding filtering options to the activity feed as well as displaying additional data. There was no room to add filtering options on the primary dashboard without making things incredibly crowded and confusing. This meant that activity would need a new home dedicated to it. In the first pass, we've simply increased the number of items displayed in the activity feed.

Activity is now displayed on its own page so that it has room to grow.

Summary

This is just the tip of iceberg, and nothing is written in stone. There's plenty of room for improvement, feedback is always welcome, and we'd love to get some additional perspectives.

A Few Hiccups and A Lot of Improvements

Fri, 15 May 2009 14:22:50 -0600

Earlier this morning, we finished making some long overdue, but behind-the-scene improvements to Sifter. We've been working on these changes since our unscheduled downtime a couple of weeks ago, and taking steps to make sure that doesn't happen again has been our top priority. Unfortunately, with these changes, there's a chance for some minor downtime now to lay a better foundation for the future.

The Hiccups

In preparing for and making the switch, we experienced a few additional windows of downtime over the last couple of days. We're really sorry for that, but long-term, we think the improvements will far outweigh the temporary inconvenience. We really appreciate everyone's patience during those times.

The Improvements

For the most part, we wanted to make things more stable and reliable, faster, safer, and easier for us to work with so that going forward we can spend our time focusing on the application and not the platform. So we've spent the last couple of weeks making dramatic improvements to our processes and platforms.

Server Stack

We've switched from Mongrels to Passenger, and we're using Ruby Enterprise Edition. We're seeing dramatic improvements in memory usage and response times, and hopefully the application feels a little snappier and more responsive. Passenger is also making life easier on us in several ways.

Source Control

We made the transition from Subversion to Git, and consequently GitHub. We also ironed out a few other source control things along the way. The bottom-line is that source control is going to be much less of a headache going forward.

Backups

It's something you always hope that you never have to think about, but backups are important. We've gone from daily snapshots to doing daily and weekly snapshots keeping the last 7 days and last 4 weeks, and we're keeping copies of those backups on S3. This is all on top of the Raid configuration. Long-story short. Our backups are now redundant and distributed. (Naturally, the backups are encrypted as well since they're being pushed to S3.)

Monitoring

We've been using Monit since we launched, but our outage two weeks ago exposed the fact that a local monitoring service wouldn't help if the entire machine goes down. So, while we're stilling using Monit, we've also begun using Pingdom as an external monitoring service dedicated to keeping an eye on everything and letting us know if anything is offline.

Summary

That's really just a quick overview of the more significant changes that we've made, but rest assured that we've done much more than that. We just don't want to bore you with details. We also owe some thanks to Ryan Schwartz for his help with some of the fiddlier bits of server administration. He definitely helped make life easier. Now that all of these improvements are behind us, we're free to focus on the application itself. Thanks for your patience during the transition. We've got some big plans for the coming months.

Unscheduled Downtime

Fri, 01 May 2009 06:02:02 -0600

Fortunately, the root cause was simple to correct. Unfortunately, we only had one layer of monitoring in place, and that monitoring was local to the machine and relied on the machine being available. Since the machine itself was affected, those alerts did not go out, and we found out about the problem much later than we would have preferred.

We now have an external monitoring service that will alert us immediately whenever there is any downtime in the future so that it can be kept to an absolute minimum. While it's almost impossible to maintain 100% uptime, we certainly don't see last night's outage as being an acceptable occurrence. We are incredibly sorry about the inconvenience, and we definitely don't take it lightly.

Phased Ambitions

Mon, 27 Apr 2009 07:07:47 -0600

Since I began thinking about adding milestones to Sifter, I've been thinking about some really cool integration throughout the interface. That thinking led to a significant amount of time sketching and designing how they would work. The more I designed, the more ambitious the implementation became. The more ambitious it was, the longer it was put on the back burner.

Recently, though, I realized that I had taken it too far. There was a way to implement milestones quickly and easily and get the functionality in everyone's hands in a matter of days. Once I saw the light, I realized that there was absolutely no reason not to implement milestones in a phased approach. The first pass would simply get the functionality into everyone's hands. The later passes will focus on more seamless integration into the interface.

This phased approach will get the rudimentary functionality out there sooner. More importantly, though, it gives me and everyone else some time to use milestones and think through how the new improvements should work. I've got some ideas, but real world usage can only help improve those ideas.

The moral of the story is that it's important to recognize when you might be trying to bite off more than you should chew right now. This is a feature that many of us want and need, and I can't justify delaying a good enough solution while chasing down the perfect solution. So, instead of trying to hit a home run and cram it all in one release, we can focus on hitting singles and making constant tiny improvements.

If all continues to go well, milestones should be available sometime this week with plenty more improvements to follow shortly. Thanks for waiting while we got it sorted.

The History of Sifter

Tue, 21 Apr 2009 17:18:14 -0600

I'm still finishing up my existing client projects, but I've stopped taking any new work. Starting in May, Sifter will begin paying all of my bills. It's not quite a full-time salary, but it's well on the way. Hopefully this will be useful for others out there looking to go out on their own as well. I've put together this timeline of our significant events as well as our revenue growth. (You can click the image for a larger view .)

During the beta we only charged $5 for paying accounts so that we were making sure the billing system was working well. The bars during those months represent the revenue that we would have made if we had been charging full-price during that time. There are a few additional points that are potentially interesting for someone else interesting in a hosted web application business.

Sifter was profitable as soon as it launched. I wasn't making a salary, but Sifter has been comfortably covering its own bills since the day we went live.
I developed Sifter part-time with some basic assistance from others. This kept our costs very low and also gave us a lot of flexibility as there wasn't a need for coordinating multiple people who were working other jobs. It also imposed a significant constraint in that I was the bottleneck for everything.
During the 16 months of working on Sifter part-time and consulting part-time, I was working about 60 hours per week and effectively making about 50% of what I made before quitting my job.
Sifter didn't pay me any money at all until April 2009 when it began paying me about a quarter salary.
All of the initial startup costs were covered by the initial investment from my business partner. Those included all of our legal work, business insurance, hosting, and some other miscellaneous expenses.
Advertising was effective and worked well for us, but I wouldn't bet the farm on it. It would take a significant outlay of advertising dollars to match the results that we've seen simply from blogging about the design process. Advertising is a good compliment to other growth strategies, but I wouldn't advise relying on it as your only growth strategy.
Next Update is profitable and debt-free. At this point, we're confident that we'll continue to grow organically with no need for any additional capital.

Long-term, we'd like to share more, but for now we're easing into it. While it hasn't been a walk in the park to get here, it's certainly possible for a web application to be developed, launched, and support an employee at a modest salary in under a year and a half. With more design or development help, we probably could have accelerated that timeline, but we'd have more employees to support and more pressure to grow. As it stands right now, we can grow sustainably without any unnecessary stress, and we don't have any debt looming over us.

Hopefully that helps provide some insight. If you have any other questions, let us know. Unless it's about the number of accounts or precise revenue numbers, we can probably answer it.

Choosing Features

Wed, 08 Apr 2009 01:25:18 -0600

Philosophical Alignment

The single most important factor for us is whether the feature matches our vision of where we want Sifter to go. We don't simply think about whether people want it, we think about whether we want it. We use Sifter ourselves, and that means that most of our decisions are based on our experience. Long-term, we want Sifter to play a significant role in supporting our business. As a result, we're more likely to add features that support web app development because that's what we know.

While this can be a contentious area, I believe it's one of the most important aspects of software and product development. It's obvious when a product has crammed in every single feature request that comes along. It's more obvious when they did it as quickly as possible without giving much thought to the value or alternatives. Trying to please everyone is a fool's game. You might widen your audience, but in doing so, you're alienating what would otherwise be your most passionate supporters.

Thanks to the breadth of the internet, even the smallest niches can comfortably support a software business. We'd like to make money off of Sifter, but we're in it first and foremost for the love of what we're doing. Oddly enough, when you do that, the money usually shows up at some point.

Business Value

The second factor is simply business value. That is, will this new feature improve the experiences of a significant number of customers or potential customers? Then, the number of customers helps determine the priority. Also, if there is a reasonable workaround without writing any additional code, there's less business value to a new feature.

For instance, Searching and uploading files are foundational features that most everyone will use at some point. They are features that will help everyone with very little drawback beyond the fact that they add complexity. They were important features with very little subjectivity.

However, adding Textile or Markdown would only help the portion of our audience that is formatting-saavy. Even among those customers, many people copy and paste error reports or portions of log files into Sifter, and Textile and Markdown would butcher those. Then, there are people who aren't familiar with text formatting options. They end up being surprised and confused when the formatting language changes their text in unintended ways. So, text formatting, implemented in a basic way, is a draw. It helps some and hurts some. So, we haven't done it. (That doesn't even go into the problem that everyone has a different preference.)

Level of Effort and Impact

The other significant facets when choosing a feature are the level of effort to implement it and the scope of the impact to the system. (There's usually a close correlation to the two but not always.) If something passes the business value test, takes only a couple of minutes to implement and only affects one page of the interface, chances are that it will make it in quickly even if there are other big features in front of it.

However, if it touches multiple pages and components and take a couple of weeks to implement, then we're going to sit on it for a while and think about it. We'll design and explore a few different options so we can understand the tradeoffs. Sometimes, we'll even create a dummy app in a sandbox just to play with our ideas.

For us, the key thing with this process is that we're not afraid to pull the plug or delay a feature at any point. Unfortunately, sometimes, even after you decide a feature is a great idea and a good fit, digging into the implementation might prove otherwise. If we think something is going to take one week, and we discover it's going to take four weeks, we reevaluate whether it's still worth it or if other features are now more valuable.

Degree of Implementation

Working hand-in-hand with level of effort is the "Degree of Implementation". That is, with most features, you can create a very simple watered-down version or you can create a really complex and fancy version. We want to aim for the sweet spot, but we also don't want to underdo it. If we have to choose between a half-ass quick and dirty implementation or delaying a feature for a month or two, we're going to delay it. We believe that if there isn't time to do it right, there won't be time to fix it.

So, while we'd love to get new features in your hands as soon as possible, we only want do push them when they're worthy of being in Sifter. We can build them quickly to make people happy with our delivery timeline, or we can take our time and make sure people are happy with the actual feature. The latter is a more sustainable model, so that's what we do.

Summary

It's a constant balancing act to please people and build a sustainable business. Time, revenue, operations, customers, and personal goals all compete for attention. If we neglect any of those, Sifter wouldn't survive for long. We can't work on features all of the time, so we strive to be very careful about where that times goes when we have it.

Suggestions for Making Feature Requests

Tue, 07 Apr 2009 23:07:03 -0600

This isn't based simply on my experience with Sifter. It's based on behavior that is visible everywhere. Whether it's software or hardware, there seems to be a limitless supply of baseless negativity from some people when the product is missing their pet feature. Instead of opening a dialogue with the creator, they just unilaterally dismiss it or create a condescending or indignant feature request. For everyone's sake, we can do better. (Note: Support requests are entirely different from feature requests.)

Making New Feature Requests

Nobody's perfect, and I'm sure that over the years I've been as guilty of the following things as the next person. However, I've learned a lot over the course of the last year developing Sifter, and it seemed it was about time to share that experience. For the most part, you might think that these are obvious common courtesies, but you would be wrong.

Make a suggestion, not a demand. This doesn't just apply to feature requests. If you approach any situation with a demand, you're starting off on the wrong foot. It's a called a feature request not a feature demand.
Recognize that the developer is balancing many requests. As of writing this post, Sifter has almost 4,000 users. That's not much in the grand scheme of things, but it's enough that I've learned that not everyone wants the same thing. Fulfilling your request may only make you happy at the expense of other customers. It's a balancing act, and while every suggestion and request is important, yours is rarely the only one that the developer has to consider.
Remember that there are humans behind the software. Any feature request you make is going to a human being, and it's more than likely that the people behind the software care deeply about making it the best that it can be. Assume that they do care and they do have the best long-term intentions in mind. Talking to them as if you feel they are actively sabotaging their product doesn't help make a case. Be friendly and civil.
Don't tell them how to run their business. Focus on your need, not their revenue. Telling a developer that a feature would "clearly lead to more customers" or that they're "missing out on additional revenue" is way oversimplifying. With each new feature, there's just as much potential for downside as their is for upside. In fact, if you feel that it's necessary to tell them that, thinking they aren't aware of the business implications, you might as well just open the request with, "I think you're incompetent." I can't speak for everyone, but that's never worked well for me.
Assume that your request is incredibly complex. Just because you're familiar with the technology or platform doesn't mean that you know the application inside and out. Every product is unique, and only the developers will know the full impact of a new feature. Hell, sometimes even the developers don't see a significant portion of the complexity until they're halfway into implementation.
Be constructive. Talk with them, not at them. Acting indignant or condescending about a feature that is "missing" because "XYZ Software has it" makes for a poor discussion. Now, referencing another application and saying, "Hey, I really like how they did this because _______. Is there any way you could offer something similar?" expresses the same thing, but accomplishes much more.
Details matter. Just because your feature request is one word, doesn't mean the solution is simple. In fact, the less information you provide, the more room you leave for misinterpretation. Mention specifics and do your best to provide clarification and details about your needs. The more information you provide, the happier you will be with the result. It also shows that you care and aren't just firing off a half-baked whimsical request.
Talk about your problem or goal instead of prescribing a solution. There's nothing wrong with suggesting a solution, but it's much more valuable to the developer if you describe the problem you're encountering or the goal you would like to accomplish. Understanding your behavior and usage of the system is a priceless component for helping the developer help you.
Offer alternatives and keep an open-mind. If you want to request a feature, suggest multiple implementation options that could solve your problem. By thinking through other options, you'll invariably be able to come up with a better solution, and you'll help the developer gain some additional insight into your needs.
Participate. If you really care about a feature request, the best way to make your case is to invest some time in it. Do a little research. Maybe mock it up. If the company has public forums, see if anyone else has made a request. See if the developer has shared any information about their decision to exclude that feature or if it's something that they've openly stated is in the works. If they don't say yes right away, and you care, continue the discussion. If you don't care enough to do more than fire off a one-sentence feature request, you're not making much of a case, and it probably won't go far.

That's probably not an exhaustive list, but those are things that come to mind. The list is based on the hundreds of requests I've read since launching Sifter as well as the hundreds of comments I've read on other developers' blogs and forums over the years.

It's important to remember that just because you've eloquently made your case and presented a flawless solution to your problem doesn't mean that it will make it into the application. Any software company still has a philosophy and vision that will influence their decision to include, exclude, or delay a feature. It doesn't mean you were wrong or that your input and feedback was any less valuable. It just means they have a diferent vision. However, that's a topic for another day.

Summary

Whether we all realize it or not, receiving user feedback is one of the most important aspects of software development. The day that people stop caring enough to share their thoughts is the day that your software has become irrelevant. However, ill-conceived, rude, or otherwise poorly worded feature requests really take the joy out of pleasing people. (Remember, these are feature requests, not support requests.) Instead of feeling like you're granting their wishes and making them happy, it feels like you're merely mitigating their misery. I think it's safe to say we'd rather spend our time granting wishes.

Choose Your Own Adventure

Sun, 05 Apr 2009 09:51:07 -0600

Features. It's such an innocuous word. New features can usually be summarized in a single word, and therein lies the problem. For instance, implementing Search was much more complicated than its one-syllable description might imply. Most features unfold in a similar fashion.

Whenever designing a new feature, there are always different paths and approaches that you can take. Which path you choose is the single most important factor in determining whether that feature is ultimately successful. Implementing features isn't binary. i.e. Implemented or Not Implemented. Implementing features has every imaginable shade between complete failure and raging success.

The due diligence that goes into the feature up front and the path you choose as a result are the best way to ensure it's on the success end of that spectrum. Whereas the best way to fail is to rush out a knee-jerk half-baked solution.

Since creating Sifter, the hardest lesson to learn is that it's alright for us to stay the course. I desperately want to crank out feature after feature to please people, but deep down, I know that while everyone might be happy with the timeline, they won't be happy with the result. Both are important, but the latter is what we really care about.

We want to add features. However, we want to add them Sifter-style. That is, we want to explore and find options that fit the vision. We're dying to make Sifter even better, and we couldn't be more passionate or excited about the future. We're as eager as anyone to get there, but we don't to be sitting on top of a house of cards when we do.

April Status Update

Thu, 02 Apr 2009 07:59:52 -0600

We're not sure of dates or timelines on any of the following features or ideas, but since we're really excited, I wanted to make sure that we're open about what's going on behind the scenes.

Client Work

For the immediate future (the next few weeks), the reality is that I'll be spending the majority of my time doing client work to pay the bills. While that means that things will be relatively quiet on the Sifter front, we're in the process of designing and prototyping some significant new features.

The more exciting news is that Sifter is getting every closer to supporting me full-time, and when that happens in the next month or two, the sky's the limit. In the meantime, though, we're having to be very mindful of how we spend our time.

How We Approach Features

Building software for the long haul isn't trivial. We spend significant amounts of time refactoring and improving code, designing and prototyping new features, and exploring new ideas to see how we can make Sifter great. The things we learn along the way invariably have an impact on our decisions.

It's a Marathon, not a Sprint

We will never rush features out the door just to get something in your hands. We'd love to crank out features overnight on request, but we're trying to be much more prudent about that. As a result, it takes longer for us to settle on what we believe is the right design and implementation. We like to release features when they're ready instead of focusing on a deadline. If we seem unresponsive with new features, it's not about neglect. In fact, it's quite the opposite. It's about being cautious and thoughtful about the impact of our decisions.

Learning and Adapating

We spend most of our time designing and prototyping features. As a result, we often learn that while those features sound great on paper, Sifter isn't really ready for them. This makes it tough because we want to be as open and transparent as possible, but our priorities are constantly in flux as we gather additional information and learn how new features will interact with existing features as well as planned upcoming features.

Our Vision for Upcoming Features

Email

It's easy to talk about "email" as a feature. It's one word, so it sounds really simple. Unfortunately, like implementing search, there are a lot of moving parts. However, that's the easy part. Our biggest challenge with email is enabling a simple, intuitive, and natural way to do more than deliver text updates. How do you update status, priority, category, or assignee via email?

We think priority and assignee should be straightforward, but status is the most important, and we haven't thought of a way to update status via email that we're satisfied with.

We have some ideas, and we've spent some time exploring them, but we didn't want to invest a significant amount of time creating a crippled feature as a supplementary interface. Email, while important in our long-term vision, wouldn't be enabling capabilities that couldn't already be performed. So, it's on the back burner for now.

Help Desk

The single biggest point of confusion with Sifter is that many people see a bug tracker and help desk as interchangeable, and Sifter isn't a help desk. Help desks are public facing whereas bug trackers are for internal teams.

We've known that we want, to some degree, to add functionality to Sifter to support some of the common help desk tasks. We need this ourselves to manage Sifter's growing customer base. This has actually become one of our top priorities for several reasons.

First and foremost, a significant portion of feature requests are indirectly related to or based on the assumption that Sifter works as a help desk. The truth is that since it wasn't designed to be a help desk, those features couldn't be added in a reasonable way. It also means that there are a significant amount of people out there that like Sifter, but can't use it because it's not a help desk. Or, they have to awkwardly concoct a workaround within Sifter. That's no good for anybody.

Additionally, if we release the other features before we add this, many people will be using Sifter in unusual ways in an effort to shoehorn it into being a help desk that would make it difficult to transition to this feature and use Sifter in a more sustainable fashion.

The one caveat with the help desk is that we are planning on treating it as an optional for-pay feature. It won't be a stand-alone product as it will be tightly integrated and reliant upon the bug and issue tracking portion of Sifter, but because it's almost a separate product.

A significant majority of our customers are consulting companies that have no need for a help desk. This makes sense because that was the exact group Sifter initially set out to help. So, after thinking through many different options, treating the help desk portion of Sifter as a for-pay add-on was the best way to support product companies without pushing what will essentially be a new product on our existing customers that have no need for it.

Sifter's origins were based on my experience and desires from years of consulting, but, since we're a product company, we've become acutely aware of Sifter's shortcoming for managing a public-facing product. This is a feature we're dying to have ourselves, and we're really excited about it.

Milestones, Categories, and Tagging

We've spent an absurd amount of time thinking about solutions for milestones. We've also been giving serious consideration to migrating from a single category model to tagging. Ultimately, the solution to all of these will be inter-twined. They present very similar challenges and the problems share several important attributes. However, the most important factor is that this is something where we're feeling the pain ourselves. We want to make this happen, but we want to do it right.

API

The final major feature on our radar is the API. It's important to us on many levels, but we also don't want to rush an API out the door just before we make sweeping changes to the foundation of Sifter. With the API, there's also the aspect of documentation. Just like we wouldn't want to release a crippled feature, the documentation will be a significant piece of the API, and we want to make sure we have a good place to put it and that it's designed in a useful and usable manner. That leads to the next "feature" that we're working on.

NextUpdate.com

This site was designed and created before we launched Sifter. We didn't even have a name or a domain for Sifter. The site worked, but we've outgrown it now that we have a working public application. Our vision is to continue sharing design decisions behind Sifter and, in most cases, share the design ideas before we've even built the features so that everyone can share their thoughts and offer feedback.

Unfortunately, the current design and content management solution don't come close to the vision that we have for it. We need a place for a status blog and a wiki for API documentation and common support questions among other things. We've finished the design and wireframes for the redesign, but we're focusing on Sifter for now.

Long-term, we see NextUpdate.com and the blog as one of the most important things we do. Sifter started by sharing some comps and ideas, and it worked really well. We'd like nothing more than to get back to sharing comps ahead of time and refining our ideas with your feedback before we push them out to the world.

Summary

We're burning the midnight oil. Between SXSW and client work, we haven't pushed any significant new feature updates in several weeks, but we're working hard on getting these features out the door and making sure they're well done. It's scary sharing information like this knowing full well that things can and will change and adapt as the process continues, but we'd rather share our plans openly than leave everyone in the dark.

Help Desks vs. Bug Trackers

Mon, 09 Mar 2009 13:23:00 -0600

Depending on the application, help desks and bug trackers have roughly 80% overlap in terms of basic functionality. However, that last 20% makes a big difference in determining whether they're truly good at one or the other.

Individuals vs. Teams

Help desks are about supporting an individual. Whether that's a question, a support request, or just comments, each support request stands alone. The individual's request history may be relevant, but it's still focused on the individual. As such, with a help desk, people's visibility should be limited to only public issues or those that belong to them.

Bug and issue trackers are about multiple people collaborating on a shared goal. So, it's important for everyone to have visibility into each other's work loads. This is the single most important defining characteristic because it affects and informs all of the other design decisions.

Limitless vs. Discrete

With help desks, you can't always predict who your users will be. For instance, if you sell software, you could have thousands or millions of users. As a result, you need to either accept anonymous or semi-anonymous requests or provide a means for users to register and create an account.

However, complete anonymity is a dangerous idea. A significant number of requests require additional details or clarification because very few people are thorough enough to submit all of the necessary information on the first request. If you can't follow up with someone, you can't get this information.

With a bug and issue tracker such as Sifter, there is a basic assumption that the people involved are on a well-defined team with a discrete number of team members. Under these circumstances, it is possible for each team member to have a manually-created account.

Since everyone in Sifter has their own account, the concept of accountability is baked deeply into Sifter. We've always felt that the best way for work to get done is for one person and one person only, the assignee, to be responsible for the solution while one person, the opener, is responsible for the problem, or, more accurately, facilitating the solution and providing the necessary information to the assignee.

With an anonymous request, there's no way to follow up for that additional information. In fact, even if you think you've fixed the problem, you have no real way of knowing whether you did indeed fix the problem for that individual.

Open-Source

The exception to the rules is open-source software which needs and involves elements of both a help desk and a bug tracker. The participants are limitless and undefined, and everyone should have access to all of the issues. Similarly, there will be active participants who both discover and fix bugs, but there will also be people who are only interested in sharing their individual needs and are not necessarily interested in the project as a whole. These are subtle but important differences that complicate things for us. Ultimately, we'd like to support open source projects, but we have to focus first and foremost on our core goals.

Summary

We'd love for Sifter to work great as a help desk, bug and issue tracker, and open-source bug and issue tracker. It would obviously increase our potential customer-base, and what business wouldn't want that? However, we also firmly believe that if you chase two rabbits, you won't catch either.

Our decision to make Sifter what it is wasn't accidental or contrived. We built the tool that we always wished we had in order to work with clients. We understood that we couldn't be everything to everybody, so we chose to focus on supporting a client/consultant or developer/stakeholder relationship for well-defined teams.

This isn't to say that Sifter can't be used as a help desk, but it's definitely not designed to behave like one. Every decision about responsibility and workflow has been carefully considered and explored, and we've had to make some decisions that impede Sifter's ability to work as a help desk in order to make sure that it really works at being a bug and issue tracker.

Implementing Search

Sat, 07 Mar 2009 09:03:48 -0600

We recently released an update to Sifter that included search, and despite the deceptively simple name of "search", it's actually a fairly complex feature. I wanted to run through everything involved in getting search up and running. This was written to help people who are completely new to search as well as providing a behind the scenes look to some who already have a deep understanding. As a result, parts of this will be very basic, some parts will be fairly technical, and other parts will even talk about some of the interface decisions.

I'm not planning on going into the decision-making process of choosing Sphinx and Thinking Sphinx for our search implementation, but I will say that it was carefully considered and researched. There are advantages and disadvantages to every option, and everyone has different needs. This combination just happens to be the one that works best for what we wanted to do.

How it All Works

Before I get too carried away talking about the technical bits, it's worth looking at the big picture of how everything fits together. We chose to use Sphinx and Thinking Sphinx for Sifter. I'm going to gloss over some of the finer points in favor of keeping this simple.

Sphinx

Sphinx, by Andrew Aksyonoff, is a full-text search engine. It includes both indexer and searchd which, unsurprisingly, handle the index creation and search requests respectively.

Thinking Sphinx

Thinking Sphinx is a Ruby library by Pat Allan that plays the go-between for ActiveRecord and Sphinx including rake tasks for configuring, indexing, and starting/stopping the search daemon.

Pieces of the Puzzle

There are several elements that come together so that everything works. I'll be focusing exclusively on our specific so setup. We're relying on delta indexing, which is optional, but since it's more complex than not using delta indexing, this is a safe superset of what can be done.

Adding search to a web application involves several moving parts that involve some external coordination.

Configuration - Much of the configuration is handled within your models, and the model-independent options are configured in a sphinx yaml file in config. (/config/sphinx.yml). Running the rake thinking_sphinx:configure rake task will generate the actual Sphinx configuration file for you based on the settings in your models and your Sphinx yaml file.
Search Daemon - This is the daemon that actually handles the search requests.
Indexer - The indexer creates the index. In order to keep the index up to date, we have to regularly run the indexer, which in the case of Thinking Sphinx, runs a full reindex each time.
Index - The index is just like the index to a book. It's a series of files created by the Sphinx indexer so that it can efficiently perform searches.
Monit - Occasionally, the search daemon will die, so Monit keeps an eye on it to ensure that it's resuscitated in these rare situations.
Delta Indexing - Because the index is maintained separately from the database, by default, it won't reflect changes to the database until the next indexing occurs. Depending on your configuration, that could be hours or days. Delta indexing enables your app to essentially update the index every time a searchable record is saved by maintaining a delta index in addition to the primary index. Then, when a search request is sent, both indexes will be searched. The disadvantage to this is saving records will inevitably slow down as the size of the delta index increases. Regularly re-indexing helps ensure that the delta stays small and keeps performance high. There are other options, but I'll be focusing on our implementation.

Installation & Setup

With all of the moving pieces, Sphinx isn't trivial to setup, but it's really not too bad either. I've tried to pull together the resources that I found most helpful at each stage of the game so that it's easier to get through the process. All of these steps are well-documented elsewhere, so instead of reinventing the wheel, I just wanted to pull together all of the relevant resources based on the process I went through setting things up for Sifter.

Installation

Installation was straightforward, and both Sphinx and Thinking Sphinx have fantastic installation instructions.

Configuring Sphinx & Thinking Sphinx

For configuration, there are extensive details on both Sphinx Configuration and Thinking Sphinx Configuration which explains how to configure Sphinx through the sphinx.yml file.

Configuring Your Searchable Models

The Thinking Sphinx usage guide covers most of the basics on how to setup your models. The most significant aspect of this configuration for us was enabling delta indexing which takes advantage of Rails callbacks to automatically update the delta index after a relevant record is saved or updated. While we went with standard delta indexing, several additional options for delta indexes are available.

Monitoring the Search Daemon

From time-to-time, the search daemon will die, so keeping an eye on it is important. Adding a section to our Monit configuration file for the search daemon was fairly straightforward. I've include the relevant section of our Monit config file below. Make sure to replace anything in all caps with the values for your system.



check process searchd with pidfile /YOUR_PATH_TO/searchd.pid
   start program = "/YOUR_PATH_TO/searchd --config /YOUR_CURRENT_PATH_TO_RAILS_APP/config/ENVIRONMENT.sphinx.conf" as uid USER and gid GROUP 
   stop program = "/YOUR_PATH_TO/searchd --stop --config /YOUR_CURRENT_PATH_TO_RAILS_APP/config/ENVIRONMENT.sphinx.conf" as uid USER and gid GROUP
   if failed host localhost port 3312 then restart
   if 3 restarts within 5 cycles then timeout

Initiating Regular Indexing

Even if you're using delta indexing, you'll still want to regularly reindex because the larger the delta gets, the slower your saves and updates will become. Cron is the natural choice here. You'll just need to setup a cron task to run the rake thinking_sphinx:index task on a regular basis and you'll be all set. The indexer is incredibly fast, on a small site, you could probably get away with running it every 5 minutes or so without any problems. However, with delta indexing turned on, frequency is less important, so we're running it once per hour.

Managing Deploys

While it wouldn't be the end of the world to reindex with every deploy, I wanted to avoid having to rebuild it every time. Instead, we're following in Wade's footsteps and keeping the index in our shared path and then creating the symlink to it on each deploy.

Wade mentions in hist post that Thinking Sphinx generates the actual Sphinx config file from your models and sphinx.yml using rake thinking_sphinx:configure. The resulting file will be used when the Sphinx search deamon is started. So, with each deploy you'll want to manually rebuild the Sphinx configuration file in order to ensure that any relevant changes in your models or sphinx.yml get updated within the file that Sphinx uses. So, just after the symlink and before we restart the Sphinx search daemon, we make sure to run rake thinking_sphinx:configure to make sure it will start itself up with the new settings.

Additional Resources

I would have never made it this far without the following resources, and of course many thanks go to Andrew Aksyonoff and Pat Allan for providing such wonderful tools to the community.

Search and Cross-project Issue Filtering

Tue, 03 Feb 2009 09:50:36 -0600

This morning we rolled out search and cross-project issue filtering. There's not much more to it than that, but I wanted to give a quick overview of the highlights from the release.

Search

One of the most challenging aspects of adding search was deciding how to scope the search. Sometimes, you might want to search all of your open projects. Sometimes, you may want to include archived projects. Other times, you may want to just search within a single project. So, depending on the page that you're on, search will function slightly differently.

If you're on a project page, the search will limit the results to just that project. However, there's always a link to expand your search to all projects. Otherwise, if you're on a global page like the dashboard or account management, Sifter will search all of your open projects. From the results page, you can choose to expand the search to include archived projects.

In addition to being a search box, it's also an issue shortcut. If you type in an issue number, the issue number exists, and you have access to it, you'll automatically be taken to the issue page.

The search box is somewhat intelligent. You can type an issue number and jump straight to that issue. By default, it searches all projects. However, if you are within a project, it will limit the results to that project.

Results and Filtering

The search results pages have some selected filtering options that will vary depending on whether you're searching one project or multiple projects. For instance, since categories can be customized for each project, it didn't make sense to enable filtering by category when you search all projects.

The cross-project issue filtering is available from the dashboard.

Additionally, when you're searching multiple projects, each search result will include a link to the project just below the issue subject. Naturally, if you're searching within a project, this link isn't there because it would be redundant.

Technical Bits

While I plan on writing a more in depth blog post about the design and technical aspects, I wanted to share a little bit for the developers out there. We're using Sphinx and Thinking Sphinx. We're taking advantage of delta indexing so that search results are immediately available, re-indexing every couple of hours via a cron job, and keeping an eye on the search daemon with Monit.

Fluid Icon for Sifter

Tue, 27 Jan 2009 23:17:32 -0600

For those of you who use Fluid to create site specific browsers for your web apps, we've finally created a high quality icon for you to use with Sifter. When we set out to create the original icon, we intentionally wanted the style to be something that would work well as a dock icon for a Fluid app, but somewhere along thew ay we forgot to think about the overall shape.

The original icon was too wide and short to fit in well with the other larger icons.

As a result, the icon never felt quite right in the dock. It haunted me, but I wasn't sure what we'd be able to do. Fortunately, Jared Christensen is a genius and came with numerous workable concepts. We waffled a bit, but pretty quickly settled on the direction that we thought was right for a dock icon. Of course, Jared has provided more detail on the design process over on his site.

The new fluid icon is a slight departure, but still carries the same basic visual style. Download the Full-size Icon

The result was something that strays a bit from reality, but we felt the creative freedom was necessary to create something that felt like it belonged on the dock. So, without further ado, download the icon and get going with Fluid. In the future we'll be adding the proper link elements to the pages so that Fluid will automatically recognize the icon. However, it will be a little while before we do our next release, and I didn't want to let that hold up sharing it.

Post-Launch Update

Wed, 10 Dec 2008 11:14:57 -0600

In terms of our vision for Sifter, we're just getting warmed up. We've got a plethora of features to implement, big ideas for improving existing features, and lots of stuff to share. While we've been pretty heads down during development, we want to share more and be incredibly transparent about our vision and what's going on with Sifter.

Getting Satisfaction

We've been receiving lots of great feedback and suggestions on features and improvements. The best way for us to gauge what we'll focus on next is if you share your thoughts and get involved in the conversation at Get Satisfaction. While we're not attaching timelines or promises to features, we're pretty open about our intentions and vision for Sifter.

Since there are only so many hours in a day, we have to prioritize the features, and the number of me-toos on Get Satisfaction will play a big role in influencing the roadmap. So, if you have any ideas or questions that aren't related specifically to your account, we strongly encourage you to share via Get Satisfaction.

Currently

Now that Sifter is live and running smoothly, we'll be spending the next couple of weeks doing some refactoring and making improvements that will lay the groundwork for our next features. There won't be many visible changes on the surface, but rest assured that I'm slaving away every day making improvements under the hood.

Upcoming

We're not sure what's next for Sifter, but we are sure that there's no shortage of work to do improving it. The way we see it, we've only laid the foundation. Email integration, search, milestones, RSS feeds, an API, source control integration, 3rd party product integration, and more. We have a long road ahead of us and can't wait to provide new features to help make bug and issue tracking less tedious for you.