The ENTP Blog

From idea to shipped feature in an hour

zenhob — Thu, 10 May 2012 15:16:00 -0700

Sometimes the time between coming up with a kickass feature, having a “product planning meeting”, writing, coding, and shipping, is one hour
— c3 (@court3nay) April 21, 2012

While we were trying to scope a new Tender feature, we accidentally stumbled across a very simple and effective UI improvement that we were able to implement, test and deploy in the space of an hour. I figured I could introduce this (very small) feature with the (very short) story of how we built and shipped it.

The issue was thus: We provide per-queue notification settings for support staff. These only affect whether the staff member in question receives email notifications for a queue. However, these notification settings also represent something else. They’re a list of queues that are important or relevant to a particular staff member. Amazingly, we don’t make use of this information anywhere else! I know!

As things wound down in the ENTP office a couple of Fridays ago, Courtenay and I became so energized by this discovery. Courtenay immediately pushed his first pass at a topic branch for this feature. I reviewed and staged it. We went over the QA routine ourselves, since everyone else had mostly gone for the weekend. There was some confusion at first because I couldn’t find the right notification settings. This is why I don’t usually do QA. We pressed on, because we are intrepid and we kept the scope very small. We verified that it worked and shipped it to production to see it in action on our own support page.

Within an hour we had the following simple, new additions:

From now on when you elect to be notified about a queue, that queue will receive priority in the support dashboard UI under “My Queues”. Since we were keen to get a useful change shipped right away, we didn’t do any more than this for the first pass.

I’m not going to lie, we were already barfing rainbows over this, so great was our excitement at this easy win. But! There was one very important bit of functionality missing, in my opinion: What I really wanted was the ability to see discussions from all of my queues in the pending view at once. So we took a little more time and added that too.

We left ourselves greater leeway on the turnaround for this one, and we were able to get our usual QA approval before shipping. We were even able to pay off some technical debt in related code at the same time! This change has been very important to me personally as I find it to be conducive to my own workflow. It gives me an easy way to check whether any queued discussions that I should know about about are waiting for a response, and nothing else.

Like I said it’s the small things, or maybe we here at ENTP are hopeless workflow nerds. Possibly both.

New: JSONP knowledge base API.

courtenay3 — Wed, 18 Apr 2012 15:38:15 -0700

I’ve just added a callback parameter to our autosuggest json endpoint. This means you can now search your knowledge base from anywhere and integrate the results into your page. Furthermore, because it’s using the autosuggest, you can plug in a bunch of unrelated words and it will simply find the best match.

For example, you could parse the contents of your page and automatically suggest the best results for that page from your KB in the sidebar.

Here’s the URL:

http://your.tenderapp.com/search/autosuggest.json?q=query&callback=function

Here’s some sample code to get you started, using a script tag:

<script type="text/javascript"> 
function kb_search(results){ 
 $('#results').html(results[0].title); 
} 
</script> 
<script src="http://monkeys.tenderapp.com/search/autosuggest.json?callback=kb_search&q=how+do+i+find+api+results" defer="true"></script>

I’ve written a more complex example, that works with a form and a jQuery jsonp call at this gist: feel free to fork it and improve it. It attempts to minimize the number of requests sent.

How to set up and manage your notifications

bemmabobemma — Fri, 13 Apr 2012 12:16:47 -0700

Good day to you!

When creating a new profile in Tender, the email preferences default to notifying the user of changes in the issues/discussions they have participated in (either by creating or commenting on), but you can curate your notifications for any given category or queues that you wish by signing in and going here: https://help.tenderapp.com/profile/edit

Scroll mid-page where you’ll find “Email Preferences” and click as you see fit. A word to the wise: choose your notifications wisely, young Jedi. If you get click happy, you may find your email inbox quickly becomes inundated with things you don’t really care about.

Occasionally, users report being spammed by Tender’s support system. If this happens, have them check their email preferences before submitting a bug report. If they have any public discussion category notifications set, the user will receive all posts, regardless of their direct involvement in the discussion.

Find all the bugs

chronicole — Fri, 30 Mar 2012 16:12:06 -0700

Even before the beginning of a pull request for Lighthouse and Tender, TATFT is enforced. Red, green, refactor as the code is written is excellent but it won’t catch everything. Scripts are vital but can sometimes lead to complacency. I believe exploratory testing is a great tool in my web application testers tool belt. Unfortunately, it’s often misunderstood. Here are three common misunderstandings of exploratory testing:

1. Testers have to find bugs. Exploratory testing is like a hunting trip and serves no purpose.

The belief that testers *must* find bugs is false. My happiest day is when I go through a pull request the first time and can say “Ship it!”. The best exploratory testing is not blind exploring, monkey clicking through the application, but rather thoughtful investigation of the application by someone who knows the code specifications.

2. Exploratory testing is only good right before deploy.

Exploratory testing can actually enhance the automated test suite, by finding new uses of the application not already covered by existing tests. I see every bug report as an opportunity to beef up our test scripts.

3. Exploratory testing is done without scripts.

The best exploratory testing is based on carefully created scripts, but the wise tester knows when to stray from the scripts and follow the bug trail where it leads; being careful to track the steps to reproduce the bug in question.

Although it is very easy to dismiss exploratory testing as an inefficient testing method, most web application users are not as predictable as programmers would hope. Exploratory testing bridges the gap between automated testing and those unpredictable users.

Tender updates for March 27th

calexicoz — Tue, 27 Mar 2012 15:50:00 -0700

Hi everyone,

So what have we been up to?

Well for starters, we fixed the spam count: we had quite some fun with this one, and it should now accurately reflect the number of messages in your spam folder.

We also added a couple of endpoints to the API, and fix a big annoyance. You can now:

Delete KB articles
Create and delete sections
Use pagination for discussion comments (at last)

We’ve added the relevant documentation as well. I know some folks have been waiting on those, so let us know how this works out for you.

That’s it for today, see you next time with more goodies!

Lighthouse updates for March 22nd

zenhob — Thu, 22 Mar 2012 11:30:00 -0700

Since this is a much shorter period than the last change log, we have less to talk about this time, but here goes!

Most notably, ticket search has improved: While it’s always been possible to specify multiple responsible: specifications in a search, responsible:none only worked in isolation. It was not possible to search for the set of tickets assigned to someone as well as unassigned tickets. Well, this has changed! So for example, if you want to make a bin of your tickets as well as stuff that nobody has picked up yet, you can do that now.

Milestone due dates could exhibit some odd behavior for project members in different time zones, so we now anchor milestone due date comparisons to the account-level time zone. This means milestones will be due/late at the same time for everyone!

In some cases it was possible to silently truncate content for a page if it was too large in size. We’ve added validation to prevent data loss, but the size restriction remains in place for now.

We are also handling (and cleaning up) more instances of failed callbacks, reducing the load on our job runners. We have also continued to improve the stability of our export process for very large data sets.

Lighthouse updates for March 15th

zenhob — Thu, 15 Mar 2012 16:58:00 -0700

Hi folks, it’s been too long since the last Lighthouse update! You have me to thank for that, but I have bravely re-learned to blog and am finally here to fill you in. We are tearing through our backlog of issues, and we have plenty of stuff to tell you about!

Here are some highlights from Lighthouse releases in the last month or so:

Audited for mass-assignment bugs in every controller. I’ll have more on this later.
There is much-improved display of ticket changes, including inline links.
We’ve fixed the display for projects with thousands of tickets, so your ticket badges don’t take up 2 lines.
We improved the fidelity of table styling in preview, to match the final comment.
Added pagination to user- and account-level activity feeds, and improved pagination behavior in milestone views.
We’ve updated our OpenID back-end, which should go a long way toward improving interoperability.
We now avoid modifying ticket creation date when changes are rolled back.
Watchers were not being added/notified in some cases, this has been fixed.
We also fixed instances of subscribed watchers being re-added to tickets accidentally.
Refined auto-linking behavior in the markdown formatter, so that auto-links do not happen inside markdown and textile links.
We’ve restored reverse-sort functionality in ticket queries.
There were instances of HTML content appearing in text email notifications, these have been fixed.
Internal links now work properly while previewing.
Fixed a JS error being experienced by some Internet Explorer 8 user.
We’ve made numerous internal improvements to our export functionality, making data exports quicker and more reliable.
Improved handling/reporting of missing resources during API requests.
Fixed tag editing on Internet Explorer 9.

Bigger improvements and new functionality are on the horizon. I expect you’re as excited about the new hotness as we are, so stay tuned for more!

Lighthouse updates for February 17th

courtenay3 — Fri, 17 Feb 2012 16:56:00 -0800

Howdy! We have been shipping a bunch of updates to Lighthouse, dealing out bugfixes and tidying up in general; basically fixing a bunch of the nigglingest issues and making up words while we’re in there. Here’s what we shipped recently:

Lighthouse has been modernized and now runs on Ruby 1.9, along with a bunch of upgraded and moderner libraries for our whole stack: memcached, mysql, and more.
Callbacks - you may know them as web hooks - now have their own UI (previously only accessible via the API.) You can make callbacks from the account-level settings page. All ticket updates get a JSON post to any URL you choose. If a callback fails continuously, we will eventually stop posting to it.
Bulk edit failure modes got improved: if you create a bulk edit that has no actual tickets (i.e. if you create it twice, or the ticket got moved), it won’t sit around hassling you that it hasn’t run. Also, you can’t select archived projects as a bulk edit target.
CSV reports are now fully Excel compliant and are UTF-16, so your non-ASCII characters will work just fine, right-to-left, und so weiter.. There’s also a dedicated Excel button (it’s actually a TSV) as well as the same old CSV.
We upgraded our markdown processing library to the latest, which may fix some annoying formatting bugs.
We also upgraded our date parsing library and fixed some minor bugs there.
Attachments on tickets with more than 100 comments now show correctly.
Pagination on milestones is fixed and now actually works.
“Late” milestones is now an accurate thing that refers to the end of day rather than beginning (you’re not late til the day is done, man).
Saving default ticket bins no longer clears the default state.

Once we get these fixes out of the way we have lots of exciting lighthousey things to share with you!

Scheduled-ish downtime

courtenay3 — Tue, 13 Dec 2011 10:40:00 -0800

Both Tender and Lighthouse will be down for up to 30 minutes tonight from 8pm Pacific while we reboot some critical servers. While we can reboot most of the instances without causing downtime (in fact, we already have), there are some services that don’t handle failover particularly gracefully and so it will be cleaner for us to take the apps down completely rather than try to do it ‘live’. For example, we could easily fail over our redis cluster to the slave instances (they’re already used for reads) but there’s a risk that a few minutes of writes would be lost.

We apologize for any inconvenience due to the late notice - we’re not entirely happy about it either.

User Classes

courtenay3 — Fri, 11 Nov 2011 18:00:00 -0800

We’ve just deployed the User Class feature. This is our way of providing the minimum amount of access control and segregation possible.

You see, many of you have two main versions of their software or application: the normal version, and a beta version with features not generally available. You want to have articles and categories available to those users without revealing that information to the public.

You also tend to have regular users, and a separate class of important users — whether they’re paying more, need special handling, or you just love ‘em because they always open nice detailed and friendly support requests.

So, we’ve enabled a few toggles on your users, KB sections, and categories. You’ll need to enable it like so:

Go to your site settings. Select “Enable beta users” and hit save.
Go to your users page (settings/users), and check some users’ checkboxes. Set the Beta Access, or Important User, flag. (also can be set in the API and SSO hash)
Go to your categories admin page, and edit or create a forum category with the same toggle.
Go to your KB sections admin page, and edit or create a KB section with the same toggle.

Any sections, KB articles, categories, or discussions flagged as beta or important won’t be visible to other users. Users who have access to see those sections and categories will have the results show up in searches, autosuggest and regular browsing.

Finally, while we haven’t enabled this on discussion filters (yet!) we HAVE shipped something else that’s even more useful. You can now design your filters around the ‘extras’ hash that is attached to each user and discussion. So, for example, lets say you set a user with the extras hash of { plan: ‘Premium’ }

You can now create a filter that will check for key/value pairs on new discussions, like “plan=Max” or “name=Joe”. You can even use regular expressions (name=~Joe) and have multiple statements (name=~Joe;plan=Premium).

This feature should be (somewhat ironically) considered beta quality for the next week or so, but I’d love to hear how you’re using it or how it doesn’t quite meet your needs. As always, open a discussion on our own forum at help.tenderapp.com or email us at help@tenderapp.com.

Scaling - a series of small, incremental evolutionary changes...

courtenay3 — Tue, 08 Nov 2011 00:55:00 -0800

Scaling - a series of small, incremental evolutionary changes interspersed with intense “oh shit!” moments.

We’ve recently undergone a significant change in architecture in response to a series of bad downtime. You can see above what it used to look like: a fairly simple setup, with a few app servers, and what it looks like now. We’ve added a few important changes to the way everything works, and we’re rolling these out in stages.

Tender has been growing at a fantastic rate (doubled in size this year) but the hardware infrastructure hasn’t really changed - as we’ve grown, we just increase the instance size of the master db and added app servers as necessary. This has worked excellently because it doesn’t require any downtime. But as you can probably tell, the old infrastructure had many points of failure. The main two issues we’ve seen are (a) the haproxy machine going down but not failing over and (b) high IO on the database master EBS disks.

We can’t fix the failover issues - that’s the domain of our webhost, and they assure us it’s fixed. Regardless, we’ve upgraded our support plan with them to super-mega-premium, so we can get expert help a bit faster (an additional $2,000 per month! There goes the summer home in the Bahamas!).

The database/EBS IO issues are unavoidable on EC2. Sometimes the disks are just slow which means random queries like ‘update sites set updated_at = NOW() where id=12254’ take fifteen to twenty seconds. In combination with the latency issue we also outgrown our web host: the largest machine that EC2 offers, is the quad extra-large has 68gb of RAM, but our largest tables (comments and discussions) each have indexes larger than this, which means a lot of mySQL hunting on disk for the data. To make things worse, Engineyard, who provides the stack we’re using on top of EC2, doesn’t support running multi-master replication or even having multiple writable database servers. And since we’re on the largest machine that EC2 offers, we’re prevented from growing with the existing strategy. If we hosted our own gear, we could easily build yet another bigger server. But we aren’t.

As it turns out, we can work around this limitation with noSQL. It involves managing a lot of the in-memory data MySQL used to handle by building things ourselves in Redis, which is ridiculously easy to scale horizontally. Here’s how we’re doing it:

Store as much as possible in Redis, which means, store as much as possible in RAM. Redis makes doing big intersections of lists (i.e. all discussions in this queue, on this site, in this state) super fast and easy.
Minimize database table and index scans, again, by building our own indexes (e.g. all discussions by this user_id) in Redis instead of mysql.
Add a bunch of redis instances and servers dedicated to running Redis. Redis and its ruby library makes this super easy again with consistent hashing. We’re making up to 32 instances per server.

Finally, we can also add readonly slaves to mySQL (and hope that replication doesn’t get too far out of sync), and move most of our reads to those. We already use background jobs to process most writes to the database already, so this was easier than expected. (on Friday, we were down due to a memory leak in the sharding library. D’oh! Two steps forward, one step back)

So, implementing these improvements include all your standard scaling progress:

Shard the database vertically so that reads go to a separate slave. Once this works reliably,
Shard the database horizontally so that each site has its own dedicated database on a server with very few other databases (better CPU to RAM ratio)
Add more Redis servers with up to 32 instances per box
Reindex all the existing redis data so that it is spread onto the new instances
Add a bunch more app servers, so the frontend is always responsive even if the master db is backed up.

Combined, this allows us to move individual chunks of both Redis and MySQL databases onto their own server to equalize load and memory usage; once one instance gets too large (at this stage, over a few hundred MB), we can move that one instance onto a new server with very little effort and almost zero downtime. It also gives us an order of magnitude more CPU per GB of RAM, which means everything’s faster.

What’s next? Hopefully all this will happen behind the scenes and everything will get faster and more reliable. We’ve just hired two more skilled developers to help us implement all this as soon as possible. Once we have the new backend fully online, we can start shipping the fully dynamic ajax frontend, live in-page updates, and more.

Welcome to the new blog.

courtenay3 — Mon, 07 Nov 2011 23:03:00 -0800

We’ve migrated to a new blog. Yay! All the old posts are still cached online at the old Hoth blog, but from now on, we’re on Tumblr. Feels shiny.

Once in a blue moon a new service comes along that is so simple...

imagetic — Tue, 11 Jan 2011 14:15:00 -0800

Once in a blue moon a new service comes along that is so simple and well executed, you walk away rethinking the meaning of software as a service. Flavors.me is just that, a personal homepage service that allows you to make a homepage in minutes.

So when Flavors.me announced this morning that they are using Tender to handle their customer support and knowledge base articles, we couldn’t help but get excited.

Does Virb do ANYTHING that isn’t awesome?

imagetic — Wed, 28 Jul 2010 21:00:00 -0700

Does Virb do ANYTHING that isn’t awesome?

The case against SLAs

entpstaff — Wed, 19 May 2010 21:00:00 -0700

We built Tender for a few reasons, most of them pretty reactionary. We needed a better way than a shared email box or public Lighthouse project to handle customer support, but the solutions out there really didn’t work for us from a user perspective. I personally think that’s because most of them were created by people who came from a customer support or “Big IT” background, rather than coming from “just being a user” background. I think this shows mostly from the concept of SLAs and automated workflows.

(Side note: this is a similar rant along the lines of why we don’t do priorities in Lighthouse. We do actually have priorities, but I stand by the assertion that priorities don’t work).

Here’s how it generally goes:

A customer opens an issue; they get an automated response back along the lines of “You are customer request number #425. We will respond to you shortly”. Conveniently turned into a number, the customer now can be conveniently turned into a trackable metric. We can start judging the performance of our employees based on the speed with which they deal with this request.

Next, a support workflow kicks in, and a support operator has a limited amount of time to respond to the query, or they (presumably) get into trouble. The operator responds, and presumably they do the support dance with the customer until there’s something to track down, a question unanswered, or there just isn’t a suitable response.

A while passes, and the support workflow again kicks in, re-escalating the request or performing some sort of automated action on the ticket. Generally, this is “close the issue after a week of no replies.” An automated response goes out informing the number, I mean customer, that their issue has been deemed solved.

I don’t agree with this at all. It’s a technical solution to a human problem. That being said, I also don’t think we’ve solved it in Tender, either, from a human OR technical standpoint.

We’re not perfect when it comes to responding to our own customer support discussions, and we’re not fully there in providing tools to our customers to help them solve this themselves. But we do get responses like this, and it makes everyone on the team happy:

“Check the Tender forums for an example of how to engage with your customers in a genuine manner.”

But we’re working on it, in a few ways.

Step zero is to be authentically human and not pretend otherwise.

Next, we choose to not offer SLAs. They make everyone feel worthless and under pressure. (We may implement this at some point in the future, but that will rely on the successful state-changes outlined below).

We also got rid of all mention of support “tickets” or “issues” and made it uniformly “discussions”. I feel that the nomenclature is important, and that the way you refer to your customer interactions actually colors the tone, emotion and ultimate success of the conversation.

Finally, we’re modifying the discussion states, so you can have something that isn’t strictly “resolved”, but also isn’t necessarily “open”, either, so it doesn’t weigh on your conscience.

So, if you’re coming to Tender from a service where it encourages you to treat your customers like machines, it’s worth taking a moment to to consider turning your intrahuman communications into something a bit more fleshy.

Post written by Courtenay Gasking, Art by Josh Piles

"It’s only been a week of working with Tender and a few days of having it live for our customers to..."

entpstaff — Mon, 22 Jun 2009 21:00:00 -0700

“It’s only been a week of working with Tender and a few days of having it live for our customers to use, but I’m thrilled with it. The discussion forum is much more structured than our old Google Groups forum—we can mark issues as open or resolved, assign priority queues to issues, and, best of all, support emails go to the forum so they can’t get lost in a cluttered inbox. It’s not as large and in charge as FogBugz, but that’s part of its appeal. We’re even more committed to offering timely and effective support as our customer base continues to grow, and Tender will help us stay on top of our game. My next blog post will talk more about this customer support tool and why it fits No Thirst Software so well.”

- Kevin Hoctor, No Thirst Software

git track is where it's at

entpstaff — Tue, 20 Jan 2009 21:00:00 -0800

Remembering git branch --track local origin/remote && git checkout local is proving to be as difficult as a Chief Justice remembering the presidential oath. On that note, here is a a quick rundown on git alias and a few simple examples to get you started.

git track foo origin/foo

[alias]
 track = !sh -c 'git branch --track "$0" "$1" && git checkout "$0"'

Here is a shorter version that assumes your local branch will be named the same as your remote branch and you’ll always be pulling from origin/*.

git track foo

[alias]
 track = !sh -c 'git branch --track "$0" "origin/$0" && git checkout "$0"'

update: git checkout -b foo origin/foo will track if you have branch.autosetupmerge=true.

Roll Your Own Git Alias

Git Aliases are a nice way to simplify your common tasks into simple commands. Not only can you chain together git commands like we did above, you can run entire scripts, whether it be bash, ruby, python, etc.

The formatting can sometimes be tricky, but one important thing to keep in mind is that unless you’re simply aliasing git’s built in commands, you’ll need to prefix your commands with the ! sign. If your entire line is in quotes you’ll want to put the ! inside the quotes, directly before your starting command only. One final note, you can accept arguments in your aliases using $n. Here are a couple more commands to give you some ideas.

git lh - find all commits affecting a Lighthouse ticket

lh = "!git log --pretty='format:%h [%ar | %aD] [%an] -- %s' | grep -i '\\[#[0-9]*'"
# == Output:
# 658f5c9 [2 months ago | Mon, 10 Nov 2008 13:46:16 -0200] [Nicolas Sanguinetti] -- Ensure managers can see a blog's profile even if it's still pending [#24 state:resolved]
# 48afbb9 [2 months ago | Mon, 10 Nov 2008 13:17:08 -0200] [Daniel Cadenas] -- Fixed. Can only change a user role if user!=current_user [#25 state:resolved]

git ignored - list all files that are ignored

ignored = "!git ls-files --others --exclude-standard"

So that’s it! Do you have any useful aliases you want to share?

Post written by Justin Palmer

Migrate from JIRA or Trac to Lighthouse

entpstaff — Wed, 23 Apr 2008 21:00:00 -0700

Looking to move from old tickets from JIRA or Trac to Lighthouse? You’re not alone. Some of our members have gotten creative with the Lighthouse API and implemented some unofficial importers.

Post written by Justin Palmer