entp hoth blog - Everything ENTP

It's tradition

2009-11-04T16:05:00Z

Zack Hobson hard at work building his own desk at the ENTP office before he starts his first day as an ENTP'er on Wednesday. The desk building thing has become a bit of a tradition.

Recent updates to Tender

2009-11-02T22:52:00Z

It’s too easy to let the little things slide by unnoticed. Bug fixes, tweaks, and additional minor functionality are pushed to Tender Support all of the time.

Below are the most recent changes and additions to Tender.

Now with FAQ ordering

You can now tweak the order of FAQ articles. To do do this, go to the Knowledge Base / FAQ area in your admin and then select an FAQ category. In there you’ll see this small change…

Just click on the arrow icons and drag the FAQ article to the location in that category where it belongs.

We also added FAQ redirections. Previous FAQ ordering consisted of constructing arbitrary permalinks that were ordered alphabetically. This led to awkward links like “/faqs/api/1_introduction”. Now, Tender will remember these and redirect old requests to the new FAQ location.

Email Headers inside comments

The second addition is that you can now view email header information from users. If you go to a discussion and click on the Edit button,

The following edit page now displays basic header data if that user emailed their discussion.

Beefed up email handling and options

Email handling is always under improvement. After all, it’s the single most awesome portion of Tender Support if you ask me.

Recent update include:

Added #ignore and #unsubscribe commands. See the Knowledge Base article for more details.
Use email headers to determine which discussion an email is replying to. The long form reply address (tender+CODE@tenderapp.com) is no longer required.
Start pushing all spam received in our public email server’s spam box into Tender’s spam queue.
Backend email processing workflow is now more reliable. Emails are tracked as they go through the public email server, the internal server, our ruby processing scripts, and Tender. We can detect if an email has passed through the chain successfully, and easily re-process missed emails.
Basic mail processing fixes for bugs, weird encoding, attachments, etc.

If you have any questions or comments, please feel free to hit us up on Twitter or fire up a discussion on http://help.tenderapp.com.

Postmortem of Tender issues

2009-10-27T08:43:00Z

As most of you are well aware, we’ve spent the last few days troubleshooting heavy response times on Tender Support.

Current status

As of 4:30 p.m. PDT, the brunt of the problem appears to be behind us. In the meantime, we’re keeping a close watch on things for when traffic escalates on Tuesday morning. We’re also still going through the data to double check that nothing was lost. So far, all data has been kept intact and delivered.

We would also like to formally apologize about the severity of the issue. We’re just as dependent on our support system as our users and felt their pain every step of the way.

For those of you curious as the the specifics of the server issues, as well as what steps we are taking to prevent it from occurring in the future, please continue reading…

The long story

When issues arose on the Tender Support server late Thursday night, we had the initial symptoms correct, but not the root cause. We thought the backlogged emails were creating the huge load when in fact it was likely due to the European users waking up and starting their day. Increased hits on Tender would lead to increased load on the servers, right?

The weekend was spent pouring through logs, identifying slow requests and optimizing them. This work actually began over a week ago, so you may notice a few minor features here and there (spoilers!).

The hardware

One of the things that seemed odd was the fact that the actual server resources were not actually spiking. Memory usage was fine and CPU usage was completely normal in comparison to previous weeks.

As you can see, CPU usage actually went down!

Tender Traffic as a whole

Google Analytics showed no signs of a spike in traffic. Current Tender Support sites were pulling in an anticipated number of hits.

Resources and setup

On Sunday night, Engine Yard bumped up our resources and added the HAProxy load balancer.

HAProxy gave us a window into the queueing situation. We can now see which application servers were busy and how many requests were waiting in the queue.

Monday on Tender Support

As Monday morning rolled around, things started to get nasty. In response, Engine Yard support technicians jumped in to help out.

One of the suggestions we received was to test out Mongrel as a ruby application server instead of Thin. Though it didn’t lead to any conclusion, it did lead to the first big discovery to make a difference in our server issues.

HAProxy was showing all of our application servers maxed out. We noticed them hanging on to requests for several seconds, which led to more requests getting backed up. As soon as the requests were completed, we downloaded the logs looking for the long request, only to come up empty handed.

Since one of the servers was now using Mongrel, we installed the mongrel_proctitle plugin. It allows me to monitor what the processes are doing while hanging from the top command:

mongrel_rails [5000]: handling GET help.tenderapp.com/login                                                        
mongrel_rails [5001]: handling GET customer.tenderapp.com/support.php                                                                     
mongrel_rails [5002]: handling GET customer.tenderapp.com/posts.rss                                                                            
mongrel_rails [5003]: \m/ (no request)

As a result, we started to see random URLs from other support systems. These looked like requests from various forums or support software that predated their Tender Support sites.

After deploying an update to optimize the 404 handling, the queues have stayed empty and response times have returned to normal.

This particular problem was more difficult to track down because the server logs neglect to show the benchmarked request time for exceptions.

Also, since a lot of our traffic comes from public sources, we tend to get a lot more of these requests than with private apps like Lighthouse.

The next steps…

In the short term, we have resolved the current issue and have more hardware on the way for tomorrow to give us a little more leg room.

We’ll also be running additional tests to find exactly why things were so slow, as well as finding and eradicating some odd caching bugs that have been cropping up from the extra measures taken over the weekend.

In addition, we’ve also started preliminary planning with Engine Yard for a major hosting change. If we have a more flexible solution in place, we could just turn the dial up to 11 to keep things steady while we diagnose the problem.

Tender status for Monday, Oct 26

2009-10-26T17:19:00Z

After a weekend of troubleshooting in response to the Tender Support server delays on Friday, Monday has shown no mercy as usage picks back up to normal.

Currently, the servers are suffering from extremely heavy response times, causing frequent timeouts and delay in email processing.

Over the weekend we took extra precautions by adding additional resources to the server as well as some extra caching measures.

We are currently working side by side with our host to resolve the issue as quickly as possible.

You can also follow us on Twitter for continuous status updates

Tender Slowdown

2009-10-23T05:34:00Z

Early this morning, I was woken up with some urgent Tender news: requests were taking an abnormally long time to run.

As I logged in, some Engine Yard technicians had already checked the basics: memory usage is stable, database is happy, the filesystem is functioning, etc.

Then we came across this graph in New Relic:

Our application processes were being slammed with requests. The culprit?

Since extending email processing in Tender to handle the spam from our public email server, we’ve been seeing a lot more email traffic. Each email is turned into a HTTP web request from our email server to the Tender servers. Most of the time this load is perfectly fine on our current hardware. However, the backlog of spam emails pushed it over the top. It’s definitely eye-opening seeing how much spam emails we receive.

The issue is taken care of for now. For the short term, we’re going to setup a dedicated application server just for emails. This would be something we’d spin up when there’s a large volume of emails to process. This way email traffic won’t affect the web traffic at all.

In the longer term, we’re looking into adding more application servers as our need dictates.

Sorry for the inconvenience with the issues tonight. If you have any questions or problems, let us know.

New in Tender: Edit Users and Analytics

2009-10-15T21:39:00Z

It’s been way too quiet on the Tender Support blog posting front lately. Have no fear though, we’ve just been hard at work making refinements to Tenders email system (we’ll have a full blog post about these changes tomorrow). It’s about time we rolled out something new to the front end though.

Edit users

Yesterday, we added a feature to allow you to edit your Tender Support user profiles.

When a member is having issues with their login or settings and trying to post issues, your support staff can now hop under the hood and make necessary tweaks such as password updates and email notification changes. It’s a lot more practical than having to come directly to us for user level support. It's a small but mighty addition.

Google Analytics

This one isn't exactly new, it was added back in August, but it's been hiding in the shadows like a good little ninja since it was never announced publicly. You can add custom Google Analytics tracking for your Tender support site using your own Analytics setup.

If you click on Site Settings and scroll down to the Domain Information area where you would set your custom domain, you may have noticed an additional field to add your Google Analytics code.

There were some pretty heavy discussions on this from our users over on our support site. Since we love our users, who love their stats, this little addition has made its way to the front lines.

We're hiring

2009-09-30T10:00:00Z

ENTP is looking for a few talented people to staff up our Portland office. We offer an excellent benefits package and competitive salaries, as well as a great workplace and an inspiring world-class team. You should be a reliable, skilled, talented practitioner who can learn from as well as teach their coworkers.

We may consider remote hires as well as contractors in exceptional circumstances.

Designer / wizard

You will be responsible for developing wireframes, comps and functioning HTML/CSS/Javascript prototypes of new web applications, and working with a small development team to bring those to life into a functioning app.

Ideally, we'd like someone with a strong eye for developing beautiful illustrator and photoshop work, and will consider this at the expense of advanced javascript skills.

You'll also be maintaining and improving the UI and user experience of our existing suite of applications, including our flagship applications Lighthouse and Tender.

Ruby programmer

We have some large applications with a challenging array of back-end ruby work. You will need to demonstrate a working knowledge of writing scalable ruby APIs, testing, and caching techniques. System administration is a bonus, as is front-end ability (html, javascript)

Junior ruby programmer

Somewhere between support and fulltime programmer there lies an interesting position for an junior Ruby programmer who can dive into code and fix things, but also answer customer support requests from a knowledgeable and technical background.

How to apply

For more information, or to apply, please email jobs @ entp.com

Tender's newly refined dashboard listings

2009-08-28T02:10:00Z

We’ve been adding a lot of fine feature polish on Tender recently – things that on their own don’t qualify as a “feature” but add up to the whole of the product experience. One of the larger pieces of that polish was redesigning the discussion listing in the dashboard context.

The old dashboard context

It suffered from some problems:

There is a lot of hidden functionality, but none of it was apparent until you hovered over select portions of the listing. It just wasn’t intuitive.
State icons were not very well defined and made it difficult to scan the list.
Low information density. There’s a lot of information we have that we weren’t showing to you.

The new dashboard context

Admittedly, the new screen is busier which I’m not a fan of, but after using this screen myself for a few days, I’m happy with the functionality and information density trade-offs. Some of the bigger points:

Bulk edit – You can now check many discussions and resolve, open, add to queue and so on.
We’ve added the number of people watching the discussion to the right hand side (this is the number of unique users who have posted to the thread). It’s a good gauge of popularity and isn’t influenced by lengthy back-and-forth discussions.
The state icons have been merged with the checkboxes and are much clearer. The gearbox dropdown on the right hand side is a lot clearer too – inviting you to see what’s below (rather than making you guess that you could hover over an ‘Actions’ label).
More information that matters – you’ll see that the date has been replaced with how old a discussion is. This is obvious, and I’m not sure why it took this long to change it. The dates are also color coded to how old they are, as they get older, they get darker and bolder, signaling that you should probably take a look.

All in all, I’m really happy with these updates. I am positive some people will not like them (especially at first glance, without using it for a while). But over time I think it’ll grow on you and feel natural.

Custom Tender Gallery

2009-08-04T18:28:00Z

One of the our core goals on Tender is making sure that your support site works perfectly with your brand and product. In line with that, we allow people to apply Custom CSS styles to their installs. We’ve got some amazing customizations out there, and I’d like to take a few minutes to go through some of our customer’s sites.

Tender allows you to use your own domain, favicon, Apple Touch icon, change text and apply custom CSS styles for everyone on the Plus and above plans.

Beanstalk

Beanstalk provides hassle-free Subversion hosting. You can see their support site at http://help.beanstalkapp.com

DabbleDB

DabbleDB helps you create online databases on the web. You can see their support site at http://support.dabbledb.com

Daily Burn

DailyBurn helps you track and reach your fitness goals. You can see their support site at http://support.dailyburn.com

Engine Yard

Engine Yard provides Ruby on Rails Hosting and Deployment in the Cloud. You can see their support site at http://community-support.engineyard.com

Newsberry

Newsberry helps you send & track email newsletters. You can see their support site at http://help.newsberry.com

SonarHQ

SonarHQ provides you a way to create gorgeous looking surveys to tap into the pulse of your community. You can see their support site at http://support.sonarhq.com

YouCastr

YouCastr helps you share high school sports and youth videos. You can see their support site at http://support.youcastr.com

New in Tender: Private Sites

2009-07-31T20:49:00Z

We recently rolled out a fairly major update to Tender Support today that allows you to turn your Tender into a password-protected private site.

If you log into your admin panel and go to Site Settings, you’ll find a new block that allows you to switch your site to private mode.

Once your site is marked private, people visiting your support site will be redirected to a login screen.

You can customize the welcome text, and even white-list email domains to allow people to automatically sign up. When you white list an email domain, anyone with an email ending in the domain will be given the option to sign up.

The new Member Management section gives you full control over who has access to your site. You can manually invite people to your install, promote them to support users, or disable their accounts.

Whether you’re a small IT team in a larger organization looking to manage requests, or looking to manage support for a private beta, we think this new feature is going to open up Tender to a whole new group of people. As usual, we’re eating our own dog food and already have it implemented for managing requests from one of our clients!

A widget for your Tender install

2009-07-23T03:48:00Z

Widgets seem to be all the rage lately, and we’re not one to be left out of the fun. If you log into your Tender account and go to the Admin -> Extras section, you’ll find a link to embed a widget into your application.

The widget will allow your customers to post issues and find Knowledge Base articles right from within your site. The widget by default comes with a tab (that shows up on the right hand side) to open it up, but if you’d like to build in your own trigger, you can find instructions on the Widget page to use your own trigger.

You can see the widget in action in Lighthouse. Just go ahead and click the upper right Help icon and play around with it!

Apologies for Invalid Emails involving CC Issues

2009-07-14T17:39:00Z

Last night you may have received an email claiming that your credit card information was incorrect on your Lighthouse account. We moved to a new payment system and inadvertently sent out emails during the migrations. You can safely ignore these emails.

If you have any problems please let us know by filing an ticket and we'll get you taken care of.

New in Tender: Enhanced Search

2009-06-23T08:36:00Z

It’s true. Until now, Tender’s search capabilities have been in need of some work. Over the weekend, we deployed the first major update to overhaul the search system.

Design-wise, we didn’t like how the search was so comment-centric. Searching for anything that matches anywhere in a comment body will now return the whole discussion just once. You can also search for names and emails to pull up discussions that the user has participated in.

Also, we wanted the search to be unified on a single page. You can now search Discussions, FAQs, and users (if you’re a support member) from a single page. We’re working on a proper FAQ + Discussion search (heavily weighted to prefer FAQs) as well.

On the backend, we’ve added filters for searching just the title or by discussion state. We’ll also be adding filters for the published/updated dates, as well as configurable sorting.

Please check out the updated search and let us know how it works out for you. If you have any feedback, please let us know on http://help.tenderapp.com or ping us on twitter.

No Thirst Software On Tender

2009-06-23T07:52:00Z

It's only been a week of working with Tender and a few days of having it live for our customers to use, but I'm thrilled with it. The discussion forum is much more structured than our old Google Groups forum—we can mark issues as open or resolved, assign priority queues to issues, and, best of all, support emails go to the forum so they can't get lost in a cluttered inbox. It's not as large and in charge as FogBugz, but that's part of its appeal. We're even more committed to offering timely and effective support as our customer base continues to grow, and Tender will help us stay on top of our game. My next blog post will talk more about this customer support tool and why it fits No Thirst Software so well.

Job Queue Issues: It's Not Lupus

2009-05-23T11:30:00Z

Yow, what happened to the queue? We had two separate issues on Thursday night and Friday night that have since been fixed.

The first one had to do with a strange error from parsing one of the many emails we receive. A bug in our email parsing library raises a critical exception that kills the queue process. I found the issue, deployed a workaround, and then submitted a patch to the library author. Typically, our queue catches and logs exceptions, instead of dying. However, the bug caused it to mistakenly raise a fatal SyntaxError, which is typically reserved for a typo in a script that prevents the application from even running.

The second problem has to do with a new job we deployed on Thursday. One of the problems we saw with accepting anonymous comments on Tender, is that you lose all connection to those comments when you create a new profile. The new job automatically links them to you when you create a profile on Tender. The problem is, these jobs started taking a bit longer than anticipated to run, holding up more critical tasks. I’ve since started using a priority system to ensure that critical tasks like spam checks or comment notifications are run before these jobs.

On a final note, it’s clear that the job queue is of utmost importance for Tender. We actually have a munin server setup for monitoring, but we’re still working with Engine Yard on getting the firewall port issues worked out. This incident has moved that ticket to our top priority so we can get on top of these issues sooner. Sorry for the inconvenience, everyone!