Brightwurks Status

Scheduled downtime for tonight at 10pm EST

Tue, 14 Feb 2012 08:24:00 -0800

Tonight (10pm EST) we have to take down the Help Scout web app for 5-15 minutes to deploy several updates. Our incoming and outgoing queues will continue to function normally, so no emails will be lost.

Take a break to enjoy some time with your valentine and we'll be back before you finish your second heart-shaped chocolate. :-)

Permalink

Outgoing Email Queue Delayed

Tue, 07 Feb 2012 10:02:00 -0800

Yesterday morning we ran into some rogue emails that temporarily clogged our outgoing email queue. No data was lost, only delayed for up to a couple of hours (some weren't delayed at all).

So sorry for the interruption! Tonight we're deploying a fix to prevent this moving forward.

Permalink

Help Scout - Slow Email Delivery

Tue, 20 Dec 2011 09:46:00 -0800

Help Scout has been experiencing sporadic delays with incoming email delivery. It's been hard to diagnose as some cases were caused by external email servers getting the email to us in time, while other cases may have in fact been caused by our own mail server. Therefore, tonight we will be making changes to our mail server to address the issues we can from our side.

Starting at 9pm CST/10pm EST, the mail server will go offline for approximately 10 minutes (or less) so that we can make some configuration changes. Emails will most likely bounce during this time but we will work as quickly as possible to prevent the loss of any emails.

We are so sorry for the inconvenience, but once this change is made, we feel our mail server will no longer play a role in the random slow deliveries.

UPDATE: 10:20 pm EST -

The mail server changes were executed this evening and it was a success. We were able to do everything ahead of time, so the originally planned offline time of 10 minutes was shortened to 15 seconds (long enough for us to perform a server restart). No emails should have been lost during this short window.

Once again, we feel this change will solve the email slowness from the Help Scout side. Please continue to watch for slow deliveries, and if you spot any, don't hesitate to let us know.

Permalink

Follow-up: Delayed Incoming Emails

Thu, 15 Dec 2011 12:58:00 -0800

It's standard operating procedure when we have blips or downtime that we look at what happened and how we can get better. Early this week, our incoming email queue was not processing emails for several hours.

What Happened

This week, Amazon Web Services went about rebooting their cloud servers as part of scheduled maintenance. After the reboot on Monday, our incoming message queue stopped communicating with the server. Between roughly 3am EST - 11am EST, all emails were being received by Help Scout, but not processed in the database.

There was no data lost during this window and Help Scout was never down. It just took a while for us to get incoming emails processed and everything in sync again.

How We're Getting Better

When Help Scout launched back in April, there wasn't a "high availability" version of the message queue we use, meaning it could be a single point of failure. More recently, a high availability version of the software has entered beta and we're going to start testing it.

Assuming we can move this new version of the message queue into production eventually, this blip would be completely avoided in the future.

Any additional questions or concerns can be addressed to us via email. Thank you!

Permalink

Incoming Help Scout emails delayed

Mon, 12 Dec 2011 06:20:00 -0800

Incoming email processing has been down for a couple of hours and we're working on restoring it. No messages have been lost, they just aren't showing up in Help Scout yet because the message queue is down.

The rest of Help Scout continues to operate normally, we'll update this post as soon as incoming mail is restored. So sorry for the problem!

Update @ 11:30am EST: All emails are now processing and will be loaded in your accounts soon. So sorry for the disruption!

Permalink

AWS Maintenance on Tue, December 13

Sat, 10 Dec 2011 14:03:00 -0800

At some point between 2:00-2:30 am EST time on Tuesday, December 13, Amazon Web Services will be performing maintenance on our database instances. This could impact availability of Help Scout for a short time. Thank you!

Permalink

Upcoming AWS Maintenance will Effect Help Scout

Mon, 01 Aug 2011 06:14:00 -0700

Amazon Web Services (AWS), where most of Help Scout is hosted, announced upcoming maintenance that will impact our services at some point between August 1-7, from 3-3:30am EST. We don't have an exact date from Amazon, but the upgrade will take place during that 30-minute window. Availability should only be effected for a few minutes.

For more on the work being done, here's a link to the official announcement from Amazon.

Permalink

Brief downtime this morning

Sat, 16 Jul 2011 11:20:00 -0700

This morning around 6:24am EST, FeedMyInbox.com, Brightwurks.com and our mail server were experiencing issues for less than an hour.

The issues were due to an outage with one of our hosting partners, Voxel.net. We've been happy customers for several years and Voxel's network is extremely reliable, but something went wrong this morning. Once we hear from them about what happened, we'll update this post.

As with every outage on any of our sites, we're working on how we can handle this better in the future and not interrupt our services. So sorry for the inconvenience.

Permalink

Scheduled Help Scout Maintenance: Friday, July 15

Wed, 13 Jul 2011 09:18:00 -0700

On Friday, July 15 from 8-9pm EST we will be performing scheduled maintenance on Help Scout.

The application and marketing site will continue running normally. However, no incoming emails will be processed and no outgoing emails will be sent during the one-hour window. Everything you send from Help Scout during the window will be sent once the maintenance is completed. Thanks everyone!

Permalink

Scheduled Feed My Inbox Email Maintenance

Sun, 03 Jul 2011 09:59:00 -0700

This afternoon we are doing some maintenance on Feed My Inbox. Only emails will be offline. The website will continue working properly.

The maintenance is expected to take 3 hours or less. We won't be sending emails during this time, but once we're back up all posts during the maintenance will be sent. No data will be lost.

Hit us up at help (at) feedmyinbox.com if you have any questions.

Permalink

Brightwurks Domain Failure

Sun, 19 Jun 2011 18:56:00 -0700

Today we lost the Brightwurks.com domain for about 10 hours. It expired early this morning and somehow we were never notified. By the time we got it renewed the nameservers were switched and it takes a while to get them switched back. This caused chaos we are very unhappy to report.

Domains are a single point of failure. If the domain goes down there isn't a backup or way to minimize damage in most cases. We made a terrible mistake and have moved all our company domains to a place where this will never happen again. It shouldn't have happened today and we're very sorry for the inconvenience.

Help Scout

We lost any incoming emails between 12pm-10pm EST today. Any customer who's message was lost received a bounce message informing them that their email was not delivered. Emails hit Help Scout and immediately bounced because our mail servers use brightwurks.com in the host name.

Feed My Inbox

Realtime and 6-hour emails that should have been delivered during the 12pm-10pm EST window were not delivered. Any entries posted during that window were not emailed out properly and thus were lost. We are migrating over some stability improvements we already have in Help Scout that will prevent data loss in this case if it were to ever happen again (which it won't).

If you have any questions or want further explanation from us, please email help (at) helpscout.net or help (at) feedmyinbox.com. We promise it won't bounce!

Permalink

Help Scout Maintenance

Sun, 05 Jun 2011 16:51:00 -0700

We're in the process of some planned Help Scout updates and had to take the application down temporarily ... hopefully 20 minutes or so. This is planned maintenance and we're tracking everything to ensure no data is lost. Thanks!

Update @ 10:46pm EST: Our maintenance took about two hours. It's too long and we'll make sure it doesn't drag out like this moving forward.

Permalink

Feed My Inbox Emails Delayed

Wed, 01 Jun 2011 17:53:00 -0700

It's taking too long for our daily digest emails to go out today ... so sorry, we're having some growing pains. Real-time and 6-hour emails are not effected, just the daily ones. We're working on improvements so we can speed this process up. Thanks!

Update (9:15pm EST on 6/2 ): We've got everything sending much faster now ... don't expect further delays on the daily digest emails.

Permalink

Feed My Inbox: Blip at 11:10am EST

Mon, 30 May 2011 08:18:00 -0700

We had about a 10-minute blip this morning when doing some database maintenance. Everything is back up and we don't expect any further issues. Sorry about that!

Permalink

Help Scout Down: 8:00am EST

Thu, 21 Apr 2011 06:02:00 -0700

Amazon Web Services (cloud service that hosts Help Scout) has brought us down for two hours thus far. We're so sorry for the inconvenience and anxiously await news from their team about bringing everything back up. Stay tuned to this post for updates.

Update (1:15pm EST): The AWS outage continues, crippling thousands of websites and applications. So sorry, folks. We're watching this page religiously to try and bring service back- http://status.aws.amazon.com/.

Update (1:00am EST on 4/22 ): We are still down thanks to a historically bad outage on AWS. We have a great plan of action moving forward so we can try to prevent such issues in the future. Had we been launched for more than 3 days before this happened, we might be more prepared, but it's a great learning experience nonetheless. A full post-mortem will be coming once we are back and can look at the data we have.

Update (9:45am EST on 4/22 ): We are up, working from only one server for now. Incoming emails are not yet working and it looks like we lost incoming emails from yesterday. We'll be touching base with all customers to provide more details. So very sorry for the problem! Now that we have suffered our first AWS outage, we know what to expect and how to circumvent any lost data. We have a list of items we'll get right next time.

Update (10:05am EST on 4/22 ): Incoming emails are now working ... rock on!

Permalink

Emergency Maintenance on Feed My Inbox: Starting Now

Mon, 11 Apr 2011 09:16:00 -0700

We've seen our database crash twice already this morning. The issues have been fixed to prevent downtime, but we have to perform emergency maintenance. We were going to try and wait until the weekend to deploy what we *think* is a fix to the downtime issues, but it can't wait.

During this time, we should be able to keep the website up and running. However, emails will not go out for 8 hours or so. Fingers crossed.

This is more of a pre-emptive outage than unexpected one. We will be deploying several changes and hopefully fixing our database issues once and for all.

We're so sorry for pausing Feed My Inbox emails again, but are hopeful this will be the last time. Our database infrastructure will be updated in the future to make sure we don't have to take the service offline to make these kinds of changes.

Please direct any feedback or questions to help (at) feedmyinbox.com and we'll be happy to assist. Thanks so much for your patience.

Update @ 4:00pm EST: We're back! Over the course of these issues we have lost some data and charged some customers that shouldn't have been charged. We'll be touching base with those people individually. Hopefully we're good to go now!

Permalink

Feed My Inbox Down: 1:00pm EST

Sun, 10 Apr 2011 11:00:00 -0700

We are down again with the same database issues, will update this page when we know more.

Update @ 4:06pm: All of our recent outages are a result of the same problem, which we haven't been able to nail down yet. While restoring the database backup (again), we're working with several people more qualified than us to solve the problem. We feel terrible about these issues, but are doing everything we can to fix them permanently.

Update @ 6:01pm: We're back! Now making some adjustments to try and avoid another hiccup.

Permalink

Feed My Inbox down: 10:30am EST

Thu, 07 Apr 2011 08:27:00 -0700

We are struggling with database problems once again. No ETA as of yet, will update this post with more as soon as we get clarification from our hosting provider.

Update @ 11:49am EST - We're looking at a downtime window similar to a few weeks ago, which could be 8 hours or so. This is an extremely bad situation for us, but we have no choice but to wait for this database to load so we can hopefully prevent any data from being lost. We will keep everyone posted.

Update @ 5:28am EST - We're back up!

Permalink

We're back!

Wed, 06 Apr 2011 22:44:00 -0700

Feed My Inbox is back up! We didn't lose any data during the restore and have been migrated to all-new (hopefully more reliable) hardware. All emails just started going out roughly 90 minutes behind the usual time, so they may be delayed slightly.

We're extremely sorry for the inconvenience caused. Please email help@feedmyinbox.com if the downtime caused any inconvenience and we'll give you a credit.

We are taking steps on this end to prevent database issues from hurting us like this again. This amount of downtime is clearly unacceptable and it will be fixed. Thanks!

Permalink

Should be up within 6 hours

Wed, 06 Apr 2011 19:20:00 -0700

Hey folks ... more database server issues on our end. We're having to restore a backup. It should take no more than 6 hours (fingers crossed, hopefully less time), so the daily emails will begin sending a few hours later than normal. No data loss is expected. So sorry!

Permalink