Earlier this morning, one of our backbone providers, Qwest, had an issue with one of their networks resulting our limited ability to send some emails that were scheduled to be sent this morning between about 6AM and 8AM PDT. Our systems staff has been monitoring and working with Qwest to resolve this matter in the quickest way possible.

During this period, we have been re-routing traffic as to minimize any impact on customers. However, some email traffic has been queued up and will be successfully sent once the network is stable.

Good news is, it looks like the network is up and running and emails that didn’t get sent are now getting sent. We apologize for this inconvenience and will keep you posted on this matter.

UPDATES

UPDATE - 8:18 AM - It appears our Qwest network is online and stable. We’re observing email traffic resume successfully. We’re still monitoring the issue.

RESOLVED - 10:22 AM – We have determined this issue is fully resolved and has been since the last update. “All clear” reports our Systems team.

At about 7:26 AM to 7:30 AM this morning, one of our datacenters lost network connectivity.

We are investigating with the provider as to the cause and will identify any opportunities for us to prevent it in the future. While the downtime was short (four minutes), we are notifying users because it was affected a majority a minority of our Phoenix-based users.

The data center is located in Tempe, AZ and our systems team is in contact with them. At this time, they confirm full network connectivity has been restored.

—
Updates:

7/7/2010 10:11 AM – We have determined the impact of this outage was isolated to only between the data center and the Cox network in the Phoenix, Arizona region. As such, a very small number of users may have been affected.

This morning, we encountered an issue where users will notice a cosmetic issue with the chart display on Email Batch Status Report. What happens is users access the report and notice only one contact has opened the message and everyone else hasn’t taken action with email messages sent.

Example:

Luckily, there’s a workaround.

To see who interacted with your message, click the pie chart to see the accurate results. Alternatively, you can do an email batch report and select the appropriate message (or period of time). From there you will be able to view the contacts under the conditions of what you’ve searched for.

We’re aware of the issue and plan to have this issue resolved tomorrow evening. This data is all there and is merely a display issue on the Email Batch Status/Broadcast Report.

Thanks for your patience and understanding in this matter; we’ll update this post with further developments.

Updates

7:00 PM 6/16/2010: This issue has been resolved.

This morning, we became aware of a system-wide issue impacting anyone using a 1.19 (Summer Release) involving HTML email templates. At this time we are aware of the full scope of the issue and there is unfortunately no workaround. We’re working hard to address this critical issue as quickly as we can.

There are two main issues at hand: Certain text like apostrophes, registered trademarks, etc. in email would turn into “'” and respective XHTML entities and not be actually rendered as designed. Another impact of the issue is when there are styling around text such as bold or emphasis, it will remove the space between the words.

From what we have tested, it appears all (UPDATE, read below!) few outbound HTML emails are impacted. Including ones from the Classic HTML email editor and the new Email Builder. We’ve isolated the cause and are testing a solution at this time.

We will update this post with progression and resolution. We apologize for this inconvenience and appreciate your patience. We aim to have this resolved over the weekend.

—

UPDATES:

1:21 PM 6/11/2010: Our development team narrowed the issue down to only users who edit a template AND click the Source tab. At this time we have a fix in place and are currently testing it. The workaround is to not click the Source tab. One caveat — we are not able to restore the spaces between words that may have been modified by HTML styles. We strongly recommend that users who edited their HTML email templates recently to confirm they are rendering the way it should.

1:27 PM 6/11/2010: We feel confident the changes will pass QA and deployed tonight. Much better than a weekend.

8:04 PM 6/11/2010: The issue has been resolved and deployed to affected applications.

8:21 AM 6/18/2010: Based on a couple reports from customers, we have re-opened this case and have an additional patch that will go out at the next maintenance release. If you’re still having trouble, it is important to contact support so they can properly diagnose the situation.

9:00 AM 6/22/2010: This has been resolved and corroborated by the lack of customer reports of the issue. Thanks for your patience.

At 10:03 PM on 6/10/2010, we’ve successfully deployed our fourth phase of our 2010 Summer Release to users. Approximately 1700 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.29

At 9:38 PM on 6/09/2010, we’ve successfully deployed our third phase of our 2010 Summer Release to users. Approximately 800 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.25

At 8:46 PM, we’ve successfully deployed our second phase of our 2010 Summer Release to users. Approximately 500 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.24

We’ve successfully deployed the latest production version of our 2010 Summer Release to our Beta users. After hitting a few snags late last week, we’re on pace to deploy to all users in the next couple weeks.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

This applies to you if you have 1.19.1.24 in the lower left corner of your application.

Today, we discovered an issue where some applications are experience higher than normal memory usage. We will be selectively rebooting various front-end servers until we resolve the underlying issue causing more memory usage. (High memory usage could severely impact performance.)

We use multiple applications on our servers so if one app behaves abnormally, others may be affected. To prevent adverse effects, our Systems team will identify troublesome servers and reboot them to release the memory. They’re super smart and dedicated and will be doing this in a way to minimize impact to you.

During the process of our server reboots, users will be logged out from their active Infusionsoft sessions and will need to log in again. From that point, there may be a short delay from being logged out to logging in again.

We’re aware of these issue as we are initiating the reboots. There is no need to contact support for this matter. Our Systems team will continue to monitor our servers and application performance and will be making improvements to keep things running smoothly.

If you are writing lengthy emails in your templates, we recommend to save a copy on your computer locally and to save frequently in the application until this issue is resolved. We’ll post updates on the blog post here so you can check back to know when we determine this to be fully resolved.

Thanks for your patience and understanding. Updates will be posted below.

—

UPDATE 5/14/10 1:31 PM: *RESOLVED* Our Systems team has identified the culprit and isolated the performance to a very small group of applications. As such, we will not be rebooting server today. Enjoy your awesome uptime and performance.

This morning we identified an issue where users would send themselves a test email in a template and they would receive an older copy of the message. Our developers have been made aware of this and are working to resolve it quickly.

This issue won’t impact any emails that sent via broadcast or from follow-up sequence. It is isolated to only the “Save and Send Test” button found on the email editor.

There is a workaround to use in the meantime so you can test your email messages. Simply go to your own contact record and send yourself an email and choose the desired template you wish to test. Alternatively, you can also send a broadcast and target yourself as the recipient.

Sorry for the inconvenience and we appreciate your patience as we fix this.

—

UPDATE 5/13/10 2:48 PM: At this time we have identified the cause and a patch to resolve this. We anticipate to deploy the fix this evening to all applications. We apologize for the delay but thank you for your patience and understanding.

RESOLVED 5/14/10: The issue has been resolved.