tag:status-api.heroku.com,2005:/feedHeroku Status2024-03-26T22:38:02ZHerokuhttp://status.heroku.comtag:status-api.heroku.com,2005:Incident/26522024-03-26T22:37:45Z2024-03-26T22:38:02ZResolved: Dynos failing to provision<p>The incident has been resolved. Impact time was from 2024-03-26 7:34 PM UTC to 2024-03-26 9:54 PM UTC</p>
tag:status-api.heroku.com,2005:Incident/26512024-03-26T17:38:04Z2024-03-26T17:38:20ZResolved: Dashboard Issues<p>The incident has been resolved. The impact start time was March 26th 2024 at 16:14 UTC and the end time was March 26th 2024 at 16:29 UTC.</p>
tag:status-api.heroku.com,2005:Incident/26492024-03-01T21:56:08Z2024-03-21T01:21:09ZResolved: Heroku Maintenance scheduled on March 21st, 2024 at 1:00AM UTC (9:00PM EDT, March 20th)<p>This maintenance has been completed. All platform functionality has returned to normal.</p>
tag:status-api.heroku.com,2005:Incident/26482024-02-23T16:07:20Z2024-02-23T16:28:27ZResolved: Dashboard Issue with changing dyno sizes via Web UI<p>This incident is resolved as of 2024-02-23 16:23 UTC</p>
tag:status-api.heroku.com,2005:Incident/26472024-02-23T12:34:53Z2024-02-23T12:48:47ZResolved: Data Dashboard Issue<p>This incident is resolved as of 2024-02-23 12:45 UTC</p>
tag:status-api.heroku.com,2005:Incident/26142024-02-15T02:15:00Z2024-03-25T13:37:02ZResolved: Heroku API experiencing higher latency<h2>Performance Degradation/Service Disruption on Heroku on February 15th, 2024</h2>
<h2>Preliminary Root Cause Analysis</h2>
<p>Published on February 26th, 2024</p>
<p>We sincerely apologize for any impact this incident may have caused you and your business. At Salesforce, Trust is our #1 value, and security is our top priority. We value transparency and want to take this opportunity to outline the facts regarding a recent service disruption that may have impacted your ability to use multiple Heroku services, based on our current understanding. Our investigation is ongoing, and we will endeavor to provide you with updated information as it becomes available.</p>
<h2>Executive Summary - What Happened?</h2>
<p>On February 15th, 2024 between 1:20 UTC and 7:40 UTC, a subset of Heroku Private Spaces customers experienced degraded performance and functionality. The incident was caused by a third-party vendor's process consuming excess CPU and memory. This was the result of a system error allowing the set thresholds or limits on the process’ consumption to be ignored. The incident was remediated by disabling the third-party vendor’s process on our fleet.</p>
<p>The final root cause of the vendor’s process causing the issue is still under investigation.</p>
<h2>How did this issue impact Heroku services?</h2>
<p>Multiple Private Spaces customers across several regions experienced degraded performance and functionality. In some cases, dynos entered a crash-looping state where they would remain broken until manual intervention by Heroku Support, Heroku engineers, or the customer took steps to replace the underlying dyno.</p>
<h2>Technical Details</h2>
<p>Detection and Initial Impact:</p>
<p>On February 15th, at 2:00 UTC, the Heroku Engineering team began seeing the Heroku API experiencing higher latency. Additionally, some Private Spaces experienced higher than normal memory and CPU consumption, causing some dynos to go offline. The Heroku Engineering team partially mitigated the problem by replacing the underlying dyno hosts. This action was executed during the incident period while Heroku Engineering continued the investigations.</p>
<p>At 3:32 UTC, Heroku’s engineers determined that the high latency on the Heroku API was being caused by high memory and CPU consumption from a third-party vendor’s process.</p>
<p>At 4:38 UTC, additional engineering teams from Salesforce and the third-party whose processing caused the issue were engaged to help understand the behavior of the process causing the issue. We confirmed no configuration or version changes were introduced causing the issue. The final root cause is still under investigation.</p>
<h2>Remediation</h2>
<p>On February 15th, at 7:40 UTC, the Heroku engineering team deployed a change to remove the third-party process from Heroku fleets. The removal of the third-party process resolved the issue.</p>
<h2>Root Cause Analysis</h2>
<p>The Technology team’s post-incident investigation and analysis determined that the incident was triggered by a known third-party vendor’s process running in our hosts.</p>
<h2>Next Steps</h2>
<p>To maintain the performance level that our customers expect from Heroku and to prevent this defect from recurring, our focus is on continuous improvement. The Technology team has identified and is implementing the following actions:
* Complete: The process that caused the issue was disabled on our hosts and will remain as it is until the underlying root cause is clear.
* In Progress: Heroku’s engineering teams are working closely with the third-party vendor to understand the root cause of the issue and decide next steps.</p>
<p>We sincerely apologize for the impact this incident may have caused you and your business. Heroku is fully committed to minimizing downtime when incidents do occur. We also continually assess and improve our tools, processes, and architecture to provide you with the best service possible.</p>
<h2>March 22th, Update</h2>
<p>The vendor issue caused a constant loop of reconnection attempts to a local port, driving up CPU and memory consumption on host. The service has been removed from our platform.</p>
tag:status-api.heroku.com,2005:Incident/26112024-02-02T18:35:50Z2024-02-02T18:42:19ZResolved: One-off dynos experiencing errors <p>This incident is now resolved. The impact start time is 2024-02-02 at 16:25 UTC and the end time is 2024-02-02 at 18:29 UTC.</p>
tag:status-api.heroku.com,2005:Incident/26102024-01-26T16:30:01Z2024-02-09T02:31:44ZResolved: Heroku Maintenance scheduled on February 9th, 2024 at 2:00AM UTC (9:00PM EST, February 8th)<p>The scheduled maintenance is now complete. All functionality has returned to normal.</p>
tag:status-api.heroku.com,2005:Incident/26092024-01-25T20:05:30Z2024-01-25T20:05:47ZResolved: Delays/failures in Release phase<p>The incident has been resolved.</p>
tag:status-api.heroku.com,2005:Incident/26062023-11-27T18:34:10Z2023-12-08T02:25:46ZResolved: Heroku Maintenance scheduled on December 8th, 2023 at 2:00AM UTC (9:00pm EST, December 7th)<p>The scheduled maintenance is now complete. All functionality has returned to normal.</p>
tag:status-api.heroku.com,2005:Incident/26052023-11-17T16:10:59Z2023-12-07T03:00:53ZResolved: Heroku Maintenance scheduled on December 7th, 2023 at 2:00AM UTC (9:00pm EST, December 6th)<p>The scheduled maintenance is now complete. All functionality has returned to normal.</p>
tag:status-api.heroku.com,2005:Incident/26042023-11-15T19:57:42Z2023-11-15T20:21:35ZResolved: Issues with Heroku Builds<p>Our engineers detected an issue where apps with <a href="https://devcenter.heroku.com/articles/release-phase">release phase</a> enabled were experiencing timeouts during deployment.</p>
<p>This issue is now resolved. The impact start time was 19:19 UTC and the end time was 19:52 UTC.</p>
tag:status-api.heroku.com,2005:Incident/26032023-11-08T16:02:36Z2023-11-08T17:04:13ZResolved: Issues with upstream DNS provider<p>The issue has been resolved.</p>
<p>The impact start and end times are identified as 2023-11-08 14:00 UTC and 2023-11-08 16:47 UTC.</p>
tag:status-api.heroku.com,2005:Incident/26022023-11-07T21:01:08Z2023-11-07T22:32:08ZResolved: Platform API Outage<p>Platform API is working as expected. The incident is now resolved. Impact start and end times are 11/07/2023 at 19:50 UTC and 11/07/2023 at 21:57 UTC.</p>
tag:status-api.heroku.com,2005:Incident/26012023-11-06T19:38:18Z2023-11-06T19:38:36ZResolved: Issue with new user signups<p>The incident is now resolved.</p>