Resolved: An issue impacting authentication and system responsiveness for a subset of end-users and administrators accessing Microsoft Office 365 has subsided. Further investigation of the issue found the errors were correlated with a service degradation reported by Microsoft (Incident MO147606). Further information is available from Microsoft via https://status.office365.com. ------------------------------- Root Cause Analysis: On Sept 4th, 2018, at approximately 6:00am PDT, Okta began receiving reports that end-users were experiencing delays and errors while attempting to authenticate to Office 365 and related services hosted by Microsoft. Upon further investigation the service disruption began at approximately 2:29am PDT and appeared to be localized to customers with Office 365 and related resources hosted within Microsoft’s South-Central US datacenter. The service disruption was reported as fully resolved by Microsoft on Sept 7th at 1:40am PDT. Throughout the duration of the incident, Okta-provided services continued to function normally. Microsoft reports a severe weather event, including lightning strikes, occurred near one of the South-Central US datacenters. This resulted in a power voltage increase which impacted cooling systems. Automated datacenter procedures designed to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process. Please refer to your [Office 365 Service Health](https://portal.office.com/adminportal/home#/servicehealth) dashboard for further details regarding this incident and mitigating steps.

2020-01-27T20:18:33.000Z

Okta is currently experiencing an issue with the service in US Cell 1. AD Agent and IWA agent states may not be correctly reflected. We are actively investigating and will update this message with more information as soon as we have it. Resolved: Okta has resolved an issue in US Cell 1 impacting AD/LDAP Agent state reporting. All on-line AD/LDAP Agents are now correctly reporting ACTIVE status. Additionally, some customers where all AD/LDAP Agents were marked NOT_CONNECTED should now see successful imports. Additional root cause information will be provided within 48 hours. Root Cause Analysis: On January 22nd at approximately ‪11:39am PST‬ Okta’s monitoring systems detected internal server errors in US Cell 1. The errors resulted in some customers' AD and LDAP agents being reported as NOT_CONNECTED to Okta Admins, even though the agents were functioning normally. Additionally, customers where all AD Agents were marked offline and who had Desktop Single Sign-On (IWA) configured using automatic fail-over may have experienced IWA failures. Okta implemented mitigation steps and the errors subsided at approximately ‪12:06pm PST‬. Okta subsequently corrected agent status for all customers by approximately ‪5:00pm PST‬. Upon investigation, Okta determined that a mistake was made when updating references to the database system during the execution of a planned routine database maintenance, This misconfiguration caused a very small number of requests to return an HTTP 500 error code and also resulted in the outdated agent status information. Okta has updated run-books and is improving the automation of maintenance verification procedures to prevent future occurrences. Okta is also investigating detection of agent status check job failures to prevent incorrect status indication to administrators.

Resolved Service Degradation

2019-12-19T02:29:21.000Z

Okta is currently experiencing an issue with the service. We are actively investigating and will update this message with more information as soon as we have it. Resolved: Okta has resolved the issue. All systems are operational issue has been mitigated. Okta will provide an RCA within 2 Business Days. Root Cause Analysis: On December 16, 2019 at approximately 6:22 AM PST, Okta detected a degradation in application performance in US Cell 4. As a result, customers may have experienced: - Longer than normal response times - A small percentage (0.25%) experienced sporadic read-only errors - Admin users may have intermittently seen the read-only banner Upon investigation, Okta determined that the root cause was unusual database activity required to process a significant number of group rules and assignments. Although our throttling and rate limiting mechanisms engaged, a portion of requests saw additional latency due to the increased utilization of our database tier while the load was being redistributed. At approximately 6:48 AM PST, Okta began to redistribute requests across the cell’s database cluster to reduce load and contention on the impacted database. By 8:09 AM PST, normal service performance was fully restored. Okta is implementing additional monitoring, optimization, and mitigation strategies to prevent similar issues in the future. Okta is also investigating asynchronous processing for group rules and performance improvements for group deletions to prevent system slowdown.

Resolved Service Degradation

2019-12-16T08:19:31.000Z

Okta experienced an issue with Agentless Desktop SSO for a subset of ORGs. We are actively investigating and will update this message with more information as soon as we have it. Resolved: Okta has resolved an issue where a subset of customers were unable to authenticate successfully via Agentless Desktop SSO in US Cell 1, US Cell 11, and EMEA Cell 1. Impacted customers may be required to clear their DNS cache to fully restore end user normalcy. Root Cause Analysis will be provided here within 2 business days. Root Cause Analysis: On December 10th, 2019 at approximately 9:30 AM PST, Okta began investigating reports by customers of problems authenticating via Agent-less Desktop SSO in US Cell 1, US Cell 11, and EU Cell 1. Okta determined that during the implementation of new load-balancers on December 4th 2019 at 3:15pm PST, DNS entries for Kerberos endpoints were not updated correctly for the impacted cells. As a result, Agent-less Desktop SSO traffic was still directed to the previous load balancers which were no longer working as intended, resulting in requests that timed out and user browser sessions being redirected to Okta login screens. While affected users were able to login, they had to type in their credentials instead of being logged in automatically. At approximately 10:48 AM, Okta updated the affected entries to reference the new load-balancers and service was restored. Some clients may have had to refresh DNS caches to restore service more quickly. Okta is working to implement additional monitoring and automation of changes to DNS records to prevent similar issues during future updates.