Brev - Notice history

https://app.brev.io - Operational

100.0% uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2026
Jul 2026
Aug 2026

Notice history

View current status

Aug 2026

Fetching mails from Microsoft accounts producing errors and preventing email based creation of Jira work items
ResolvedPartial outage10 hours 49 minutes
  • ResolvedUTC

    We're now seeing full recovery across Atlassian services that have been impacted, and this incident is now resolved.

    For those continuing to experience issues relating to this incident we encourage you to please check https://status.cloud.microsoft/ as they continue to provide further updates about additional services.

  • MonitoringUTC

    Jira Incoming Emails are experiencing elevated failures when fetching mail from Microsoft accounts. This affects email-based creation and updates of Jira work items. Users may see 500 internal server errors when email fetch attempts occur.

    We are tracking updates from Microsoft and continuing to review the impact. These updates are available publicly via https://status.cloud.microsoft/

  • IdentifiedUTC

    Impact

INC20000188
ResolvedMajor outage2 hours 24 minutes
  • UpdateUTC

    Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site: https://community.snowflake.com/s/article/INC20000188

  • ResolvedUTC

    Current status: We've monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community or the Support page in Snowsight.

    Customer experience: Customers hosted in the specified region may have been unable to access or use multiple core Snowflake services and features, including signing in to Snowsight, executing queries, loading and unloading data, and using Snowpark Container Services. Some customers may have also experienced delays with Snowpipe Streaming ingestion and elevated error rates when accessing Snowsight.

    The majority of impact was resolved at 17:25 UTC; however, some customers may have continued to experience some residual errors or increased latency until approximately 18:10 UTC.

    Incident start time: 16:58 UTC August 31, 2026
    Incident end time: 18:10 UTC August 31, 2026

    Preliminary root cause: A recent configuration change degraded backend processing due to a compatibility issue with how the affected systems handled requests. As a result, the infrastructure experienced resource contention and subsequent impact to multiple Snowflake services and features.

    A root cause analysis (RCA) will be conducted and a summary will be shared on the Snowflake status page within 5 business days.

  • MonitoringUTC

    Current status: We've implemented a fix to address the residual impact, and we'll continue to monitor the environment until we're confident all services are functioning properly.

    Customer experience: Customers hosted in the specified region may have been unable to access or use multiple core Snowflake services and features, including signing in to Snowsight, executing queries, loading and unloading data, and using Snowpark Container Services. Some customers may have also experienced delays with Snowpipe Streaming ingestion and elevated error rates when accessing Snowsight.

    The majority of impact was resolved at 17:25 UTC; however, some customers may have continued to experience some residual errors or increased latency until approximately 18:10 UTC.

    Incident start time: 16:58 UTC August 31, 2026
    Incident end time: 18:10 UTC August 31, 2026

    Preliminary root cause: A recent change on the affected infrastructure inadvertently resulted in an increase in resource contention and subsequent impact to multiple Snowflake services and features.

  • UpdateUTC

    Current status: We are continuing to monitor service recovery, and have confirmed that the majority of service impact has been resolved as of 17:25 UTC with some residual impact to Snowsight access and Snowpipe Streaming. We'll provide an update within 30 minutes.

    Customer experience: Customers hosted in the specified regions may be unable to access or use multiple core Snowflake services and features. Affected users may be unable to sign in, execute queries, or manage data.

    ETA: Most services have returned to a healthy state. We expect any residual impact to Snowsight access and Snowpipe Streaming to be resolved within the next hour.

    Incident start time: 16:58 UTC August 31, 2026

  • IdentifiedUTC

    Current status: We identified and have since reverted a problematic change as a potential mitigation. Initial telemetry indicates that the service is recovering. We will continue to monitor to confirm recovery. We'll provide an update within 30 minutes.

    Customer experience: Customers hosted in the specified regions may be unable to access or use multiple core Snowflake services and features. Affected users may be unable to sign in, execute queries, or manage data.

    ETA: Customers should be starting to see recovery, and our current estimate is that service will be restored within 1 hour.

Asana full outage for some users (2 hours, 25% outage)
ResolvedPartial outage2 hours 4 minutes
  • UpdateUTC

    We’ve been working to add a caching layer to our update pipeline, tuning it carefully and rolling it out gradually. On Wednesday, August 26 we enabled it for most use cases, and it initially performed well.

    On Monday, August 31, a combination of unrelated infrastructure changes and peak traffic pushed the cache past its scaling limits. Once that threshold was crossed, the cache became unusable, and many pods serving read traffic for the Asana application could no longer serve it.

    We mitigated the incident by reverting the system to use the previous, non-cached code path. The revert was successful, but recovery took longer than we would expect for this class of issue.

    A fuller analysis is underway. We will follow up with root causes, action items, and improvements, including why recovery took as long as it did.

    We were fully down for about 25% of our users, for 2 hours, 15 minutes.

  • ResolvedUTC

    User-facing symptoms have recovered. We'll continue to monitor, and will prioritize a retrospective to understand and prevent similar incidents in the future.

  • MonitoringUTC

    We've applied a fix are seeing signs of recovery.

  • UpdateUTC

    We have made configuration changes and see partial recovery, but we continue to see some elevated errors.

  • UpdateUTC

    We are continuing to investigate the issue; we have reverted recent changes, and are working to identify the source of the errors.

  • InvestigatingUTC

    We are investigating alerts for slow performance and application errors.

Jul 2026

Jun 2026

INC20000052
ResolvedMajor outage1 hour 44 minutes
  • UpdateUTC

    Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site: https://community.snowflake.com/s/article/inc20000052

  • ResolvedUTC

    Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community or the Support page in Snowsight.

    Customer experience: Customers hosted in the specified region may have been unable to access or use Snowflake services and features using Snowsight. A subset of affected users in the region may also have been unable to execute queries, manage data, or may have experienced increased latency or performance issues with database operations.

    Incident start time: 15:10 UTC June 24, 2026
    Incident end time: 15:50 UTC June 24, 2026

    Preliminary root cause: A configuration issue caused an unexpected increase in load on our metadata database layer, resulting in cascading impact to dependent services like Snowsight and database operations.

    A root cause analysis (RCA) will be conducted and a summary will be shared on the Snowflake status page within 5 business days.

  • MonitoringUTC

    Current status: We've implemented the fix for this issue, and we'll continue to monitor the environment until we're confident all services are functioning properly.

    Customer experience: Customers hosted in the specified region may have been unable to access or use Snowflake services and features using Snowsight. A subset of affected users in the region may also have been unable to execute queries, manage data, or may have experienced increased latency or performance issues with database operations.

    Incident start time: 15:10 UTC June 24, 2026
    Incident end time: 15:50 UTC June 24, 2026

    Preliminary root cause: A configuration issue caused an unexpected increase in load on our metadata database layer, resulting in cascading impact to dependent services like Snowsight and database operations.

  • UpdateUTC

    Current status: We've identified the source of the issue, and we've implemented a fix to restore service. We're assessing the recovery of affected components and we'll provide another update within 30 minutes.

    Customer experience: Customers hosted in the specified regions may have been unable to access or use multiple core Snowflake services and features. Affected users may have been unable to sign in, execute queries, or manage data. Following initial recovery actions, sign in functionality has been restored, but customers may observe some increased latency or performance issues until service has fully recovered.

    ETA: Our current estimate is that service is recovering and will be fully restored within the next 30 minutes.

    Incident start time: 15:10 UTC June 24, 2026

  • IdentifiedUTC

    Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 30 minutes.

    Customer experience: Customers hosted in the specified regions may be unable to access or use multiple core Snowflake services and features. Affected users may be unable to sign in, execute queries, or manage data.

Scheduled exports and materializations are delayed
ResolvedDegraded performance6 hours 5 minutes
  • UpdateUTC

    Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis (RCA) is available on the Sigma Community site: Postmortem for Delayed Scheduled Exports and Materializations

  • ResolvedUTC

    The mitigation measures have proven effective, and our monitoring indicates that this issue is resolved. We will continue monitoring closely this incident for any recurrence in the next hours. We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in the coming days.

    Impacted Cloud Regions & Services: AWS US-WEST-2

    User Experience: We expect users to no longer experience delayed scheduled exports and materializations. Users should have received previously delayed scheduled exports and delayed materializations should have ran.

    Incident Start Time: Approximately 05:00 UTC June 24, 2026

    Incident End Time: Approximately 17:30 UTC June 24, 2026

  • UpdateUTC

    Mitigation measures have proven effective and our monitoring indicates that delayed scheduled exports and materializations are now being executed.

    Impacted Cloud Regions & Services: AWS US-WEST-2

    User Experience: We expect users to start seeing their delayed exports and materializations come through and new ones process.

    Incident Start Time: Approximately 05:00 UTC June 24, 2026

  • MonitoringUTC

    Our engineers have identified scheduled exports and materializations being delayed due to slow running queries in our database causing scheduled tasks to be backed up. Mitigations measures have been taken to circumvent the issue causing this backup and our engineers are monitoring their effect closely.

    Impacted Cloud Regions & Services: AWS US-WEST-2

    User Experience: Scheduled exports and materializations remain delayed. Users are advised to manually run scheduled exports as needed in the meantime.

    Incident Start Time: Approximately 05:00 UTC June 24, 2026

  • InvestigatingUTC

    We have received reports of scheduled exports and materializations being delayed and not executing on their planned schedules. We are investigating the issue at the highest priority.

    Our preliminary investigations indicate the following:
    Impacted Cloud Regions & Services: AWS US
    User Experience: Scheduled exports not being received at planned time and materializations not being created at scheduled time.
    Incident Start Time: Approximately 5:00 UTC June 24, 2026

    We will provide updates on this page when we have pertinent status updates.

Degraded performance for Directory Provisioning for Authkit
ResolvedPartial outage55 hours 35 minutes
  • UpdateUTC

    Summary

    Between June 22, 2026 at 21:36 UTC and June 25, 2026 at 05:12 UTC, Directory Provisioning for AuthKit was interrupted. During this period, directory updates continued to be received and processed by WorkOS, but downstream AuthKit user and organization membership changes were not applied as expected.

    The issue was caused by an internal data migration being run in production with parameters that unintentionally changed the AuthKit provisioning state for existing customer directories. Once identified, we restored the affected directory provisioning state, backfilled missed user provisioning updates, and deactivated memberships that had not been deactivated during the incident window. No directory events were lost or dropped during this period; however, events received during the incident period were not immediately propagated to AuthKit. Directories created after the migration were unaffected.

    Background

    Directory Provisioning for AuthKit works in two steps: WorkOS receives directory events from a customer's identity provider, then propagates those events downstream to create or update AuthKit users and organization memberships. That second step only happens if the directory is marked as eligible for downstream provisioning. If that eligibility is disabled, WorkOS continues receiving directory events normally, but the corresponding AuthKit updates are not applied.

    What Happened

    On June 22, 2026 at 21:36 UTC, an internal data migration was executed through an administrative migration tool. The migration changed the provisioning status for customer directories in a way that unintentionally disabled Directory Provisioning in AuthKit for affected customers.

    As a result:

    • New directory users created during the incident window were not consistently provisioned into AuthKit.
    • Some directory user updates, including attributes and roles, were not propagated to corresponding organization memberships.
    • Some users who were removed or deactivated in the directory were not fully deactivated in AuthKit during the incident period. This meant that, for the duration of the incident, some users who should have lost access to an organization may have retained it. Based on our review, we found no evidence that this residual access was exercised or misused during the incident window.

    The incident was reported on June 24, 2026 after customer reports of provisioning failures, approximately 42 hours after the migration ran. We did not have sufficient monitoring in place at the time that was sensitive enough to detect an abnormal drop in volume for Directory Provisioning in AuthKit, which is why detection relied on customer reports rather than internal alerting. Our team investigated the behavior, identified the migration as the cause, and immediately worked to restore affected directory provisioning state.

    We then performed additional remediation to address downstream effects from the incident window. This included provisioning missed users, refreshing affected membership attributes and roles, and deactivating stale memberships that should no longer have been active.

    Customer Impact & Recommended Actions

    Customers with Directory Provisioning enabled for AuthKit during the incident window may have experienced delayed user provisioning, attribute/role updates, or deactivation between June 22, 2026 21:36 UTC and June 25, 2026 05:12 UTC. All known gaps from this window have since been backfilled as part of our remediation.

    If you have reason to believe your AuthKit users or organization memberships were affected in a way not addressed by this remediation, please reach out to our support team so we can investigate your specific environment.

    Timeline

    • June 22, 2026 21:36 UTC – An internal data migration unintentionally disabled Directory Provisioning for affected directories.
    • June 24, 2026 15:49 UTC – The incident was reported after customer provisioning failures were identified.
    • June 24, 2026 17:23 UTC – We restored provisioning status for all affected directories, allowing new provisioning activity to resume.
    • June 24, 2026 23:30 UTC – We completed the first remediation pass to propagate provisioning updates missed during the incident period.
    • June 25, 2026 02:10 UTC – We completed remediation to refresh affected role and custom attribute data.
    • June 25, 2026 05:12 UTC – We completed deactivation remediation and considered the incident fixed.

    Remediation

    Immediate Fixes

    We took the following steps to restore service and correct affected data:

    • Restored the provisioning status for affected directories to resume Directory Provisioning in AuthKit.
    • Backfilled user creations, updates, and deactivations made by the identity provider during the incident window that had not been propagated to AuthKit users and organization memberships.
    • Removed the unsafe migration path that caused the incident so it could not be run again.

    Near-Term Improvements

    Additionally, we are making the following changes to reduce the likelihood and impact of similar incidents:

    • Improved monitoring and detection: We are adding additional monitoring and alerting for Directory Provisioning in AuthKit behavior, so we can more quickly respond if a similar issue were to occur in the future.
    • Harden internal migration tooling: We are reviewing access and safeguards for internal migration tools, including stronger controls for destructive operations and clearer scoping requirements.

    Conclusion

    Reliable user provisioning and deactivation are critical parts of identity infrastructure, and this incident did not meet the standard we hold ourselves to. We regret the impact this had on affected customers and their users.

    The incident has been resolved, affected data has been remediated, and we are implementing the controls and monitoring described above to help prevent similar incidents in the future. Please reach out to our team if you have additional questions on the impact to your environments.

  • ResolvedUTC

    The remaining out-of-sync user records are now back in sync.

  • UpdateUTC

    Synchronization has continued to catch up. We are continuing to monitor the few out of sync changes that are remaining.

  • MonitoringUTC

    Provisioning has caught up for user creation, and new directory activity is syncing normally. We are still working through and monitoring the remaining synchronization catch-up, which is not yet fully complete. Another update to follow.

  • UpdateUTC

    We have corrected the affected directories, and they are back to a provisioning status.

  • IdentifiedUTC

    The underlying issue that has lead to a delay in provisioning has been identified, and a fix is being prepared

  • InvestigatingUTC

    We are seeing a delay in Directory Provisioning for Authkit, for any directories created before June 22, 21:36:17 UTC. Any directories created after this time are working as expected.

Previous

Jun 2026 to Aug 2026

Next