tag:cirun.instatus.com,2005:/history Cirun.io Status - Incident history 2026-02-20T00:10:28.521+00:00 Cirun.io tag:cirun.instatus.com,2005:Incident/cmlu4vuay0in9qmpzr37hpf6c 2026-02-20T00:10:28.521+00:00 2026-02-20T00:10:28.528+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Feb 20, 00:10:28 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Feb 20, 00:11:40 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmlu2f3n00amme0oecb6iqy1b 2026-02-19T23:01:28.236+00:00 2026-02-19T23:01:28.250+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Feb 19, 23:01:28 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Feb 19, 23:02:44 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmkgdn1fi01wbezttap3o5w9v 2026-01-16T04:27:05.597+00:00 2026-01-16T04:27:05.608+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Jan 16, 04:27:05 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Jan 16, 04:28:07 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmk2wvg2s0091134nclwgv3rs 2026-01-06T18:16:44.067+00:00 2026-01-06T18:17:12.480+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Affected Components: Webhook

Jan 6, 18:17:12 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

Jan 6, 18:16:44 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmiyfz7180oagrwl6ru9wm93j 2025-12-09T10:32:58.460+00:00 2025-12-09T10:32:58.467+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 4 minutes

Affected Components: Webhook

Dec 9, 10:32:58 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Dec 9, 10:36:30 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmiydmmqy02qsxocbue759wp9 2025-12-09T09:27:13.066+00:00 2025-12-09T09:27:13.079+00:00 Task Queue - Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 19 minutes

Affected Components: Webhook

Dec 9, 09:27:13 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Dec 9, 09:46:12 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmir99v7j0234gq7qjgwsngs0 2025-12-04T09:50:55.807+00:00 2025-12-04T09:50:55.824+00:00 Task Queue - Webhook had degraded performance Type: Incident

Duration: 6 minutes

Affected Components: Webhook

Dec 4, 09:50:55 GMT+0
Investigating - Task Queue - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Dec 4, 09:56:48 GMT+0
Resolved - Task Queue - Webhook is now operational! This update was created by an automated monitoring service..

Dec 4, 10:15:02 GMT+0
Resolved - Incoming webhook traffic increased faster than the autoscaling configuration could provision additional capacity. The delay between load detection and new instances becoming available caused request processing latency to spike. This has been resolved..

]]> tag:cirun.instatus.com,2005:Incident/cmhkpedz60203jyjx11leug7w 2025-11-04T15:08:15.041+00:00 2025-11-04T15:08:15.048+00:00 Webhook Processing Outages - October 31 - November 4, 2025 Type: Incident

Duration: 6 minutes

Affected Components: Webhook

Nov 4, 15:08:15 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 4, 15:13:49 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

Nov 10, 00:38:03 GMT+0
Postmortem - **Postmortem - November 10, 2025** Between October 31 and November 4, we experienced three separate webhook processing outages that disrupted GitHub webhook delivery to Cirun. This caused delays and failures in CI/CD pipeline triggers for our customers. Total affected time was approximately \~7 hours across the three incidents. **What happened:** Our webhook processing system became unstable following infrastructure changes we deployed on October 30\. The issues manifested as intermittent worker crashes, causing webhooks to queue up or fail entirely. **Root cause:** We made three simultaneous changes to our infrastructure: 1. Migrated our Redis cluster to new infrastructure 2. Upgraded Celery (our distributed task queue library) to the latest version 3. Switched worker concurrency model from pre-fork (multi-process) to gevent (single-process, green threads) for better memory efficiency The gevent migration exposed critical bugs in Celery (#8030, #9091) that weren't caught in our testing. Under production load, workers would crash and fail to restart properly, leaving webhooks unprocessed. **Why it happened multiple times:** Our initial fixes on October 31 were partial rollbacks that didn't address the core issue. We were trying to keep some performance improvements while fixing stability, which backfired. We should have done a complete rollback immediately. **Final resolution:** * Complete rollback to pre-fork worker model * Reverted Celery to previous stable version * Isolated the Redis migration as a separate change (which wasn't causing issues) * Improved worker health checks and automatic restart mechanisms * Implemented better alerting for webhook queue depth **What we're doing differently:** 1. No more simultaneous infrastructure changes - one at a time, with proper bake time 2. Production-scale load testing before any concurrency model changes 3. Faster rollback decision-making - don't try to fix forward during an active incident 4. Better status page messaging that explains customer impact, not just system status 5. Automatic escalation when webhook processing falls behind by more than 10 minutes This was our worst operational week in 5 years. We're apologies for the inconvenience caused. We've learned from this and have made changes to ensure it doesn't happen again. If you have questions about this incident or how it affected your workflows, reach out to us directly..

]]> tag:cirun.instatus.com,2005:Incident/cmhkojz1m01nx9c7syrkzhz47 2025-11-04T14:44:36.010+00:00 2025-11-04T14:44:36.030+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 13 minutes

Affected Components: Webhook

Nov 4, 14:44:36 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 4, 14:58:00 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhknwoap011hc3ryj7wfpstn 2025-11-04T14:26:28.992+00:00 2025-11-04T14:26:29.018+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 8 minutes

Affected Components: Webhook

Nov 4, 14:26:29 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 4, 14:34:32 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhjgcamj01lq8zjja4m5d3b4 2025-11-03T18:06:54.666+00:00 2025-11-03T18:06:54.682+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 3 minutes

Affected Components: Webhook

Nov 3, 18:06:54 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 18:10:16 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhje4lpm019ck7sg61elvwfm 2025-11-03T17:04:56.553+00:00 2025-11-03T17:04:56.567+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 2 minutes

Affected Components: Webhook

Nov 3, 17:04:56 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 17:06:39 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhjbvnku00zd5lopztsob5la 2025-11-03T16:01:59.838+00:00 2025-11-03T16:01:59.861+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 3 minutes

Affected Components: Webhook

Nov 3, 16:01:59 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 16:04:42 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhjblqsn00fxq09t7x7i936l 2025-11-03T15:54:17.447+00:00 2025-11-03T15:54:17.473+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 3 minutes

Affected Components: Webhook

Nov 3, 15:54:17 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 15:56:59 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhjaq0og00ngi20qf17td7zf 2025-11-03T15:29:37.263+00:00 2025-11-03T15:29:37.277+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 4 minutes

Affected Components: Webhook

Nov 3, 15:29:37 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 15:34:01 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhjail7m01apsvbj7avcx1ja 2025-11-03T15:23:50.625+00:00 2025-11-03T15:23:50.647+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Nov 3, 15:23:50 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 15:24:37 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhj2yrv00099ueij7xfrwxjq 2025-11-03T11:52:28.812+00:00 2025-11-03T11:52:54.817+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Affected Components: Webhook

Nov 3, 11:52:54 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

Nov 3, 11:52:28 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhj2l15q0083wkpmqd6n2k1y 2025-11-03T11:41:47.677+00:00 2025-11-03T11:41:47.699+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Nov 3, 11:41:47 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 11:42:21 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhj21a8a006hwkpmzuec4dpc 2025-11-03T11:26:26.314+00:00 2025-11-03T11:26:26.336+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Affected Components: Webhook

Nov 3, 11:26:26 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 11:26:42 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhizno8v00m010mx1o5ofgin 2025-11-03T10:19:52.062+00:00 2025-11-03T10:19:52.081+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 1 minute

Affected Components: Webhook

Nov 3, 10:19:52 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 10:21:00 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhizgrdk018xuqlnctbpviso 2025-11-03T10:14:29.528+00:00 2025-11-03T10:14:29.541+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Affected Components: Webhook

Nov 3, 10:14:29 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 3, 10:14:51 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmhe156ix005820v2j18ypwza 2025-10-30T23:02:37.640+00:00 2025-10-31T04:47:21.773+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 5 hours and 45 minutes

Affected Components: Webhook

Oct 31, 04:47:21 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

Oct 31, 05:10:35 GMT+0
Resolved - This incident has been resolved..

Oct 30, 23:02:37 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmh3qd2ai00oqm2y7emb0wn5i 2025-10-23T18:03:07.865+00:00 2025-10-23T18:03:07.879+00:00 Website isn't accessible at the moment (automated) Type: Incident

Duration: 5 minutes

Affected Components: Website

Oct 23, 18:03:07 GMT+0
Investigating - Website cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 23, 18:08:14 GMT+0
Resolved - Website is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmh3j5zw6000hv7ew079dcio9 2025-10-23T14:41:40.853+00:00 2025-10-23T14:41:40.877+00:00 Website isn't accessible at the moment (automated) Type: Incident

Duration: 8 minutes

Affected Components: Website

Oct 23, 14:41:40 GMT+0
Investigating - Website cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 23, 14:49:14 GMT+0
Resolved - Website is now operational! This update was created by an automated monitoring service..

]]> tag:cirun.instatus.com,2005:Incident/cmh3j4qso002ovflhc9de8ix2 2025-10-23T14:40:42.408+00:00 2025-10-23T14:40:42.423+00:00 Webhook isn't accessible at the moment (automated) Type: Incident

Duration: 2 minutes

Affected Components: Webhook

Oct 23, 14:40:42 GMT+0
Investigating - Webhook cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 23, 14:42:22 GMT+0
Resolved - Webhook is now operational! This update was created by an automated monitoring service..

]]>