Service Outage
Resolved
Jun 14 at 07:08pm CDT
Root Cause Analysis of outage on June 13, 2023
BLINK experienced an outage on June 13th from approximately 1:52PM to 2:20PM Central time.
Impact Assessment:
The functions impacted were primary redirect services, multiple API endpoints, and web console functions. This outage impacted both self-service and enterprise customers.
Timeline:
(All times central. GMT -5)
1:52pm - initial alerts notified of 502 Bad Gateway response
2:08pm - AWS confirmed issues identified in the US-East Region were impacting all lambda functions (which BLINK utilizes in numerous places)
2:17pm - BLINK modified our services to skip the lambda services and process all traffic through another technology. All services started to recover.
2:20pm - All alerts cleared and systems were fully operational again.
4:00pm - AWS indicated that services were restored, but backlogged. BLINK monitored the situation until we felt confident that lambda services were operational.
5:15pm - BLINK reintroduced lambda services back into production. No issues were encountered.
Root Cause Identification:
The issue stemmed from the lambda functions being used to serve redirects and API calls. BLINK utilizes lambda as a primary application layer in our platform. While BLINK does not depend exclusively on this service, the failover to bypass the lambda layer is manual and required engineer intervention to transition.
Action Plan:
BLINK has been actively building a new structured architecture over the last 10 months that will provide a fully-dispersed footprint that utilizes multiple regions with immediate, automatic failover. Customers already on this new platform were not impacted by the outage providing real-world validation of the new architecture. This system is active today and we are working to transition to this new infrastructure in the near future. BLINK will begin sharing more details about this update in the coming months. Any immediate questions may be directed to help@bl.ink
Affected services
BL.INK Enterprise: USA
BL.INK Core Platform
Updated
Jun 13 at 03:00pm CDT
AWS services appear to be operational again, however an official update has not yet been announced so we are still operating without the lambda services. All functions are operating normally and without issue. We are considering this outage Resolved at this time and will reintroduce Lambda services back into our stack once we are confident the issue has been resolved. There are no anticipated outages during the Lambda reactivation so a formal announcement will not be made. A root cause analysis of this issue will be posted within 24 hours with additional details.
Affected services
BL.INK Enterprise: USA
BL.INK Core Platform
Updated
Jun 13 at 02:25pm CDT
We have identified the source of the outage and confirmed it is related to the AWS Lambda outage in the US-East region. We have reconfigured our systems to bypass the Lambda services and all functions appear to be restored. We are currently reviewing all systems to confirm operations. We will post another update by 3pm Central time with additional information.
Affected services
BL.INK Enterprise: USA
BL.INK Core Platform
Created
Jun 13 at 02:05pm CDT
We are currently experiencing a service outage across redirects and management functions. We will post additional information as soon as it becomes available.
Affected services
BL.INK Enterprise: USA
BL.INK Core Platform