Incident: Team Valley Packet Loss

Incident Report for Wildcard Networks

Postmortem

This incident was caused by a faulty SFP module in our suppliers core network causing a port to flap, resulting in small bursts of packet loss.

Posted Jun 10, 2025 - 10:37 BST

Resolved

No further packet loss has been observed, we now believe this to be resolved.

An RFO will be provided on this page once available.
Posted Jun 08, 2025 - 13:45 BST

Monitoring

Hardware has been replaced at the exchange - Monitoring for improvements
Posted Jun 08, 2025 - 04:33 BST

Update

Engineers are en route to the Gateshead exchange to investigate a potential fault with core hardware.

They're expected on site at 1am, with updates expected within an hour of their arrival.
Posted Jun 08, 2025 - 00:24 BST

Identified

Bursts of packet loss are still being observed - Our supplier has engaged additional teams to find the root cause.
Posted Jun 07, 2025 - 21:10 BST

Monitoring

Hardware has been replaced - We're monitoring for improvements
Posted Jun 07, 2025 - 18:59 BST

Update

Our supplier is expecting to commence work to replace hardware at their exchange at 17:30

Please be aware this will cause circuits to be hard down, and will also impact some other leased lines with Virgin tail circuits,
Posted Jun 07, 2025 - 16:47 BST

Update

Our supplier has identified a flapping port at the head end that may be responsible for the spikes in packet loss - An engineer is being dispatched to replace modules and clean fibres.
Posted Jun 07, 2025 - 13:03 BST

Identified

A small amount of packet loss is still being observed - We're working to determine the next steps
Posted Jun 07, 2025 - 09:35 BST

Monitoring

Further hardware has been replaced, we are monitoring for improvements.
Posted Jun 07, 2025 - 00:53 BST

Identified

A small amount of packet loss has been observed again, and our supplier is still seeing errors on the link.

Further replacement hardware has been ordered with a view to replacing around between 22:00 - 23:00.
Posted Jun 06, 2025 - 20:35 BST

Monitoring

Following engineer work, we are monitoring this for improvements.
Posted Jun 06, 2025 - 19:17 BST

Update

Our supplier will be commencing a hardware replacement and intrusive circuit test within the hour.

Downtime of 30 minutes should be expected.
Posted Jun 06, 2025 - 18:23 BST

Identified

CRC errors are still being observed on our backhaul, as such our supplier is sourcing replacement hardware.

A short outage should be expected later whilst hardware is replaced - Updates will be provided once an ETA is known
Posted Jun 06, 2025 - 16:48 BST

Monitoring

We have replaced hardware and are monitoring for improvements
Posted Jun 06, 2025 - 14:26 BST

Identified

We're aware of an issue impacting the Team Valley backhaul circuit causing this loss.

We're dispatching an engineer to replace hardware at the end of the circuit - Users may experience a brief outage (<1m) whilst this happens.
Posted Jun 06, 2025 - 12:34 BST

Investigating

Impact Period: Ongoing
Service Impact: Packet Loss / Intermittent Connectivity

We're currently aware of issues in the Team Valley area causing intermittent packet loss for leased line and FTTP customers. This is being investigated.
Posted Jun 06, 2025 - 09:19 BST
This incident affected: Broadband Services (Fibreconnect Full-Fibre) and Wildcard Local Fibre Network (Gateshead).