Notice #485269
Recovering

NKE Storage Backend Issues

Began:

We are still seeing recurring issues, but we’ve implemented a workaround to limit impact on customer workloads. Since the issue has reappeared, we are continuing to work with the vendor to implement a permanent fix.

The issue has been resolved, and any pending tasks that could not complete have been worked through. According to our monitoring, everything is back to normal. We will keep a close eye on the situation and will close the status tomorrow morning if it remains stable.

The issue is still ongoing and we are actively working to resolve it with the vendor. For now, we recommend not launching new deployments or updates to existing workloads until further notice.

The problem has been identified and we're working on a fix together with the vendor.

Due to an issue with our hypervisor, we're currently experiencing problems managing PVCs (Persistent Volume Claims) on NKE clusters. Kubernetes based workloads with storage requirements (NKE workloads/Deploio/vClusters/certain managed on-demand services) may be impacted. We've opened a ticket with the vendor and are actively working on a resolution.

  • General
  • Virtualization
  • Managed Kubernetes
  • Network
  • External
  • E-Mail
Upcoming Maintenance

[Scheduled] Multiple planned maintenances of the virtualization infrastructure

  • Virtualization

We are upgrading some software components of our virtualization environment in order to improve its stability and reliability.

No service disruptions are expected and there will be no downtime of the virtual servers or platform services.

Maintenance work will only be performed during the night, the planned time windows are listed below.

  • 23.02.2026 (23:00 - 06:00): Colozüri CZ41 (part 1/4)
  • 24.02.2026 (23:00 - 06:00): Colozüri CZ41 (part 2/4)
  • 25.02.2026 (23:00 - 06:00): Colozüri CZ41 (part 3/4)
  • 26.02.2026 (23:00 - 06:00): Colozüri CZ41 (part 4/4)

[Scheduled] Planned maintenance of the NKE files service

  • Virtualization
  • Managed Kubernetes

We will perform a software upgrade on the NKE files service (NFS server) on the ES34 cluster, this affects the Read Write Many (RWX) storage.

Due to restarts of the services, connection interruptions and momentary service disruptions are to be expected. Everything should recover automatically once the services are running again.

[Scheduled] Planned Object Storage maintenance

  • General
  • Virtualization

We will perform a software upgrade on our S3 storage systems in both datacenter locations.

Connection interruptions and momentary service disruptions are to be expected during the maintenance window.

Previous Incidents

[] Multiple planned maintenances of the virtualization infrastructure

Began: Ended: Duration:
  • Virtualization

We are upgrading some software components of our virtualization environment in order to improve its stability and reliability.

No service disruptions are expected and there will be no downtime of the virtual servers or platform services.

Maintenance work will only be performed during the night, the planned time windows are listed below.

  • 09.02.2026 (23:00 - 06:00): NTT ES34 (part 1/4)
  • 11.02.2026 (23:00 - 06:00): NTT ES34 (part 3/4)
  • 12.02.2026 (23:00 - 06:00): NTT ES34 (part 4/4)

The scheduled maintenance is now underway. We'll keep you updated on our progress.

Part one of the maintenances has completed, we will continue with part two as planned tonight at 23:00 CET.

Due to the issues that resulted from last night's upgrade, tonight's maintenance window (part 2/4) has been postponed.

The maintenance scheduled for tomorrow night (part 3/4) should proceed as planned, starting at 23:00 CET.

Following issues encountered during the first upgrade, we have decided to cancel the remaining maintenance windows this week and reschedule them. This allows us to monitor the environment and make sure everything runs smoothly before proceeding.

We will provide you with a new schedule as soon as possible.

[Resolved] Managed Grafana issues

Began: Ended: Duration:
  • Managed Kubernetes

We are currently having issues with our managed on-demand Grafana instances and are investigating into the problem.

We identified the underlying issue and found a more generic NKE storage backend problem being the cause for this. A separate status entry was created for it, where we will track progress in.

The issue has been resolved, and any pending tasks that could not complete have been worked through. According to our monitoring, everything is back to normal. We will keep a close eye on the situation and will close the status tomorrow morning if it remains stable.

[Resolved] BGP Routing Updates not correctly processed in CZ41 and ES34

Began: Ended: Duration:
  • Network

We are currently investigating why route updates in our BGP Core Network are not processed correctly in the locations CZ41 and ES34.

After a fade out of the BGP sessions in ES34 the situation in ES34 stabilized. After enabling a few sessions again, routing update processing is stuck again. We will fade out all traffic on this router to prepare for an emergency reboot.

The reboot did not fix the issue. We disabled now affected IPv4 BGP Upstream Transits. Currently the situation is stable and traffic is flowing through other Transits. We currently collect information for a case with the vendor.

We've now resolved the incident together with our vendors TAC. Thanks for your patience.

No further notices from the past 30 days.