Post-mortem: October 2026 dstack guest-agent vulnerability and host isolation fixes

Summary

Between October 8 and October 12, 2026, an attacker exploited a vulnerability in the dstack guest-agent to read container logs and environment variables from a small set of customer applications. On one host node, the attacker also used a network isolation gap between Confidential VMs (CVMs) and host management services to steal a gateway admin token. The attacker used that token to change the certificate-management configuration on that one gateway.

Our automated checks detected the attack on October 12 at 19:28 UTC. We isolated the affected host node within 40 minutes. Within 3 hours, we blocked the vulnerable guest-agent port across all host nodes. We completed all platform fixes by October 15.

Your TEE keys, KMS keys, customer disks, and host nodes are safe. The attacker never accessed enclave keys, sealing keys, on-chain KMS keys, customer filesystems, or host SSH.

The root causes were in our platform, not in customer configuration. We have notified every affected customer with the exact applications, time windows, and data volumes involved. We apologize for the impact and for the disruption caused by the required CVM upgrades.

Customer impact

Affected

• Container logs and environment variables for a small set of customer applications. The attacker read container configurations, which include environment variables (Config.Env). We sent an individual notice to the owner of every affected application with the exact read window and byte count. If we did not notify you about an application, the attacker did not read its logs or environment variables.
• Certificate-management configuration on one gateway. The attacker used a stolen gateway admin token to change DNS-credential and ACME settings on one gateway. Active TLS certificates remained valid, and certificate issuance was not interrupted. We revoked and replaced the Cloudflare DNS token with IP restrictions on the same day.

Not affected

• TEE key material and KMS services. Our Key Management Service (KMS) and key-provider endpoints returned zero bytes to unauthorized requests on every node. Enclave keys, sealing keys, and on-chain (Base and Ethereum) KMS operations are unaffected. We upgraded and re-attested the on-chain KMS on October 14.
• Customer disks and filesystems. The vulnerability exposed only container logs and container configuration. The attacker could not read files on customer virtual disks.
• Host nodes. The attacker never gained SSH or remote-shell access to any host node. All observed login attempts failed.
• All other gateways. We inspected the configuration of every other gateway in the fleet and confirmed that none were changed.

Many customers and internal tools read their own application logs at high volume. During our analysis, we separated this normal traffic from attacker traffic line by line. Everything reported above reflects confirmed cross-tenant access.

Actions required from customers

1. If you received an exposure notice. Rotate any secrets stored in the environment variables of the listed applications. Applications without secrets in their environment variables do not require key rotation.
2. All customers. Upgrade your CVMs to a fixed dstack OS image (v0.6.1 or v0.5.12) by November 19, 2026.
   • Starting November 20, 2026, we will automatically upgrade remaining v0.5.x and v0.6.0 CVMs in batches. We will announce each batch on our status page at least 24 hours in advance.
   • CVMs on v0.3.x OS images have no direct upgrade path. We will retire v0.3.x CVMs on November 20, 2026. Enterprise plans are excluded from automatic retirement, and our team is working directly with those customers to migrate their workloads.

What happened

System context

Each customer application runs inside a Confidential VM (CVM) on a shared host node. Inside each CVM, an in-guest management agent called the guest-agent provides management and TEE attestation functions.

Phala Cloud is a verifiable compute platform. Anyone can remotely verify what code runs inside a deployment without trusting Phala and without needing platform credentials. To support this, the per-node gateway exposes the guest-agent read-only endpoints on the public internet at <app>-8090.<gateway-domain>.

On this public route, the guest-agent serves three things:

1. Info endpoint. Returns the application identity (name, app ID, and instance ID), its TEE measurements (MRTD, RTMRs, mr_aggregated, OS-image hash, and compose hash), and the TDX event log when public_tcbinfo is enabled (the platform default).
2. Quote endpoint. Returns a TDX attestation quote that cryptographically binds to the application signing key. Together, the Info and Quote endpoints let application owners and end users verify that unmodified code runs inside a genuine TEE.
3. Logs endpoint (/logs/<container>) and dashboard. Lets operators read container logs remotely when public_logs is enabled in the application compose file.

Separately, host management services (telemetry and firewall control) run on local interfaces on the host node. These host management services were never meant to be reachable from tenant CVMs.

How the vulnerability worked

Two weaknesses combined in this incident.

1. Missing container-name validation in the guest-agent (Root cause 1)

The public guest-agent route was built as a read-only attestation and log interface. However, in dstack OS images v0.3.1 through v0.6.0, the /logs/<container> endpoint did not validate the <container> name before passing it to bollard, the upstream Rust client library for the Docker API.

The bollard library inserts the container name directly into the Docker API request path without escaping special characters. By sending a crafted container name, the attacker turned a log request into a Docker container-inspect call. That call returned the container’s full configuration, including Config.Env, where applications commonly store secrets in environment variables.

We fixed this vulnerability on October 13 in dstack#1442 (released in v0.6.1) and backported the fix in dstack#1443 (released in v0.5.12). We also reported the request-path handling issue to the bollard maintainers in bollard#832. A CVE ID for the guest-agent vulnerability is being assigned.

2. Insufficient network isolation between tenant CVMs and host management services (Root cause 2)

To steal the gateway admin token on one host node, the attacker combined the guest-agent vulnerability with a host network isolation gap:

• Tenant CVMs using user-mode (SLIRP) networking could reach local management ports on the host node.
• Host firewall rules filtered only forwarded external traffic. They did not filter traffic originating from processes on the host itself, which is the path a local CVM uses.
• Some local host management APIs did not require authentication.

Attack sequence

1. The attacker submitted code to a customer application that is designed to execute user-submitted code.
2. On October 11 at 22:03 UTC, from inside that application’s CVM, the attacker scanned the host node’s internal network and reached an unauthenticated host management API.
3. The attacker used that host management API to change firewall rules on that single host node.
4. Next, the attacker exploited the guest-agent vulnerability against the node’s gateway container. This returned the gateway container’s configuration and exposed the gateway admin token.
5. Between October 11 at 23:01 UTC and October 12 at 01:04 UTC, the attacker used the stolen gateway admin token to change DNS-credential and ACME settings on that single gateway.
6. Separately, between October 12 at 00:47 UTC and 17:55 UTC, the attacker exploited the guest-agent vulnerability over the public gateway route to read container logs and environment variables from other tenant applications.

Contributing factors

We also identified and addressed four factors that increased the impact of this incident:

• Shared gateway admin tokens. Several gateways shared the same admin token. Stealing the token from one node put other gateways at risk, even though the attacker modified only one gateway. Every gateway now uses a unique admin token.
• public_logs enabled by default in console templates. The underlying dstack platform sets public_logs to off by default, but our web console template set public_logs=true. As a result, most applications had public log routes enabled without a deliberate choice by the developer.
• Missing source-IP logs and tokens in URLs. The gateway admin API did not log caller source IPs. In addition, an admin token passed in a URL query string was written to a container log that the guest-agent vulnerability could read.
• 7-day security log retention. We retained security logs for 7 days. Seven days covered the confirmed attack window (October 8 to October 12), but we are extending log retention to support longer forensic lookbacks.

During our investigation, we also found a Server-Side Request Forgery (SSRF) vulnerability in our ACME HTTP-01 relay. Log evidence shows that the attacker used it only to read 238 KB of logs from one application. We found no evidence that it was used to enter any host node. We disabled the ACME HTTP-01 relay across the fleet on October 12, fixed the URL validation vulnerability, and re-enabled the hardened relay on October 13.

Detection and response

An automated reconciliation check detected the attack on October 12 at 19:28 UTC, when certificate and CAA reconciliation failed with HTTP 400 on the affected gateway. This check triggered roughly 17.5 hours after the first malicious action on the host node and 1.5 hours after the last cross-tenant log read.

Our team executed the response in three phases around the clock.

Day 0, October 12 (Emergency containment)

• Within 40 minutes of detection, we isolated the affected host node.
• Within 3 hours of detection, we blocked the guest-agent port at the load balancer across the fleet, stopped the ACME HTTP-01 relay, banned the attacker IP addresses, and rotated the Cloudflare DNS token.

Day 1, October 13 (Fixed releases and core infrastructure cutover)

• We merged and released fixed dstack OS images (v0.6.1 and v0.5.12) and updated our host management services.
• We completed fleet-wide log forensics, which confirmed the exact set of affected tenant applications and proved that KMS and TEE key material were unaffected.
• We upgraded core infrastructure node by node across the fleet: Cloud KMS, gateways, and host management services. On every host node, we verified KMS key delivery, gateway re-registration, and application health.
• During our fleet sweep, we found a second, unrelated gateway admin port exposure and closed it the same day.

Day 2, October 14 to October 15 (Gap closure and hardening)

• We set host firewalls to default-deny across the fleet.
• We deployed a host-level CVM-to-host access guard in enforce mode to block tenant CVMs from connecting to host services.
• We deployed per-node guest-agent deny lists and restricted new CVM creation to fixed OS images.
• We assigned unique admin tokens to all gateways, upgraded and re-attested the on-chain KMS (Base and Ethereum), restored the affected gateway configuration, and rotated internal Phala secrets.

Timeline (UTC)

Remediation and follow-up

Every weakness identified in this incident maps to a concrete engineering change.

Lessons learned

What went well

• Automated detection caught the attack. Our certificate reconciliation check detected the gateway configuration change automatically before any customer report.
• Fast initial containment. Once the alert fired, we isolated the affected host node within 40 minutes and blocked the vulnerable port across the fleet within 3 hours.
• Verified per-customer forensics. By analyzing the full retained log set, we produced an exact, verified exposure list for each affected customer rather than an estimate.

What went wrong

• We trusted an upstream library without auditing how it built request paths. The guest-agent vulnerability persisted from dstack v0.3.1 to v0.6.0 because we passed unvalidated user input directly into bollard.
• Our host network did not treat tenant CVMs as untrusted. A tenant CVM should never be able to reach a local host management port.
• We shared gateway admin tokens across nodes. A single read on one node exposed a credential used by multiple gateways.
• 7-day security log retention was too short. Seven days covered this incident, but a slower attack could have exceeded our retention window.

Where we need to improve

• Shorten the gap between intrusion and detection. Although an automated certificate check caught the attack, 17.5 hours passed between the attacker’s first action on the host node and our alert. Our log inspection and anomaly checks did not run frequently enough. We are increasing our log inspection frequency and adding real-time alerts on gateway admin actions, firewall changes, and abnormal guest-agent requests so we can detect intrusions within minutes rather than hours.
• Speed up fleet-wide infrastructure rollouts. While we blocked the vulnerable port within 3 hours on Day 0, rolling out the permanent fixes, default-deny firewalls, and host access guards across every node took nearly three days (October 13 to October 15). Much of this rollout required node-by-node manual verification. We are automating our host fleet deployment and verification pipeline so we can roll out security updates across all host nodes much faster.

Where we got lucky

• Although the stolen token was shared across multiple gateways, the attacker used it on only one gateway and altered only certificate settings rather than intercepting traffic.
• Because the exploit required a specific crafted container name, the attacker’s read volume stayed bounded.

None of this luck is a security control. Every failure above maps to a permanent fix in our remediation table.

Next steps

All platform-side fixes are live across our fleet. Our remaining focus is helping customers upgrade their CVMs before the November 19, 2026 deadline. We will publish a final update after November 20, 2026, once the fleet-wide CVM upgrade program is complete.

If you have questions or need help upgrading your CVMs, please contact [email protected].