Reliability: Trace Uploads, Tunnel DNS, Rollout Stability

This week focused on infrastructure reliability—faster trace handling, more stable tunnels, and fewer dropped rollouts.

Automatic Trace Uploads for Large Payloads

Traces over 100KB are now automatically uploaded to blob storage via presigned URLs. This prevents HTTP timeouts when running verifiers with large session histories:

client = GraphCompletionsClient(base_url, api_key, auto_upload_traces=True)
 
# Large traces (>100KB) upload automatically, verifier receives a reference
result = await client.verify_with_rubric(session_trace=large_trace, rubric=rubric)

Max upload size is 50MB. Traces under 100KB still inline as before.

Tunnel DNS Improvements

Cloudflare tunnel verification now uses direct DNS queries to 1.1.1.1 instead of Railway's system DNS. This cuts task app verification from 90+ seconds to under 5 seconds in most cases.

We also stopped resolving quick tunnel URLs to IPs—this preserves SNI for TLS and fixes connection failures on *.trycloudflare.com endpoints.

More Stable Rollouts

Fixed LocalProtocolError failures on Cloudflare quick tunnels caused by:

• Stale pooled connections exceeding Cloudflare's idle timeout (~120s)
• Manual Content-Length headers conflicting with chunked encoding

Rollouts now use dedicated non-pooling clients for tunnel URLs with automatic retry on protocol errors.

Faster Trace Processing

Large rollouts now hydrate 10-20x faster. Jobs with many traces no longer hang during verification.

Streaming Progress for GEPA

Jobs now stream progress via SSE instead of polling:

result = pl_job.stream_until_complete(timeout=3600.0)
# [00:15] running | score: 0.72
# [00:30] running | score: 0.78

Other Fixes

• Image payloads up to 100MB now supported
• MIPRO and GEPA jobs now connect more reliably to local task apps, reducing "network unreachable" failures