Refractive RPC: pooled, self-healing endpoints for Robinhood Chain

A provider endpoint is a shared resource with a private failure mode. When it rate limits, it returns 429 to everyone at once. When a region degrades, latency rises before errors do, so naive health checks continue to report success while user-visible performance collapses. When a node falls behind, it answers correctly for a block height that is no longer current - the worst failure, because it is indistinguishable from success.

Applications compensate with client-side lists and retries. This works until it does not: retries amplify load against an already-failing provider, timeouts are tuned once and never revisited, and no single application has enough traffic to learn which provider is actually fastest right now.

• Transparency. Identical wire format to a bare node. No client library, no proprietary envelope.
• Safety over availability for writes. A transaction is submitted to exactly one upstream, or the failure is returned.
• Availability over latency for reads. A read may be retried across providers until the deadline is exhausted.
• Monotonic chain view. A client must never observe the chain height move backwards because two upstreams disagree.
• Bounded overhead. Routing must cost materially less than a millisecond at p50.

Four layers. The edge terminates TLS, validates the API key, applies the key’s method allowlist and rate limit, and parses the JSON-RPC envelope including batches. The core holds routing, breakers, the retry policy, in-flight deduplication and the deterministic-result cache. The transport owns keep-alive connection pools, the WebSocket manager and its subscription registry, and validates every response before it is returned. The upstream layer is the set of providers, each with independent health state per region.

State that must be shared between edge nodes - breaker state, rolling latency windows, quota counters - is replicated with a short TTL. A partition between edge nodes degrades routing quality, never correctness: an edge node with stale health data still refuses to route to a provider its own probes have failed.

Selection is a scoring function evaluated per request over the set of providers whose breaker is closed. Weights are configurable per API key, which is how the four named strategies are expressed: round robin, random and weighted are degenerate cases of the same function with latency weight set to zero.

score(p) =
    w_latency  * normalize(p50_rolling(p))
  + w_error    * error_rate_5m(p)
  + w_quota    * quota_burn(p)
  + w_affinity * region_distance(p, caller)

// Weights are per-key. The default profile is read-optimised:
//   w_latency = 0.55, w_error = 0.30, w_quota = 0.10, w_affinity = 0.05
// Providers with an open breaker are excluded before scoring.

Latency is tracked as a rolling p50 over a sliding five-minute window per provider per region, which is responsive enough to follow a degradation but slow enough not to oscillate. Because Prism observes every request from every tenant, its estimate of a provider’s current performance converges far faster than any single application’s could.

Each provider carries a circuit breaker in one of three states. In closed, traffic flows and failures accumulate in a rolling window. Crossing the threshold moves it to open, where the provider is removed from scoring entirely and requests reroute without penalty. After a cooldown it becomes half-open and admits a single probe; success closes it, failure reopens it with an extended cooldown.

Retries are governed by method classification rather than by error code alone. Reads are idempotent and may be replayed on a different provider with exponential backoff and full jitter. State-changing submissions are never replayed automatically: a timeout on a raw transaction submission is returned to the caller, because a duplicate submission is a worse outcome than an ambiguous one.

Pooling introduces a hazard a single node does not have: two upstreams at different block heights. A client that reads height N from a leading provider and then reads state from a lagging one observes the chain moving backwards.

Prism tracks a per-key high-water mark of the greatest block height it has returned to that key. Providers reporting a height below the mark, beyond a small tolerance for propagation, are excluded from routing for that key until they catch up. A provider persistently behind is treated as unhealthy and its breaker trips on drift alone, without a single error being returned.

Routing overhead is the time between the request being parsed and the upstream call being issued: scoring, breaker checks and cache lookup. Measured at p50 this stays under a millisecond, which is one to two orders of magnitude below the network round trip it is choosing between - the selection pays for itself whenever it avoids a provider that is even slightly slower.

Deduplication and caching remove work rather than distribute it. Under bursty read patterns - a page load fanning out identical eth_call requests - collapsing in-flight duplicates reduces upstream volume substantially without changing what any caller observes.

API keys are scoped: each carries a method allowlist, an optional origin lock and an independent rate limit, so a key embedded in a browser can be restricted to reads. Prism never holds private keys and never signs. Transaction payloads arrive pre-signed and are forwarded unmodified; the layer cannot alter a transaction without invalidating its signature, which is the property that makes a shared endpoint acceptable for tokenized real-world assets.

Request bodies are not retained. Metrics record method, latency, status and provider - never parameters, addresses or results.

• Per-method provider scoring: upstreams differ sharply on wide eth_getLogs ranges, and a single score per provider loses that.
• Predicting degradation from the slope of the latency curve rather than waiting for an error threshold to be crossed.
• Whether the high-water mark should be per-key or per-session; per-key is conservative and occasionally excludes a healthy provider unnecessarily.
• Fair-sharing scarce provider quota across tenants during a partial outage.