12 — Testing, sizing and release acceptance

12 — Testing, sizing and release acceptance#

Handbook index

Evidence, not architecture promises#

Test layers#

Mandatory acceptance matrix#

For T-02/T-03, inspect the controlled upstream's actual received bytes, not only the Zotline decision response. For privacy tests, seed unique canary strings and search every configured persistence/export destination, including exception logs and model diagnostics.

Client support matrix template#

Before GA, replace TBD entries with named versions and evidence:

Do not promote a working demo to broad compatibility. Provider payloads evolve; maintain synthetic adapter fixtures and a controlled compatibility check process. Separate “connects through proxy” from “content inspection and rewriting are certified.”

Performance workload model#

Collect expected users, peak requests/second, concurrency, median/p95/max body size, upload distribution, TLS handshake rate, response streaming duration, semantic-rule fraction, retention and acceptable latency. Separate burst rate from daily average.

Measure deterministic overhead separately from model queueing/inference and provider response time. Report p50/p95/p99 under sustained and burst load, with error/coverage rate. Throughput claims are valid only for the tested payload, rules, model and hardware profile.

Approximate steady-state inspected concurrency as arrival rate × inspection duration. This relationship is a starting estimate; burst queues, long streams and TLS state also consume resources. Estimate in-flight body memory using concurrent buffered requests × bounded decompressed size plus parser/rewriter overhead. A 10 MiB cap and 100 simultaneously buffered bodies already imply roughly 1 GiB of body storage before overhead.

Worked audit sizing example#

Assume 1,000,000 events/day at 1,500 bytes/event and 30-day retention. Raw event payload is 45,000,000,000 bytes, about 45 GB decimal. This excludes indexes, row overhead, WAL, replicas, archives and backups. Measure those factors with the chosen schema rather than assigning an unexplained universal multiplier.

At a peak of 100 events/second and an 8-hour ingestion outage, raw queued events occupy 4.32 GB decimal. Add record/encryption/filesystem overhead and headroom; also test replay throughput while normal traffic continues. These are illustrative inputs, not capacity guarantees for Zotline.

Initial lab sizing#

A single-node deterministic test environment can start at 4 vCPU, 16 GiB RAM and 100 GiB durable storage. This is not a supported minimum until measured. Semantic inference hardware depends on model, precision, context, batching and concurrency; size it separately. Reserve capacity for one-node failure in HA profiles rather than using 100% of aggregate capacity during normal operation.

Accuracy evaluation#

Use labeled synthetic/publicly authorized fixtures covering true positives, benign lookalikes, multilingual text, long contexts, escaped/encoded strings and adversarial instructions. Report per-category false-positive and false-negative rates with sample sizes and known gaps. Natural-language policy quality and regex detection quality are different measurements.

Any model/detector change can change enforcement. Treat it as a versioned release with regression analysis and customer canary, not a transparent dependency patch.

Release evidence package#

Include completed requirement-to-test mapping, support matrix, security scan findings/disposition, privacy egress capture, protocol artifacts, upgrade/restore timings, regional installation results and commercial-flow results. Security and product owners sign off remaining limitations. No critical unmitigated prevention/isolation failure can be hidden as an operational note.