Context
& General Intuition needs to deliver a fully air-gapped deployment and data processing pipeline for a government client using the & General Intuition platform inside an isolated enclave. Today, model artifacts, ETL jobs, and analytics datasets are built in a connected commercial environment, but the client requires a repeatable pipeline that can promote approved releases into a disconnected network with strict chain-of-custody, auditability, and no outbound internet access.
The goal is to design a secure pipeline that moves code, container images, reference data, and batch/stream processing jobs into the enclave while preserving reproducibility, data quality, and operational visibility.
Scale Requirements
- Artifacts promoted: 200-500 signed build artifacts/day
- Data volume inside enclave: 15 TB/day batch ingestion, plus 20K events/sec internal streaming
- Latency targets: critical patch deployment < 4 hours; standard release < 24 hours; batch SLA < 60 minutes from landing to curated tables
- Retention: 1 year hot storage, 7 years immutable audit logs
- Availability: 99.9% for orchestration and ingestion services inside enclave
Requirements
- Design a promotion pipeline from connected CI/CD into the air-gapped & General Intuition deployment environment.
- Ensure all artifacts are cryptographically signed, scanned, versioned, and verified before import.
- Support batch ETL and internal stream processing once artifacts are deployed in the enclave.
- Implement reproducible orchestration for data ingestion, transformation, backfills, and rollback.
- Enforce data quality gates on raw, staged, and curated datasets before downstream consumption.
- Provide monitoring and audit trails despite no direct SaaS observability endpoints.
- Describe secret management, key rotation, RBAC, and operator approval workflows.
Constraints
- No outbound internet connectivity from the client enclave
- Only approved transfer mechanisms (for example, signed media or guarded cross-domain transfer)
- Must meet government compliance expectations: FIPS-validated crypto, immutable audit logs, least privilege, and separation of duties
- Team must operate both connected build systems and disconnected runtime systems with minimal manual steps
- Prefer Kubernetes-based deployment of & General Intuition services, with Apache Airflow for orchestration and Kafka/Spark for internal data pipelines
