You are responsible for breaking a growing application into microservices that will run on Kubernetes and expose internal APIs to each other and a small set of external clients. The system will handle customer data, model artifacts, and operational metadata, and several teams will deploy services independently. Recent incidents in similar environments have included over-permissive service accounts, leaked secrets in CI/CD, and poor visibility into east-west traffic. You need an architecture that improves team velocity without turning the cluster into a flat trust zone.
How would you design the microservices architecture so that service-to-service communication, identity, secrets, policy enforcement, and observability are secure by default? Explain the main trade-offs you would make, the threats you are prioritizing, and how you would verify the controls actually work in production.