API Security across microservices
The first choice for API security in the microservice architecture is OpenID Connect 1.0 and JWT. It’s important to know that there many ways to use JWT incorrectly.
- The JWT created as a result of interactive user authentication is copied to next (nested) API calls. It’s very easy to copy full HTTP Authorization header to next HTTP requests. The problem is that initially created JWT usually will have very wide context of permissions (claims) what contradicts the idea of atomic business/technical scope of a single microservice. JWT level 0 would allow to view and modify user address data, view and modify basket/orders, view transactional history, etc. In case JWT 0 is copied to lower level API invocations microservices below UI have got too much power. It gives a possibility to abuse JWT 0 in deeper tree of API calls by calling originally not intended operations also reaching higher levels. What’s more — read only aggregation service at level N may call N+1 service which would call N-1 write service. With a tree of API call we totally lose access control.
- Some developers think that they can get rid of JWT 0 abuse by retrieving JWTs dedicated for a particular API call — I want to call GetAddress API (HTTP/1.1 GET https://address), it means I need to authorize current microservice (with static user and password) to posses read-address permission (JWT claim). With this static approach we lose user context. GetAddress should be allowed only to return data for a particular user id. The caller authorized to invoke GetAddress shouldn’t be trusted to read data for any user put in the request without prior control.
The only correct solution for delegated and impersonated API calls (in the context of user) is OAuth 2.0 token exchange: https://www.rfc-editor.org/rfc/rfc8693.
resource — A URI that indicates the target serviceaudience — The logical name of the target servicescope — space-delimited list of intended services to be calledsubject_token — identity of the party on behalf of whom the request is being made, usually initial useractor_token — identity of the acting party, so the service using exchanged token
Initial JWT 0 is created with some scope in mind — a set of business domains (Domain Driven Development) which are allowed to be contacted. When an API call is to be executed the JWT 0 should be exchanged for a next JWT with narrowed down set of domains, maybe even just one. With every nested call the scope set should be smaller. Thanks to this we limit the possibility to abuse access rights.
In the communication between microservices we should have API Gateway to verify correctness of HTTPS URL stored in resource field with relation to the audience. API Gateway should allow API calls only matching resource URL (it should be external DNS name of API GW) and logical service name. Microservices should allow connections only from API GW. This protects us from the case when API call is forwarded to a malicious microservice from a malicious container image. This might be an attack to stole JWT.
The subject token should always be used to verify the entity on which the API call is executed. The subject token and entity id in API request should be the same or related (like user and his/her order, address, etc.). The relation should be backed up by primary key or foreign key in the data.
The actor token should be verified and audited if the caller is expected and allowed to execute API.
Every nested call should use own JWT acquired via token exchange. Every nested API call should consume JWT not allowing for it to be reused. JWTs should have an appropriate life time. The expiration embedded into JWT also allows for blocking timed out requests and improve data locking/consistency. If a token is stolen and reused the API call with the attached JWT would be blocked. It’s important not only to check signature using cached public key of IDP issuing JWT, but validate JWT fully by this IDP (see: https://github.com/keycloak/keycloak-documentation/blob/main/authorization_services/topics/service-rpt-token-introspection.adoc). This behavior should be mandatory for crown jewels. Less important or hardened in a different way services can use off-line JWT verification.
Now let’s get back to our initial wrong ways of applying JWT security. In case there is a typo in yaml files for Kubernetes and DevOps deploy a malicious container image instead of a correct one and our JWT 0 (or service-to-service JWT) is stolen we allow for vast number of uncontrolled API calls. When we build API security upon OAuth 2.0 Token Exchange with proper expiration and token disposal after consumption we limit the attack surface or even totally block the attack.
How to prevent an internal attacker from API consumption? We can use firewall and/or network policies. We can also add mTLS. It the JWT was leaked (copied or generated) we still can block an API call from the wrong requestor by validating client’s X509 certificate.
How to secure batch processing when we can’t propagate long lived tokens? Example: Some microservice pushes data to a Kafka topic.
A dedicated API Gateway for Kafka should validate HTTPS sender and it’s payload. In JWT claims there should be Kafka topic names (probably abstracted). API Gateway should verify if claims have got matching topic. A message digest can be computed and added to new JWT attached as Kafka message header. The Kafka consumer should read header, unpack JWT, verify if audience and scope matches. If there is a need for API call there should be next token exchange for every invocation. JWT attached as Kafka header should be used internally only by producer and consumer. The consumer would use JWT to validate that the sender had design-time permissions to send the data. A Kafka topic should have correct ACLs preventing data processing by not allowed parties.
Decisions about access can be offloaded to Open Policy Agent: https://www.openpolicyagent.org/docs/edge/external-data/, https://www.openpolicyagent.org/docs/latest/http-api-authorization/.