TL;DR If an endpoint is behind AWS API Gateway, it can’t return WWW-Authenticate header since it will be remapped to x-amzn-remapped-WWW-Authenticate due to a limitation on AWS’ side. As workaround on the caller side, the caller could be modified to send the Authorization header with credentials preemptively, so that the WWW-Authenticate header never must be returned by the endpoint. If modifying the caller is not possible, the only option is to move the authentication to a lambda authorizer and setting up a gateway response.

When working at a fast pace that using cloud technologies can (but not always) make possible, one can start to assume that many things will just work. From configuring and starting a virtual machine or a database to using speech to text as a service, many things can be accomplished quickly and simply. However, we have to be ready for surprises too.

The problem to solve

Since in reality not all actions can be performed immediately (e.g. think bringing an item to a workshop for repair), managing workflows can often benefit from an asynchronous approach. Rather than waiting for the result of the action (sometimes referred to as busy wait), we can create systems, where the initiator of the action is notified when there is a result (you get a phone call that the repaired item can be picked up). As result, the initiator can do meaningful work in the meantime instead of waiting.

This approach can be found in software engineering at many levels ranging from calling functions at code level to interactions between systems. As an example for the latter, we’re going to look at a case, where once some action is ready, the system that executed it must announce the action’s completion by sending an HTTP request to some of the initiator’s systems.

A service that can be used to easily implement this, in case we are in an AWS cloud environment, is SNS.

Using SNS

SNS is a publish-subscribe messaging service where messages can be published to topics, and messages from the topics are delivered to each topic’s subscribers via a variety of methods like conventional SMS messages, email, HTTP etc. It offers a wide range of capabilities (which would deserve their own article) which can save a lot of effort for the teams that are using it. For example in many cases having a dead letter topic and delivery retries with backoff are a must, but all these are already built in into SNS.

The architecture

In our case a lambda function, to announce an action’s completion, will publish a message to SNS topic A, to which destination A and B are subscribed using HTTPS protocol. Undeliverable messages will be sent to a dead letter queue, to which if messages arrive, it will trigger an email notification to be sent to the development teams.

Destination A and B will need to receive these notifications, and to do so, they will expose an HTTPS endpoint. They will have to do the following:

  • expose a public endpoint accessible by SNS
  • support basic authentication, be able to return HTTP status code 401 in case the credentials are incorrect or not present
  • be able to return a 2xx status code to acknowledge the notification’s receival, be able to return codes outside the range 2xx-4xx to indicate failed receival, for which SNS will retry the delivery
  • validate the notification’s signature generated by SNS

As a twist, in this scenario the destination endpoints also happen to be using an AWS service: they are publicly exposed via API Gateway.

Implementing the HTTP endpoint isn’t difficult, neither is the creation of a subscription. One just needs to keep in mind that if the target endpoint has basic authentication, then the URL has to be in the format https://username:password@site.com.

Unexpected difficulties

After creating the new subscription and waiting for the first, the confirmation request to arrive from SNS, we face the following in the incoming request’s logs:

Headers: 
Accept-Encoding=[gzip,deflate],
Content-Type=[text/plain; charset=UTF-8],
Host=[<...>.execute-api.us-east-1.amazonaws.com],
User-Agent=[Amazon Simple Notification Service Agent],
x-amz-sns-message-id=[d21eca99-eb95-4014-afb8-42279fcfd620],
x-amz-sns-message-type=[SubscriptionConfirmation],
x-amz-sns-topic-arn=[arn:aws:sns:us-east-1:<...>:notifications],
X-Amzn-Trace-Id=[Root=1-6363ff93-2cc15c4b270cbb080cfbf1b4],
X-Forwarded-For=[<...>],
X-Forwarded-Port=[443],
X-Forwarded-Proto=[https]

There is no Authorization header, there are no credentials included as we would expect.

This is because SNS doesn’t send the authentication credentials preemptively (despite the fact that they have been specified in the subscription URL), it expects the destination endpoint to return first the WWW-Authenticate header to specify the authentication method to use in a next, repeated request as defined in RFC 2617.

Adding a response header for the no credentials supplied case isn’t difficult, let’s have a look at the result:

Headers:
Date=[Thu, 03 Nov 2022 18:33:24 GMT]
Content-Type=[application/json],
Content-Length=[0],
Connection=[keep-alive],
x-amzn-RequestId=[99f72c6c-9984-439f-95fc-a727c5b6016c],
x-amzn-Remapped-content-length=[0],
x-amz-apigw-id=[bCZqKGwvoAMFtGg=],
x-amz-Remapped-WWW-Authenticate=[Basic],
X-Amzn-Trace-Id=[Root=1-63640974-61e564fc05b16a982dd01dd3;Sampled=0]

We can notice that the WWW-Authenticate header returned by the destination endpoint as response to the unauthenticated request has been remapped to x-amz-Remapped-WWW-Authenticate by API Gateway. Of course, this remapped header won’t trigger SNS to resend the request with basic authentication, as we would expect.

We can try to rewrite the headers passing through API Gateway, but it won’t work. The remapping of this header is simply a limitation of this service. The reason for this is that theoretically, both the gateway and the service behind it could ask for authentication, therefore return WWW-Authenticate, so there has to be a way to differentiate between these scenarios, hence the remapping.

There is a workaround

Fortunately there is a possibility to return a WWW-Authenticate header, however it means modifying the setup, which in many cases implies an inconvenient complexity increase. In this workaround, instead of the service behind API Gateway doing the authentication, this logic must be moved to a lambda authorizer function, which is a special lambda that is called to validate the request before allowing it to reach the endpoint behind the gateway.

First, a lambda function has to be created that will validate the received credentials and as response returns a policy that allows or denies access. As an overly simplistic example:

exports.handler = async (event) => {
    let response = new Object();
    if (event.authorizationToken === "Basic <username:password here in base64>") {
        console.log("Allowed");
        return {
          "policyDocument": {
            "Version": "2012-10-17",
            "Statement": [{
              "Action": "execute-api:Invoke",
              "Effect": "Allow",
              "Resource": "arn:aws:execute-api:us-east-1:<...>:<...>/*/*/*"
            }]
          }
        };
    } else {
        console.log("Denied");
        return {
          "policyDocument": {
            "Version": "2012-10-17",
            "Statement": [{
              "Action": "execute-api:Invoke",
              "Effect": "Deny",
              "Resource": "arn:aws:execute-api:us-east-1:<...>:<...>/*/*/*"
            }]
          }
        };
    }
};

Then, a lambda authorizer has to be created, that will make use of the previously created function:

Finally, a gateway response must be added, that will be triggered in case the lambda authorizer has denied access; here finally the WWW-Authenticate header to return can be specified:

The resulting flow would be

  1. SNS sends request without Authentication header
  2. API Gateway receives request, then calls the lambda authorizer
  3. lambda authorizer denies access
  4. API Gateway returns status code 401 with header WWW-Authenticate
  5. SNS repeats requests with Authentication header and credentials
  6. API Gateway receives request, the lambda authorizer allows it
  7. API Gateway forwards request to the service serving the endpoint

Conclusions

Even though many things can be easily accomplished using cloud services, one has to be ready (and allocate time) for potential difficulties. Considering the maturity of these services one has to wonder that in case of SNS and API Gateway why certain decisions have been made:

  • why not have as default (or at least optional) behavior sending basic credentials preemptively
  • why not give the option to bypass the remapping of commonly used headers in API Gateway

Having the first point alone would have addressed the whole problem. Do you have any idea, insight on why these choices have been made? Feel free to let us know in the comments.