Application Programming Interfaces or API’s are responsible for majority of system integration and functional components of modern computing landscape in both consumer and enterprise environments.
Properly constructed secure API’s provide significant benefits during initial build, integration with other systems and during entire application lifecycle while protecting sensitive information stored in business systems.
Based on author's experience designing and implementing API’s for variety of clients in financial, insurance, telecom and public sectors, security is often overlooked in favor of simplistic, vendor/product specific solutions.
This article addresses best practices for API security design in product neutral manner to help architects to plan and build easy to work with and secure API's.
Recommended approach is to separate API security from its business functionality and allow back-end developers to concentrate only on business functions. Once business logic for an API is ready, it can be published using common security components described in this article.
This article does not provide any product specific recommendations, but any modern API security/governance platform will be able to satisfy majority of suggested requirements using out-of-the-box functionality.
API Traffic Flow
Typical traffic flow for an API is presented on the diagram below:
Client (typically an application) sends a request to defined API endpoint(s) with some type of authentication/authorization which is then routed to business services where actual processing occurs. As shown on the diagram, security and optional functionality of request/response processing is located outside of core business functionality of the back-end in separate layer.
Recommended approach is to have a set of standard functional blocks in the API policy flow which will improve API security, usability and better protect back-end business services with following common components:
Let's review proposed functions in more detail:
API description
This function provides client an ability to request API descriptions in machine readable (YAML, XML, JSON) or human readable (HTML) formats using simple HTTP GET call.
This is technically not an API security component, but it is very useful function which can greatly simplify API consumption and integration efforts by a client. Having both human and machine readable documentation within API itself has proven to be great resource for those who seek current API information, request/response samples, JSON and XML schemas and other info.
Client can request an API description in different formats for automatic client code configuration or for functionality review by a developer.
Following table provides suggestions on request formats and expected responses:
Request format (HTTP query parameter) | Response format |
?yaml | OpenAPI description in YAML |
?json | OpenAPI description in JSON |
?wsdl | API web service description WSDL in XML |
?wadl | API rest service description WADL in XML |
?html | OpenAPI description in HTML |
?soap | WSDL description in HTML |
?rest | WADL description in HTML |
Machine readable API descriptions (OpenAPI, WSDL, WADL) are typically hard-coded into API policy at the policy deployment stage, but human readable versions in HTML can be generated automatically during the request from machine readable versions or also hard-coded into API policy when automatic conversion is not practical.
For example, there are few free XSL templates which can be used to automatically convert WSDL or WADL documents into easy to read HTML versions. OpenAPI JSON or YAML descriptions can be also presented in readable HTML format using various tools like Swagger editor.
In some cases, this functionality may be provided by integrating with an API portal, but for the purposes of this document we will keep it at generic API flow level.
Request content type verification
Request content type verification is the first security verification step. It is required to limit request messages of incorrect format from reaching service back-end. It is also allowing other security checks to determine what error message format will be required in the response should any of them fail to validate request.
For a typical API, valid request content type is application/json for REST JSON, application/xml for REST/XML and text/xml for SOAP.
TLS connection enforcement
It is a good security practice to encrypt request and response API traffic to protect information privacy in transit over untrusted networks. Common industry practice is to use SSL/TLS encryption protocol.
Security enforcement should include following steps:
Verification that traffic is coming via TLS tunnel and using latest protocol version
TLS tunnel is using safe set of ciphers, digital signatures and hash functions. This requirement is technology dependent and will require careful selection of safest parameters supported by server and all clients which will need to connect to API.
Optional verification of client certificate for 2-way TLS tunnels.
Client Authentication and Authorization
This function will authenticate and authorize incoming API requests using appropriate authentication methods or a combination of them:
User ID/Password (direct or via security frameworks like WS-Security)
API key in HTTP header
Client certificate
Federated identity (SAML, JWT OIDC)
OAuth
It not recommended to have API management layer to hold any client identities, it should rely on external authentication service where all identity management and security policies can be properly enforced.
Payload Validation
This function will validate that request payload does not include any malicious content before passing it to back-end service. Types of checks performed here will depend on the request format (XML or JSON) and can include following items:
JSON document structure validation (object length, object entry count, array entry count, entry name length)
XML document structure (Contiguous text length, attribute value length, attribute name length, element nesting depth)
Common SQL injection attacks in request message body, request URL and request URL query string
Other injection payload detection for OS command injection, LDAP injection etc.
Application specific payload security validations
These checks are essential for protection of back-end service from malicious or malformed payload.
HTTP Method Validation
This function will validate that API request is received with correct HTTP method (GET, POST, PUT etc.). The same API can provide support for multiple methods and operations on the same endpoint so this validation may need to take into account other request parameters like request URL and HTTP query.
Message Rate Validation
This function will validate global API request rate or specific rate for requested API operation. This validation is providing protection for back-end services against denial of services attacks or just limits request rate to protect it from overload.
Message rate can be configured as a number of concurrent messages per second or in a specified period of time to support API quotas per client.
Request Size Validation
This function will validate that request payload size does not exceed configured limits for requested API operation. It is very important to stop incorrect (large) requests from reaching back-end service as it can lead to multiple undesired issues.
In most cases, typical request payload should be below few kilobytes unless operation requires larger payloads.
Validate Request Payload against Schema
This function will validate request payload against appropriate XML or JSON schema document to make sure request is semantically correct for requested API operation. Since request schema validation can be CPU intensive, validating it here can remove this overhead from back-end business service and ensure that payload is configured correctly before reaching back-end system.
Obviously, in cases where multiple request formats are accepted, API policy must be configured to use correct schema depending on request payload type (XML or JSON).
Request Transformation
This optional function supports use cases where request payload must be transformed to a format accepted by back-end service. For example, common case would be when client will send request with JSON payload from browser based AJAX or mobile application and it will need to be converted to SOAP XML request supported by back-end service.
Back-end Connection Handling
This function is handling API security layer connectivity to back-end system. It is needed to inform client about any connectivity issues encountered during network or application level connections. We will discuss more details when discussing connection timeout configurations later in this document. As a rule, this function will respond to the client with specific error message and HTTP error code 503 to indicate that connection to back-end system failed.
Response Validation
This optional function allows validation of back-end response. In most cases, response is trusted and can be sent back to the client unless its format can cause issues and needs to be validated to make sure it doesn't cause problems on the client side.
Response Transformation
This optional function is needed when back-end response format needs to be converted to other message format understood by the client. Similar to request transformation, this function may be used to transform SOAP XML response to JSON before sending it to the client.
Rule/Condition Verification
This is optional placeholder where response can be checked for any additional condition in case some custom logic needs to be applied.
Timeout Configuration
Correctly configured service timeouts are very important for proper error handling and troubleshooting. They are very often ignored or misconfigured causing multiple issues and API outages.
With API security layer in place, correct system timeout configuration should not allow client/requester connection to fail before endpoint system:
In this configuration API security layer timeout value is at least one second higher then service back-end and client connection timeout one second higher than API security layer and two seconds higher then back-end service.
If timeouts are misconfigured, in some cases valid responses from back-end service and/or API layer will be lost leading to failure of client application.
Error Handling
Good portion of API security layer logic is dedicated to correct error handling which will aid client application integration and troubleshooting.
Error codes for API security layer are configured using appropriate HTTP 400 and 500 error codes as indicated on the flow diagram above.
Obviously error responses must be in the format appropriate for the received request. SOAP request will require SOAP fault error format, REST/XML and REST/JSON will require XML and JSON errors respectively.
Error responses must be informative and should provide enough information for troubleshooting, but do not expose any sensitive information to unauthenticated clients.
At minimum, each error response should contain following information:
API and operation description
Error code
Error origin and description with appropriate details
Timestamp in human readable format
Unique message transaction ID
It is important to distinguish between API security layer errors and errors from back-end service itself. In many cases API security layer will process request successfully, but back-end service will return an error.
One of the recommended ways to deal with API security layer and back-end errors is to use separate schema namespaces for security layer and back-end services. That way it will be obvious which system is responding with error.
Below are sample error messages in XML and JSON with clear identification that they are coming from API security layer (APISEC):
<?xml version="1.0" encoding="UTF-8"?>
<APISEC>
<Status>
<code>401</code>
<message>Unauthorized – client failed to present valid api key</message>
<api>Get Customer Detail</api>
<id>0000015e5892aaa3-23a3</id>
<time>Wed Sep 13 04:44:01 2017</time>
</Status>
</APISEC>{
"APISEC":{
"Status":{
"code":"401",
"message":"Unauthorized – client failed to present valid api key using SSL/TLS connection",
"api":"Get Customer Detail",
"id":"0000015e5892aaa3-23a5",
"time":"Wed Sep 13 04:45:18 2017"
}
}
}Security Event Logging
API Security event logging is critical component to support system troubleshooting, API performance and historical trend analysis. Every API security platform has it's own set of security and performance events, but following minimum of parameters are common and should be recorded for every transaction:
Message identifier (success/error)
Timestamp in ISO8601 format (for interoperability with external systems)
Request URL
Destination service routing URL
Request HTTP method
Request HTTP content type
Message response code sent to the client
Error/success message description
API service name/operation
Authenticated user ID
Time between request and response
Client source IP address
Transaction ID (API security layer will typically generate it for each message)
Current rate limit for an API
Request message size
Response message size
Additional parameters specific to API security technology in use can be included.
API Security Platform Specific Considerations
Modern API security platforms include on premise systems like MuleSoft Anypoint, IBM DataPower SOA Gateway, Broadcom Layer7 API gateway, OKTA Kong API gateway and others which are designed to address all the functionality described in this document.
For cloud API deployments, all major cloud providers (Azure, AWS, GCP) have cloud based API management components which can support most of the required functionality also.
It is recommended to only use out-of-the-box functionality of above platforms staying away from running custom code as it will defeat the purpose of having separate API security driven system with low configuration and support costs.
API Security Anti-patterns
As a conclusion, I would like to provide examples of bad practices as it relates to API security implementation.
Following design patterns should be avoided:
Application session management that requires persistent session creation and storage and is not a good fit for API security gateway technology. Application session creation and management needs to be done by application servers and/or dedicated identity management systems.
Stateful message processing. Any stateful message processing on the API security gateway will require message attribute storage and retrieval on the API gateway platform which can compromise its performance and manageability. In some special cases, gateway may be used for message replay protection using memory cache storage.
Asynchronous message processing. Asynchronous message flow requires API gateway to generate persistent message identifiers and processing of asynchronous responses which is not natural function of API gateway platform and should be left to enterprise service bus to handle.