Unboxing JSON Schema Validator 320
WSO2 APIM-3.2.0 Vs Microgateway 3.2.0
This or That? Left or Right? Yes or No? Black or White?
Always a huge dilemma right? The same conundrum exists when it comes to choosing the best option for API security. Today’s digital era has wrapped almost all its solutions in the format of API. So protecting your APIs has a huge impact on the business and thus inevitable. An API can be protected in different ways concerning various factors. Some of the commonly used API security methods are as follows.
- Authentication with tokens, API keys, etc.
- Usage of encryption and signatures
- Quotas and throttling
Apart from the above-mentioned methods, there are plenty of methods available to protect your APIs. In this blog let’s see how we can protect APIs by securing the request and response payloads from malicious attacks.
Let’s begin the game!
Why protect the payloads?
One of the API attacks that target the payload is the “Malicious Payload Attack” where an unintended 3rd party will send malicious payloads in order to collapse the system. Changing the behavior of the system, inserting unwanted or garbage values to the system, fetching the details from the system are some of the impacts of the malicious payload attack. This will be a bad hit on the business and can break the business functions which ends up losing the customers. Therefore, by validating the payloads the APIs and the systems can be protected in order to have a smooth business flow.
How to protect your systems from “Malicious Payload Attacks”: JSON Schema Validator
In WSO2, the APIM-3.2.0 and Microgateway-3.2.0 (MGW-3.2.0) provide us the JSON schema validation feature, which is a client-side security mechanism to protect the systems from malicious attacks, by validating the payloads against an API definition. This is a two-way validation where both the request and response payloads can be validated.
Basic JSON schema validation flow
The JSON Schema Validator is completely handled by the gateway or MGW component. The above diagram illustrates both the request and response payload validations. Upon the validation results of the payloads, the gateway or MGW will decide whether to direct the API request/response to the relevant target (backend or client application).
The complete step by step guidance on creating the APIs and enabling the JSON schema validation feature in APIM can be referred to in this link. You can get the overall idea of enabling the JSON schema validator in MGW-3.2.0 by referring to this link.
One-to-One comparison
Here we will be looking at the latest versions, APIM-3.2.0 and Microgateway-3.2.0. As mentioned earlier, the JSON schema validator feature is handled by the gateway and MGW components. Therefore, this section elaborates on how some factors are handled in different ways in both APIM-3.2.0 gateway as well as in MGW-3.2.0. The comparisons are as follows:
1. Architecture
- Microgateway: Adheres to the microservices architecture.
- Gateway: Adheres to the monolithic architecture.
2. Runtime artifacts
- Microgateway: To work with the JSON schema validator, all the jars, dependencies, and API definitions need to be built as an executable prior to the server startup.
- Gateway: No need to create any runtime artifacts to enable or work with the JSON schema validator.
3. Runtime footprint
- Microgateway: Lightweight, because only the required jars and dependencies are bundled for the runtime. So can run even on low-end computers.
- Gateway: Heavy, because contains all the jars and dependencies during the runtime despite whether all of those jars are needed or not. Therefore, it consumes high CPU and memory. Recommended running on high-end computers.
4. Enabling the JSON Schema Validator feature
- Microgateway: In order to enable the JSON schema validator feature in MGW you need to perform the following 2 steps prior to the server startup.
1. Copy all the .jar files from <tool-kit-home>/lib/dependencies/validation directory to the <project>/lib directory before building the project.2. The request and response schema validations needs to be enabled seperately. In order to enable both the request and response validations you need to copy following configuration from default-micro-gw.conf.template to micro-gw.conf file.[validationConfig]
enableRequestValidation = true
enableResponseValidation = true
- Gateway: The JSON schema validator feature should be enabled only during the runtime, i.e: after the server startup. The schema validation feature can be enabled by navigating to the Publisher portal → Select the API → Runtime Configurations → Schema Validation and toggle it.
When the Schema validation is enabled in the publisher, both request and response will be enabled at once.6. Packages that handle the feature
Even though both gateways support the JSON schema validator feature, the logics implemented for this feature are handled in different backend source codes (packages).
- Microgateway: The logic is implemented in the micro-gateway core[1]
- Gateway: The logic is implemented in carbon apimgt[2]
Hassles and Hints!
JSON Schema validator is a new feature introduced in the 3.X.X series of WSO2 APIM and Microgateway. Since this feature highly depends on OpenAPI specification and JSON schema specification, the feature should be updated from time to time to align with these specs. While using this feature, we might come across various errors that require proper fixes. This section of the blog will help you to identify such issues and the proper way to fix them. Let’s begin!
The following OpenAPI definition will be used in the following cases.
1. Empty payload with JSON schema validator
When invoking an API call, we can send the request or receive the response with a payload or body. In general, for POST, PUT, and PATCH API request calls we can have the request payload. Always remember that the payload may or may not be sent along with these calls.
Deciding whether an API call should have the payload, relies on the sending and receiving parties and it is not decided by the HTTP(S) protocol. Therefore, you can send a JSON payload or an empty payload or no payload at all for the above-mentioned methods. While designing the API resources in the WSO2 APIM Publisher or in the API definition schema, you can make the request payload a mandatory or optional field.
If you are to validate the request payload then it is mandatory to send the payload in the API call request and you have to enable set the requestBody as a required field while designing the API resource. Also, you have to add the relevant schema object in the API definition.
post:
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/User'If you send an empty payload or send the payload without making it as a required field with the JSON schema validator enabled, you will encounter the following error in both APIM devportal and in carbon logs.
2. Email validation
If you want to validate the email parameter in the payload and if you find an error as below, it indicates that the validation jar is missing.
Download the jar(commons-validator-1.6.jar) from https://mvnrepository.com/artifact/commons-validator/commons-validator/1.6 and add it to <APIM_HOME>/repository/components/lib for APIM-3.2.0 and in <project>/lib for MGW.
3. Restrict the API calls based on the payload parameter count
Apart from validating the JSON payload parameters for the data type or formats, you also have the privilege to validate the API calls based on the payload parameter count. This is to restrict the API calls with more or fewer parameter counts than expected. This can be achieved using the following keys in the OpenAPI definition.
1. additionalProperties
2. minProperties
3. maxProperties- Known payload parameter count
This case is applicable if you know the exact count of payload parameters you are expecting in the request or response payload. In this case, you can use the minProperties and maxProperties keys to define the number of payload parameters. You can have either one of the keys or both the keys in the definition. These keys accept integer values. The following is the syntax you use to define these keys in the OpenAPI definition.
components:
schemas:
Intern:
maxProperties: 3
minProperties: 2If the payload has more parameters than maxProperties or fewer parameters than minProperties you will encounter the following errors.
Error occuring scenarios:
------------------------
minProperties > actual payload paramater count
maxProperties < actual payload paramater count
- Unknown payload parameter count
This case is applicable if you don’t know the exact count of payload parameters you are expecting in request or response payload. In this case, you can use the additionalProperties key to restrict the payloads to have more parameters than the ones defined in the components → schemas → schema object. The additionalProperties key accepts boolean values. By default, the additionalProperties are set as true. Using this key you can restrict the payload from having more parameters than defined. The following is the syntax you use to define this key in the OpenAPI definition. Please note that if you are using the OpenAPI I have shared in this blog, then you need to remove the dob key in this as mentioned below and use http://run.mocky.io/v3/c7f53f3e-a78a-4a7e-ab27-513603bfa62b as the server URL which will return only one parameter in the payload.
components:
schemas:
User:
type: object
properties:
employee-id:
pattern: '^emp-\d{3}-\d{2}-\d{4}$'
type: string
uuid:
type: string
format: uuid
name:
type: string
email:
type: string
format: email
additionalProperties: falseIn this case, you cannot have a payload parameter count of more than 4(employee-id, UUID, name, and email). If you have more values then you will get the following error message in the carbon log.
The error message will have the additional property details in the error log.
ERROR - SchemaValidator Schema validation failed in the Response: #: extraneous key [dob] is not permitted org.everit.json.schema.ValidationException: #: extraneous key [dob] is not permitted4. Unable to validate payload with additional values in Content-type
According to the current implementation in [3,4], the request or response validation will take place only if the Content-Type: application/json. If there are any additional suffix added to it such as Content-Type: application/json; charset=UTF-8 the validation will not take place.
In this case, let’s analyze the response flow to explain this issue. This can be solved by removing the charset from the Content-Type header and set the content type as application/json. This can be achieved by making the following configuration in the <APIM-Home>/repository/conf/deployement.toml . By default the headers are preserved for requests, so have to explicitly make the below changes for response headers.
[passthru_http]
"http.response.headers.preserve"="Content-Type"After making the above configuration change you need to add a custom sequence in the response flow as follows to remove the charset encoding and to set the content type.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="content-type-sequence">
<property name="URL" expression="$ctx:REST_FULL_REQUEST_PATH"/>
<log level="custom">
<property name="path" expression="get-property('URL')"/>
</log>
<filter source="get-property('URL')" regex=".*/employee">
<log level="custom">
<property name="test" value="sample"/>
</log>
<then>
<property name="messageType" value="application/json" scope="axis2" type="STRING"/>
<property name="ContentType" value="application/json" scope="axis2" type="STRING"/>
<property name="setCharacterEncoding" value="false" scope="axis2" type="STRING"/>
</then>
</filter>
</sequence>After these changes, the JSON schema validation for response flow will work as expected. A sample response has been attached below.
<am:fault xmlns:am="http://wso2.org/apimanager">
<am:code>500</am:code>
<am:message>Bad Response</am:message>
<am:description>Schema validation failed in the Response: #: extraneous key [response] is not permitted, </am:description>
</am:fault>5. Unable to create API from OpenAPI archive file in Mac OS
In APIM-3.2.0 a new feature has been introduced to create APIs with archived OpenAPI definitions. In order to achieve this, you have to have the main API definition named swagger.json or swagger.yaml inside a root folder. The root folder should be archived. When you upload this folder to the publisher portal you will be able to create the API. Inside the root folder, you can have multiple folders containing the OpenAPI definitions.
If you are a Mac OS user then you will encounter the following error while uploading the archived file which was archived in Mac OS.
When you zip/archive a folder in Mac OS, by default, it will add __MACOSX and some hidden files such as .DS_Store. You can list this with the following command by navigating to the target folder or file.
ls -d .*Based on the current implementation [5], during the API creation if you upload a zip file, recursively the file will be checked for all the folders and files inside it including the hidden folders. Since the hidden folders don’t contain any many OpenAPI definitions with the name “swagger.json” or “swagger.yaml” the Mac OS users might encounter this error. This error will only occur if you zip the folder using Mac OS. If you zip the folder in some other OS that doesn’t add any hidden folders or metadata folders and upload the same zip file in Mac OS you will not face this issue.
The following are the solutions to fix this issue in Mac OS.
- If you are creating the zip folder in Mac OS from scratch you can use the following terminal command to recursively remove the __MACOSX (-x “MACOSX”)and any hidden files (-x “.*”). Since the hidden files’ names start with “.” we use the wild card character “.*” to remove all the hidden files.
zip -r <Target.zip path> <Source folder path> -x “.*” -x “__MACOSX”2. If you have already created the zip file in Mac OS then you can remove the existing __MACOSX and hidden files with the following command.
zip -d <Target.zip path> __MACOSX .*Now you can upload the zip files created in Mac OS without any problems to create the API in the publisher.
6. Wrong naming conventions for the main OpenAPI definition during API creation
When you try to upload the archived OpenAPI definition and if you don’t name the main OpenAPI definition as “swagger.json” or “swagger.yaml” you will encounter the following error.
In order to overcome this rename your master or main API definition file as “swagger.yaml” or “swagger.json”.
RECAP!
In this post, we initially discussed the need to do JSON schema validation and the workflow of it. Then we checked on how this JSON schema validator impacts APIM-3.2.0 gateway and MGW-3.2.0 along with their major differences. We also analyzed some important issues we might come across while using this feature and the proper fixes for them to overcome those issues. I hope that this blog post will be useful for all those who are interested in this feature of WSO2!
Happy Learning!
Stay Safe!
