Step-by-step guidance on WSO2 API Manager-3.0.0 JSON Schema Validation Feature
During this era, many applications communicate with each other with the aid of APIs. Despite the domain, the scale of the business and user base, APIs are being used to transfer both sensitive and insensitive data. Nevertheless, it is essential to validate the data being passed through these APIs. The man-in-the-middle security attacks are very common and expected threats of a system and it requires a significant amount of effort to overcome that.
When a request is being sent from the client-side, it should be validated against a predefined model. Upon successful validation, the request should be directed to the corresponding backend. If not the attackers can send malicious payloads and break the API Gateway which poses security threats to the system. This safety measure not only applies to requests but also for the responses to reduce backend failure. This provides more security to the APIs.
WSO2 API Manager 3.0.0 into the picture
WSO2 API Manager(APIM) is an open-source application that handles the API life cycles and provides a secure platform for different applications to communicate with each other through APIs. Already the previous versions of APIM had mechanisms to detect malicious content if passed during the request-response flows such as XML schema validations, identifying malicious XML content (XML bombs), identifying SQL injections, and identifying malicious JSON content over attributes such as max array size and max element size.
As another milestone in enhancing malicious content detection, APIM-3.0.0 was introduced with JSON Schema Validation feature. This is used to validate the payload against a predefined JSON schema. Unless the request/response complies with this schema, the content will be blocked to protect the system.
Assume a bottle manufacturing company that manufactures bottles based on a particular model. If a manufactured bottle doesn’t comply with the model, it will be classified as a defected product and be rejected. The rejected bottle cannot be passed to the next level for further processing. The JSON schema validation scenario can be compared to this manufacturing process for better understanding.
Enabling the JSON schema validation feature in APIM-3.0.0
API creation in the Publisher portal
Let’s enable the JSON schema validation feature in APIM-3.0.0 by creating and publishing an API from scratch.
- Start the APIM-3.0.0 server and go to API Publisher Portal
- Create an API in the publisher. Since I have a swagger file with API definition I’m using it to create the API. You can create the API based on any of the available ways.
3. Since I’m going to upload the swagger file, I’m selecting “OpenAPI File” as the Input type and click “NEXT”.
The used API definition is as follows,
4. Create a dummy backend. I’ve used Mocky to create the backend. If your backend is ready you can use that endpoint URL.
5. In step 3, once “NEXT” is clicked you will be redirected to the following UI to provide API details, that include API name, context, version, endpoint URL, and business plan.
Let’s take a look at each of these elements in detail.
a. Name — Name of the API and it’s a mandatory field to be filled
b. Version — Version of the API and it’s a mandatory field
c. Context — A unique name fronted by a “/”. In a URL path, the context path is a part that is used to select the relevant context to which the request should direct to. This is also a mandatory field.
d. Endpoint — The endpoint URL that directs to the actual backend
e. Business plan(s) — This allows enabling subscription level throttling. The default throttling tiers are,
- Bronze: 1000 requests per minute
- Silver: 2000 requests per minute
- Gold: 5000 requests per minute
- Unlimited: Allows unlimited access
When a user is subscribed to any of these throttling levels the user can send requests based on that rate limit. This protects the APIs by limiting the number of successful hits during a specific time.
6. Once the “CREATE” button is clicked you will be directed to the overview UI
7. Click on “Runtime Configurations” to enable the JSON schema validation. Toggle on the “Schema Validation”.
8. Select “YES” on the Caution box and Click on the “SAVE” button on Runtime Configurations UI.
9. Go to Overview UI and click on the “PUBLISH” button to move the API from Create state to Publish state. Once the API is published it can be viewed from the Deve Portal.
Accessing API from the Dev portal
Once the API is published, it can be accessed via the Dev portal. Dev portal can be accessed in 2 ways.
a. Through https://localhost:9443/devportal URL
b. By clicking the “View in Dev Portal” in Publisher you can go to the Dev portal.
- Sign in to the Dev Portal
- The newly created API will be listed in the API list view
3. Create a new application to try out the JSON schema validation feature. Therefore, click on the “Application” button.
4. In the Applications UI click on the “ADD NEW APPLICATION” button.
5. In the “Create an application” UI provide the application name, per token quota, token type and application description. Then click on the “SAVE” button.
6. Go to the API list view and selected created API. Now the API should be subscribed to an application to get communicated with other applications/users. Therefore, click the “SUBSCRIBE TO AN APPLICATION” button in the API overview UI.
7. Next, select the newly created application from the dropdown list and click on the “SUBSCRIBE” button.
When creating the API I have only enabled the unlimited subscription tier. Therefore, in the Dev portal, only the unlimited tier is being displayed in the throttling policy.
8. Once the application is subscribed the subscription details will be listed in the same UI as follows.
9. To generate the keys in the production environment click on the “PROD KEYS” and the key generation UI will be displayed in the same UI as follows.
Since this is a demo application I have not provided any Callback URL. This is the URI where the client(web browser) should be redirected to after the authorization server grants access.
I didn’t make any changes to the grant types which were selected by default.
10. Click on the “GENERATE KEYS” button.
11. Next, the following UI will be displayed with the Production key, secret and access token. This access token is valid for 3600 seconds. After that the access token becomes invalid and you can click on the “GENERATE ACCESS TOKEN” button to get a new access token. Now you can click on the “UPDATE” button.
12. Click on the “TRY OUT” option on the left panel to invoke API calls and test the JSON schema validator feature in APIM-3.0.0. Select the application, key type, environment, and the access token will be displayed automatically since you have updated it by clicking the update button in the credentials UI.
13. The JSON schema validates against the request body. Therefore, I’ll be invoking the POST request as it has the payload. So click on the “Try it out” button.
14. Provide the relevant payload and click on the “Execute” button.
15. Since the provided payload complies with the API definition, we’ll get the success code as 200.
The background story of JSON Schema Validator in WSO2 APIM-3.0.0
The APIM-3.0.0 follows the OpenAPI-3.0 specifications. The schema object in the API definition is used to validate against the payload sent along with the request. In the API definition, the “schema” object holds the validations for each attribute in the payload. The schema validation can be done in 2 ways. They are,
- External schema referred by $ref keyword
- In-line schema
In OpenAPI-3.0 the request body can be defined using body and formData parameters. The body parameter content type is application/json where the form data’s content type can be application/x-www-form-urlencoded or multipart/form-data, which are commonly used to submit HTML forms. In this blog post, we will be concentrating only on the body parameter.
1. External schema referred by $ref keyword
The $ref is recommended as the best practice in OpenAPI-3.0 to refer the schema for the bodyRequest. This is mainly for the re-usability of the schema object in an API definition. Let’s dive deep into get more insights of $ref in APIM-3.0.0.
In this example, we will be testing the basic data type “string” along with some special formats such as email, UUID and date. As the first step, the schemas object needs to be defined under the components object in the API definition as mentioned below.
As you can see under components we have a “schemas” object that has a schema called “User”. The User schema has properties that will be used to validate the corresponding request payload. To validate special formats under the specific property the relevant data type should be mentioned along with the special format type with the keyword “format”.
Once the schema has been implemented it should be referred to the relevant request path(/POST) as follows.
The $ref keyword is referred to in the POST requests, which implies when the POST request is invoked with the payload will be fetched and validated against the User schema that has been referred to as “#/components/schemas/User”. In such a manner you can implement any schemas and re-use them with $ref.
A sample valid request payload is as follows,
{
“id”: “3fa85f64–5717–4562-b3fc-2c963f66afa6”,
“name”: “google”,
“email”: “google@gmail.com”,
“dob”: “2019–12–05”
}2. In-line schema
In-line schema is nothing but having the schema inside the relevant HTTP method under the “paths” element. A sample for the in-line schema is as follows,
Even though the in-line schema is an accepted way in OpenAPI-3.0, the APIM-3.0.0 support only the schemas with $ref.
Therefore, if you try to execute the API definition with the in-line schema you will end up with the following error in the carbon console.
Bonus Bonanza
- In case if you find an error as below, it indicates that the validation jar is missing. Download the jar(commons-validator-1.6.jar) from https://mvnrepository.com/artifact/commons-validator/commons-validator/1.6 and add it to <APIM_HOME>/repository/components/lib
2. Assume you are trying to pass a payload with the wrong email format as below.
{
“id”: “3fa85f64–5717–4562-b3fc-2c963f66afa6”,
“name”: “google”,
“email”: “email3618”,
“dob”: “2019–12–05”
}You will get an error message as follows in the carbon console. This confirms that the Schema validation you have enabled is validating the payload against the provided schema.
Seeking more formats to be validated?
What if you are not happy with the existing pre-defined formats available in OpenAPI and want to create your format? It is possible with the “Pattern” keyword available in OpenAPI-3.0.0. This is nothing but expressing the regular expression template for the string value. For example,
properties:
employee-id:
type: "string"
pattern: '^[A-Za-z]*-\d{3}-\d{2}-\d{4}$'The pattern ‘^[A-Za-z]*-\d{3}-\d{2}-\d{4}$’ indicates that the employee id should be started with alphabetic values and not case sensitive followed by a “-” 3 digits, “-” 2 digits, “-” and ends with 4 digits. For example,
"employee-id": "empID-234-34-7382"The results will be as follows in the Dev portal.
If the provided value is in the wrong format as follows,
"employee-id": "879-empID-86796-8677r6r6"The response will be “400” bad request,
From APIM-3.0.0 the schema validation has become very efficient that almost the widely used scenarios are covered up in it.
A quick recap…
In this blog, we have covered the new feature introduced in the APIM-3.0.0 product which was expected by many of its users all these years. Also, we have seen how to enable this feature in the publisher in a very detailed manner. Next, we tested these features with basic data types, special format and with regular expressions. These are some of the widely used scenarios in a day to day business environment. Some of the known issues you might come across when using this feature and the relevant fixes are also shared in this post.
Since APIM-3.0.0 got released a couple of weeks back and the documentations are still getting aligned to the right track, I thought of sharing my experience in using this feature with all of you. As many of you might have many issues in trying this feature, thought that this might be a quick start for all the APIM users out there… Do try this feature and share your experience, issues with others… See you in another interesting blog post… Happy Learning buddies….!!!
