Step-by-step guidance on WSO2 API Manager-3.0.0 JSON Schema Validation Feature

Image for post
Image for post
[Source: https://i.imgflip.com/gs54m.jpg]

During this era, many applications communicate with each other with the aid of APIs. Despite the domain, the scale of the business and user base, APIs are being used to transfer both sensitive and insensitive data. Nevertheless, it is essential to validate the data being passed through these APIs. The man-in-the-middle security attacks are very common and expected threats of a system and it requires a significant amount of effort to overcome that.

When a request is being sent from the client-side, it should be validated against a predefined model. Upon successful validation, the request should be directed to the corresponding backend. If not the attackers can send malicious payloads and break the API Gateway which poses security threats to the system. This safety measure not only applies to requests but also for the responses to reduce backend failure. This provides more security to the APIs.

WSO2 API Manager 3.0.0 into the picture

Image for post
Image for post
[Source: https://wso2.com/api-management/]

WSO2 API Manager(APIM) is an open-source application that handles the API life cycles and provides a secure platform for different applications to communicate with each other through APIs. Already the previous versions of APIM had mechanisms to detect malicious content if passed during the request-response flows such as XML schema validations, identifying malicious XML content (XML bombs), identifying SQL injections, and identifying malicious JSON content over attributes such as max array size and max element size.

As another milestone in enhancing malicious content detection, APIM-3.0.0 was introduced with JSON Schema Validation feature. This is used to validate the payload against a predefined JSON schema. Unless the request/response complies with this schema, the content will be blocked to protect the system.

Assume a bottle manufacturing company that manufactures bottles based on a particular model. If a manufactured bottle doesn’t comply with the model, it will be classified as a defected product and be rejected. The rejected bottle cannot be passed to the next level for further processing. The JSON schema validation scenario can be compared to this manufacturing process for better understanding.

Image for post
Image for post
[Source: https://lowres.cartooncollections.com/conveyor_belts-factories-manufacture-manufacturers-comedy_glasses-industry-CC124551_low.jpg ]

Enabling the JSON schema validation feature in APIM-3.0.0

API creation in the Publisher portal

Let’s enable the JSON schema validation feature in APIM-3.0.0 by creating and publishing an API from scratch.

  1. Start the APIM-3.0.0 server and go to API Publisher Portal
  2. Create an API in the publisher. Since I have a swagger file with API definition I’m using it to create the API. You can create the API based on any of the available ways.
Image for post
Image for post
Fig 1: Create REST API from the swagger definition

3. Since I’m going to upload the swagger file, I’m selecting “OpenAPI File” as the Input type and click “NEXT”.

Image for post
Image for post
Fig 2: Upload the swagger definition

The used API definition is as follows,

swaggerSample.yaml

4. Create a dummy backend. I’ve used Mocky to create the backend. If your backend is ready you can use that endpoint URL.

Image for post
Image for post
Fig 3: Mocky.io to create a dummy backend

5. In step 3, once “NEXT” is clicked you will be redirected to the following UI to provide API details, that include API name, context, version, endpoint URL, and business plan.

Image for post
Image for post
Fig 4: UI to provide details about the API

Let’s take a look at each of these elements in detail.

a. Name — Name of the API and it’s a mandatory field to be filled

b. Version — Version of the API and it’s a mandatory field

c. Context — A unique name fronted by a “/”. In a URL path, the context path is a part that is used to select the relevant context to which the request should direct to. This is also a mandatory field.

d. Endpoint — The endpoint URL that directs to the actual backend

e. Business plan(s) — This allows enabling subscription level throttling. The default throttling tiers are,

  • Bronze: 1000 requests per minute
  • Silver: 2000 requests per minute
  • Gold: 5000 requests per minute
  • Unlimited: Allows unlimited access

When a user is subscribed to any of these throttling levels the user can send requests based on that rate limit. This protects the APIs by limiting the number of successful hits during a specific time.

6. Once the “CREATE” button is clicked you will be directed to the overview UI

Image for post
Image for post
Fig 5: Overview UI

7. Click on “Runtime Configurations” to enable the JSON schema validation. Toggle on the “Schema Validation”.

Image for post
Image for post
Fig 6: Runtime Configurations

8. Select “YES” on the Caution box and Click on the “SAVE” button on Runtime Configurations UI.

Image for post
Image for post
Fig 7: Runtime Configurations

9. Go to Overview UI and click on the “PUBLISH” button to move the API from Create state to Publish state. Once the API is published it can be viewed from the Deve Portal.

Image for post
Image for post
Fig 8: Publish the API by clicking the “PUBLISH” button in the overview UI

Accessing API from the Dev portal

Once the API is published, it can be accessed via the Dev portal. Dev portal can be accessed in 2 ways.

a. Through https://localhost:9443/devportal URL

b. By clicking the “View in Dev Portal” in Publisher you can go to the Dev portal.

Image for post
Image for post
Fig 9: Publish the API by clicking the “PUBLISH” button in the overview UI
  1. Sign in to the Dev Portal
  2. The newly created API will be listed in the API list view
Image for post
Image for post
Fig 10: API list view in Dev portal

3. Create a new application to try out the JSON schema validation feature. Therefore, click on the “Application” button.

Image for post
Image for post
Fig 11: Create a new application

4. In the Applications UI click on the “ADD NEW APPLICATION” button.

Image for post
Image for post
Fig 12: Click on Add new application button

5. In the “Create an application” UI provide the application name, per token quota, token type and application description. Then click on the “SAVE” button.

Image for post
Image for post
Fig 13: Create and save the application

6. Go to the API list view and selected created API. Now the API should be subscribed to an application to get communicated with other applications/users. Therefore, click the “SUBSCRIBE TO AN APPLICATION” button in the API overview UI.

Image for post
Image for post
Fig 14: Subscribe to the newly created application

7. Next, select the newly created application from the dropdown list and click on the “SUBSCRIBE” button.

Image for post
Image for post
Fig 15: Click on the subscribe button

When creating the API I have only enabled the unlimited subscription tier. Therefore, in the Dev portal, only the unlimited tier is being displayed in the throttling policy.

8. Once the application is subscribed the subscription details will be listed in the same UI as follows.

Image for post
Image for post
Fig 15: Subscribe details

9. To generate the keys in the production environment click on the “PROD KEYS” and the key generation UI will be displayed in the same UI as follows.

Image for post
Image for post
Fig 16: Generate prod keys

Since this is a demo application I have not provided any Callback URL. This is the URI where the client(web browser) should be redirected to after the authorization server grants access.

I didn’t make any changes to the grant types which were selected by default.

10. Click on the “GENERATE KEYS” button.

11. Next, the following UI will be displayed with the Production key, secret and access token. This access token is valid for 3600 seconds. After that the access token becomes invalid and you can click on the “GENERATE ACCESS TOKEN” button to get a new access token. Now you can click on the “UPDATE” button.

Image for post
Image for post
Fig 17: Update the access token in the “TRY OUT” UI by clicking the update button in the credentials UI

12. Click on the “TRY OUT” option on the left panel to invoke API calls and test the JSON schema validator feature in APIM-3.0.0. Select the application, key type, environment, and the access token will be displayed automatically since you have updated it by clicking the update button in the credentials UI.

Image for post
Image for post
Fig 18: Select the relevant application, key, and environment details

13. The JSON schema validates against the request body. Therefore, I’ll be invoking the POST request as it has the payload. So click on the “Try it out” button.

Image for post
Image for post
Fig 19: Click on Try it out button

14. Provide the relevant payload and click on the “Execute” button.

Image for post
Image for post
Fig 20: Click on the execute button to invoke the POST call

15. Since the provided payload complies with the API definition, we’ll get the success code as 200.

Image for post
Image for post
Fig 21: Successful response for the invoked POST request

The background story of JSON Schema Validator in WSO2 APIM-3.0.0

The APIM-3.0.0 follows the OpenAPI-3.0 specifications. The schema object in the API definition is used to validate against the payload sent along with the request. In the API definition, the “schema” object holds the validations for each attribute in the payload. The schema validation can be done in 2 ways. They are,

  1. External schema referred by $ref keyword
  2. In-line schema

In OpenAPI-3.0 the request body can be defined using body and formData parameters. The body parameter content type is application/json where the form data’s content type can be application/x-www-form-urlencoded or multipart/form-data, which are commonly used to submit HTML forms. In this blog post, we will be concentrating only on the body parameter.

1. External schema referred by $ref keyword

The $ref is recommended as the best practice in OpenAPI-3.0 to refer the schema for the bodyRequest. This is mainly for the re-usability of the schema object in an API definition. Let’s dive deep into get more insights of $ref in APIM-3.0.0.

In this example, we will be testing the basic data type “string” along with some special formats such as email, UUID and date. As the first step, the schemas object needs to be defined under the components object in the API definition as mentioned below.

components.yaml

As you can see under components we have a “schemas” object that has a schema called “User”. The User schema has properties that will be used to validate the corresponding request payload. To validate special formats under the specific property the relevant data type should be mentioned along with the special format type with the keyword “format”.

Once the schema has been implemented it should be referred to the relevant request path(/POST) as follows.

Image for post
Image for post
Fig 22: Usage of $ref

The $ref keyword is referred to in the POST requests, which implies when the POST request is invoked with the payload will be fetched and validated against the User schema that has been referred to as “#/components/schemas/User”. In such a manner you can implement any schemas and re-use them with $ref.

A sample valid request payload is as follows,

{
“id”: “3fa85f64–5717–4562-b3fc-2c963f66afa6”,
“name”: “google”,
“email”: “google@gmail.com”,
“dob”: “2019–12–05”
}

2. In-line schema

In-line schema is nothing but having the schema inside the relevant HTTP method under the “paths” element. A sample for the in-line schema is as follows,

inline.yaml

Even though the in-line schema is an accepted way in OpenAPI-3.0, the APIM-3.0.0 support only the schemas with $ref.

Therefore, if you try to execute the API definition with the in-line schema you will end up with the following error in the carbon console.

Image for post
Image for post
Fig 23: Error while trying to invoke POST request with in-line schema in APIM-3.0.0

Bonus Bonanza

  1. In case if you find an error as below, it indicates that the validation jar is missing. Download the jar(commons-validator-1.6.jar) from https://mvnrepository.com/artifact/commons-validator/commons-validator/1.6 and add it to <APIM_HOME>/repository/components/lib
Image for post
Image for post
Fig 24: NoClassDefFoundError

2. Assume you are trying to pass a payload with the wrong email format as below.

{
“id”: “3fa85f64–5717–4562-b3fc-2c963f66afa6”,
“name”: “google”,
“email”: “email3618”,
“dob”: “2019–12–05”
}

You will get an error message as follows in the carbon console. This confirms that the Schema validation you have enabled is validating the payload against the provided schema.

Image for post
Image for post
Fig 25: Wrong format error

Seeking more formats to be validated?

What if you are not happy with the existing pre-defined formats available in OpenAPI and want to create your format? It is possible with the “Pattern” keyword available in OpenAPI-3.0.0. This is nothing but expressing the regular expression template for the string value. For example,

format.yaml
properties:           
employee-id:
type: "string"
pattern: '^[A-Za-z]*-\d{3}-\d{2}-\d{4}$'

The pattern ‘^[A-Za-z]*-\d{3}-\d{2}-\d{4}$’ indicates that the employee id should be started with alphabetic values and not case sensitive followed by a “-” 3 digits, “-” 2 digits, “-” and ends with 4 digits. For example,

"employee-id": "empID-234-34-7382"

The results will be as follows in the Dev portal.

Image for post
Image for post
Fig 26: Successful output in Dev portal for “Pattern” keyword example

If the provided value is in the wrong format as follows,

"employee-id": "879-empID-86796-8677r6r6"

The response will be “400” bad request,

Image for post
Image for post
Fig 27: Bad Request for wrong formatted input for the employee id parameter

From APIM-3.0.0 the schema validation has become very efficient that almost the widely used scenarios are covered up in it.

A quick recap…

In this blog, we have covered the new feature introduced in the APIM-3.0.0 product which was expected by many of its users all these years. Also, we have seen how to enable this feature in the publisher in a very detailed manner. Next, we tested these features with basic data types, special format and with regular expressions. These are some of the widely used scenarios in a day to day business environment. Some of the known issues you might come across when using this feature and the relevant fixes are also shared in this post.

Since APIM-3.0.0 got released a couple of weeks back and the documentations are still getting aligned to the right track, I thought of sharing my experience in using this feature with all of you. As many of you might have many issues in trying this feature, thought that this might be a quick start for all the APIM users out there… Do try this feature and share your experience, issues with others… See you in another interesting blog post… Happy Learning buddies….!!!

Image for post
Image for post
[Source: https://media1.tenor.com/images/49af2056717bdc97ea295b1329ad294b/tenor.gif?itemid=11488359]

Written by

Inquisitive

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store