A Guide to run the OIDC Conformance Suite with WSO2 Identity Server

7 min readNov 7, 2021

Testing vector created by stories — www.freepik.com

Having trouble in understanding how the OIDC conformance suite works? Looking for a way to set it up locally and run it against your Identity Provider? Here’s a simple guide to help you understand the test configurations and test flows of the OIDC conformance suite using the WSO2 Identity Server.

First of all, what is this OIDC Conformance Suite?

The OIDC Conformance Suite is an open source project run by the OIDC Foundation. It is basically a set of tests that can be used to check the compliance of Identity Providers (and Relying Parties) with the OIDC specification. The test suite tests different scenarios based on a set of profiles corresponding to the OpenID Connect Conformance Profiles.

Understanding the Conformance Suite

The Conformance Suite can be run in 2 ways:

Using the production deployment
Running locally inside a docker container

The production deployment is available at https://www.certification.openid.net and you can use the information available at the OIDC Foundation website to configure the test suite to run against a hosted Identity Provider.

The test suite can also be cloned and run locally inside a docker container if we need to test a locally hosted Identity Provider. This blog will guide you through the procedure of configuring a basic test plan and running the tests.

Follow up this tutorial to setup the conformance-suite locally and configure it to run against the WSO2 Identity Server.

Let us try to understand what happens here.

When we start the docker containers by running the command docker-compose -f docker-compose-dev.yml up, the test suite will be accessible at https://localhost:8443/.

Here, you are given the option to create a new test plan or to view available test logs and test plans. Let us first create a new test plan and run a test against the IS server.

Test Configurations

When you click on the Create a new test plan button, you will have to choose the type of test plan first. The drop down list for the Test Plan field will have a list of available test plan types depending on different conformance profiles and the parties tested(Identity Provider/Relying Party).

For example: If we need to test the basic profile for an Identity Provider we should choose OpenID Connect Core: Basic Certification Profile Authorization server test. Since in the above tutorial we are creating a test plan to test the compliance of WSO2 Identity Server, a test plan under Test an OpenID Provider / Authorization Server should be chosen.

Next, choose static and static_client for the next fields and provide the following Test Information.

The Server metadata location field allows us to choose the method of configuring the server endpoints. The dynamic option is recommended if the server supports discovery, so that it is not necessary to manually enter the server endpoints. But for simplicity, in this tutorial we will choose the static option and provide the following endpoint details under the Server section.

issuer: https://localhost.com:9443/oauth2/token
jwks_uri: https://localhost.com:9443/oauth2/jwks
authorization_endpoint: https://localhost.com:9443/oauth2/authorize
token_endpoint: https://localhost.com:9443/oauth2/token
userinfo_endpoint: https://localhost.com:9443/oauth2/userinfo

The Client Registration Type field allows us to choose the method to register service providers that are required during the test flow. For example: In the test plan — Basic Certification Profile Authorization server test, 2 service providers are required to carry out the test flow.

If dynamic_client is chosen, the test-suite will use Dynamic Client Registration to create new service providers when required. In the static_client option, the service providers should be created earlier and their client Id and secret should be entered as below.

Note: The callback URL of these service providers should always be https://localhost.emobix.co.uk:8443/test/a/<alias>/callback.
In this case, the same client details can be used for Client for client_secret_post and Client. Read up for more details.

Running Tests

Once all the configurations are added, we can start running the tests. Click Create Test Plan and you will be directed to the following page.

In this page you will see the set of tests available under the created test plan. The variant field shows the different parameters used in the tests and the leftmost box shows the test status after it is run. Each test can be manually run by clicking the Run Test button and then you will be directed to a page as follows.

The colour coded box at the top left corner indicates the status of the test. If you scroll down, you can see the steps of the test flow as logs in the test.

For example: The above logs describe the steps taken when creating the authorization request. By clicking on More button additional information on each executed step can be found such as the added state parameter, generated authorization request, received response for this request, etc.

For our convenience, the relevant section of the OIDC specification is also linked at steps where compliance is checked. For example: In the following log, the c_hash claim of the id_token is validated against the code extracted from the authorization response. Therefore the logs contain the expected value, actual value and the link to the OIDC specification section related to the c_hash claim.

We can easily use these logs to find the cause for test failures since we can follow this flow and find the exact place where the flow was interrupted and compare the expected and actual outcomes of each step. All these logs can be downloaded from the Download Logs button.

Once the test is finished running, you can click the Continue Plan button to run the next test or click the Return to Plan button to go back to the test plan page.

You can use the View Config button to view the configurations you added earlier and the Edit configuration button to change any configuration if needed.

Certain tests pause at a REVIEW state, where in most of the cases it is required to submit a screen shot of a specific page to proceed through the test. A WARNING or FAILED status will be received if there is a violation of the OIDC specification depending on how critical the violation is. If all goes well, you will receive a PASSED status!

Apart from creating a test plan, the home page provides you options to view previous tests plans and logs.

If you click View all available test plans button, the following page will display a summary of all the test plans that you have run in this container.

The coloured boxes under Modules represent the status of each test run under the particular test plan (Ex: Green — Passed, Red — Failed, Orange — Warning). The logs of all the tests of the test plan can be viewed by clicking the View Plan icon under the Instance ID column.

If you click View all available test logs button in the home page, you will see the following page with a summary of all the logs of tests that you have run so far.

As discussed in this blog, the OIDC Conformance Suite is a convenient tool that can be used to check the compliance of Identity Providers with the OIDC specification. There are a number of other test plans that can be run against the IS server using similar configurations as discussed in this blog. Use this knowledge and try out other profiles as well!

Happy testing! :)

A Guide to run the OIDC Conformance Suite with WSO2 Identity Server

Understanding the Conformance Suite

Test Configurations

Running Tests

References

Written by Imalsha Gunasekara