Integrating the OIDC Conformance Suite with WSO2 Identity Server Pipeline
OIDC Conformance Suite is an open source project that can be used to check the compliance of Identity Providers and Relying Parties with the OIDC specification. If you are new to the Conformance Suite take a look at this article to get a basic understanding about this test-suite or checkout their website for more information.
The OIDC Conformance Suite is a very convenient tool to test different flows related to profiles of your Identity Provider and check conformance with the OIDC specification. A special feature of the conformance suite is the support to be integrated into a development pipeline. This allows the tests to be automated and be used in a continuous integration system.
The oidc-conformance-tests project in the product-is repository automates the process of configuring the WSO2 Identity Server and running 6 test plans of the conformance suite against it. The README file of the project contains the basic instructions needed to run the tests locally or as a Github action. This blog will take you through the process of setting up this project locally.
Prerequisites
- Java 11
- Python 3
- Git
- Maven
- Docker
Setting up the Conformance Suite
This section is on local setting up of the Conformance Suite. For a more thorough guide, view this tutorial on Running OIDC Conformance Suite Locally with WSO2 Identity Server.
- Build the Conformance Suite
Clone the conformance-suite repository and build it. (In this tutorial I have used release-v4.1.37)
git clone https://gitlab.com/openid/conformance-suite.git
cd conformance-suite
mvn clean package -Dmaven.test.skip=true2. Configure Docker
Add the following to each of the 3 docker containers: mongodb, httpd, server available under the services section in the docker-compose-dev.yml file.
extra_hosts:
- "localhost:host-gateway"Your docker-compose-dev.yml file should look like below:
version: '3'
services:
mongodb:
image: mongo:4.2
volumes:
- ./mongo/data:/data/db
extra_hosts:
- "localhost:host-gateway"
httpd:
build:
context: ./httpd
dockerfile: Dockerfile-static
ports:
- "8443:8443"
extra_hosts:
- "localhost:host-gateway"
volumes:
- ./src/main/resources/:/usr/local/apache2/htdocs/
depends_on:
- server
server:
build:
context: ./server-dev
ports:
- "9999:9999"
extra_hosts:
- "localhost:host-gateway"
volumes:
- ./target/:/server/
command: >
java
-Xdebug -Xrunjdwp:transport=dt_socket,address=*:9999,server=y,suspend=n
-jar /server/fapi-test-suite.jar
--fintechlabs.base_url=https://localhost.emobix.co.uk:8443
--fintechlabs.devmode=true
--fintechlabs.startredir=true
links:
- mongodb:mongodb
depends_on:
- mongodb
logging:
# limit logs retained on host
driver: "json-file"
options:
max-size: "500k"
max-file: "5"max-file: "5"
3. Start the Docker Containers
Use the following command to start the 3 docker containers: mongodb, httpd, server and you will find the conformance suite available at https://localhost:8443/.
docker-compose -f docker-compose-dev.yml upBefore moving on to automation, if you feel like you need to try running a test manually, have a look at this tutorial which will help you understand the conformance suite better!
Setting up the Integration Project
The oidc-conformance-tests project which integrates the OIDC Conformance Suite with the WSO2 IS Server is available at the WSO2 product-is repository. This section will guide you through the process of setting up the oidc-conformance-tests project locally.
- Clone the product-is repository and go inside the oidc-conformance-tests directory.
git clone https://github.com/wso2/product-is.git
cd product-is/oidc-conformance-tests2. Get a zip file of the latest IS pack
You can download the zip file of the latest IS pack from the product-is releases or since you have the the product-is repository cloned, build it and get the zip file available at product-is/modules/distribution/target (In this tutorial I have used WSO2 Identity Server 5.12.0 Alpha9).
3. Configure the automation scripts
Open the test_runner.sh file of the oidc-conformance-tests directory in a text editor and add the following configurations.
- CONFORMANCE_SUITE_PATH : add the path to the conformance-suite project directory you cloned earlier where the scripts directory with the run-test-plan.py file is available.
- PATH_TO_SCRIPTS : add the path to the oidc-conformance-tests project directory where the test-runner.sh file is available.
- IS_LOCAL : set it to true
- PRODUCT_IS_ZIP_PATH : add the path to the zip file of the IS server you downloaded/built earlier.
Remove the sudo keyword from the following line to prevent errors during local setup.
sudo python3 ./configure_is.py $PRODUCT_IS_ZIP_PATH4. Run the automation script.
Use the following command inside the oidc-conformance-tests directory and run the test_runner.sh script to start the tests.
bash test_runner.sh5. Check the status of running tests.
Go to the conformance suite running at https://localhost:8443/ and got to the View all available test plans to check the status of the running test plans.
Understanding the Integration Project
The oidc-conformance-tests project will run and configure an IS server and use the run-test-plan.py python script of the conformance-suite project to run the automated tests against this IS server.
Let me break down this procedure to you…
The test_runner.sh script of the oidc-conformance-tests project will carry out the following tasks.
- Run the configure_is.py python script
configure_is.py script takes the path to the IS pack zip file as an input and carries out the following tasks:
- Extract the IS server from the zip file and starts the server.
- Register 2 service providers for each test plan
- Add necessary configurations for authentication
- Set user claims and service provider claims needed for the tests
- Generate JSON config files for each test plan
2. Run the run-test-plan.py script for each of the following test plans
- Basic certification test plan
- Implicit certification test plan
- Hybrid certification test plan
- Form post basic certification test plan
- Form post implicit certification test plan
- Form post hybrid certification test plan
Here, the run-test-plan.py python script takes the following inputs:
- Name of the test plan
- Server metadata location
- Client registration type
- Path to the JSON config file
If you have gone through the previous tutorial, you would have quite an understanding about the configurations needed to execute a test plan.
Go to the conformance suite running at https://localhost:8443/ and click on Create a new test plan.
The first 3 inputs to the run-test-plan.py corresponds to the first 3 fields of this page. The Configure Test section gives us the option to enter the configs as a form or as a JSON object. Here in the python script, we are providing the configurations through the JSON file generated from the configure_is.py script.
If you have successfully run the test_runner.sh script at least once, you will have test plans available at https://localhost:8443/plans.html. Click on View Config button of one of the test plans to see the configs added by the JSON file.
Apart from the configurations we normally add during the manual setup for server and client sections, here, you can see another configuration named “browser” which provides the configurations needed for browser control during test automation.
The browser block should be configured based on the URLs called and the response pages received during the test flow. These configurations are added through the JSON config files generated from the configure_is.py script. This script calls the config/browser_configuration.py file at the oidc-conformance-tests home directory where these configs related to the browser control should be added. Please follow the wiki pages of the conformance-suite to find instructions on this configuration.
Once you have successfully run the tests, go to https://localhost:8443/plans.html where a summary of the 6 test plans are available and start analyzing!
References
https://openid.net/certification/about-conformance-suite
https://gitlab.com/openid/conformance-suite
https://gitlab.com/openid/conformance-suite/-/wikis/Design/BrowserControl
