Property-based and Generative testing for Microservices

shahzad bhatti
9 min readDec 20, 2022

--

The software development cycle for microservices generally include unit testing during the development where mock implementation for the dependent services are injected with the desired behavior to test various test-scenarios and failure conditions. However, the development teams often use real dependent services for integration testing of a microservice in a local environment. This poses a considerable challenge as each dependent service may be keeping its own state that makes it harder to reliably validate the regression behavior or simulate certain error response. Further, as the number of request parameters to the service or downstream services grow, the combinatorial explosion for test cases become unmanageable. This is where property-based testing offers a relief as it allows testing against automatically generated input fuzz-data, which is why this form of testing is also referred as a generative testing. A generator defines a function that generate random data based on type of input and constraints on the range of input values. The property-based test driver then iteratively calls the system under test to validate the result and assert the desired behavior, e.g.

def pre_condition_test_input_param(kind):
### assert pre-condition based on type of parameter and range of input values it may take

def generate_test_input_param(kind):
### generate data meeting pre-condition for the type

def generate_test_input_params(kinds):
return [generate_test_input_param(kind) for kind in kinds]

for i in range(max_attempts):
[a, b, c, ...] = generate_test_input_params(type1, type2, type3, ...)
output = function_under_test(a, b, c, ...)
assert property1(output)
assert property2(output)
...

In above example, the input parameters are randomly generated based on a precondition. The generated parameters are passed to the function under test and the test driver validates result based on property assertions. This entire process is also referred as fuzzing, which is repeated based on a fixed range to identify any input parameters where the property assertions fail. There are a lot of libraries for property-based testing in various languages such as QuickCheck, fast-check, junit-quickcheck, ScalaCheck, etc. but we will use the api-mock-service library to demonstrate these capabilities for testing microservice APIs.

Following sections describe how the api-mock-service library can be used for testing microservice with fuzzing/property-based approaches and for mocking dependent services to produce the desired behavior:

Sample Microservices Under Test

A sample eCommerce application will be used to demonstrate property-based and generative testing. The application will use various microservices to implement online shopping experience. The primary purpose of this example is to show how different parameters can be passed to microservices, where microservice APIs will validate the input parameters, perform a simple business logic and then generate a valid result or an error condition. You can view the Open-API specifications for this sample app here.

Customer APIs

The customer APIs define operations to manage customers who shop online, e.g.:

Product APIs

The product APIs define operations to manage products that can be shopped online, e.g.:

Payment APIs

The payment APIs define operations to charge credit card and pay for online shopping, e.g.:

Order APIs

The order APIs define operations to purchase a product from the online store and it will use above APIs to validate customers, check product inventory, charge payments and then store record of orders, e.g.:

Defining Test Scenarios with Open-API Specifications

In this example, test scenarios will be generated by api-mock-service based on open-api specifications ecommerce-api.json by starting the mock service first as follows:

docker pull plexobject/api-mock-service:latest
docker run -p 8000:8000 -p 9000:9000 -e HTTP_PORT=8000 -e PROXY_PORT=9000 \
-e DATA_DIR=/tmp/mocks -e ASSET_DIR=/tmp/assets api-mock-service

And then uploading open-API specifications for ecommerce-api.json:

curl -H "Content-Type: application/yaml" --data-binary @ecommerce-api.json \
http://localhost:8000/_oapi

It will generate mock APIs for all microservices, e.g. you can produce result of products APIs, e.g.:

curl http://localhost:8000/products

to produce:

[
{
"id": "fd6a5ddb-35bc-47a9-aacb-9694ff5f8a32",
"category": "TOYS",
"inventory": 13,
"name": "Se nota.",
"price":{
"amount":2,
"currency": "USD"
}
},
{
"id": "47aab7d9-ecd2-4593-b1a6-c34bb5ca02bc",
"category": "MUSIC",
"inventory": 30,
"name": "Proferuntur mortem.",
"price":{
"amount":23,
"currency": "CAD"
}
},
{
"id": "ae649ae7-23e3-4709-b665-b1b0f436c97a",
"category": "BOOKS",
"inventory": 8,
"name": "Cor.",
"price":{
"amount":13,
"currency": "USD"
}
},
{
"id": "a3bd8426-e26d-4f66-8ee8-f55798440dc3",
"category": "MUSIC",
"inventory": 43,
"name": "E diutius.",
"price":{
"amount":22,
"currency": "USD"
}
},
{
"id": "7f328a53-1b64-4e4f-b6a6-7a69aed1b183",
"category": "BOOKS",
"inventory": 54,
"name": "Dici utroque.",
"price":{
"amount":23,
"currency": "USD"
}
}
]

Above response is randomly generated based on the properties defined in Open-API and calling this API will automatically generate all valid and error responses, e.g. calling “curl http://localhost:8000/products" again will return:

< HTTP/1.1 400 Bad Request
< Content-Type:
< Vary: Origin
< X-Mock-Path: /products
< X-Mock-Request-Count: 1
< X-Mock-Scenario: getProductByCategory-07ef44df0d38389ca9d589faaab9e458bd79e8abe7d2e1149e56c00820fac1fb
< Date: Tue, 20 Dec 2022 04:54:58 GMT
< Content-Length: 122
<
{ [122 bytes data]
{
"errors": [
"category_gym_bargain",
"expand_tuna_stomach",
"cage_enroll_between",
"bulk_choice_category",
"trend_agree_purse"
]
}

Applying Property-based/Generative Testing for Clients of Microservices

Upon uploading the Open-API specifications of microservices, the api-mock-service automatically generated templates for producing mock responses and error conditions, which can be customized for property-based and generative testing of microservice clients by defining constraints for generating input/output data and assertions for request/response validation.

Client-side Testing for Listing Products

You can find generated mock scenarios for listing products on the mock service using:

curl -v http://localhost:8000/_scenarios|jq '.'|grep "GET.getProductByCategory"

which returns:

"/_scenarios/GET/getProductByCategory-1a6d4d84e4a8a1ad706d671a26e66c419833b3a99f95cc442942f96d0d8f43f8/products": {
"/_scenarios/GET/getProductByCategory-6e522e565bb669ab3d9b09cc2e16b9d636220ec28a860a1cc30e9c5104e41f53/products": {
"/_scenarios/GET/getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1/products": {
"/_scenarios/GET/getProductByCategory-9ed14ecd11bbeb9f7bfde885d00efcbf168661354e4c48fe876c545e9a778302/products": {

and then invoking above URL paths, e.g.

curl -v http://localhost:8000/_scenarios/GET/getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1/products

which will return randomly generated response such as:

method: GET
name: getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1
path: /products
description: ""
order: 1
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{}'
path_params: {}
query_params:
category: '[\x20-\x7F]{1,128}'
headers:
"Content-Type": "application/json"
contents: ""
response:
headers: {}
contents: '[{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 10000 10000}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"{{RandStringMinMax 0 0}}"}}]'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":".+","id":"(__string__\\w+)","inventory":".+","name":"(__string__\\w+)","price.amount":".+","price.currency":"(__string__\\w+)"}'
wait_before_reply: 0s

We can customize above response contents using builtin template functions in the api-mock-service library to generate fuzz response, e.g.

headers:
"Content-Type":
- "application/json"
contents: >
[
{{- range $val := Iterate 5}}
{
"id": "{{UUID}}",
"category": "{{EnumString `BOOKS MUSIC TOYS`}}",
"inventory": {{RandNumMinMax 1 100}},
"name": "{{RandSentence 1 3}}",
"price":{
"amount":{{RandNumMinMax 1 25}},
"currency": "{{EnumString `USD CAD`}}"
}
}{{if lt $val 4}},{{end}}
{{ end }}
]
status_code: 200

In above example, we slightly improved the test template by generating product entries in a loop and using built-in functions to randomize the data. You can upload this scenario using:

curl -H "Content-Type: application/yaml" --data-binary \
@fixtures/get_products.yaml http://localhost:8000/_scenarios

You can also generate a template for returning an error response similarly, i.e.,

method: GET
name: error-products
path: /products
description: ""
order: 2
group: products
predicate: '{{NthRequest 2}}'
request:
headers:
"Content-Type": "application/json"
query_params:
category: '[\x20-\x7F]{1,128}'
response:
headers: {}
contents: '{"errors":["{{RandSentence 5 10}}"]}'
contents_file: ""
status_code: {{EnumInt 400 415 500}}
match_contents: '{"errors":"(__string__\\w+)"}'
wait_before_reply: 0s

Invoking curl -v http://localhost:8000/products" will randomly return both of those test scenarios so that client code can test for various conditions.

Client-side Testing for Creating Products

You can find mock scenarios for creating products that were generated from above Open-API specifications using:

curl -v http://localhost:8000/_scenarios|jq '.'|grep "POST.saveProduct"

You can then customize scenarios as follows and then upload it:

method: POST
name: saveProduct
path: /products
description: ""
order: 0
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{"category":"(BOOKS|MUSIC|TOYS)","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
path_params: {}
query_params: {}
headers:
"Content-Type": "application/json"
contents: |
category: {{EnumString `BOOKS MUSIC TOYS`}}
id: {{UUID}}
inventory: {{RandNumMinMax 5 500}}
name: {{RandSentence 3 5}}
price:
amount: {{RandNumMinMax 1 50}}
currency: "{{EnumString `USD CAD`}}"
response:
headers: {}
contents: '{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 5 500}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"$"}}'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":"(__string__(BOOKS|MUSIC|TOYS))","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
wait_before_reply: 0s

And then invoke above POST /products API using:

curl -H "Content-Type: application/yaml" --data-binary @fixtures/save_product.yaml http://localhost:8000/_scenarios
curl  http://localhost:8000/products -d \
'{"category":"BOOKS","id":"123","inventory":"10","name":"toy 1","price":{"amount":12,"currency":"USD"}}'

The client code can test for product properties and other error scenarios can be added to simulate failure conditions.

Applying Property-based/Generative Testing for Microservices

The api-mock-service test scenarios defined above can also be used to test against the microservice implementations. You can start your service, e.g. we will use sample-openapi for testing purpose and then invoke test request for server-side testing using:

curl -H "Content-Type: application/yaml" --data-binary @fixtures/get_products.yaml \
http://localhost:8000/_scenarios
curl -H "Content-Type: application/yaml" --data-binary @fixtures/save_product.yaml \
http://localhost:8000/_scenarios
curl -k -v -X POST http://localhost:8000/_contracts/products -d \
'{"base_url": "http://localhost:8080", "execution_times": 5, "verbose": true}'

Above command will submit request to execute all scenarios belonging to products group five times and then return:

{
"results": {
"getProducts_0": {},
"getProducts_1": {},
"getProducts_2": {},
"getProducts_3": {},
"getProducts_4": {},
"saveProduct_0": {
"id": "895f584b-dc65-4950-982e-167680bcd133",
"name": "Opificiis misera dei."
},
"saveProduct_1": {
"id": "d89b6c16-549c-4baa-9dca-4dd9bb4b3ecf",
"name": "Ea sumus aula teneant."
},
"saveProduct_2": {
"id": "15dd54eb-fe89-4de8-9570-59fca20b9969",
"name": "Vim odor et respondi."
},
"saveProduct_3": {
"id": "e3769044-2a19-4e86-b0aa-9724378a0113",
"name": "Me tua timeo an."
},
"saveProduct_4": {
"id": "07ee20b9-df9a-487d-9ff9-cf76bef09a8f",
"name": "Ruminando latinae omnibus."
}
},
"metrics": {
"getProducts_counts": 5,
"getProducts_duration_seconds": 0.007,
"saveProduct_counts": 5,
"saveProduct_duration_seconds": 0.005
},
"errors": {},
"succeeded": 10,
"failed": 0
}

You can also add custom assertions to validate the response in the save-product scenario:

method: POST
name: saveProduct
path: /products
description: ""
order: 0
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{"category":"(BOOKS|MUSIC|TOYS)","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
path_params: {}
query_params: {}
headers:
"Content-Type": "application/json"
contents: |
category: {{EnumString `BOOKS MUSIC TOYS`}}
id: {{UUID}}
inventory: {{RandNumMinMax 5 500}}
name: {{RandSentence 3 5}}
price:
amount: {{RandNumMinMax 1 50}}
currency: "{{EnumString `USD CAD`}}"
response:
headers: {}
contents: '{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 5 500}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"$"}}'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":"(__string__(BOOKS|MUSIC|TOYS))","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
pipe_properties:
- id
- name
assertions:
- VariableGE contents.inventory 5
- VariableContains contents.category S
- VariableContains contents.category X
wait_before_reply: 0s

If you try to run it again, the execution will fail with following error because none of the categories include X:

{
"results": {
"getProducts_0": {},
"getProducts_1": {},
"getProducts_2": {},
"getProducts_3": {},
"getProducts_4": {}
},
"errors": {
"saveProduct_0": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_1": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_2": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_3": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_4": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'"
},
"succeeded": 5,
"failed": 5
}

Summary

Using unit-testing and other forms of testing methodologies don’t rule out presence of the bugs but they can greatly reduce the probability of bugs. However, with large sized test suites, the maintenance of tests incur a high development cost especially if those tests are brittle that requires frequent changes. The property-based/generative testing can help fill in gaps in unit testing while keeping size of the tests suite small. The api-mock-service tool is designed to mock and test microservices using fuzzing and property-based testing techniques. This mocking library can be used to test both clients and server side implementation and can also be used to generate error conditions that are not easily reproducible. This library can be a powerful tool in your toolbox when developing distributed systems with a large number services, which can be difficult to deploy and test locally. You can read more about the api-mock-library at “Mocking and Fuzz Testing Distributed Micro Services with Record/Play, Templates and OpenAPI Specifications” and download it freely from https://github.com/bhatti/api-mock-service.

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

--

--

shahzad bhatti

Lifelong learner, technologist and a software builder based in Seattle.