Property-based and Generative testing for Microservices
The software development cycle for microservices generally include unit testing during the development where mock implementation for the dependent services are injected with the desired behavior to test various test-scenarios and failure conditions. However, the development teams often use real dependent services for integration testing of a microservice in a local environment. This poses a considerable challenge as each dependent service may be keeping its own state that makes it harder to reliably validate the regression behavior or simulate certain error response. Further, as the number of request parameters to the service or downstream services grow, the combinatorial explosion for test cases become unmanageable. This is where property-based testing offers a relief as it allows testing against automatically generated input fuzz-data, which is why this form of testing is also referred as a generative testing. A generator defines a function that generate random data based on type of input and constraints on the range of input values. The property-based test driver then iteratively calls the system under test to validate the result and assert the desired behavior, e.g.
def pre_condition_test_input_param(kind):
### assert pre-condition based on type of parameter and range of input values it may take
def generate_test_input_param(kind):
### generate data meeting pre-condition for the type
def generate_test_input_params(kinds):
return [generate_test_input_param(kind) for kind in kinds]
for i in range(max_attempts):
[a, b, c, ...] = generate_test_input_params(type1, type2, type3, ...)
output = function_under_test(a, b, c, ...)
assert property1(output)
assert property2(output)
...In above example, the input parameters are randomly generated based on a precondition. The generated parameters are passed to the function under test and the test driver validates result based on property assertions. This entire process is also referred as fuzzing, which is repeated based on a fixed range to identify any input parameters where the property assertions fail. There are a lot of libraries for property-based testing in various languages such as QuickCheck, fast-check, junit-quickcheck, ScalaCheck, etc. but we will use the api-mock-service library to demonstrate these capabilities for testing microservice APIs.
Following sections describe how the api-mock-service library can be used for testing microservice with fuzzing/property-based approaches and for mocking dependent services to produce the desired behavior:
Sample Microservices Under Test
A sample eCommerce application will be used to demonstrate property-based and generative testing. The application will use various microservices to implement online shopping experience. The primary purpose of this example is to show how different parameters can be passed to microservices, where microservice APIs will validate the input parameters, perform a simple business logic and then generate a valid result or an error condition. You can view the Open-API specifications for this sample app here.
Customer APIs
The customer APIs define operations to manage customers who shop online, e.g.:
Product APIs
The product APIs define operations to manage products that can be shopped online, e.g.:
Payment APIs
The payment APIs define operations to charge credit card and pay for online shopping, e.g.:
Order APIs
The order APIs define operations to purchase a product from the online store and it will use above APIs to validate customers, check product inventory, charge payments and then store record of orders, e.g.:
Defining Test Scenarios with Open-API Specifications
In this example, test scenarios will be generated by api-mock-service based on open-api specifications ecommerce-api.json by starting the mock service first as follows:
docker pull plexobject/api-mock-service:latest
docker run -p 8000:8000 -p 9000:9000 -e HTTP_PORT=8000 -e PROXY_PORT=9000 \
-e DATA_DIR=/tmp/mocks -e ASSET_DIR=/tmp/assets api-mock-serviceAnd then uploading open-API specifications for ecommerce-api.json:
curl -H "Content-Type: application/yaml" --data-binary @ecommerce-api.json \
http://localhost:8000/_oapiIt will generate mock APIs for all microservices, e.g. you can produce result of products APIs, e.g.:
curl http://localhost:8000/productsto produce:
[
{
"id": "fd6a5ddb-35bc-47a9-aacb-9694ff5f8a32",
"category": "TOYS",
"inventory": 13,
"name": "Se nota.",
"price":{
"amount":2,
"currency": "USD"
}
},
{
"id": "47aab7d9-ecd2-4593-b1a6-c34bb5ca02bc",
"category": "MUSIC",
"inventory": 30,
"name": "Proferuntur mortem.",
"price":{
"amount":23,
"currency": "CAD"
}
},
{
"id": "ae649ae7-23e3-4709-b665-b1b0f436c97a",
"category": "BOOKS",
"inventory": 8,
"name": "Cor.",
"price":{
"amount":13,
"currency": "USD"
}
},
{
"id": "a3bd8426-e26d-4f66-8ee8-f55798440dc3",
"category": "MUSIC",
"inventory": 43,
"name": "E diutius.",
"price":{
"amount":22,
"currency": "USD"
}
},
{
"id": "7f328a53-1b64-4e4f-b6a6-7a69aed1b183",
"category": "BOOKS",
"inventory": 54,
"name": "Dici utroque.",
"price":{
"amount":23,
"currency": "USD"
}
}
]Above response is randomly generated based on the properties defined in Open-API and calling this API will automatically generate all valid and error responses, e.g. calling “curl http://localhost:8000/products" again will return:
< HTTP/1.1 400 Bad Request
< Content-Type:
< Vary: Origin
< X-Mock-Path: /products
< X-Mock-Request-Count: 1
< X-Mock-Scenario: getProductByCategory-07ef44df0d38389ca9d589faaab9e458bd79e8abe7d2e1149e56c00820fac1fb
< Date: Tue, 20 Dec 2022 04:54:58 GMT
< Content-Length: 122
<
{ [122 bytes data]{
"errors": [
"category_gym_bargain",
"expand_tuna_stomach",
"cage_enroll_between",
"bulk_choice_category",
"trend_agree_purse"
]
}Applying Property-based/Generative Testing for Clients of Microservices
Upon uploading the Open-API specifications of microservices, the api-mock-service automatically generated templates for producing mock responses and error conditions, which can be customized for property-based and generative testing of microservice clients by defining constraints for generating input/output data and assertions for request/response validation.
Client-side Testing for Listing Products
You can find generated mock scenarios for listing products on the mock service using:
curl -v http://localhost:8000/_scenarios|jq '.'|grep "GET.getProductByCategory"which returns:
"/_scenarios/GET/getProductByCategory-1a6d4d84e4a8a1ad706d671a26e66c419833b3a99f95cc442942f96d0d8f43f8/products": {
"/_scenarios/GET/getProductByCategory-6e522e565bb669ab3d9b09cc2e16b9d636220ec28a860a1cc30e9c5104e41f53/products": {
"/_scenarios/GET/getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1/products": {
"/_scenarios/GET/getProductByCategory-9ed14ecd11bbeb9f7bfde885d00efcbf168661354e4c48fe876c545e9a778302/products": {and then invoking above URL paths, e.g.
curl -v http://localhost:8000/_scenarios/GET/getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1/productswhich will return randomly generated response such as:
method: GET
name: getProductByCategory-7ede8f15af851323576a0c864841e859408525632eb002a1571676a0d835a0e1
path: /products
description: ""
order: 1
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{}'
path_params: {}
query_params:
category: '[\x20-\x7F]{1,128}'
headers:
"Content-Type": "application/json"
contents: ""
response:
headers: {}
contents: '[{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 10000 10000}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"{{RandStringMinMax 0 0}}"}}]'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":".+","id":"(__string__\\w+)","inventory":".+","name":"(__string__\\w+)","price.amount":".+","price.currency":"(__string__\\w+)"}'
wait_before_reply: 0sWe can customize above response contents using builtin template functions in the api-mock-service library to generate fuzz response, e.g.
headers:
"Content-Type":
- "application/json"
contents: >
[
{{- range $val := Iterate 5}}
{
"id": "{{UUID}}",
"category": "{{EnumString `BOOKS MUSIC TOYS`}}",
"inventory": {{RandNumMinMax 1 100}},
"name": "{{RandSentence 1 3}}",
"price":{
"amount":{{RandNumMinMax 1 25}},
"currency": "{{EnumString `USD CAD`}}"
}
}{{if lt $val 4}},{{end}}
{{ end }}
]
status_code: 200In above example, we slightly improved the test template by generating product entries in a loop and using built-in functions to randomize the data. You can upload this scenario using:
curl -H "Content-Type: application/yaml" --data-binary \
@fixtures/get_products.yaml http://localhost:8000/_scenariosYou can also generate a template for returning an error response similarly, i.e.,
method: GET
name: error-products
path: /products
description: ""
order: 2
group: products
predicate: '{{NthRequest 2}}'
request:
headers:
"Content-Type": "application/json"
query_params:
category: '[\x20-\x7F]{1,128}'
response:
headers: {}
contents: '{"errors":["{{RandSentence 5 10}}"]}'
contents_file: ""
status_code: {{EnumInt 400 415 500}}
match_contents: '{"errors":"(__string__\\w+)"}'
wait_before_reply: 0sInvoking “curl -v http://localhost:8000/products" will randomly return both of those test scenarios so that client code can test for various conditions.
Client-side Testing for Creating Products
You can find mock scenarios for creating products that were generated from above Open-API specifications using:
curl -v http://localhost:8000/_scenarios|jq '.'|grep "POST.saveProduct"You can then customize scenarios as follows and then upload it:
method: POST
name: saveProduct
path: /products
description: ""
order: 0
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{"category":"(BOOKS|MUSIC|TOYS)","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
path_params: {}
query_params: {}
headers:
"Content-Type": "application/json"
contents: |
category: {{EnumString `BOOKS MUSIC TOYS`}}
id: {{UUID}}
inventory: {{RandNumMinMax 5 500}}
name: {{RandSentence 3 5}}
price:
amount: {{RandNumMinMax 1 50}}
currency: "{{EnumString `USD CAD`}}"
response:
headers: {}
contents: '{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 5 500}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"$"}}'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":"(__string__(BOOKS|MUSIC|TOYS))","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
wait_before_reply: 0sAnd then invoke above POST /products API using:
curl -H "Content-Type: application/yaml" --data-binary @fixtures/save_product.yaml http://localhost:8000/_scenarioscurl http://localhost:8000/products -d \
'{"category":"BOOKS","id":"123","inventory":"10","name":"toy 1","price":{"amount":12,"currency":"USD"}}'The client code can test for product properties and other error scenarios can be added to simulate failure conditions.
Applying Property-based/Generative Testing for Microservices
The api-mock-service test scenarios defined above can also be used to test against the microservice implementations. You can start your service, e.g. we will use sample-openapi for testing purpose and then invoke test request for server-side testing using:
curl -H "Content-Type: application/yaml" --data-binary @fixtures/get_products.yaml \
http://localhost:8000/_scenarios
curl -H "Content-Type: application/yaml" --data-binary @fixtures/save_product.yaml \
http://localhost:8000/_scenarioscurl -k -v -X POST http://localhost:8000/_contracts/products -d \
'{"base_url": "http://localhost:8080", "execution_times": 5, "verbose": true}'Above command will submit request to execute all scenarios belonging to products group five times and then return:
{
"results": {
"getProducts_0": {},
"getProducts_1": {},
"getProducts_2": {},
"getProducts_3": {},
"getProducts_4": {},
"saveProduct_0": {
"id": "895f584b-dc65-4950-982e-167680bcd133",
"name": "Opificiis misera dei."
},
"saveProduct_1": {
"id": "d89b6c16-549c-4baa-9dca-4dd9bb4b3ecf",
"name": "Ea sumus aula teneant."
},
"saveProduct_2": {
"id": "15dd54eb-fe89-4de8-9570-59fca20b9969",
"name": "Vim odor et respondi."
},
"saveProduct_3": {
"id": "e3769044-2a19-4e86-b0aa-9724378a0113",
"name": "Me tua timeo an."
},
"saveProduct_4": {
"id": "07ee20b9-df9a-487d-9ff9-cf76bef09a8f",
"name": "Ruminando latinae omnibus."
}
},
"metrics": {
"getProducts_counts": 5,
"getProducts_duration_seconds": 0.007,
"saveProduct_counts": 5,
"saveProduct_duration_seconds": 0.005
},
"errors": {},
"succeeded": 10,
"failed": 0
}You can also add custom assertions to validate the response in the save-product scenario:
method: POST
name: saveProduct
path: /products
description: ""
order: 0
group: products
predicate: ""
request:
match_query_params: {}
match_headers: {}
match_contents: '{"category":"(BOOKS|MUSIC|TOYS)","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
path_params: {}
query_params: {}
headers:
"Content-Type": "application/json"
contents: |
category: {{EnumString `BOOKS MUSIC TOYS`}}
id: {{UUID}}
inventory: {{RandNumMinMax 5 500}}
name: {{RandSentence 3 5}}
price:
amount: {{RandNumMinMax 1 50}}
currency: "{{EnumString `USD CAD`}}"
response:
headers: {}
contents: '{"category":"{{EnumString `BOOKS MUSIC TOYS`}}","id":"{{RandStringMinMax 0 0}}","inventory":"{{RandNumMinMax 5 500}}","name":"{{RandStringMinMax 2 50}}","price":{"amount":{{RandNumMinMax 0 0}},"currency":"$"}}'
contents_file: ""
status_code: 200
match_headers: {}
match_contents: '{"category":"(__string__(BOOKS|MUSIC|TOYS))","id":"(__string__\\w+)","inventory":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","name":"(__string__\\w+)","price.amount":"(__number__[+-]?(([0-9]{1,10}(\\.[0-9]{1,5})?)|(\\.[0-9]{1,10})))","price.currency":"(USD|CAD)"}'
pipe_properties:
- id
- name
assertions:
- VariableGE contents.inventory 5
- VariableContains contents.category S
- VariableContains contents.category X
wait_before_reply: 0sIf you try to run it again, the execution will fail with following error because none of the categories include X:
{
"results": {
"getProducts_0": {},
"getProducts_1": {},
"getProducts_2": {},
"getProducts_3": {},
"getProducts_4": {}
},
"errors": {
"saveProduct_0": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_1": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_2": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_3": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'",
"saveProduct_4": "failed to assert '{{VariableContains \"contents.category\" \"X\"}}' with value 'false'"
},
"succeeded": 5,
"failed": 5
}Summary
Using unit-testing and other forms of testing methodologies don’t rule out presence of the bugs but they can greatly reduce the probability of bugs. However, with large sized test suites, the maintenance of tests incur a high development cost especially if those tests are brittle that requires frequent changes. The property-based/generative testing can help fill in gaps in unit testing while keeping size of the tests suite small. The api-mock-service tool is designed to mock and test microservices using fuzzing and property-based testing techniques. This mocking library can be used to test both clients and server side implementation and can also be used to generate error conditions that are not easily reproducible. This library can be a powerful tool in your toolbox when developing distributed systems with a large number services, which can be difficult to deploy and test locally. You can read more about the api-mock-library at “Mocking and Fuzz Testing Distributed Micro Services with Record/Play, Templates and OpenAPI Specifications” and download it freely from https://github.com/bhatti/api-mock-service.