Basic Format

In its simplest form, a test definition requires two things:: a request to make, and some form of validation to perform on the response. Jikken allows you to define tests ranging from the most basic scenario to much more complex scenarios.

Note

Jikken tests are defined in a YAML/JSON format. Test filenames should end with the .jkt extension.

Some of the capabilities of the test definition format:

1. Tests are made up of requests and expected responses
2. They also can be set up to make two separate requests and compare them to each other
3. Tests can manipulate the JSON objects returned before comparing them to the expected results
4. Tests can be comprised of multiple stages, with any number of requests/comparison requests and expected responses
5. Tests can have a setup stage and/or a cleanup stage
6. The cleanup stage can define a step to run only on success, only on failure, or to always run
7. Tests can extract data from a response and store the data in a variable
8. Tests can inject values stored in variables into subsequent requests
9. Tests can randomly generate values for variables based on defined criteria
10. Tests can depend on other tests to execute before running, even beyond the boundaries of stages, within a single test definition

We have a number of planned enhancements to the test format, all of which will be backward compatible. As we extend and enhance the test definition to support more capabilities, we target remaining backward compatible so existing tests do not break. If we determine a change will be too damaging to be backward compatible, we will version the test definition format. At this time we do not see any upcoming need to do so.

While there are many capabilities and sections to test definition files, there are only a few unique structures which are reused as needed. Once you are familiar with these structures, it becomes easy to describe a wide range of test scenarios.

Section Outline

Test fields can be segmented into multiple sections, each corresponding to a specific desired behavior. Below is a simple skeleton view of the test definition to highlight these sections.

# metadata
name:
id:
description:
project:
env:
tags:
requires:
iterate:

# basic request fields
request:
compare:
response:

# multi-staged request fields
setup:
stages:
- name:
  request:
  compare:
  response:
  variables:
  delay:
cleanup:
  onsuccess:
  onfailure:
  always:

# variables
variables:

Metadata

The metadata section is made up of entirely optional fields. You can define the majority of common test scenarios without including any fields from the metadata section. We do recommend leveraging a few of the fields for ease of use.

# metadata
name:
id:
description:
project:
env:
tags:
requires:
iterate:

Field	Description
name	(Optional) A name used for providing a user-friendly label in test execution output and the dashboard. If this field is not present, the test will be referred to by its number in the execution order, or its id. It is recommended to populate this field to make it easier to interpret test results.
id	(Optional) A unique identifier for tracking a test across many runs, and for referencing a test in dependent tests. This identifier can be any string, but we recommend using a UUID. When creating tests via the jk new command, an id will be generated automatically. If this field is not present, a hash of the file’s contents will serve as the unique identifier, so any change to the test will result in a new id. The id changing over time may impact tracking the test execution history in the Jikken webapp. It is also required to provide this field if you want to depend on the execution of this test in another test (as the ID field is what links them).
description	(Optional) A description for the test. This can be visualized in the test dashboard but it is not leveraged by the CLI test executor.
project	(Optional) The project to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard.
env	(Optional) The environment to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard. Generally speaking, it is often recommended for the environment setting to be passed in via configuration (such as environment variables), but for tests that always have a specific environment target, you can forcibly set the environment in the test definition.
tags	(Optional) A list of tags that apply to this test. Tags are used for selecting tests to be run, and for filtering in the dashboard. Tags are provided as a white-space-delimited string.
requires	(Optional) The id value of another test that is required to run before this one. This will force the test execution order so that, for example, variables extracted in the requires test are available to be embedded into this test. If you have a specific set of steps you want to run in order, we recommend leveraging the “multistage” test design.
iterate	(Optional) The number of times this test should be repeated per run. When variables are defined for the test, based on the generative nature of those variables, each iteration can pass in different values. This is useful if you want to test the same API multiple times with different parameters - you can do so with a single test file.

Basic Request Fields

The basic request fields allow definition of a request with a comparer and/or a response. If you define a request, you must also define a comparer and/or a response block. You cannot have a compare or response block without a request block.

# basic request fields
request:
compare:
response:

Field	Description
request	The API request to make. This supports a HTTP method, URL, query parameters, header definitions, and a JSON body definition.
compare	The compare block is where you can define a second API to call with which you want to compare the response from the API called in the request definition. It supports all of the same fields as the request definition, with some additional fields for flexibility.
response	The response block allows you to define an expected response (status code, headers, JSON body), to be compared with the actual response(s). It also allows you to define the extraction of data from the response.

Request Fields

The request block defines an API request to make. The same request structure is used anywhere you can specify an API call, including stage definitions, setup, cleanup, and compare calls.

request:
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:

Field	Description
method	(Optional) The HTTP method to use when making the request. Jikken currently supports Get, Post, Put, Patch, and Delete. Will default to Get if not defined.
url	The URL of the API to call. This field supports variable embeddings. Use of variables is recommended to leverage the same test definition across environments as part of your CI/CD process.
params	(Optional) The params field allows you to define URL Query Parameters to attach to the defined URL. Params are defined by a key (param) and value. The values for params support variable embeddings.
headers	(Optional) The list of HTTP headers to send with the request. They are defined with a key (header) and value. The values for headers support variable embeddings.
body	(Optional) The JSON body to include in the request. This can be literal JSON or a variable referencing literal JSON or a file.

Compare Fields

The compare block allows you to specify an additional API request to be made, whose response will be compared with the original request. Compare supports all fields from request, but also supports some additional fields. All listed fields are optional.

Note

Compare requests automatically inherit the headers and params blocks from the request definition. However, if you define the headers block in the compare request, it will NOT inherit any headers from the request definition. Likewise, if you define the params block in the compare request, it will NOT inherit any params from the request definition. We provide add and ignore fields to modify the inherited definitions.

compare:
  # request fields
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:

  # additional fields
  addParams:
  - param:
    value:
  ignoreParams:
  -
  addHeaders:
  - header:
    value:
  ignoreHeaders:
  -

Field	Description
addParams	The addParams field allows you to add additional parameters to the compare request, beyond those specified in the original request.
ignoreParams	The ignoreParams field allows you to remove a parameter from the list of params in the original request definition.
addHeaders	The addHeaders field allows you to add additional headers to the compare request, beyond those specified in the original request.
ignoreHeaders	The ignoreHeaders field allows you to remove a header from the list of headers in the original request definition.

Tip

If you combine the ignore and add fields, you are able to replace a value for an existing definition. It will first ignore the header/parameter from the original collection and then add it back in with whatever new value you provide.

Response Fields

The response block provides a means to validate the response by defining expected status codes, body, and headers. It provides an ignore list to prune out fields of the JSON response before comparing the results. Values from the response to be stored into variables for use in future requests can by specified using the extract field. All listed fields are optional.

response:
  status:
  headers:
  - header:
    value:
  body:
  ignore:
  -
  extract:
  - name:
    field:

Field	Description
status	The expected status code. This will be used to validate both the request and the compare request (if one is defined).
body	The expected JSON body. This will be used to validate both the request and compare request (if one is defined). This can be literal JSON or a variable referencing literal JSON or a file.
headers	A list of expected header objects. Any header defined here must match the headers returned for the request and compare calls. The request may have additional headers which are not defined here, and not be considered a failure.
ignore	A list of JSON fields which will be pruned from the response bodies before comparing them.
extract	A list of variable extraction definitions. This allows you to store a value from the request’s response body into a variable that can be embedded into later stages and/or tests.

Tip

If your API calls require authentication, it is recommended to always define the response status code. If an auth failure occurs, it is likely both the request and the compare endpoints will return the same failure, which could cause the test to erroneously pass. However, if you specify the expected status code in the response definition, then the test will fail appropriately.

Multi-Staged Request Fields

Multi-staged tests allow you to define a series of operations in a single test definition file. It supports a setup stage which runs prior to any normal stages, as well as a clean-up stage which can run only on failure, only on success, or always run.

Note

It is technically possible to define a normal test definition (non-staged) and a staged definition in the same file. If you define the parent level request, compare, and response fields, they will simply be prepended to the stages list,and will execute prior to all requests defined in the stages block.

setup:
  request:
  response:
stages:
  - name:
    request:
    compare:
    response:
    variables:
cleanup:
  onsuccess:
  onfailure:
  always:

Field	Description
setup	(Optional) The setup field allows you to define a request and response which executes prior to any stage. If the setup block defines a response which triggers it to fail, the stages will not be executed. It is common to leverage this block with variable extraction for auth tokens, which can be embedded into test stages.
stages	The stages field allows you to define a list of requests. Each request supports compare, response, and variables definitions. The stages are executed in the order that they are defined. If a stage fails, the test will terminate immediately, so subsequent stages will not execute.
cleanup	(Optional) The cleanup field allows you to define a request that only runs on success, only runs on failure, or always runs.
cleanup.onsuccess	This field is a standard request definition. It will only execute if the test succeeds.
cleanup.onfailure	This field is a standard request definition. It will only execute if the test fails.
cleanup.always	This field is a standard request definition. It will always run regardless of the outcome of the test. If an onsuccess or onfailure is also defined, this will execute AFTER the other cleanup request.

Variables

Variables are what give tests the ability to be dynamic. They support generating values for fuzz testing, generating values that are relative (for example, dates relative to TODAY), cycling through multiple pieces of input data to verify more scenarios in the same test definition, or even extracting data from one request so it can be embedded into another request.

variables:
- name:
  dataType:
  value:
  file:
  modifier:
    operation:
    value:
    unit:
  format:

Field	Description
name	The name of the variable. This is used to reference the variable for embedding. Variable names should to be unique or they will overwrite each other. If a global variable is defined with a given name and a local variable has the same name, the local variable will overwrite the global variable.
dataType	The data type of the variable. Currently supported values are: Int, String, and Date. The data type is used when handling value generation and modifier operations. Our plan for the future is to remove the requirement of this field and to allow mix-mode types for sequence values (arrays). For now, you can use String to cover mixed data types when using arrays.
value	(Optional) The value to store in the variable when embedding it. This supports single values or JSON sequences (arrays), as defined by a comma delimited list surrounded by []. When a sequence is present, each iteration will grab the next value in the sequence. Either value or file must be provided.
file	(Optional) The absolute or relative path of a file whose contents should be stored in the variable. Either value or file must be provided.
modifier	(Optional) The modifier field allows you to define simple operations to apply on the variable value. This is useful when leveraging extracted variables or globals. This structure is currently only supported for Date types but we plan to expand this to other data types.
modifier.operation	The modifier operation indicates the operation to perform. Currently this supports add and subtract.
modifier.value	The modifier value indicates the amount to modify the variable’s value. Currently this only supports unsigned integers.
modified.unit	The modifier unit indicates the unit of value to modify the variable by. Currently this supports days, weeks, and months. This allows a test to start with the value for TODAY and then add/subtract a specified number of days, weeks, or months from the starting date. We plan to expand this capability with a number of operations for other data types.
format	(Placeholder) This field is a placeholder and is not currently used. It will allow you to modify the output of the variable’s value.