Basic Format

In its simplest form, a test definition requires two things: a request to make, and some form of validation to perform on the response. Jikken allows you to define tests ranging from the most basic scenario to much more complex scenarios.

Note

Jikken tests are defined in a YAML/JSON format. Test filenames should end with the .jkt extension.

Some of the capabilities of the test definition format:

1. Tests are made up of requests and expected responses
2. They also can be set up to make two separate requests and compare them to each other
3. Tests can manipulate the JSON objects returned before comparing them to the expected results
4. Tests can be comprised of multiple stages, with any number of requests/comparison requests and expected responses
5. Tests can have a setup stage and/or a cleanup stage
6. The cleanup stage can define a step to run only on success, only on failure, or to always run
7. Tests can extract data from a response and store the data in a variable
8. Tests can inject values stored in variables into subsequent requests
9. Tests can randomly generate values for variables based on defined criteria
10. Tests can depend on other tests to execute before running, even beyond the boundaries of stages, within a single test definition

We have a number of planned enhancements to the test format, all of which will be backward compatible. As we extend and enhance the test definition to support more capabilities, we target remaining backward compatible so existing tests do not break. If we determine a change will be too damaging to be backward compatible, we will version the test definition format. At this time we do not see any upcoming need to do so.

While there are many capabilities and sections to test definition files, there are only a few unique structures which are reused as needed. Once you are familiar with these structures, it becomes easy to describe a wide range of test scenarios.

Note

This is only a subset of all of the possible sections and fields supported by the .jkt format. For an exhaustive reference, see the Full Format page.

Test outline

Test fields can be segmented into multiple sections, each corresponding to a specific desired behavior. Below is a simple skeleton view of the test definition to highlight these sections.

# metadata
name:
id:
project:
env:
tags:
requires:

# basic request fields
request:
compare:
response:

# multi-staged request fields
setup:
stages:
cleanup:

# variables
variables:

Metadata

The metadata section is made up of entirely optional fields. You can define the majority of common test scenarios without including any fields from the metadata section. We do recommend leveraging a few of the fields for ease of use.

# metadata
name:
id:
project:
env:
tags:
requires:

Field	Description
name	(Optional) A name used for providing a user-friendly label in test execution output and the dashboard. If this field is not present, the test will be referred to by its number in the execution order, or by its id or platformId. It is recommended to populate this field to make it easier to interpret test results.
id	(Optional) A unique identifier used to reference a test in dependent tests (using the requires field). If present, must contain only alphanumeric characters, hyphens, and underscores.
project	(Optional) The project to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard.
env	(Optional) The environment to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard. Generally speaking, it is often recommended for the environment setting to be passed in via configuration (such as environment variables), but for tests that always have a specific environment target, you can forcibly set the environment in the test definition.
tags	(Optional) A list of tags that apply to this test. Tags are used for selecting tests to be run, and for filtering in the dashboard. Tags are provided as a white-space-delimited string.
requires	(Optional) The id value of another test that is required to run before this one. This will force the test execution order so that, for example, variables extracted in the requires test are available to be embedded into this test. If you have a specific set of steps you want to run in order, we recommend leveraging the “multistage” test design.

Basic request fields

The basic request fields allow definition of a request with a comparer and/or a response. If you define a request, you must also define a comparer and/or a response block. You cannot have a compare or response block without a request block.

# basic request fields
request:
compare:
response:

Field	Description
request	The API request to make. This supports a HTTP method, URL, query parameters, header definitions, and a JSON body definition.
compare	The compare block is where you can define a second API to call with which you want to compare the response from the API called in the request definition. It supports all of the same fields as the request definition, with some additional fields for flexibility.
response	The response block allows you to define an expected response (status code, headers, JSON body), to be compared with the actual response(s). It also allows you to define the extraction of data from the response.

Request

The request block defines an API request to make. The same request structure is used anywhere you can specify an API call, including stage definitions, setup, cleanup, and compare calls.

request:
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:

Field	Description
method	(Optional) The HTTP method to use when making the request. Jikken currently supports Get, Post, Put, Patch, and Delete. Will default to Get if not defined.
url	The URL of the API to call. This field supports variable embeddings. Use of variables is recommended to leverage the same test definition across environments as part of your CI/CD process.
params	(Optional) The params field allows you to define URL Query Parameters to attach to the defined URL. Params are defined by a key (param) and value. The values for params support variable embeddings.
headers	(Optional) The list of HTTP headers to send with the request. They are defined with a key (header name) and value. The values for headers support variable embeddings.
body	(Optional) The JSON body to include in the request. This can be literal JSON or a variable referencing literal JSON or a file.

Compare

The compare block allows you to specify an additional API request to be made, whose response will be compared with the original request. Compare supports all fields from request, but also supports some additional fields.

Note

Compare requests automatically inherit the headers and params blocks from the request definition.

compare:
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:

Response

The response block provides a means to validate the response by defining expected status codes, body, and headers. It provides an ignore list to prune out fields of the JSON response before comparing the results. Values from the response to be stored into variables for use in future requests can by specified using the extract field.

response:
  status:
  headers:
  - header:
    value:
  body:
  ignore:
  -
  extract:
  - name:
    field:

Field	Description
status	(Optional) The expected status code. This will be used to validate both the request and the compare request (if one is defined). This field also supports specifying constraints instead of a literal value, see the Full Format page for details.
headers	(Optional) A list of expected header objects. Any header defined here must match the headers returned for the request and compare calls. The request may have additional headers which are not defined here, and not be considered a failure.
body	(Optional) The expected JSON body. This will be used to validate both the request and compare request (if one is defined). This can be literal JSON or a variable referencing literal JSON or a file.
ignore	(Optional) A list of JSON fields which will be pruned from the response bodies before comparing them.
extract	(Optional) A list of variable extraction definitions. This allows you to store a value from the request’s response body into a variable that can be embedded into later stages and/or tests.

Tip

If your API calls require authentication, it is recommended to always define the response status code. If an auth failure occurs, it is likely both the request and the compare endpoints will return the same failure, which could cause the test to erroneously pass. However, if you specify the expected status code in the response definition, then the test will fail appropriately.

Multi-staged request fields

Multi-staged tests allow you to define a series of operations in a single test definition file. It supports a setup stage which runs prior to any normal stages, as well as a clean-up stage which can run only on failure, only on success, or always run.

Note

It is technically possible to define a normal test definition (non-staged) and a staged definition in the same file. If you define the parent level request, compare, and response fields, they will simply be prepended to the stages list,and will execute prior to all requests defined in the stages block.

setup:
  request:
  response:
stages:
  - name:
    request:
    compare:
    response:
    variables:
cleanup:
  onsuccess:
  onfailure:
  always:

Field	Description
setup	(Optional) The setup field allows you to define a request and response which executes prior to any stage. If the setup block defines a response which triggers it to fail, the stages will not be executed. It is common to leverage this block with variable extraction for auth tokens, which can be embedded into test stages.
stages	The stages field allows you to define a list of requests. Each request supports compare, response, and variables definitions. The stages are executed in the order that they are defined. If a stage fails, the test will terminate immediately, so subsequent stages will not execute.
cleanup	(Optional) The cleanup field allows you to define a request that only runs on success, only runs on failure, or always runs.
cleanup.onsuccess	This field is a standard request block. It will only execute if the test succeeds.
cleanup.onfailure	This field is a standard request block. It will only execute if the test fails.
cleanup.always	This field is a standard request block. It will always run, regardless of the outcome of the test. If an onsuccess or onfailure is also defined, this will execute AFTER the other cleanup request.

Variables

Variables are what give tests the ability to be dynamic. They support generating values for fuzz testing, generating values that are relative (for example, dates relative to TODAY), cycling through multiple pieces of input data to verify more scenarios in the same test definition, or even extracting data from one request so it can be embedded into another request.

variables:
- name:
  type:
  value:
  valueSet:
  file:

Field	Description
name	The name of the variable. This is used to reference the variable for embedding. Variable names should to be unique or they will overwrite each other. If a global variable is defined with a given name and a local variable has the same name, the local variable will overwrite the global variable.
type	(Optional) The data type of the variable. Valid values are Int, String, Date, Float, Object, List, Boolean, Email, Name, and Datetime. The type must be specified when generating a random variable value. Mutually exclusive with value, valueSet, and file.
value	(Optional) The value to store in the variable when embedding it. Mutually exclusive with valueSet, type, and file.
valueSet	(Optional) The value to store in the variable when embedding it. This supports JSON sequences (arrays), as defined by a comma delimited list surrounded by []. When a sequence is present, each iteration will grab the next value in the sequence. If the iteration is longer than the sequence length, it will repeat starting from the first item. Mutually exclusive with value, type, and file.
file	(Optional) The absolute or relative path of a file whose contents should be stored in the variable. Mutually exclusive with value, valueSet, and type.