Full Format

Jikken tests are defined in a YAML/JSON format. Below is an example of the full structure with all possible fields.

This format is subject to change over time as we add more capabilities to Jikken.

name:
id:
platformId:
description:
project:
env:
tags:
requires:
disabled:
iterate:
request:
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:
compare:
  method:
  url:
  params:
  - param:
    value:
  addParams:
  - param:
    value:
  ignoreParams:
  -
  headers:
  - header:
    value:
  addHeaders:
  - header:
    value:
  ignoreHeaders:
  -
  body:
  strict:
response:
  status:
  time:
  headers:
  - header:
    value:
  body:
  bodySchema:
  strict:
  ignore:
  -
  extract:
  - name:
    field:
setup:
  request:
    method:
    url:
    params:
    - param:
      value:
    headers:
    - header:
      value:
    body:
  response:
    status:
    time:
    headers:
    - header:
      value:
    body:
    bodySchema:
    strict:
    ignore:
    -
    extract:
    - name:
      field:
stages:
- name:
  request:
    method:
    url:
    params:
    - param:
      value:
    headers:
    - header:
      value:
    body:
  compare:
    method:
    url:
    params:
    - param:
      value:
    addParams:
    - param:
      value:
    ignoreParams:
    -
    headers:
    - header:
      value:
    addHeaders:
    - header:
      value:
    ignoreHeaders:
    -
    body:
    strict:
  response:
    status:
    time:
    headers:
    - header:
      value:
    body:
    bodySchema:
    strict:
    ignore:
    -
    extract:
    - name:
      field:
  delay:
cleanup:
  onsuccess:
    method:
    url:
    params:
    - param:
      value:
    headers:
    - header:
      value:
    body:
  onfailure:
    method:
    url:
    params:
    - param:
      value:
    headers:
    - header:
      value:
    body:
  always:
    method:
    url:
    params:
    - param:
      value:
    headers:
    - header:
      value:
    body:
variables:
- name:
  type:
  value:
  valueSet:
  file:
  min:
  max:
  pattern:
  oneOf:
  noneOf:
  anyOf:
  length:
  minLength:
  maxLength:
  schema:
  modifier:
    operation:
    value:
    unit:
  format:

Field definitions

Test definition

name:
id:
platformId:
description:
project:
env:
tags:
requires:
disabled:
iterate:
request:
compare:
response:
setup:
stages:
cleanup:
variables:

Field	Example	Description
name	`User Login`	(Optional) A name used for providing a user-friendly label in test execution output and the dashboard. If this field is not present, the test will be referred to by its number in the execution order, or by its `id` or `platformId`.
id	`auth`	(Optional) A unique identifier used to reference a test in dependent tests (using the `requires` field). If present, must contain only alphanumeric characters, hyphens, and underscores.
platformId	`01J7GW3YEJ7WB7PA4HS48HQXQ5`	(Optional) A ULID used to uniquely identify and track a test in the Jikken cloud platform. This field is required when sending telemetry data via an API key, and should not be changed once the test is created, as this will impact your ability to view the history of the test’s executions over time. This value can be generated via the `jk new` command, or the `jk validate` command using the `--generate-platform-ids` option.
description		(Optional) A description for the test. This can be visualized in the test dashboard but it is not leveraged by the CLI test executor.
project		(Optional) The project to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard.
env	`prod`	(Optional) The environment to be associated with the test. This is used for organizing, filtering, and monitoring test runs in the dashboard.
tags	`regression` `smoke` `login`	(Optional) A list of tags that apply to this test. Tags are used for selecting tests to be run, and for filtering in the dashboard. Tags are provided as a white-space-delimited string.
requires	`7431857f-7e00-40b7-ac2c-077d230dff1c`	(Optional) The `id` value of another test that is required to run before this one. This will enforce an order of execution where the required tests are executed prior to their dependents. This is useful for variable extraction, where a value returned from one request is injected into a future request in a different test file. Currently this only supports a single value, but we plan to support more robust dependency graphing of test execution.
disabled	`true`	(Optional) A mechanism for specifying that a test should not be executed. By default, this is false. If `true` the test will not execute even if the filters match when running `jk run`
iterate	`5`	(Optional) The number of times this test should be repeated per run. When variables are defined for the test, based on the generative nature of those variables, each iteration can pass in different values. This is useful if you want to test the same API multiple times with different parameters - you can do so with a single test file.
request		The request structure defines the API request to be made, including the URL and parameters. See the Request section for details.
compare		(Optional) The compare structure defines an additional API request to be made, for which the response will be compared against the result of `request`. This is useful for validating two different environments or versions of an API. You can define `request` to call an API in a QA/Staging environment and `compare` to call that same API in Production, for example, and then compare the results to see if there are any unintended changes in the response. See the Compare section for details.
response		(Optional) The expected response values to validate. Though the `compare` and `response` fields are both optional, if at least one is not defined, then no validation is performed and nothing is being tested. See the Response section for details.
setup		(Optional) The setup structure defines a special stage that occurs prior to all other requests/stages. If the setup stage fails, subsequent stages and request definitions will not execute. See the Setup section for details.
stages		(Optional) Stages allow you to define multiple steps to be performed during a single test. Each stage can extract data from a response, which can then be injected into future requests. See the Stages section for details.
cleanup		(Optional) The cleanup structure defines a special stage that occurs after all other requests/stages. The cleanup stage can run only on success, only on failure, or always. See the Cleanup section for details.
variables		(Optional) A list of locally-defined variables. These variables allow you to generate and embed values into requests. Variable embedding is currently supported by the following fields: `Request.Url`, `Request.Headers`, `Request.Params`, `Compare.Url`, `Compare.Headers`, `Compare.AddHeaders`, `Compare.Params`, and `Compare.AddParams`. We plan to expand the scope of what variables can do, as well as where they can be injected. See the Variables section for details.

Request

request:
  method:
  url:
  params:
  - param:
    value:
  headers:
  - header:
    value:
  body:

Field	Example	Description
method	`Post`	(Optional) The HTTP method for the request. Supported values are: `Get`, `Post`, `Put`, `Patch`, and `Delete`. If this field is not present, it is defaulted to `Get`.
url	`http://api.myurl.com`	The URL to be called for the request. This field supports injection of both global and local variables.
params		(Optional) A list of query parameters to add to the URL request. See the Params section for details.
headers		(Optional) A list of HTTP headers to send in the request. See the Headers section for details.
body	`{ "test": "response" }`	(Optional) The JSON body to send in the request. This field supports literal JSON and variable embedding.

Compare

compare:
  method:
  url:
  params:
  - param:
    value:
  addParams:
  - param:
    value:
  ignoreParams:
  -
  headers:
  - header:
    value:
  addHeaders:
  - header:
    value:
  ignoreHeaders:
  -
  body:
  bodySchema:
  strict:

Field	Example	Description
method	`Post`	(Optional) The HTTP method for the request. Supported values are: `Get`, `Post`, `Put`, `Patch`, and `Delete`. If this field is not present, it is defaulted to `Get`.
url	`http://api.myurl.com`	The URL to be called for the request. This field supports injection of both global and local variables.
params		(Optional) A list of query parameters to add to the URL request. For the `compare` structure, if this field is not provided, it will be defaulted to the same value defined in the base `request.params` field. This simplifies the common case of testing different API versions by removing the need to define parameters twice. If this field is provided, the base `request` params will not be included. See the Params section for details.
addParams		(Optional) A list of additional query parameters to add to the URL request. This is used to add more query parameters in addition to those specified in the `request`. See the Params section for details.
ignoreParams	`- foo`	(Optional) A list of query parameters to be excluded from the `compare` request. This field allows you to specify parameters defined in the base `request.params` object that you do not wish to include in the `compare` request.
headers		(Optional) A list of HTTP headers to send in the request. For the `compare` structure, if this field is not provided, it will be defaulted to the same value defined in the base `request.headers` field. This simplifies the common case of testing different API versions by removing the need to define headers twice. If this field is provided, the base `request` headers will not be included. See the Headers section for details.
addHeaders		(Optional) A list of additional HTTP headers to send in the request. This is used to add more headers in addition to those specified in the `request`. See the Headers section for details.
ignoreHeaders	`- Authorization`	(Optional) A list of HTTP headers to be excluded from the `compare` request. This field allows you to specify headers defined in the base `request.headers` object that you do not wish to include in the `compare` request.
body	`{ "test": "response" }`	(Optional) The JSON body to send in the request. This field supports literal JSON and variable embedding.
strict	`true`	(Optional) When true, body fields in the request response that are missing from the compare response body result in an error. By default, this is true.

Response

response:
  status:
  headers:
  - header:
    value:
  ignore:
  -
  extract:
  - name:
    field:
  body:
  bodySchema:
  strict:

Field	Example	Description
status	`200`	(Optional) The expected HTTP status code. We recommend defining this in most cases, as it will be a great indicator of unforeseen problems such as authorization issues, gateway/proxy issues, etc. This field can hold a literal integer, or an object which supports all integer advanced validation fields (but does not require the `type` field, as it is always an integer). For example: status: anyOf: [200, 201]
time	`500`	(Optional) The expected response time in milliseconds. This field can hold a literal integer (in which case, it represents the maximum allowed time), or an object which supports all integer advanced validation fields (but does not require the `type` field, as it is always an integer). For example: time: min: 500 max: 1000
headers		(Optional) A list of expected HTTP headers. Any header defined here must match the headers returned for the request and compare calls. The request may have additional headers which are not defined here, and not be considered a failure.
ignore	`- data.users.lastLogin`	(Optional) A list of JSON fields to ignore when performing `body` validation. This can be useful when there are runtime-dependent fields in the response JSON, for example, if an API has a timestamp indicating the time of the request, This allows you to prune out sections of JSON in the received response prior to comparing the data between the `request` and `compare` responses and/or the `response.body` definition. This notation supports traversing both arrays and objects, so if a field is an array, the ignored field will be pruned from all objects in the array.
extract		(Optional) A list of variable extraction terms. This field allows you to extract fields from a JSON response and store them into a variable, which can then be injected into subsequent tests.
extract.name	`authToken`	The name of the variable to store the extracted field into.
extract.field	`token`	The JSON field to be extracted. In this case, the expected response structure is an object with a first-level field called `token`. This path does support nested path traversal similar to the `response.ignore` definitions, so you could extract a field at `data.user.token` if the structure had nested objects that contained the field you wish to extract.
body	`{ "test": "response" }`	(Optional) The expected response body. This field supports literal JSON and variable embedding. Mutually exclusive with `bodySchema`.
bodySchema		(Optional) The schema to be used when validating the response body. Mutually exclusive with `body`. See the Body Schema section for details.
strict	`true`	(Optional) When true, body fields in the request response that are missing from the expected response result in an error. By default, this is true.

Setup

setup:
  request:
  response:

Field	Description
request	The API request to be made prior to all other stages. See the Request section for details.
response	(Optional) The expected response values to validate. If the setup stage fails, no other stages will be executed. See the Response section for details.

Stages

stages:
- name:
  request:
  compare:
  response:
  delay:

Field	Description
name	(Optional) The name for the stage. This is used in console output and dashboard messaging.
request	The API request to be made. See the Request section for details.
compare	(Optional) The comparison request to be made. See the Compare section for details.
response	(Optional) The expected response values to validate. See the Response section for details.
delay	(Optional) The millisecond duration to pause after executing the stage. If not provided, there is no delay.

Cleanup

cleanup:
  onsuccess:
  onfailure:
  always:

Field	Description
onsuccess	(Optional) An API request to be made only when the test has succeeded. See the Request section for details.
onfailure	(Optional) An API request to be made only when the test has failed. See the Request section for details.
always	(Optional) An API request to be made every time this test executes. This runs whether the test passes or fails. If you define `onsuccess` and/or `onfailure` as well, this request will trigger after the `onsuccess`/`onfailure` requests execute. See the Request section for details.

Variables

variables:
- name:
  type:
  value:
  valueSet:
  file:
  min:
  max:
  oneOf:
  noneOf:
  anyOf:
  length:
  minLength:
  maxLength:
  schema:
  modifier:
    operation:
    value:
    unit:
  format:

Field	Example	Description
name	`foo`	The name of the variable. This is used to reference the variable in the places where you want it to be injected. Variable names must be unique or they will overwrite each other. If a global variable is defined with a given name and a local variable has the same name, the local variable will overwrite the global one. Must be at least one character in length, and contain only alphanumeric characters, hyphens, and underscores.
type	`String`	(Optional) The data type of the variable. Valid values are `Int`, `String`, `Date`, `Float`, `Object`, `List`, `Boolean`, `Email`, `Name`, and `Datetime`. The type must be specified when generating a random variable value. Mutually exclusive with `value`, `valueSet`, and `file`.
value	`my_value`	(Optional) The value to store in the variable when embedding it. Mutually exclusive with `valueSet`, `type`, and `file`.
valueSet	`[1, 5, 23143]`	(Optional) The value to store in the variable when embedding it. This supports JSON sequences (arrays), as defined by a comma delimited list surrounded by `[]`. When a sequence is present, each iteration will grab the next value in the sequence. If the iteration is longer than the sequence length, it will repeat starting from the first item. Mutually exclusive with `value`, `type`, and `file`.
file		(Optional) The absolute or relative path of a file whose contents should be stored in the variable. Mutually exclusive with `value`, `valueSet`, and `type`.
min	`15`	(Optional) The minimum possible value for a numeric type. Only applies to variables generated using the `type` field with a value of `Int` or `Float`.
max	`100`	(Optional) The maximum possible value for a numeric type. Only applies to variables generated using the `type` field with a value of `Int` or `Float`.
oneOf	`[10, 20, 30]`	(Optional) The possible values for a variable. Only applies to variables generated using the `type` field.
noneOf	`[1, 2, 3]`	(Optional) The excluded values for a variable. Only applies to variables generated using the `type` field.
length	`15`	(Optional) The exact length for a String or List. Only applies to variables generated using the `type` field with a value of `String` or `List`.
minLength	`15`	(Optional) The minimum possible length for a String or List. Only applies to variables generated using the `type` field with a value of `String` or `List`.
maxLength	`30`	(Optional) The maximum possible length for a String or List. Only applies to variables generated using the `type` field with a value of `String` or `List`.
schema		(Optional) Used to add constraints to the structure of generated List and Object types. For Objects, this is an object where the names refer to literal JSON values or Jikken variables. For Lists, one or more `anyOf`, `oneOf` values or length constraints can be specified. Only applies to variables generated using the `type` field with a value of `Object` or `List`.
modifier		(Optional) The modifier object allows you to define simple operations to be performed against the variable value. This is useful when leveraging extracted variables or globals. This structure is currently only supported for `Date` and `DateTime` types, but we plan to expand this capability with a number of operations for other data types. This allows a test to start with the value for `${TODAY}` and then to add/subtract a provided number of days, weeks, or months from the starting date. Must be used alongside `value`.
modifier.operation	`subtract`	The operation to perform. Currently this supports `add` and `subtract`.
modifier.value	`3`	The amount to modify the variable’s value. Currently this supports unsigned integers (positive whole numbers).
modifier.unit	`days`	The unit to be applied to `value`. This currently supports `days`, `weeks`, and `months`.
format	`%Y-%m-%d`	(Optional) The format to be applied to a generated variable. This is currently only supported for `Date` and `Datetime` types. For all supported format tokens, see the chrono documentation page.

Params

params:
- param:
  value:

Field	Example	Description
param	`foo`	The query parameter name.
value	`bar`	The query parameter value.

With the provided example, the request URL would become <url>?foo=bar.

Headers

headers:
- header:
  value:

Field	Example	Description
header	`Authorization`	The HTTP header key.
value	`eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9`	The HTTP header value.

Body Schema

bodySchema:
  type:
  schema:
    "field1": 0
    "field2": "value"
    "field3": {"field": "value"}
    "field4":
      type:
      min:
      max:
      ...

Field	Example	Description
type	`Object`	The data type of the body. See the Variables section for a list of valid types.
schema		A representation of the schema to be used when generating or validating a body.
schema.”field”	`"field": "value"`	A field to be included in the body. This is a key-value pair, where the key is a string indicating the name of the field. The value can be a literal numeric or string value, a literal JSON value, or a Jikken variable specification. See the Variables section for details on the Jikken variable format.