Jikken CLI v0.8 Released!

Published on 2024-09-17 by Louis Russo

It’s been a few months, but it’s finally here… the release of Jikken v0.8! This release adds some powerful new features which greatly expand the capabilities of Jikken’s API test runner. Jikken now supports more complex data validation, random data generation, and additional data types for variables.

Instead of explicitly defining expected response values, you can now provide constraints around types, ranges, and collections of possible valid values. Jikken can also dynamically generate values based on these same constraints. This not only allows you to be more flexible in how you validate responses, but also enables new forms of fuzz testing to increase confidence in your APIs. All of this functionality requires minimal input when writing your tests.

We’ve worked with a number of users to refine the new features, but as with all things, input from a wider audience may inspire us to adjust them further. As always, if you have any feedback (good or bad), feel free to contact us, open an issue on Github, or even fork the repo and create a pull-request.

Thanks for all your support!

New Features

  1. Advanced JSON Validation: The ability to describe response validation in more flexible terms than just equality. These checks are dictated by the type of the field (the same types that are described in the feature below) and include options such as min, max, pattern, noneOf, oneOf, and more.

  2. Variable Data Generators: Instead of providing a value or file source for a variable, now you can specify a type along with some optional constraints, and Jikken will automatically generate an appropriate value. Available types include String, Integer, Float, Date, DateTime, Object, List, Name, Email, and Boolean. By leveraging these types and constraints, you can generate a wide range of values to enable powerful fuzz testing with minimal input required in test writing. See the docs for details and examples.

  3. Secrets: Similar to Global Variables, Secrets allow you to define values which are embedded in your test definitions. Unlike global variables, secret values are automatically masked in any console output. And if you’re using our paid analytics platform, secrets are also masked in any telemetry data sent back to us. For more details, check out the configuration docs.

  4. OpenAPI Spec Improvements: As we announced in v0.7, Jikken gained the ability to generate tests based on OpenAPI specs. In v0.8, we’ve enhanced this functionality to also support ref links. And since we’ve added Advanced JSON Validation and Variable Data Generators, Jikken now auto-populates schema validation and data generation based on example data from your OpenAPI specs. As always, we will continue to look for ways to improve test generation and save you time, so keep checking back for the latest improvements. If you’d like to request specific improvements, feel free to contact us or open an issue on Github.

  5. JKT Schema Improvements: We’ve made a number of improvements to the Jikken test file schema. These include:

    • Support for variable embeddings in setup, cleanup, and multi-stage response bodies.
    • A new name field in stages to specify a display name for each stage in multi-stage tests.
    • A new bodySchema field for defining Advanced JSON Validation for response bodies.

  6. New CLI Commands: We’ve added a few commands to the Jikken CLI tool:

    • The format command will autoformat JKT files similar to tools like go fmt or cargo fmt. This will re-order the definition fields to a deterministic state and remove unnecessary fields.
    • The validate command will provide some basic validation for your test files. It will also allow you to automatically generate platformId ulids for existing tests if they are missing via a CLI flag.

    In the future we hope to expand these commands’ capabilities to be leveraged by our VSCode extension as well as use in our web platform.

Changes

  1. Console Output Improvements: We’ve made several changes to the default CLI output.

    • Test runs now provide each test’s runtime so you can see the performance of your APIs at a glance, as well as the total time for the entire test session execution.
    • Displays a warning when user-specified config file is not found.
    • Displays an error and skips the test if the test definition contains unknown fields.

  2. VSCode Extension:

    • VSCode extension now recognizes the project keyword (as well as all new keywords added in this release).
    • VSCode extension has better support for embedded variables.

  3. Telemetry:

    • Telemetry now includes test stage names.
    • Telemetry now receives skipped test information.
    • Telemetry now receives projects and environments on session creation to support conditional alerts.
    • Telemetry no longer sends global variables attached to test data, but instead sends them as part of the session.

  4. Variable Name Restrictions: There are now some restrictions when naming variables. Variable names must be at least one character long, and may only contain letters, numbers, hyphens, and underscores.

Bugfixes

  • CLI output now properly flushes prior to test completion (for long-running tests).
  • For test reporting, each run of a multi-iteration test now correctly counts as a distinct test.
  • In telemetry, test definitions no longer include global variables, which fixes an issue that could result in inconsistently generated test IDs.

What’s next

We plan to iterate on and improve our new Advanced JSON Validation and Variable Data Generators features based on user feedback. We’d also like to deliver parallel test execution soon. Oh, and… Docs, Docs, Docs. We have a bunch of plans around improvements to documentation, guides, and providing more example tests.

<- Previous Post Next Post ->