How Meeshkan works

Testing is the core of what Meeshkan does and it's important that we're transparent about how it all works. This section covers some of the Meeshkan fundamentals.

In this section:

  • Running the tests
  • Testing frequency
  • Reporting bugs and issues
  • Suggested fixes

Running the tests#

Meeshkan uses an OpenAPI specification to understand your project's expected inputs and outputs. This OpenAPI specification provides a framework for Meeshkan to generate test properties. From there, Meeshkan runs property-based tests on your project.

If you don't already have an OpenAPI specification, we can generate one for you with our HTTP Mocking Toolkit (HMT). HMT collects recordings of server traffic and generates a JSON Lines file based on the logs. It then builds an OpenAPI schema from that file.

Testing frequency#

During the alpha release (which we're in now), there are two test schedules:

  1. Your project is out of the box compatible. Using the checks API, Meeshkan will automatically run on every commit where the app is installed.
  2. Your project needs some manually configuration by the Meeshkan team to build and run. Meeshkan will test all registered repositories on a weekly basis. We decided on weekly because property-based testing is very resource-intensive. Not to mention that the volume of bugs discovered in a single run can be significant.

We'll soon be introducing paid plans with options to test repositories more often. If you're interested in this, let us know.

Reporting bugs and issues#

We're currently experimenting with the most effective way to report bugs. By default, checks happen on a per commit basis using the GitHub Checks API.

Branch checks on GitHub

Other options we're exploring include email digests, Slack notifications, and filing an issue on the registered repository.

Suggested fixes#

In some cases, Meeshkan is able to suggest a fix for the reported bug. This is a sparse feature at the moment, but it's being actively developed. Please note that, in any case, these are suggestions and not a certain solution to the problem.

Here's an example of what they could look like (anonymized for privacy). The suggestion is the last sentence of the issue description:

A screenshot of an example issue filed by the Meeshkan GitHub bot. At the very bottom it says, 'To resolve this issue, we recommend URL encoding all paths and queries before making HTTP calls'.

Last updated: 25th May 2020