diff --git a/building/tooling/README.md b/building/tooling/README.md index dcbc5e53..245a626d 100644 --- a/building/tooling/README.md +++ b/building/tooling/README.md @@ -46,5 +46,5 @@ Track tooling is usually (mostly) written in the track's language. ```exercism/caution While you're free to use additional languages, each additional language will make it harder to find people that can maintain or contribute to the track. -We recommend using the track's language where possible, only using additional languages when it cannot be avoided. +We recommend using the track's language where possible, because it makes maintaining or contributing easier. ``` diff --git a/building/tooling/test-generators.md b/building/tooling/test-generators.md index 9b4066ea..b66c289c 100644 --- a/building/tooling/test-generators.md +++ b/building/tooling/test-generators.md @@ -1,134 +1,308 @@ # Test Generators -A Test Generator is a piece of software that creates a practice exercise's tests from the common [problem specifications](https://github.com/exercism/problem-specifications). -Some tracks also create tests for concept exercises from a similar track-owned data source. +A Test Generator is a track-specific piece of software to automatically generate a practice exercise's tests. +It does this by converting the exercise's JSON test cases to tests in the track's language. -A Test Generator give us these advantages: +## Benefits -1. They allow adding exercises more quickly without writing much boilerplate code. -2. Contributors can focus on the **design** of an exercise immediately. -3. Along the track life, automatic updates of existing tests can lower maintainer workload. +Some benefits of having a Test Generator are: -## Contributing to Test Generators +1. Exercises can be added faster +2. Automates "boring" parts of adding an exercise +3. Easy to sync tests with latest canonical data -Each language may have its own Test Generator, written in that language. -It adds code and sometimes files to what [`configlet`](/docs/building/configlet) created / updated. -The code usually is rendered from template files, written for the tracks preferred templating engine. -You should find all the details in the tracks contribution docs or a `README` near the test generator. +## Use cases -You should also know: +In general, one runs a Test Generator to either: -- what [`configlet create`](/docs/building/configlet/create) or [`configlet sync`](/docs/building/configlet/sync) do. -- what [`canonical-data.json` in problem specifications](https://github.com/exercism/problem-specifications?tab=readme-ov-file#test-data-canonical-datajson) may provide. -- why ["creating from scratch" is different from "reproducing for updates"](#from-scratch-vs-updating). +1. Generate the tests for a _new_ exercise +2. Update the tests of an _existing_ exercise -## Creating a Test Generator from scratch +### Generate tests for new exercise -There are various test generators in Exercism's tracks. -These guidelines are based on the experiences of these tracks. +Adding a Test Generator for a new exercise allows one to generate its tests file(s). +Provided the Test Generator itself has already been implemented, generating the tests for the new exercise will be (far) less work than writing them from scratch. -Even so test generators work very similar, they are very track specific. -It starts with the choice of the templating engine and ends with additional things they do for each track. -So a common test generator was not and will not be written. +### Update tests of existing exercise -There were helpful discussions [around the Rust](https://forum.exercism.org/t/advice-for-writing-a-test-generator/7178) and the [JavaScript](https://forum.exercism.org/t/test-generators-for-tracks/10615) test generators. -The [forum](https://forum.exercism.org/c/exercism/building-exercism/125) also is the best place for seeking additional advice. +Once an exercise has a Test Generator, you can re-run it to update/sync the exercise with its latest canonical data. +We recommend doing this periodically, to check if there are problematic test cases that need to be updated or new tests you might want to include. -### Things to know +## Starting point -- `configlet` cache with a local copy of the problem specifications is stored in a [location depending on the users system](https://nim-lang.org/docs/osappdirs.html#getCacheDir). - Use `configlet info -o -v d | head -1 | cut -d " " -f 5` to get the location. - Or fetch data from the problem specifications repository directly (`https://raw.githubusercontent.com/exercism/problem-specifications/main/exercises/{{exercise-slug}}/canonical-data.json`) -- [`canonical-data.json` data structure](https://github.com/exercism/problem-specifications?tab=readme-ov-file#test-data-canonical-datajson) is well documented. There is optional nesting of `cases` arrays in `cases` mixed with actual test cases. -- The contents of `input` and `expected` test case keys of `canonical-data.json` vary largely. These can include simple scalar values, lambdas in pseudo code, lists of operations to perform on the students code and any other kind of input or result one can imagine. +There are two possible starting points when implementing a Test Generator for an exercise: -### From Scratch vs. Updating +1. The exercise is new and doesn't yet have any tests +2. The exercise already exists and has existing tests -There are 2 common tasks a test generator may do, that require separate approaches: +```exercism/caution +If there are existing tests, implement the Test Generator such that the tests it generates do not break existing solutions. +``` + +## Design + +Broadly speaking, test files are generated using either: + +- Code: the tests files are (mostly) generated via code +- Templates: the tests files are (mostly) generated using templates + +We've found that the code-based approach will lead to fairly complex Test Generator code, whereas the template-based approach is simpler. + +What we recommend is the following flow: + +1. Read the exercise's canonical data +2. Exclude the test cases that are marked as `include = false` in the exercise's `tests.toml` file +3. Convert the exercise's canonical data into a format that can be used in a template +4. Pass the exercise's canonical data to an exercise-specific template -- [Creating tests from scratch](#creating-tests-from-scratch) -- [Reproducing tests for updates](#reproducing-tests-for-updates) +The key benefit of this setup is that each exercise has its own template, which: -The reason for this distinction is "designing the exercise" vs. "production-ready code". +- Makes it obvious how the test files are generated +- Makes them easier to debug +- Makes it safe to edit them without risking breaking another exercise -When creating tests from scratch the test generator should provide all the information contained in `canonical-data.json` in the resulting files. -This enables contributors to simply open up the generated test file(s) and find all relevant information interwoven with the tracks boilerplate code. -They then design the exercise's tests and student facing code based on these files rather than on the original `canonical-data.json`. -As there is no knowledge of exercise specific things, yet, a one-fits-all template targeting the boilerplate code can be used. +```exercism/caution +Some additional things to be aware of when designing the test generator -When the exercise is already in production, changes in `canonical-data.json` are rarely a reason to change the design of the exercise. -So reproducing tests for updates is based on the existing design and should result in production-ready code. -Much of the additional data presented when creating the exercise from scratch is no longer part of the result. +- Minimize the pre-processing of canonical data inside the Test Generator +- Try to reduce coupling between templates +``` -Instead, very often additional conversion of test case data is required, which is specific to this exercise. -Most tracks opt for having at least one template per exercise for this. -This way they can represent all the design choices in that template without complicating things too much for further contribution. +## Implementation -### Creating tests from scratch +The Test Generator is usually (mostly) written in the track's language. -This is more productive in the beginning of a tracks life. -It is way more easy to implement than the "updating" part. +```exercism/caution +While you're free to use additional languages, each additional language will make it harder to find people that can maintain or contribute to the track. +We recommend using the track's language where possible, only using additional languages when it makes maintaining or contributing easier. +``` -Doing only the bare minimum required for a first usable test generator may already help contributors a lot: +### Canonical data -- Read the `canonical-data.json` of the exercise from `configlet` cache or retrieve it from GitHub directly -- Preserve all data (including `comments`, `description` and `scenarios`) -- If the tracks testing framework supports no nested test case groups, flatten the nested data structure into a list of test cases -- Dump the test cases into the one-fits-all boilerplate template(s) - - Preserve the test case grouping for nested test case groups, e.g. - - using the test frameworks grouping capability - - using comments and code folding markers (`{{{`, `}}}`) - - concatenating group `description` and test case `description` - - Show all data (including `comments`, `description` and `scenarios`) +The core data the Test Generator works with is an exercise's [`canonical-data.json` file](https://github.com/exercism/problem-specifications?tab=readme-ov-file#test-data-canonical-datajson). +This file is defined in the [exercism/problem-specifications repo](https://github.com/exercism/problem-specifications), which defines shared metadata for many Exercism's exercises. ```exercism/note -Don't try to produce perfect production-ready code! -Dump all data and let the contributor design the exercise from that. -There is way too much variation in the exercises to handle all in one template. +Not all exercises have a `canonical-data.json` file. +In case they don't, you'll need to manually create the tests, as there is no data for the Test Generator to work with. +``` + +#### Structure + +Canonical data is defined in a JSON object. +This object contains a `"cases"` field which contains the test cases. +These test cases (normally) correspond one-to-one to tests in your track. + +Each test case has a couple of properties, with the description, property, input value(s) and expected value being the most important ones. +Here is a (partial) example of the [canonical-data.json file of the leap exercise](https://github.com/exercism/problem-specifications/blob/main/exercises/leap/canonical-data.json): + +```json +{ + "exercise": "leap", + "cases": [ + { + "uuid": "6466b30d-519c-438e-935d-388224ab5223", + "description": "year not divisible by 4 in common year", + "property": "leapYear", + "input": { + "year": 2015 + }, + "expected": false + }, + { + "uuid": "4fe9b84c-8e65-489e-970b-856d60b8b78e", + "description": "year divisible by 4, not divisible by 100 in leap year", + "property": "leapYear", + "input": { + "year": 1996 + }, + "expected": true + } + ] +} ``` -There are optional things a test generator might do: +The Test Generator's main responsibility is to transform this JSON data into track-specific tests. +Here's how the above JSON could translate into Nim test code: -- Provide code for a simple test case (e.g. call a function with `input`, compare result to `expected`) -- Provide boilerplate code for student code file(s) or additional files required by the track -- Respect `scenarios` for grouping / test case selection -- Skip over "reimplemented" test cases (those referred to in a `reimplements` key of another test case) -- Update `tests.toml` with `include=false` to reflect tests skipped by `scenarios` / `reimplements` +```nim +import unittest +import leap + +suite "Leap": + test "year not divisible by 4 in common year": + check isLeapYear(2015) == false + + test "year divisible by 4, not divisible by 100 in leap year": + check isLeapYear(1996) == true +``` -### Reproducing tests for updates +The structure of the `canonical-data.json` file is [well documented](https://github.com/exercism/problem-specifications?tab=readme-ov-file#test-data-canonical-datajson) and it also has a [JSON schema](https://github.com/exercism/problem-specifications/blob/main/canonical-data.schema.json) definition. -This may become more relevant over track life time. -It is much harder to implement than the "from scratch" part. -If you need to invest much effort here, maybe manual maintenance is more efficient. -Also keep in mind: maintaining the test generator adds to the maintainers workload, too. +##### Nesting + +Some exercises use nesting in their canonical data. +This means that each element in a `cases` array can be either: + +1. A regular test case (no child test cases) +2. A grouping of test cases (one or more child test cases) ```exercism/note -Choose a flexible and extensible templating engine! -The test cases vary largely between exercises. -They include simple scalar values, lambdas in pseudo code, lists of operations to perform on the students code and any other kind of input or result one can imagine. +You can identify the types of an element by checking for the presence of fields that are exclusive to one type of element. +Probably the best way to do this is using the `"cases"` key, which is only present in test case groups. +``` + +Here is an example of nested test cases: + +```json +{ + "cases": [ + { + "uuid": "e9c93a78-c536-4750-a336-94583d23fafa", + "description": "data is retained", + "property": "data", + "input": { + "treeData": ["4"] + }, + "expected": { + "data": "4", + "left": null, + "right": null + } + }, + { + "description": "insert data at proper node", + "cases": [ + { + "uuid": "7a95c9e8-69f6-476a-b0c4-4170cb3f7c91", + "description": "smaller number at left node", + "property": "data", + "input": { + "treeData": ["4", "2"] + }, + "expected": { + "data": "4", + "left": { + "data": "2", + "left": null, + "right": null + }, + "right": null + } + } + ] + } + ] +} ``` -Doing the bare minimum required for a usable updating test generator includes: +```exercism/caution +If your track does not support grouping tests, you'll need to: + +- Traverse/flatten the `cases` hierarchy to end up with only the innermost (leaf) test cases +- Combine the test case description with its parent description(s) to create a unique test name +```` + +#### Input and expected values + +The contents of the `input` and `expected` test case keys vary widely. +In most cases, they'll be scalar values (like numbers, booleans or strings) or simple objects. +However, occasionally you'll also find more complex values that will likely require a bit of preprocessing, such as lambdas in pseudo code, lists of operations to perform on the students code and more. + +#### Scenarios + +Test cases have an optional `scenarios` field. +This field can be used by the test generator to special case certain test cases. +The most common use case is to ignore certain types of tests, for example tests with the `"unicode"` scenario as your track's language might not support Unicode. + +The full list of scenarios can be found [here](https://github.com/exercism/problem-specifications/blob/main/SCENARIOS.txt). + +#### Reading canonical-data.json files + +There are a couple of options to read the `canonical-data.json` files: + +1. Fetch them directly from the `problem-specifications` repository (e.g. `https://raw.githubusercontent.com/exercism/problem-specifications/main/exercises/leap/canonical-data.json`). +2. Add the `problem-specifications` repo as a Git submodule to the track repo. +3. Read them from the `configlet` cache. + The [location depends on the user's system](https://nim-lang.org/docs/osappdirs.html#getCacheDir), but you can use `configlet info -o -v d | head -1 | cut -d " " -f 5` to programmatically get the location. + +#### Track-specific test cases + +If your track would like to add some additional, track-specific test cases (which are not found in the canonical data), one option is to creating an `additional-test-cases.json` file, which the Test Generator can then merge with the `canonical-data.json` file before passing it to the template for rendering. + +### Templates + +The template engine to use will likely be track-specific. +Ideally, you'll want your templates to be as straightforward as possible, so don't worry about code duplication and such. + +The templates themselves will get their data from the Test Generator on which they iterate over to render them. + +```exercism/note +To help keep the templates simple, it might be useful to do a little pre-processing on the Test Generator side or else define some "filters" or whatever extension mechanism your templates allow for. +``` + +### Using configlet + +`configlet` is the primary track maintenance tool and can be used to: + +- Create the exercise files for a new exercise: run `bin/configlet create --practice-exercise ` +- Sync the `tests.toml` file of an existing exercise: run `bin/configlet sync --tests --update --exercise ` +- Fetch the exercise's canonical data to disk (this is a side-effect of either of the above commands) + +This makes `configlet` a great tool to use in combination with the Test Generator for some really powerful workflows. + +### Command-line interface + +You'll want to make using the Test Generator both easy _and_ powerful. +For that, we recommend creating one or more script files. + +```exercism/note +You're free to choose whatever script file format fits your track best. +Shell scripts and PowerShell scripts are common options that can both work well. +``` + +Here is an example of a shell script that combines `configlet` and a Test Generator to quickly scaffold a new exercise: + +```shell +bin/fetch-configlet +bin/configlet create --practice-exercise +path/to/test-generator +``` + +## Building from scratch + +Before you start building a Test Generator, we suggest you look at a couple of existing Test Generators to get a feel for how other tracks have implemented them: + +- [C#](https://github.com/exercism/csharp/blob/main/docs/GENERATORS.md) +- [Clojure](https://github.com/exercism/clojure/blob/main/generator.clj) +- [Common Lisp](https://github.com/exercism/common-lisp/blob/main/bin/lisp_exercise_generator.py) +- [Crystal](https://github.com/exercism/crystal/tree/main/test-generator) +- [Emacs Lisp](https://github.com/exercism/emacs-lisp/blob/main/tools/practice-exercise-generator.el) +- [F#](https://github.com/exercism/fsharp/blob/main/docs/GENERATORS.md) +- [Perl 5](https://github.com/exercism/perl5/tree/main/t/generator) +- [Pharo Smalltalk](https://github.com/exercism/pharo-smalltalk/blob/main/dev/src/ExercismDev/ExercismGenerator.class.st) +- [Python](https://github.com/exercism/python/blob/main/docs/GENERATOR.md) +- [Rust](https://github.com/exercism/rust/blob/main/docs/CONTRIBUTING.md#creating-a-new-exercise) +- [Swift](https://github.com/exercism/swift/tree/main/generator) + +If you have any questions, the [forum](https://forum.exercism.org/c/exercism/building-exercism/125) is the best place to ask them. +The forum discussions [around the Rust](https://forum.exercism.org/t/advice-for-writing-a-test-generator/7178) and the [JavaScript](https://forum.exercism.org/t/test-generators-for-tracks/10615) test generators might be helpful too. + +### Minimum Viable Product + +We recommend incrementally building the Test Generator, starting with a Minimal Viable Product. +A bare minimum version would read an exercise's `canonical-data.json` and just pass that data to the template. -- Read the `canonical-data.json` of the exercise from `configlet` cache or retrieve it from GitHub directly -- If the tracks testing framework supports no nested test case groups, flatten the nested data structure into a list of test cases -- Render the test cases into the exercise specific template(s) located in an exercise's `.meta/` folder - - Render production-ready code that matches the manually designed exercise - - Skip over "reimplemented" test cases (those referred to in a `reimplements` key of another test case) - - Render only test cases selected by `tests.toml` (or another track-specific data source) +Start by focusing on a single exercise, preferrably a simple one like `leap`. +Only when you have that working should you gradually add more exercises. -There are different strategies for respecting test case changes like "replace always", "replace when forced to", "use `tests.toml` to ignore replaced test cases" (works like a baseline for known test issues). -None of them is perfect. +And try to keep the Test Generator as simple as it can be. ```exercism/note -Don't try to have a versatile one-fits-all template! -There is way too much variation in the exercises to handle all in one template. +Ideally, a contributor could just paste/modify an existing template without having to understand how the Test Generator works internally. ``` -There are optional things a test generator might do: +## Using or contributing -- Provide a library of templates and / or extensions to the template engine -- Maintain or respect another track-specific data source than `tests.toml` -- Maintain student code file(s) or additional files required by the track -- Handle `scenarios` for grouping / test case selection -- Have a check functionality (e.g. to run after `configlet sync`) to detect when updating is required +How to use or contribute to a Test Generator is track-specific. +Look for instructions in the track's `README.md`, `CONTRIBUTING.md` or the Test Generator code's directory. diff --git a/building/tracks/new/implement-tooling.md b/building/tracks/new/implement-tooling.md index 8977b06c..bab2f645 100644 --- a/building/tracks/new/implement-tooling.md +++ b/building/tracks/new/implement-tooling.md @@ -52,7 +52,7 @@ Track tooling is usually (mostly) written in the track's language. ```exercism/caution While you're free to use additional languages, each additional language will make it harder to find people that can maintain or contribute to the track. -We recommend using the track's language where possible, only using additional languages when it cannot be avoided. +We recommend using the track's language where possible, because it makes maintaining or contributing easier. ``` ## Deployment