Reviews and quality

Tests and automation: code, unit tests and automation tests in every story

Choose your project's unit-test and test-automation frameworks, or let Yavaren follow what your repository already uses. Every story comes with tests that prove its acceptance criteria.

Every story Tulaian delivers has three parts: the production code, unit tests for the code it adds or changes, and automation tests that prove the story's acceptance criteria. Automation tests can be UI, API, mobile, performance or BDD tests, whichever fits the change.

Choosing the frameworks

Under Projects › Delivery preferences › Tests and automation, an admin chooses the project's frameworks:

• a unit-test framework for each repository;
• the UI, API, mobile, performance and BDD automation frameworks;
• where automation lives: a folder of each repository (automation by default), or a separate automation repository of the project.

Only frameworks that suit the project's stacks are offered. Leave a kind on Follow what the repository uses to keep what is already there.

For a new product, the new-product wizard asks the same question on its Stacks step.

An existing repository

Tulaian first looks at what your repository already uses, from its configuration files and dependencies: Jest, Vitest, pytest, JUnit, xUnit, NUnit, RSpec, Playwright, Cypress, Selenium, WebdriverIO, Appium, REST Assured, Karate, Postman/Newman, k6, Cucumber and many more. It then follows the repository's folders, file naming, page objects and fixtures.

What is already there wins. If you chose Playwright but the repository uses Cypress, Tulaian writes Cypress tests and tells you so. To add Playwright alongside Cypress, tick Add my choice alongside. Tulaian then sets Playwright up in its own folder and leaves the Cypress tests exactly as they are.

The project page shows what each repository was found to use, and what a story there will use.

What you see on each story

The draft pull request has a Tests and automation section. Each acceptance criterion is listed with its unit tests and its automation tests. If a criterion has no automation test, the section says why. The section also says what actually ran.

Automation tests run headless on the runner when it has what they need: a browser, a device, or the running application. When it does not, Tulaian only lists, compiles or dry-runs them, and says plainly that they were not run, and why. Tests are never reported as run when they were not.

An acceptance criterion with no automation test, or automation tests that fail, goes back to Tulaian to fix. What is still open is listed in the pull request's quality scorecard. The same view is under Recent work › Details › Tests and automation.

Functional validation: which criteria are proven

A criterion is verified only when every test linked to it ran in the story's job and passed. A test that was only written, a test run that passed as a whole without a result per test, or tests that were only listed are shown as such: Tests written, Suite passed or Not run. They are never shown as verified.

Under Projects › Delivery preferences › Functional validation you see, for each work item, every acceptance criterion with its tests, the last result of each test, its status, the pull request and the job. You also see how many criteria are verified. This is functional coverage, not code coverage: Yavaren does not measure code coverage.

A change that removes assertions, skips tests or deletes test files is sent back to Tulaian to fix. If it is not fixed, the pull request says the definition of done is not met.

Waivers. An admin can waive one criterion of one work item, for example one that is checked by hand. A reason is required, and an end date is optional. The criterion then shows Waived, with who waived it, when and why. It never shows as verified. A waiver changes nothing else: the story, its checks and its approvals stay as they are. An admin can revoke it at any time, and it stops applying after its end date.

Test-only stories

Label a story work_type:test to have Tulaian write automation tests for an application that already exists, for example to work through an automation backlog. Tulaian does not change production code in such a story.

When automation lives in a separate repository, each story's pull request carries the code and unit tests. A test-only job then writes that story's automation tests in the automation repository.