Return code if tests fail

Is it implemented or designed to throw another return code then 0 in shell when tests failed?

Why?
Because it would be nice to have failed stages in CI system when tests failed. We haven’t to implement for content checks for *.xml for some content snippets …

Thanks.

Hi!

When you say “if tests fail”, do you mean test scenarios that are run

  1. multiple times as a load test via XLT Master Controller, or
  2. a single time as a functional test via Maven/Gradle/Ant/other build tool?

Thanks!

For example if we ran multiple tests with gradle / testsuite - gradle should fail with returncode <>0.
With this, it is possible to identify a problem within a pipeline stage.

Stage / Result

  • build → successful
  • test → failure
    with
    ./gradlew test -PfilterTests=“org.xceptance.xlt.tests.Loadtests” -PfilterBrowsers=chrome

Result: Pipeline will fail

I’m not familiar with Gradle, but according to their docs, it should already be working the way you want it. See test task properties ignoreFailures and maybe also failFast.

However, it looks as if Gradle does not distinguish between test failures and other errors. It always return either 0 for success or 1 for any issue. See this still open issue: https://github.com/gradle/gradle/issues/12374

Thanks.
We implemented a dirty way and this will work, not nice …

./gradlew test -PfilterTests=“org.xceptance.xlt.tests.Loadtests” -PfilterBrowsers=chrome
grep -L "<failure" build/test-results/*/*.xml