Exclude test cases from test run


#1

Imagine you have a complete test suite with scenarios for browsing, searching, account creation, and ordering, and maybe more test cases. You might come into a situation when it’s necessary to run the scripts of this test suite against a LIVE site.

In that case we must not have any transactional scenario included (such as TRegister, TCheckout, TOrder …) as it would spoil the database with test data, may cause additional costs if external services are involved, or even trigger fulfillment. To avoid these problems you need to exclude some test cases from the run by removing them from the test properties file:

Additionally, It’s recommended to add the @Ignore annotation to the corresponding test class, to cover also accidental execution of these test cases. It’s too easy to run the wrong test case or even the complete tests folder from an IDE.


#2

Is it possible to add the @Ignore annotation if a test case is set to disabled via the Script Developer?
Currently, this does not happen. In case I disable a test case via Script Developer but let the test cases run on Jenkins, a warning is displayed that the test case is disabled but no ignore was set.


#3

You are right, the Script Developer updates the underlying XML only. When running the test case (even if it is disabled) with Jenkins the command(s) will not be executed. This is similar to running the @Ignore-annotated test from within an IDE. In both cases you are protected from accidental execution of code.

Anyway, disabled test cases are not marked as “Ignored” in Jenkins. We tracked a corresponding improvement request, so there might be an @Ignore feature in a future release.