3

I have problems with the flakiness of some of my UI tests. I want to be able to run one particular test multiple times in multiple simulators. The idea is to apply this solution for localhost and then for CI servers.

A perfect solution would be if there is any option that I can mark some test in my PR as flaky and then during xcodebuild test run them without explicitly specify their name.

I know that there is an option to run tests in parallel but I want to run one test multiple times on copies of a simulator, not distribute tests between them. It should be simpler, I guess.

I found Can I run an XCTest suite multiple times? topic where described is how to use measureBlock to run continuous tests, but I'm not sure how to specify the number of runs. My background is C# where I have access just to Repeat(x) and case Category which I can specify during a test run. I'm looking for something similar.

Mark test category as Flaky-> apply attribute Retry(10) -> use command tool to run all tests with Flaky category

Looking for some idea what can I do to implement this kind of solution for my iOS tests.

Emil
  • 171
  • 1
  • 8

1 Answers1

0

The short answer to the question of re-running flaky UI tests is no, XCTest & xcodebuild do not support this option. So it is not possible natively.

The longer answer is - yes, it is possible, just not easy to implement.

You will need:

  1. a CI environment, which will run the xcodebuild command with your full set of tests
  2. python script to get failing tests out of your tests results (or other such language)
  3. administration access to your CI runner (or R/W access at least)

How to do it?

Your tests will run (and some will fail in the process) in the first xcodebuild test.... You will let these finish and you will parse (with a script, which was written by you) those tests, that failed. You can get this info from e.g. StandardOutputAndError.txt file (how do you find it?), thats why you'll need full access to your CI runner.

After your script parses failing tests from the results file, you can construct new xcodebuild command withing that script, which would run only the tests, that were parsed, because they failed. The new xcodebuild would use only-testing: flags (yes, you can have multiple in one xcodebuild) to specify the test, it should run (how to use these flags?).

After this, your new, constructed xcodebuild command would be returned to the CI environment and executed there. I was dealing with something similar (constructing xcodebuild within python script to only run tests according to changes in iOS project and executing it in CI) and my question should help you with the execution and CI environment.

To sum this: no, there is not native support of this but yes it can be done, if you are brave enough :-)

Hope, this helped.

E: oh and no - you cannot (and should not) use measureBlock for this. These are meant for performance tests and have no such functionality as rerunning failing tests.

Václav
  • 990
  • 1
  • 12
  • 28
  • Thanks for answer. Actually I'm looking for best option to multirun one test. I know test name. I just need to run it multiple times on CI. Should I just run `xcodebuild` multiple times in loop or there is other option? – Emil Jul 05 '19 at 04:03
  • Hi @Emil, sorry for the late answer - no there is no other option how to multirun one test, other than looping the `xcodebuild` in the CI environment. If you were to do so, I would recommend writing a script in e.g. Python/Shell with input parameter being the test name (names) and multiruning it from within the script, since having loops directly in the CI `script:` (Gitlab) is pain in the *** to maintain and set. – Václav Jul 07 '19 at 09:44