Sharding/parallel execution improvements

[bartekpacia]

Support for parallel execution has been added in PR #1732, but I see space for more improvement.

Some ideas:

• Add possibility to test in parallel without sharding (run the whole test on every device) #1844
  Run in parallel without sharding. In other words, run the whole test on every device.

• Allow for some kind of selection what tests can run on what platform.

  Related: Pass different, platform-specific appIds in a single flow #1472
  If the flow didn't have a key for one of the platforms, it would be skipped on that platform.

• Improve parallel/sharded CLI output Improve parallel/sharded CLI output #1862

  Better CLI output, to allow understanding test status at glance.

[bartekpacia]

Maybe it would also be cool if maestro record supported many devices at once?

[bartekpacia]

Request from a user on Slack:

What's the expected workflow to get a single build onto the generated simulators (iOS)?

Right now it has to be done manually with adb install <apk_path>/xcrun simctl install booted <app_path>, but it's obviously not convenient enough.

Maybe Maestro should have a separate install command, and have list of paths to app binaries to install in .maestro/config.yaml? Open for discussion.

[tokou]

Request from a user on Slack:

What's the expected workflow to get a single build onto the generated simulators (iOS)?

Right now it has to be done manually with adb install <apk_path>/xcrun simctl install booted <app_path>, but it's obviously not convenient enough.

Maybe Maestro should have a separate install command, and have list of paths to app binaries to install in .maestro/config.yaml? Open for discussion.

Linked #1665 #1854

[dsyrstad]

When running tests with --shard-split, Maestro tries to evenly split tests between devices. However, there are often tests that run shorter or longer than others. What we end up with is a device that finishes way ahead of the others, and another device that's trying to finish its allocation of tests while the other sit idle.

My suggestion is to have a "shard queue" where all of the tests go into a single queue. When a device (shard) is idle, it pulls the next test from the queue. Perhaps this could be a --shared-queue option if you don't want to change the behavior of --shard-split, but I think this would just be an improvement to --shard-split.