(TL;DR) Remaining issues

🚨Fixture upload failing (?) on Jenkins for the jupiter test (MGNLTEST-5)

• see a recording of PagesCoreFunctionalTests.editComponent(PageObjects) and notice that lorem-ipsum fixture node is not there
• https://jenkins.magnolia-cms.com/job/platform/job/ce/job/master/293/
• https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html (the most promising so far! 💪)

🚨Conflict between Resources UI and non-UI tests partially remains (MGNLTEST-80)

• ResourcesAppUITest#deletingAndEditingFile touches file-should-originate-from-jcr fixture, which is sensitive to the ResourcesIntegrationTest#resourceLoadOrderIsRespected
  • VERY strange thing: ResourcesAppUITest#deletingAndEditingFile is actually quarantined and shouldn't be executed (wth?)

🚨DX core effort is partially blocked by the consistently failing test

• see e.g. here: https://jenkins.magnolia-cms.com/job/platform/job/dx-core/job/PR-101/53/#showFailuresLink
  • WorkflowUITest and CookieAppUITest failures seem to related to problems with refreshing the table component upon changes (i.e. refresh status of a node is not properly reflected for WF test, a cookie node is not disappearing in cookie app test)
    • MGNLEE-620
    • MGNLEE-621
  • License test suffers from a weird problem of a legacy AbstractMgnlUItest#setUp failing to fill the login form after actually entering the license

⚠️RetryOnFailure is not implemented for junit 5 tests

Enable re-running of failing JUnit 5 tests

• only the property is set, but on its own it does nothing

• unlikely to land in stock JUnit5

  Introduce support for retrying failed flaky tests · Issue #1558 · junit-team/junit5

• some inspiration the Jupiter way:

  • @RetryingTest from JUnit Pioneer
  • https://stackoverflow.com/questions/46181026/junit5-how-to-repeat-failed-test
  • https://github.com/artsok/rerunner-jupiter

• some inspiration the Surefire way:

  Rerun Failing Tests with JUnit 5

  Surefire / Failsafe rerun failed tests functionality fails with JUnit 5 if using @DisplayName

Resolved problems

✅Disk space issues

• Jenkins tests run on EC2 machine with 20GB disk space + we attach 30GB disk which acts as a cache for (especially) Maven and Docker storage. Array of such disks is maintained over time and is available to multiple builds.
  • we typically re-link /var/lib/docker to the 'cache' disk in (attach-volume.sh script)
  • with latest issues with Docker daemon not being up during tests the attachment, however, stopped happening consistently (our fault)

✅Test framework artifacts leaking into the webapps (MGNLCE-249)

• test-fw declared in default (compile) scope in CE