This test fails very often in CI. This should hopefully improve it.
With a downclocked CPU I've got 2 errors in 10 tries. After adding
```
await page.waitForLoadState();
```
I got no errors in 20 tries. But my testing methodology is unreliable so there's no guarantee that this helps:
* `Locator.click()` documentation says that it already waits by default for caused navigation to finish
* I added some `date` time measurement and `waitForLoadState` was clocking in about 5-15 ms, which shouldn't be causing the issue for `expect` that times out in 3000ms
...but e2e issues were `element(s) not found` on timeout, not wrong state, so my assumption is that somehow it is asserting the wrong snapshot of a page that isn't fully loaded
Additionally, I've converted _revertion_ logic to _make consistent before testing_, which is better at handling test retries in case that the _revertion_ part of the code was never ran in previous failure. In such case test retry is guaranteed to fail as well, at least when testing locally, not sure about CI.
Additionally, a completely unrelated comment fix in Playwright config.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/12871
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Webkit and Mobile safari are comically unreliable, will fail for unexplainable reasons and are very hard to run locally in comparison with the other supported platforms. I do not remember the last time where these two platforms were able to catch a regression where the other platforms did not.
I would like to stress, for the historical record, that many hours has been devoted into adjusting the tests and following best practices to make these two platforms more stable but despite those, IMO wasted, efforts these two platforms are causing many hours of wasted CPU time simply because they are flaky and make (new) contributors nervous if their change contains a regression or not.
To my knowledge, the tests are not broken for these two platforms. If you go to the issue tracker you will not find issues by users that use these two platforms and report that Forgejo is broken. It does not reflect reality.
This is the sunk cost fallacy, bite the bullet and agree that these platforms will not contribute positively to Forgejo's excellent test suite.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10103
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
Goals:
- speedup
- less flakiness
- best practices and more use
- documentation
config:
- sync ports in Makefile and playwright config
(otherwise, some tests fail locally because they assert the full URL including the (wrong) port)
- even more generous timeouts
- limit workers to one again (because I finally understand how
Playwright works)
- allow nested functions to group them together with the related test
all:
- deprecate waitForLoadState('networkidle')
- it is discouraged as per https://playwright.dev/docs/api/class-page#page-wait-for-load-state
- I could not find a usage that seems to require it actually (see
added documentation in README)
- adding an exception should be made explicitly
- it does not do what you might expect anyway in most cases
- only log in when necessary
webauthn:
- verify that login is possible after disabling key
- otherwise, the cleanup was not necessary after the previous refactor to create a fresh user each
issue-sidebar / WIP toggle:
- split into smaller chunks
- restore original state first
- add missed assertion to fix race condition (not waiting
before state was reached)
- explicitly toggle the state to detect mismatch earlier
issue-sidebar / labels:
- restore original state first
- better waiting for background request
