♻️ use playwright for e2e

[thomas-lebeau]

Motivation

Improve reliability and speed of our e2e tests.

Changes

We now uses Playwright as a e2e test runner. There is small changes to the API but things remains very similar.

Speed 🚀

	Local	CI	BrowserStack*
Before	~3m50	~8m	~24m
After	~21s	~3m75s	~10m **

(*) BrowserStack timing does not account for the time spend waiting for BS resources.
(**): Temporary estimation

Playwright already allow e2e to be faster, however there is a few other optimizations:

• unit-bs and e2e-bs jobs run as soon as unit and e2e jobs are finished respectively. This is so we don't wait for the slower test-performance job.
• e2e-bs job will retry each test up to two time to be more resilient for flakiness, however, the job will stop as soon as a single test fails in order to free up browserstack resources.

new useful commands:

• yarn test:e2e --ui run playwright locally in UI mode
• yarn test:e2e test/e2e/scenario/rum/init.scenario.ts to run one test file
• yarn test:e2e recorder to run all test files that have recorder in the name
• yarn test:e2e -g unhandled rejections to grep tests by name
• yarn test:e2e --debug to playwright in debug mode
• yarn test:e2e --project firefox to run tests in firefox. available projects: chromium (default), firefox, webkit, android, and * (all)
• yarn test:e2e --repeat-each 3 to run each test 3 times, useful for catching flaky tests
• BS_USERNAME=johdoe_abcD BS_ACCESS_KEY=askldjhfn yarn test:e2e:bs to run the tests in Browserstack

Important

yarn test:e2e does not build the SDK automatically, run yarn build:bundle if you have made change to the source code of the SDK.

new useful method:

• expect([1, 2, 3]).toHaveLength(3), will show a better error message when failing, including the actual array content.
• await page.pause() to use with --debug

Testing

Locally, run yarn test:e2e --ui

• Local
• Staging
• Unit
• End to end

---

I have gone over the contributing documentation.

[cit-pr-commenter]

Bundles Sizes Evolution

📦 Bundle Name	Base Size	Local Size	𝚫	𝚫%	Status
Rum	146.77 KiB	146.77 KiB	0 B	0.00%	✅
Logs	51.19 KiB	51.19 KiB	0 B	0.00%	✅
Rum Slim	105.52 KiB	105.52 KiB	0 B	0.00%	✅
Worker	24.50 KiB	24.50 KiB	0 B	0.00%	✅

🚀 CPU Performance

Action Name	Base Average Cpu Time (ms)	Local Average Cpu Time (ms)	𝚫
addglobalcontext	0.003	0.002	-0.001
addaction	0.069	0.021	-0.049
addtiming	0.002	0.005	0.003
adderror	0.114	0.020	-0.094
startstopsessionreplayrecording	0.011	0.001	-0.010
startview	0.513	0.005	-0.508
logmessage	0.023	0.019	-0.004

🧠 Memory Performance

Action Name	Base Consumption Memory (bytes)	Local Consumption Memory (bytes)	𝚫 (bytes)
addglobalcontext	29.62 KiB	27.40 KiB	-2273 B
addaction	60.72 KiB	57.20 KiB	-3608 B
addtiming	29.52 KiB	25.77 KiB	-3836 B
adderror	65.76 KiB	57.73 KiB	-8222 B
startstopsessionreplayrecording	28.91 KiB	26.34 KiB	-2624 B
startview	416.26 KiB	419.72 KiB	3.46 KiB
logmessage	63.72 KiB	59.71 KiB	-4101 B

🔗 RealWorld

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 93.56%. Comparing base (4f7e0e4) to head (c16ff1a).

Files with missing lines	Patch %	Lines
packages/rum/test/mutationPayloadValidator.ts	25.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3159      +/-   ##
==========================================
- Coverage   93.61%   93.56%   -0.05%     
==========================================
  Files         291      291              
  Lines        7691     7694       +3     
  Branches     1751     1754       +3     
==========================================
- Hits         7200     7199       -1     
- Misses        491      495       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomas-lebeau]

Split tearDownTest and tearDownPassedTest so that when a test fails no more expect than the one failing is executed. This is because if the log shows only the last failing expect, hence if the teardown expect fails, it will show only this log which make debugging failing test difficult.

[thomas-lebeau]

This add the property @test.fixme in CI visibility, this is very useful to create dashboards