[code-infra] Update package layout for better ESM support #43264

Janpot · 2024-08-11T12:44:54Z

addresses #44055
addresses #43980
addresses #44265
addresses #44180
addresses #44993
addresses #45018

Might close:

Add a new mode to the build:

Removes nested package.json files
Stores ESM files in esm subfolder with type:'module' in a package.json
Add exports field to package.json
Modern builds go in modern subfolder

Notes:

type:module and package.json exports are supported from node 12.17 onward

Our ESM bundles don't work in Next.js due to

../../packages/mui-system/build/Box/index.mjs
Error: It's currently unsupported to use "export *" in a client boundary. Please use named exports instead.
    at Object.transformSource (/Users/janpotoms/Projects/material-ui/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]_babel-plugin-_fn2ktr3o4nwwaygw7yjiv6t4d4/node_modules/next/dist/build/webpack/loaders/next-flight-loader/index.js:87:31)

Import trace for requested module:
../../packages/mui-system/build/Box/index.mjs
../../packages/mui-system/build/index.mjs
../../packages/mui-material/build/styles/index.mjs
./src/app/layout.tsx

https://github.com/mui/material-ui/actions/runs/10578363192/job/29308312240?pr=43264#step:7:1960

Another issue that is likely caused by our lack of ESM support [RSC] The entire @mui/joy/node bundle is included in the client bundle when Joy UI component is being imported into a RSC #37934

To Do:

Bundle size increase

The bundle size increase can be fully attributed to the inclusion of the following ESM/CJS interop helper in the webpack output:

Code

/******/ 	/* webpack/runtime/create fake namespace object */
/******/ 	(() => {
/******/ 		var getProto = Object.getPrototypeOf ? (obj) => (Object.getPrototypeOf(obj)) : (obj) => (obj.__proto__);
/******/ 		var leafPrototypes;
/******/ 		// create a fake namespace object
/******/ 		// mode & 1: value is a module id, require it
/******/ 		// mode & 2: merge all properties of value into the ns
/******/ 		// mode & 4: return value when already ns object
/******/ 		// mode & 16: return value when it's Promise-like
/******/ 		// mode & 8|1: behave like require
/******/ 		__webpack_require__.t = function(value, mode) {
/******/ 			if(mode & 1) value = this(value);
/******/ 			if(mode & 8) return value;
/******/ 			if(typeof value === 'object' && value) {
/******/ 				if((mode & 4) && value.__esModule) return value;
/******/ 				if((mode & 16) && typeof value.then === 'function') return value;
/******/ 			}
/******/ 			var ns = Object.create(null);
/******/ 			__webpack_require__.r(ns);
/******/ 			var def = {};
/******/ 			leafPrototypes = leafPrototypes || [null, getProto({}), getProto([]), getProto(getProto)];
/******/ 			for(var current = mode & 2 && value; typeof current == 'object' && !~leafPrototypes.indexOf(current); current = getProto(current)) {
/******/ 				Object.getOwnPropertyNames(current).forEach((key) => (def[key] = () => (value[key])));
/******/ 			}
/******/ 			def['default'] = () => (value);
/******/ 			__webpack_require__.d(ns, def);
/******/ 			return ns;
/******/ 		};
/******/ 	})();

This is caused by the following import

material-ui/packages/mui-system/src/useMediaQuery/useMediaQuery.ts

Line 75 in eab1b9e

    
           const maybeReactUseSyncExternalStore: undefined | any = (React as any)['useSyncExternalStore' + ''];

A potential switch to using use-sync-external-store has been explored in #43476 but ultimately it makes sense to keep it, it manifests in the webpack runtime.

Resolutions

To try out the changes use following resolutions:

"resolutions": {
    "@mui/internal-markdown": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/internal-markdown",
    "@mui/base": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/base",
    "@mui/codemod": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/codemod",
    "@mui/core-downloads-tracker": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/core-downloads-tracker",
    "@mui/docs": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/docs",
    "@mui/icons-material": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/icons-material",
    "@mui/lab": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/lab",
    "@mui/material-nextjs": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/material-nextjs",
    "@mui/material": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/material",
    "@mui/material-pigment-css": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/material-pigment-css",
    "@mui/private-theming": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/private-theming",
    "@mui/styled-engine-sc": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/styled-engine-sc",
    "@mui/styled-engine": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/styled-engine",
    "@mui/styles": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/styles",
    "@mui/system": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/system",
    "@mui/types": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/types",
    "@mui/utils": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/utils",
    "@mui/internal-babel-plugin-resolve-imports": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/internal-babel-plugin-resolve-imports",
    "@mui/internal-docs-utils": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/internal-docs-utils",
    "@mui/internal-scripts": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/internal-scripts",
    "@mui/internal-test-utils": "https://pkg.csb.dev/mui/material-ui/commit/56c2982a/@mui/internal-test-utils",
    "@mui/x-license": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-license",
    "@mui/x-data-grid": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-data-grid",
    "@mui/x-data-grid-pro": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-data-grid-pro",
    "@mui/x-data-grid-premium": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-data-grid-premium",
    "@mui/x-data-grid-generator": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-data-grid-generator",
    "@mui/x-date-pickers": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-date-pickers",
    "@mui/x-date-pickers-pro": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-date-pickers-pro",
    "@mui/x-charts": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-charts",
    "@mui/x-charts-pro": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-charts-pro",
    "@mui/x-charts-vendor": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-charts-vendor",
    "@mui/x-tree-view": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-tree-view",
    "@mui/x-tree-view-pro": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-tree-view-pro",
    "@mui/x-internals": "https://pkg.csb.dev/mui/mui-x/commit/ffb75046/@mui/x-internals"
  },

mui-bot · 2024-08-11T12:47:28Z

Netlify deploy preview

https://deploy-preview-43264--material-ui.netlify.app/

Menu: parsed: +3.73% , gzip: +3.83%
@material-ui/unstyled: parsed: +0.92% , gzip: +1.01%
Select: parsed: +3.16% , gzip: +3.17%
Unstable_Popup: parsed: +3.05% , gzip: +2.63%
Unstable_NumberInput: parsed: +5.11% , gzip: +5.90%
@material-ui/core/useMediaQuery: parsed: +5.09% , gzip: +4.53%
MenuItem: parsed: +6.67% , gzip: +6.83%
Option: parsed: +6.97% , gzip: +7.37%
Tab: parsed: +7.47% , gzip: +7.53%
TabPanel: parsed: +12.28% , gzip: +11.45%
useAutocomplete: parsed: +5.06% , gzip: +5.47%
TablePagination: parsed: +7.54% , gzip: +8.23%
Tooltip: parsed: +0.44% , gzip: +0.48%
SpeedDialAction: parsed: +0.37% , gzip: +0.40%
@mui/joy/Menu: parsed: +0.41% , gzip: +0.50%
TextField: parsed: +0.34% , gzip: +0.31%
@mui/joy/Tooltip: parsed: +0.52% , gzip: +0.51%
RadioGroup: parsed: +0.68% , gzip: +0.73%
@mui/joy/ListSubheader: parsed: +0.77% , gzip: +0.72%
LoadingButton: parsed: +0.51% , gzip: +0.49%
and 27 more changes

Bundle size report

Details of bundle changes (Toolpad)
Details of bundle changes

Generated by 🚫 dangerJS against 6cdfa8c

This reverts commit 6e2bbcc.

DiegoAndai

A couple of questions:

Why create a subfolder for ESM, instead of maintaining the current convention where the subfolder is for CJS and ESM is at the root?
Can (should) we create tests for the package layout/build process? To ensure, for example, that the ATTW check is not broken, or that the package layout is not changed unknowingly.

package.json

packages/mui-base/src/Select/Select.test.tsx

packages/mui-icons-material/package.json

scripts/build.mjs

scripts/copyFilesUtils.mjs

packages/mui-icons-material/scripts/create-typings.mjs

packages/mui-material/package.json

Janpot · 2025-01-24T15:12:50Z

Why create a subfolder for ESM, instead of maintaining the current convention where the subfolder is for CJS and ESM is at the root?

To be able to mark all .js files in esm/ as ESM with a package.json containing type: 'module'. It's interesting to be able to keep the .js extension because that means we can just copy the typings from one folder to the other. Also to keep some parity with base UI.

Can (should) we create tests for the package layout/build process? To ensure, for example, that the ATTW check is not broken, or that the package layout is not changed unknowingly.

I also fixed up the existing bundler tests and added an extra case for the next.js issue we saw.

DiegoAndai · 2025-01-27T19:07:23Z

Thanks for replying to my questions @Janpot. Everything looks good from my side. Feel free to mark this as ready for review whenever you're ready.

Janpot · 2025-01-28T10:32:46Z

@DiegoAndai Only problem left seems to be that the github action that builds @apps/next-app has become a lot heavier on macos. It takes about twice the time to run in CI, locally it runs out of memory for me (have to use NODE_OPTIONS="--max-old-space-size=8192"). windows/linux seem to be fine. cc @brijeshb42 in case you have an idea?

brijeshb42 · 2025-01-28T12:18:58Z

What we can do is disable this app temporarily in this repo and move it to Pigment where I can take a look later to unblock.

DiegoAndai · 2025-01-28T12:26:42Z

What we can do is disable this app temporarily in this repo and move it to Pigment where I can take a look later to unblock.

I think it would be better first to investigate if this is due to the next-app's specific config or if this would impact everyone using next with this updated package layout.

So, there is no rush to take it out now, but let's work together to find the root cause.

brijeshb42 · 2025-01-28T14:38:16Z

What's the average build time on master/other PRs ?

Let me check this locally.

DiegoAndai · 2025-01-28T14:50:25Z

What's the average build time on master/other PRs ?

I don't have the specifics for the next-app, but:

The last 5 master test-dev (macos) builds (pnpm build:ci) took between 4 and 7 minutes

The last one on this PR was over 2 hours, the one before that was 14 minutes.

There's definitely something going on.

brijeshb42 · 2025-01-28T16:38:14Z

Just checked locally on both master and ^ this branch. master built in 2 mins but this branch's build failed with some heap memory error.
I'll have to bisect commit by commit to identify the issue. I'll update once I have some result.

brijeshb42 · 2025-01-29T13:29:41Z

@Janpot what was the issue?

Janpot · 2025-01-29T21:52:50Z

It's not solved yet, this action just took 1 hour to run.

Janpot · 2025-01-30T15:57:20Z

I think it would be better first to investigate if this is due to the next-app's specific config or if this would impact everyone using next with this updated package layout.

I tried removing pigment from the app, but kept as much as possible of the original pages and it reduces runtime drastically, see https://github.com/mui/material-ui/actions/runs/13055965609/job/36426957328?pr=45152
This suggests it's caused by some combination of pigment css with the new package layout. It's also almost 3 to 4 times faster than master, goes to show the overhead it could create.

DiegoAndai · 2025-01-30T17:48:03Z

@Janpot which option should we follow:

Investigate the issue before merging
Remove the Pigment build and investigate separately

?

I worry that the increase in build time might uncover some edge cases with the new layout we haven't seen. But if you think it's more likely to be caused by Pigment's configuration, then we could unblock this and investigate separately. This would get the layout change to users faster, so we got feedback earlier.

This suggests it's caused by some combination of pigment css with the new package layout. It's also almost 3 to 4 times faster than master, goes to show the overhead it could create.

I didn't understand this.

Janpot added 4 commits August 11, 2024 13:53

WIP

0c2d530

exclude icons

f0cc38e

MUI_PACKAGE_EXPORTS

3955055

more

33c43c1

Janpot added the scope: code-infra Specific to the core-infra product label Aug 11, 2024

Janpot changed the title ~~[core] Try new build mode that creates package exports~~ [core] Update package layout for better ESM support Aug 11, 2024

Janpot added 2 commits August 11, 2024 15:17

Update copyFilesUtils.mjs

5888018

remove deep export

642fdcd

Janpot changed the title ~~[core] Update package layout for better ESM support~~ [code-infra] Update package layout for better ESM support Aug 12, 2024

Janpot added 3 commits August 12, 2024 09:15

bring back flag

868a81d

Update ci.yml

e03039f

WIP

f5bb040

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Aug 12, 2024

Janpot mentioned this pull request Aug 14, 2024

[code-infra] Babel plugin to fully resolve imported paths #43294

Merged

Janpot added 2 commits August 23, 2024 17:32

Merge branch 'next' into esm-exports

da26652

Update build.mjs

4229626

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Aug 23, 2024

Janpot added 8 commits August 25, 2024 08:32

remove this

c11b8e5

fix snapshot

a98520b

vite icons alias

60e04ca

remove 'use client' from index.* fles

6e2bbcc

Update package.json

9348956

Update package.json

860a36e

Update package.json

904f65b

Revert "remove 'use client' from index.* fles"

0de01ee

This reverts commit 6e2bbcc.

This was referenced Aug 26, 2024

[system] createTheme() shouldn't have "use client" #42750

Open

[code-infra] Build size snapshots from installed packages #43452

Merged

Janpot added 2 commits August 27, 2024 11:03

bring changes to snashots

5172a89

Update webpack.config.js

9eaa517

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 20, 2025

Janpot added 2 commits January 23, 2025 11:14

Merge remote-tracking branch 'upstream/master' into esm-exports

0f32163

backwards compatibility mode

3ed9f16

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 23, 2025

Merge branch 'master' into esm-exports

93d37f5

DiegoAndai reviewed Jan 24, 2025

View reviewed changes

packages/mui-material/package.json Show resolved Hide resolved

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 27, 2025

Janpot added 2 commits January 28, 2025 09:56

Merge remote-tracking branch 'upstream/master' into esm-exports

e7cc867

do skipEsmPkg

56c2982

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 28, 2025

Janpot marked this pull request as ready for review January 28, 2025 10:32

Update ci.yml

7e54d18

Janpot added 4 commits January 29, 2025 10:37

stream logs to try get a clue where next gets stuck

877947a

disable sourcemaps

9251e8f

use esm for next config

e997060

bring these references back?

6cdfa8c

DiegoAndai mentioned this pull request Jan 31, 2025

Problems with module and vite ssr #45149

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[code-infra] Update package layout for better ESM support #43264

[code-infra] Update package layout for better ESM support #43264

Janpot commented Aug 11, 2024 •

edited by DiegoAndai

Loading

mui-bot commented Aug 11, 2024 •

edited

Loading

DiegoAndai left a comment

Janpot commented Jan 24, 2025

DiegoAndai commented Jan 27, 2025

Janpot commented Jan 28, 2025 •

edited

Loading

brijeshb42 commented Jan 28, 2025

DiegoAndai commented Jan 28, 2025

brijeshb42 commented Jan 28, 2025 •

edited

Loading

DiegoAndai commented Jan 28, 2025

brijeshb42 commented Jan 28, 2025

brijeshb42 commented Jan 29, 2025 •

edited

Loading

Janpot commented Jan 29, 2025 •

edited

Loading

Janpot commented Jan 30, 2025 •

edited

Loading

DiegoAndai commented Jan 30, 2025

[code-infra] Update package layout for better ESM support #43264

Are you sure you want to change the base?

[code-infra] Update package layout for better ESM support #43264

Conversation

Janpot commented Aug 11, 2024 • edited by DiegoAndai Loading

Bundle size increase

Resolutions

mui-bot commented Aug 11, 2024 • edited Loading

Netlify deploy preview

Bundle size report

DiegoAndai left a comment

Choose a reason for hiding this comment

Janpot commented Jan 24, 2025

DiegoAndai commented Jan 27, 2025

Janpot commented Jan 28, 2025 • edited Loading

brijeshb42 commented Jan 28, 2025

DiegoAndai commented Jan 28, 2025

brijeshb42 commented Jan 28, 2025 • edited Loading

DiegoAndai commented Jan 28, 2025

brijeshb42 commented Jan 28, 2025

brijeshb42 commented Jan 29, 2025 • edited Loading

Janpot commented Jan 29, 2025 • edited Loading

Janpot commented Jan 30, 2025 • edited Loading

DiegoAndai commented Jan 30, 2025

Janpot commented Aug 11, 2024 •

edited by DiegoAndai

Loading

mui-bot commented Aug 11, 2024 •

edited

Loading

Janpot commented Jan 28, 2025 •

edited

Loading

brijeshb42 commented Jan 28, 2025 •

edited

Loading

brijeshb42 commented Jan 29, 2025 •

edited

Loading

Janpot commented Jan 29, 2025 •

edited

Loading

Janpot commented Jan 30, 2025 •

edited

Loading