Allow code components to import from npm packages

Up until now, the XB team has been following a pseudo-scrum/pseudo-kanban process, but we're now shifting into more conventional scrum. We started a new 2-week sprint last Thursday (Jan 16). I'm tagging our current sprint's issues for visibility. This one I'm tagging with "spike" rather than "sprint", because our goal isn't to complete the implementation of it this sprint, but to figure out if the proposed approach is viable or if we need to change it in some way.

@effulgentsia do you have something in mind to parse the JS and flag the imports - e.g. an AST/parsing library?

If not, I can look into something like babylon, which is the parser used by babel.

Does @jspm/generator not do that? The CLI's jspm link can be pointed to an HTML file, but not sure whether it's only the CLI that can parse or if the generator library can as well.

Babylon seems fine if we need a separate parser. In case it helps, we'll be using https://swc.rs/docs/usage/wasm to compile. I don't know if it has any API to access the imports, but perhaps it makes sense to parse the compiled JS to find the imports, in which case something lightweight like Acorn might suffice?

I have a POC of using babylon to parse out imports here

Paste this in to the text box

import * as Accordion from "@radix-ui/react-accordion";

export default function MyComponent() {
  return (
    <Accordion.Root>
    </Accordion.Root>
  )
}

I'll continue with this tomorrow by adding in jspm/generator code to see if we can get an importmap

Ah, this is literally point 1 in the issue summary. But I'm surprised that #3499927 was created without taking this into account and this issue's title doesn't convey that it's a spike for future functionality and it doesn't link that issue. 😅

1. For now, we can hard-code a CDN (e.g., https://esm.sh/) to map them to. In the future, we can make that CDN configurable.

Blindly trusting a single external site is a serious risk: performance, reliability but certainly also security! Especially when they're maintained by a single person. I'd not want to do this unless we at minimum do https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implemen...

But apparently supporting this for import maps is pretty much brand new: https://jspm.org/js-integrity-with-import-maps, published August 5, 2024, released in Chrome 127. But that's only got 78.7% global support right now; it's not supported at all in Firefox for example: https://caniuse.com/mdn-html_elements_script_type_importmap_integrity

Removing the security tag because it makes this issue listed on the project page as as a publicly disclosed security issue which this isn't since the security implication has to do with what's being done in this issue.

Removing the security tag because it makes this issue listed on the project page as as a publicly disclosed security issue which this isn't since the security implication has to do with what's being done in this issue.

🤯 Since when is that the case?! 🤪

https://github.com/mozilla/standards-positions/issues/1010 is the issue to follow for updates on Firefox's plans to support importmap integrity.

#7 ✨ Allow code components to import from npm packages Active :

But I'm surprised that #3499927 was created without taking this into account and this issue's title doesn't convey that it's a spike for future functionality and it doesn't link that issue. 😅

I would recommend that we land ✨ Config entity for storing code components Active with its current scope to unblock the handful of issues it blocks. Given that this issue is in the spike stage, I would also recommend waiting with creating additional issues or adding relationships to existing ones before we know what our approach will be. I'm happy to update 🌱 [Meta] Plan for code components Active once we're there.

The POC in #6 ✨ Allow code components to import from npm packages Active is really cool! 👏🏻

I have a suggestion to also explore this from a different angle. Instead of parsing imports from the code, like it's done in #6 ✨ Allow code components to import from npm packages Active , what if we maintained a list of supported packages, and designed UI to add import statements to a code component? We could even go as far as showing/hiding the import statements in the code editor, e.g. displaying them in a collapsible, non-editable area.

What does this approach get us? Most importantly, it addresses Wim's concerns ( #7 ✨ Allow code components to import from npm packages Active ) about performance, reliability, and security. A curated list of packages that we can include in our app's bundle. (We really need to start thinking about code splitting, but let's leave that as a future problem to solve. 🙂)

Additionally, it might even be a nice UX to be able to browse a list of quality, recommended packages to use in your components. It is not the freedom to use anything from npmjs.org, but this might be fine for the target audience of code components. Radix Primitives would be a great start, they're commonly used, even tools like v0.dev generate code that (indirectly) depends on them. We could then open this up for modules, as a new extension point, to provide new sets of packages.

#12++ … and that ties in elegantly with your proposal at #3499933-18: Storage for CSS shared across in-browser code components (and PageTemplate config entities in the distant future) → to introduce config-defined in-browser code libraries — that "list of quality, recommended packages" could be defined entirely as one or more in-browser asset libraries.

P.S.: literally yesterday:

The specification for this has landed in HTML and the implementation landed in Chromium and WebKit.

— https://github.com/mozilla/standards-positions/issues/1010#issuecomment-...

Related: ✨ Add an API for importmaps Active — being worked on by @larowlan :)

I have a POC going in this commit

The issue is we can't use this import map in the current page because you can't change the import map in the current page at run-time
But I guess if we make the preview component use an iframe, we could inject the import map into there.

Here's a video. I think this is enough to conclude the spike.

Findings:

• Parsing of HTML in @jspm/generator wasn't working because it doesn't like JSX
• I was unable to get acorn + acorn-jsx to parse the example component (below)
• Using babylon (the parser under the hood of babel) gave us the nice advantage of syntax checking

Example component text I was parsing

import * as Accordion from "@radix-ui/react-accordion";

export default function MyComponent() {
  return (
    <Accordion.Root>
    </Accordion.Root>
  )
}

@balintbrews: AFAICT you intend this to not block ✨ Storage for global CSS Active , right? (I tried to guess your rationale over at #3499933-31: Storage for CSS shared across in-browser code components (and other use cases in the future) → .)

I think there's also a world where we use jspm in the editor to build an import map but then have Drupal fetch the resultant files, validate their resource integrity and serve them locally from inside the assets:// stream wrapper (aka sites/default/files/js or equivalent).

Could even be lazy like core's asset optimisation so that we only fetch it when it is requested

Let me try to summarize where we are, and suggest next steps.

What's been proven so far:

1. We can parse import statements from the user-provided code using babylon, the parser of [Babel](https://babeljs.io).
2. By passing those import statements to @jspm/generator we can generate an import map in the browser.

What I suggested in #12 ✨ Allow code components to import from npm packages Active is that instead of doing 1) above, we may want to consider a different UX where we would allow code component authors to pick from a predefined list, and generate the import statements based on their selection.

As @larowlan pointed out in #18 ✨ Allow code components to import from npm packages Active , the predefined list idea can still be relevant even if we decide to parse import statements from the code. This would also nicely support the use case of ✨ CLI tool to manage code components Active .

I think the immediate next step would be to decide on the UX:

A) Do we parse import statements from the user-provided code? Or B) do we provide UI for adding imports?

This could influence our decision on how we approach storing the import statements — if at all.

Which would be better for enabling people to generate code components with AI , or by exporting from Figma via a Figma-to-React plugin?

@effulgentsia: Can you please confirm that my understanding is correct about the proposed js_imports and js_import_dependencies properties?

1. We get the entire import map from @jspm/generator when processing the JS source code of the code component (first with babylon so it's not JSX).
2. Parsing the import map we produce lists of simple module names for js_imports and js_import_dependencies to store.
3. Those are expanded into an import map again (and merged with other entries) when generating the import map on the backend, so it looks like this: https://generator.jspm.io/#Y2NhYGBkDM0rySzJSU1hcChKTMms0C3N1C9KTUwu0U1MT....
4. We hardcode esm.sh as the CDN for now. It supports omitting package versions and entrypoint paths. (Based on a quick testing, omitting those is not supported by jspm.io's CDN, so we e.g. need https://ga.jspm.io/npm:@radix-ui/react-accordion@1.2.2/dist/index.mjs, and can't simply have https://ga.jspm.io/npm:@radix-ui/react-accordion.)

As I was thinking through this and writing #22 ✨ Allow code components to import from npm packages Active , I looked into esm.sh. I discovered that if we use esm.sh, we don't need to worry about generating the entire import map to cover the dependencies of the dependencies.

esm.sh rewrites import specifiers in the package code it serves: they are transformed to reference the dependencies also served by esm.sh. This means that we can simply do this:

import * as Accordion from "https://esm.sh/@radix-ui/react-accordion";

Take a look at what https://esm.sh/@radix-ui/react-accordion returns: all the dependencies are there, imported from esm.sh: 😳

/* esm.sh - @radix-ui/react-accordion@1.2.2 */
import "/@radix-ui/primitive@1.1.1/es2022/primitive.mjs";
import "/@radix-ui/react-collapsible@1.1.2/es2022/react-collapsible.mjs";
import "/@radix-ui/react-collection@1.1.1/es2022/react-collection.mjs";
/* ... */
export * from "/@radix-ui/react-accordion@1.2.2/es2022/react-accordion.mjs";

After seeing this, I wanted to validate our use case, so I created a POC that uses Preact, React linked to Preact (as @effulgentsia suggested before on a call), and @radix-ui/react-accordion as an example — all imported from esm.sh via an import map:

🧪 https://github.com/balintbrews/poc-import-from-esm-sh

It works beautifully! 🎉 The AccordionDemo component was copied straight from the Radix UI docs, no changes were required to the import statements.

What all of this means is that as long as we are happy with esm.sh only, we don't need to store anything on the config entity, we can easily parse the import specifiers in the JS source, and replace them, so e.g.:

import * as Accordion from "@radix-ui/react-accordion";

becomes:

import * as Accordion from "https://esm.sh/@radix-ui/react-accordion?external=react,react-dom";

(See why the ?external... in the README.md of my POC)

Then we just need this import map:

<script type="importmap">
  {
    "imports": {
      "preact": "https://esm.sh/preact",
      "preact/": "https://esm.sh/preact/",
      "react": "https://esm.sh/preact/compat",
      "react/": "https://esm.sh/preact/compat/",
      "react-dom": "https://esm.sh/preact/compat",
      "react-dom/": "https://esm.sh/preact/compat/"
    }
  }
</script>

Yes, #22 is exactly what I had mind. For #1 of that, I don't have a preference on whether to use babylon on the source or whether to compile with SWC first and then parse for imports on compiled JS. If there's no downsides to the former, then that's great.

I think #23 is fine as an interim step, but eventually we'll need to support other CDNs. Also, eventually, we'll need the site owner to be able to control when to update versions, so I think for that we'll need the imports and their dependencies in the config entity. We can bump all that to followups though, so long as the PoC in this issue's fork doesn't get deleted, so we can refer to it later.

What's next here? Is this still relevant?

✨ Bundle a small selection of packages in the hydration library for code components Active is not mentioned here at all, and brought in a select predefined list of importable packages — AFAICT this issue is about generalizing it. If so: the issue summary needs to be updated to reflect how this changes the status quo that #3508734 created. 🙏