[Meta] Vertical Slice POC

We should add localization to our vertical slice. It's a major feature that Drupal State never added, and should be considered from the beginning of this project.

> We should add localization to our vertical slice. It's a major feature that Drupal State never added, and should be considered from the beginning of this project.

+1

Chiming in here because I am 1) working on a related initiative, particularly 🌱 [Meta, Plan] Pitch-Burgh: JSON field storage & JSON field schema support Active for generating Open API schemas out of the box; 2) actively building and growing a product (Meet Kinksters) that uses JSON:API and JSON-RPC in Drupal quite extensively, and 3) a maintainer of Simple OAuth (vis-a-vis authentication questions). In the course of building my product, I've also had to effectively implement a POC on my own for integrating with Drupal as an API client without totally burning my sweat equity budget and reinventing the wheel.

I also have some strong (but hopefully medium-strength held, open to challenging) opinions that much of what seeks to be accomplished in this initiative can be done through the thoughtful integration of prior art. Over at #3364947: Evaluate Orbit.js → I suggest that perhaps the official API client could be a superset or opinionated implementation of Orbit.js. Regardless of the building blocks, here's what I am doing in my current project that I think could be a guide for such a POC. I don't intend for this to be construed as Brad making as much noise as possible about Orbit.js, but I do think it would help us get like, years ahead of the curve. So just take this as one example of how we can piggyback and get these items in short order:

Data fetching - Out of the box support for this in Orbit.js, including custom resources implemented with JSON:API Resources → (well, GET from them at least.) This does require a Drupal-specific customization of the library's URL builder, but this is to be expected as one of JSON:API's features is that it makes no assumptions about your URL structure. This still requires some structure by convention (that is, a resource named some--resource would be at /jsonapi/some/resource) but you get the idea. Basically, done.

Authentication - This is an interesting one. A common misunderstanding I see commonly in the #contenta channel in Drupal slack by people new to decoupled is to assume that your choice of API is coupled to your authentication. This couldn't be further from the truth. Rather, I think it's accurate to say that your API client must properly integrate with your authentication method... and furthermore this might be different for different consumers. E.g., Meet Kinksters has both native phone app clients (React Native) and a web UI (RN Web.) On the web side, I made the conscious decision to use cookie sessions, for simplicity's sake, since my Drupal endpoint is on a higher-level host from my web UI and they can share cookies. In addition, users can do what I'll call "direct" OAuth login flow on web, but because of the... strange and unique ways that some native app authentication flows operate, I am doing a kind of two-step flow with OAuth token exchange.

All of that is by way of example that 1) authentication is hard and 2) I think it would be unwise to too closely couple this initiative with authentication. In my case, I'm having good success with having authentication handled out of band from the API client but doing things like on-demand token refresh or CSRF header injection by way of a middleware'd fetch that is passed to the client as a transport layer. In that way, your API client doesn't need to know about or care about your authentication.

Further on this point, JSON:API (or, to be inclusive, GraphQL or whatever the client would speak) is not the only way that you might do API integrations with Drupal. I'm not sure it's in scope of an initial integration, but I could see a situation where we also provide a default JSON-RPC client for things that are not appropriate to do over JSON:API. I'm using client-js however it could best be described as "minimally maintained." It's also super basic so we could legitimately fork off the HTTP transport code and improve it from there. The Drupal JSON-RPC → module is pretty mature and is maintained by people who are still well-connected in this ecosystem. I mention all this here because if you follow the pattern of having the authenticated transport separate from the API client, you can use the same middleware'd fetch for both clients. I'm happy to share example code.

For OAuth refresh, I'm using https://github.com/badgateway/oauth2-client which has involved maintainers and has the added benefit of having 0 external dependencies.

Data Serializer - I'm not entirely sure what the scope for this is. If I squint a little and make some assumptions about what it could mean, I think this is the kind of thing that has been handled by prior art in many circumstances. If the intent is to somehow replicate Drupalisms like entity types and bundles on the client side, I think this is unnecessary and a potentially dangerous comingling of information domains. Speaking from experience on a complex JSON:API Drupal-backed project, the strength of using JSON:API is that it provides a backend-agnostic way of representing data without the client needing to care much that it's Drupal behind the scenes. On top of that, with the ability to rename, hide and/or transform responses for the API, I think using JSON:API (or some equivalent, abstracted) value objects on the client side is just fine. Or perhaps I'm misunderstanding the underlying requirement.

Local Cache - This is the memory data source in Orbit, which can act as both a cache/offline mode data store as well as a foundational element in a forking strategy (which is advanced stuff, but possible.)

What's not answered directly by any API client I'm aware of is a client-side awareness of Drupal cache tags and contexts, or TTLs. I have messed around with this just a little bit in so far as I have to handle signed URLs for user profile images (the signatures on which have a TTL) but also the fact that the image you're sent is dependent on contexts such as whether you're matched with the person (e.g., you might get a blurred photo if not.) Caching is notoriously hard and there is also a potential role here for HTTP semantics to play a role, but yeah... that's a lot. IMO the caching that the client should focus on doing should be the ability to load an entity (JSON:API resource object) from memory on the client side, potentially with no freshness validation. However if you do an operation such as update the entity by PATCHing it, your local cache's copy is updated (in Orbit.js, this is because you update the memory copy first, and then it's replicated to the remote endpoint.)

Logging - Not entirely sure what the scope is here but Orbit.js has logging by way of coordinator strategies. Think of coordinators as the glue between data sources, e.g. you set up a coordinator to pull from the remote, update the memory source and serve the result from there.

Localization - Probably one of the toughest parts of this puzzle. This isn't even fully supported in core's JSON:API implementation 🌱 [META] Formalize translations support Needs work , yet. Honestly since the server side is a moving target/not yet stable, I might suggest replacing this goal with some basic ability to consume Drupal's JSON:API 1.1 profile. I think #2794431-43: [META] Formalize translations support → sums up the blockers well. Put another way, this requires work on the server side before we can really do anything on the client side. But making sure we're ready for proper JSON:API profile support would be a good step in this direction... or just, wait.

Thanks for the thoughtful response @bradjones1. I have thoughts and comments, but first I wanted to spend a little time to actually try Orbit hands on. Walked through the getting started example which helped me understand the power of orbit better, but I'm getting stuck when I actually try to use the JSONApiSource.

I have a codesandbox here: https://codesandbox.io/s/orbit-json-api-8pl25k?file=/src/index.mjs - am I missing something obvious? I can't seem to get results back and don't really get any feedback as to why.

Alternatively, if you have a working Drupal based example that you could share, that could do the trick as well.

Also check me on one basic assumption here - Orbit and specifically the JSON:API Source always require a pre-defined schema, right?

On Orbit.js at a higher level:

After looking more at Orbit.js I think some of the misalignment here is actually due to two different valid but somewhat conflicting use cases.

The current plan for the API Client is focused around lower level utilities. It will ship a JSON:API client that will be useful out of the box, but much of the focus here is also on creating a base class that will support other API types, and making the JSON:API client itself highly configurable. My hope is that it will be possible to build on top of this work, providing a base that can be useful with the wildly evolving JS ecosystem.

This phase also does not specifically focus on a strictly typed or schema driven approach. The topic was discussed and I think it is an important use case, but it seemed like something that would be for a future phase.

Orbit seems to be very explicitly focused on this use case. And beyond that, seems to be targeting the needs of a complete / mature application. Those things are obviously more in line with your current focus - a schema driven approach, building a fully featured application. And in service of that, it does some things that I think would be very difficult (foolish?) to implement from scratch. Everything it does around coordination and syncing between multiple sources would be a giant hill to climb for example.

My theory is that both use cases are important and worth Drupal investing in.

- JSON:API Client use case - the basics, underlying utilities, less opinionated, more configurable.

- Orbit.js use case - schema driven, strongly typed, highly opinionated, focused on complex applications.

Forced to choose, I believe the API client use case is the place to start.

But I don’t think it prevents the Orbit.js type use case in the future.

- Perhaps outlining what the API client won’t do would be helpful? As mentioned above, I don’t know that it should ever should evolve into something like Orbit.
- And if Orbit is determined to be the best way to address this schema driven, application focused use case, I’m not convinced that it should be built on top of the API client work. Could be a documentation issue, but I don’t come away with the feeling that Orbit is intended to be drastically overridden, or that the internals are really intended to be used in a non-Orbit context.

Thanks for the insight... I think this is a super important conversation to have at the outset, since it seems there's still a lot of discussion to be had around scope.

I will also be fully transparent in acknowledging my skepticism around the need for such a reference client to begin with. I don't say that to be inflammatory - I think there's a lot of good work to be done here - but because I think much of the point of supporting something like JSON:API in core is that there shouldn't need to be an official, Drupal-specific API client for most basic use-cases. Because we're implementing a standard, any number of clients (yes, like Orbit.js or Kitsu) are capable of interfacing with Drupal exactly because the server side is standards-compliant.

That's exactly what you see in the wild now; people are successfully integrating with API-first Drupal because it's not tightly coupled to a client or SDK.

This is in part why I am focusing on discoverability and schemas in my work, something that is complimentary to the out-of-the-box JSON:API integration in Drupal and would allow for a real, industry-leading HATEOAS interface for clients - official or not.

All that said, I think this is a worthwhile initiative - if the goal is to not reinvent the wheel and to focus narrowly on the areas where the client has a need to interface with Drupal per se.

The current plan for the API Client is focused around lower level utilities. It will ship a JSON:API client that will be useful out of the box...

This is in line with my saying, this basically already exists and we shouldn't spend any time as a community reinventing the wheel. Putting aside my client of choice, you're basically describing Kitsu.

...much of the focus here is also on creating a base class that will support other API types, and making the JSON:API client itself highly configurable. My hope is that it will be possible to build on top of this work, providing a base that can be useful with the wildly evolving JS ecosystem.

I'll respectfully disagree here that an MVP should seek to implement anything other than JSON:API. Here's why. The reason JSON:API was selected as the "official" core API implementation is specifically because it makes some opinionated (but not too constrained) decisions around the structure of an API for relational data. As the JSON:API spec states itself, it paints your bikeshed in ways that help development teams avoid reinvention of wheels and distracting decisions around request and response structures. There is still plenty of choice in specific implementations (e.g., Drupal's filtering query parameter structure) and that flexibility has been codified in JSON:API 1.1 with profiles.

GraphQL, on the other hand, is much more about bringing the power of flexible SQL-like querying from the server-side into the client. (I understand this is a gross over-generalization, but bear with me.) What's more, it requires an expert level of skill to implement, and AFAIK does not provide any working default implementation out of the box, unlike core's JSON:API Implementation.

That aside, there are great GraphQL clients in the wild like Apollo, and this is IMO an area where if you are implementing GraphQL, you "know what you're doing" and don't need an official client. Or put another way, is there any concrete benefit of expending energy on an official client for Drupal when you 1) need to do a non-trivial amount of implementation on the server side, specific to your data model and 2) can use more mature clients with an existing and vibrant maintainer base, e.g. Apollo?

That gets us back to the big picture of the goal of an official client. If we apply the same considerations to JSON:API, Drupal ships a server implementation with sensible, yet slightly opinionated defaults out of the box. It is these opinionated defaults, within the guardrails of the JSON:API spec, which is where I think the energy of this initiative should focus. E.g., build a client that implements the Drupal JSON:API 1.1 profile features, but doesn't reinvent the wheel in other areas. This is why I'm pushing for building on some sort of existing foundation - whether that's an Orbit.js or something leaner like Kitsu.

Respectfully, I think there is some conflict between the goals listed in the IS and the concept of a lean client that is:

the basics, underlying utilities, less opinionated, more configurable.

Of the list in the IS, I would probably consider "data fetching," "authentication," and possibly "logging" to be considered "the basics." Honestly, all of this exists in the wild today. If we don't want to wade into anything that is "strictly typed," by which I think you mean the client being schema-aware, then we could do this initiative in a few weeks by extending Kitsu with an OAuth2-middleware fetch library (like the one I linked above) and call it a day. That's a bit reductionist, sure, but that would achieve much of what is considered "the basics" without much opinion.

This phase also does not specifically focus on a strictly typed or schema driven approach. The topic was discussed and I think it is an important use case, but it seemed like something that would be for a future phase.

Building on my last point, this is why I am disagreeing with the general idea of the client not being an opinionated implementation. If this is going to be an out of the box client you can get going with almost no customization ("point it at your Drupal JSON:API endpoint") then those opinions should simply match those that were made in the implementation of the Drupal core JSON:API server. This is also why the module's authors went to great pains to make so much of the API final and non-extendable; the HTTP API _is_ the module API. @bbrala, the JSON:API maintainer, raised this at DrupalCon Portland and there was good discussion at 🌱 [META] What customizations of JSON:API do you have/need Active - and in many cases it turned out that what people needed from JSON:API wasn't so much customization, but better schemas and (yes) a default client that matched the server's opinionated defaults.

TL;DR: I don't think this initiative should wade into any API shape but JSON:API. It should build upon an existing lower-level library for transport and (probably) authentication, and things that are not in the core JSON:API spec like filtering should be implemented as a profile.

I'll follow up as time allows on how I think Orbit.js might still be a good candidate for doing the above... but hopefully this helps center the conversation on the scope, regardless of the underlying tech.

> I think this is a super important conversation to have at the outset, since it seems there's still a lot of discussion to be had around scope.

Agree that conversations like this are important - a big part of going the pitch competition route was to increase visibility and inspire these types of conversations. I don’t completely agree that there is still a lot of discussion to be had around scope though. I think the scope of what was proposed for this project is relatively clear. Having the discussion we’re having just requires envisioning a different scope or the possibility of changing scope.

> I will also be fully transparent in acknowledging my skepticism around the need for such a reference client to begin with. I don't say that to be inflammatory

Understood. And welcome this conversation even if our opinions can’t align.

> That's exactly what you see in the wild now; people are successfully integrating with API-first Drupal because it's not tightly coupled to a client or SDK.

There are absolutely people having success specifically for the reasons you outline. There are also people who are not having success for essentially the same reasons. Both audiences exist, and it seems like we’re focused on different ones.

> I'll respectfully disagree here that an MVP should seek to implement anything other than JSON:API.

Sorry if any of my comments unintentionally implied that. This MVP will not implement anything other than JSON:API. We are making the architectural choice to split things between a base class and a JSON:API class that extends it. That obviously is setting us up for other things in the future. But for what we’re committing to now, 1.x is just JSON:API.

> GraphQL … what's more, it requires an expert level of skill to implement, and AFAIK does not provide any working default implementation out of the box, unlike core's JSON:API Implementation.

A bit of a rabbit hole here, but contrib GraphQL 3.x did offer an out of the box implementation. 4.x does not. But projects like https://www.drupal.org/project/graphql_compose → are aiming to change that.

> If we don't want to wade into anything that is "strictly typed," by which I think you mean the client being schema-aware, then we could do this initiative in a few weeks by extending Kitsu with an OAuth2-middleware fetch library (like the one I linked above) and call it a day. That's a bit reductionist, sure, but that would achieve much of what is considered "the basics" without much opinion.

I see Kitsu as having a decent amount of opinion. It uses Axios rather than fetch. It is a set of dependencies we then rely on. Is that the right set of opinions? Maybe. Maybe not. Maybe I’m over prioritizing flexibility here - possibly being naive. But I also really want to avoid getting boxed into something.

The good news is that I think the POC structure we’re aiming for really helps with something like this. We should use it as a checkpoint to evaluate this. Were we successful or did we bite off more than we can chew? With a working example do people actually see value in this direction, or would we be better off extending an existing solution (and I do agree we could spin up a Kitsu example quickly)

> … and (yes) a default client that matched the server's opinionated defaults.

I think we’re on the same page here but something is getting lost in translation. ‘a default client that matched the server's opinionated defaults.’ is 100% what I see this JSON:API client as from a functional perspective.

> things that are not in the core JSON:API spec like filtering should be implemented as a profile.

Don’t completely agree with that though. I specifically see this as a client focused on Drupal’s JSON:API implementation. Drupal’s opinionated defaults in this case.

To close the loop here, some quick, probably sloppy comments on things from #7.

> A common misunderstanding I see commonly in the #contenta channel in Drupal slack by people new to decoupled is to assume that your choice of API is coupled to your authentication. This couldn't be further from the truth. Rather, I think it's accurate to say that your API client must properly integrate with your authentication method... and furthermore this might be different for different consumers.

For the record, strongly agree with this as well.

> All of that is by way of example that 1) authentication is hard and 2) I think it would be unwise to too closely couple this initiative with authentication.

I still think there are reasonable things that can be offered here that would make common Drupal authentication scenarios easier and would provide a lot of value specifically because of #1.

> If the intent is to somehow replicate Drupalisms like entity types and bundles on the client side, I think this is unnecessary and a potentially dangerous comingling of information domains.

It’s actually closer to the opposite - converting a JSON:API response into a simpler JS object. Many comparable clients offer this.

> What's not answered directly by any API client I'm aware of is a client-side awareness of Drupal cache tags and contexts, or TTLs.

We’ve actually implemented a Pantheon focused variation of this that works well. Some more detail here: https://decoupledkit.pantheon.io/docs/frontend-starters/nextjs/nextjs-dr...

Nothing like this would be directly part of the 1.0 scope, but some of the things like being able to override fetch and controlling outgoing headers would help make this possible in the future.

> **Localization**

We should work through this in the related child issue. I also need to better understand the limitations. Not having any way to easily get different translated versions of content has been an issue for other clients - I’d expect it to be a problem here even if there are still issues on the server side.

> Profiles.

Realizing that in previous comments I misinterpreted references to JSON:API Profiles. I also need to get up to speed on what those are :)

Thanks for the comments - and I hope I'm not hijacking any broader process here but rather getting us closer to clarity on what you want the POC to be. A few thoughts regarding items where we might have differences of opinion/approach:

I see Kitsu as having a decent amount of opinion. It uses Axios rather than fetch. It is a set of dependencies we then rely on. Is that the right set of opinions? Maybe. Maybe not. Maybe I’m over prioritizing flexibility here - possibly being naive. But I also really want to avoid getting boxed into something.

Kitsu basically has two external dependencies: Axios and a pluralization library. Axios is super common in the wild and reasonable minds can disagree on whether it's more useful than depending on fetch. Let's also keep in mind that fetch support in node.js (related question: are you targeting node or the browser or both?) is like, a year old, so it's not surprising to see Axios as a fetch alternative in a lot of libraries where fetch might be chosen on a greenfield implementation today. Minor issue IMO.

There may be some justified discussion around how much "opinion" is prudent in an official client. I use Kitsu as an example of, IMO, a rather vanilla and unopinionated client. It basically takes an options object that maps 1:1 to reserved request query parameters, makes the request, and returns POJOs.

Speaking of POJOs (plain old javascript objects):

It’s actually closer to the opposite - converting a JSON:API response into a simpler JS object. Many comparable clients offer this.

I'm curious if you could point at examples of a "simpler" object in the wild and how it's advantageous to add an additional normalization process. In both the cases of example clients we've discussed - Kitsu and Orbit.js - both return the JSON:API resource object basically straight from the remote source, and takes input of the same shape. I actually think that doing any further bespoke normalization on these objects, which are really basic to begin with, adds unnecessary complexity.

I'm trying to think of a situation in which adding complexity to the client-side resource/entity object is advantageous. I guess there's the case of adding some easier getters (but not setters!) for the Drupal entity ID vs. UUID... but honestly it might be better to make the "official" default be to ignore these altogether in favor of UUIDs. But I'm not sure any of that mandates having a client-side object that isn't basically the same as the resource object in JSON:API format.

It also, I think, complicates doing operations like PATCHing or PUTing of new entities. Would you start by creating the bespoke object, then the client has to convert it to JSON:API resource object, and then doing the reverse when it's returned? This is another case where the JSON:API spec, if we just defer to it, lets us avoid this kind of bikeshedding.

We’ve actually implemented a Pantheon focused variation of [client-side cache tag awareness] that works well

Thanks for the pointer, I'll have to check it out. At first glance though it looks to require a pretty specific configuration on both client and (most importantly) server side, and as I think we agree the MVP here should basically integrate with a Drupal site with no special contrib dependencies.

Re: Localization, for the reasons I expressed above, I honestly think this is something that needs to get hashed out with support in core first... and is likely to just be a huge can of worms without that. I'd recommend supporting what does work now and not making any efforts to expand support within the scope of the client MVP.

Re: profiles in JSON:API 1.1, see 📌 Spec Compliance: JSON API's profile/extention (Fancy Filters, Drupal sorting, Drupal pagination, relationship arity) needs to be explicitly communicated Active . It's a good summary of all the Drupal-specific stuff that a client needs to know, plus I guess stuff that isn't really specified in a profile (I don't think) like default URL mapping from resource types. I think this is an excellent paradigm through which to view the scope of the Drupal-ly things that need to be accounted for - in particular fancy filters. To do so, I think starting from scratch means spending a lot of time on boilerplate that we can either take (e.g., forking Kitsu wouldn't be a huge lift, it's small, but still you're forking) or build upon (e.g. in the Orbit.js example.)

Re: authentication, agree it probably makes sense to support something like OAuth2 and cookies out of the box, and handling CSRF tokens for mutable operations for the latter.

I'll also make another plug for perhaps shipping a basic JSON-RPC implementation too, that automatically shares the authentication middleware. Re: the less advanced developer who is looking for a turnkey solution for working with Drupal, that would mean you have one package that gets you OAuth2 + JSON:API client for entities + JSON-RPC for non-entity operations.

I'll make a last case for doing this as a custom source type for Orbit.js; the library is broken up into a number of subpackages, so we can reduce the total size and scope of our direct dependencies there. I'm pretty sure we can also disable schema validation altogether (though we might want to further discuss if this is desirable.)

Onward!

POC has been published! https://www.drupal.org/project/api_client/issues/3391145#comment-15298742 →

Thanks to all who contributed along the way - would love any feedback on this POC that you have.

On to 1.0!

Automatically closed - issue fixed for 2 weeks with no activity.