Built in tracking utilities for common collections

Summary

This RFC proposes making the set of collections from tracked-built-ins built in to the framework, in a byte-wise opt-in in a brand new package (@ember/reactive) and able to be used without new.

Additionally, these APIs can unblock the implementation of RFC#1000: Make array built-in in strict mode and RFC#999: make hash built in in strict mode (but this is not the main focus of this RFC and only could be an outcome).

Motivation

tl;dr:

• performance
• discoverability
• aiming towards better cohesion

Because tracked-built-ins is built on top of public APIs, in particular, ember-tracked-storage-polyfill, we can expect to gain performance benefits by implementing the tracked collections directly into the framework, as we can eliminate ~2 layers of abstraction/wrapping.

Additionally, tracked-built-ins not being built in to the framework, or properly documented in the ember guides has had some negative consequences on folks apps.

For example, this often-inefficient pattern of re-assigning the whole reference.

@tracked value = [];
addItem = (x) => {
    this.value = [...this.value, x];
}

For large sets of data, rendered in a list (often tables), this pattern causes unneeded work in the reactivity-system.

We now know that tracked-built-ins' TrackedArray would be a good way to only append an item to the array, and thus append DOM to our UI, without the reactive system doing anything to the data that hasn't changed.

Some may argue that it's our renderer's responsibility to detect this situation, and optimize best it can, and while there are opportunities we can find to optimize rendering, we also can't make an assumption that either re-assigning or tracked collection usage is going to be the most performant. Developers can measure in their own app.

This is outside the scope of this RFC, but for some underlying motivation, another motivation is along the lines of reigning in our imports over time, potentially by eventually reclaiming the 'ember' package, so that there is a simple package.json that can be the framework, which aligns with real imports (or re-exports) so that we don't require build-system gymnastics in order to build ember apps.

import Route from '@ember/routing/route';
import Service, { service } from '@ember/service';
import Component from '@glimmer/component';
import { tracked, cached } from '@glimmer/tracking';

import Route from 'ember/routing/route';
import Service, { service } from 'ember/service';
import Component from 'ember/glimmer';
import { tracked, cached } 'ember/reactive';

Detailed design

while most of this already implemented, here is the behavior we expect when using any tracked wrapper:

• all property accesses should "entangle" with that property
• all property sets should "dirty" that property
• changes to the length, or overall collection, is represented by an invisible-to-users "collection" internal tracked property, so that iteration can be dirtied
• changes to a collection (add, insert, delete, etc) should cause iteration (each, each-in) to only render what changed and not cause unneeded renders
• changes to a collection copy the original passed in data -- keeping inline with the existing tracked-built-ins behavior
• deleting an entry in a collection should relieve memory pressure
• deleting an entry in a collection should dirty the "collection"
• prototype and instanceof checks should still work, e.g.: a TrackedArray should still return true from Array.isArray, and an instance of TrackedSet should be an instanceof Set.
• no @dependentKeyCompat, see: @ember-compat/tracked-built-ins

How do we handle when the platform adds new APIs?

For example, Set has had new APIs added recently, and tracked-built-ins had to be updated to support those, so if possible, it would be ideal to rely on deferring to the underlying implementation as much as possible, rather than re-implementing a class-wrapper for all known methods -- proxies are particularly good at this -- and while folks have had complaints about proxies in the past, the user-facing API and underlying implementation of all these proxies would be the exact same, so the proxy isn't hiding anything.

Additionally, unlike in tracked-built-ins, we would not expose the constructor APIs; Mirroring the Cell proposal, also allowing customizable equality checking.

The import

import { 
    // able to be used in templates, no 'new'
    trackedObject, trackedArray, 
    trackedMap, trackedWeakMap,
    trackedSet, trackedWeakSet
} from '@ember/reactive';

trackedObject, trackedArray, trackedMap, etc

These utilities wrap the call to their respective constructors. For example, for trackedObject, the implementation and type declaration may look like this:

export function trackedObject<Value>(
    data?: Value, 
    options?: { 
        equals?: (a, b) => boolean;
        description?: string;
    }
): NonNullable<Value> {
    return new TrackedObject(
        data, 
        { 
            equals: options.equals ?? Object.is,
            description: options.description,
        }
    );
}

Some examples assuming implementation of RFC#998: Make fn built-in in strict-mode as well as RFC#997: Make on built-in in strict-mode:

Example trackedArray

import { trackedArray } from '@ember/reactive';

const nonTrackedArray = [1, 2, 3];
const addTo = (arr) => arr.push(Math.random());

<template>
    {{#let (trackedArray nonTrackedArray) as |arr|}}
        {{#each arr as |datum|}}
            {{datum}}
        {{/each}}

        <button {{on 'click' (fn addTo arr)}}>Add Item</button> 
    {{/let}}
</template>

[!NOTE]
Since RFC#1000: Make Array built-in in strict mode is stalled due to the original implementation of (array) being underspecified, the new implementation of the built in (array) could use this trackdArray implementation instead of re-defining the specification of how (array) works -- and this new implementation would probably more align with how folks expect (array) to work.

With RFC#1000, the above example would be behaviorally equivalent to:

const nonTrackedArray = [1, 2, 3];
const addTo = (arr) => arr.push(Math.random());

<template>
    {{#let (array nonTrackedArray) as |arr|}}
        {{#each arr as |datum|}}
            {{datum}}
        {{/each}}

        <button {{on 'click' (fn addTo arr)}}>Add Item</button> 
    {{/let}}
</template>

Example trackedObject

import { trackedObject } from '@ember/reactive';

const nonTrackedObject = { a: 1 };
const addTo = (obj) => obj[Math.random()] = Math.random();

<template>
    {{#let (trackedObject nonTrackedObject) as |obj|}}
        {{#each-in obj as |key value|}}
            {{key}} => {{value}}</br>
        {{/each-in}}

        <button {{on 'click' (fn addTo obj)}}>Add Pair</button>
    {{/let}}
</template>

[!NOTE]
Since RFC#999: Make hash built-in in strict mode is stalled due to the original implementation of (hash) being underspecified, the new implementation of the built in (hash) could use this trackedObject implementation instead of re-defining the specification of how (hash) works -- and this new implementation would probably more align with how folks expect (hash) to work.

With RFC#999, the above example would be behaviorally equivalent to:

const nonTrackedObject = { a: 1 };
const addTo = (obj) => obj[Math.random()] = Math.random();

<template>
    {{#let (hash nonTrackedObject) as |obj|}}
        {{#each-in obj as |key value|}}
            {{key}} => {{value}}</br>
        {{/each-in}}

        <button {{on 'click' (fn addTo obj)}}>Add Pair</button>
    {{/let}}
</template>

@ember/reactive

The process of making libraries support wide-ranges of ember-source is known. ember-source has recently been adapting its release process to use release-plan, so that the ember.js repo can publish multiple packages seemslessly, rather than always bundle everything under one package.

With those new release capabilities within the ember.js repo, Instead of a polyfill for older versions of ember, @ember/reactive, the package (at the time of this RFC, does not exist, but would have the two exported utilities from it), would be published as its own type=module package and included with ember-source, as to not add more dependencies to the package.json going forward.

Why type=module?

This is a requirement for some optimization features of packages (webpack / vite), such as proper treeshaking -- without type=module, the best optimization we can get is "pay for only what you import". For large projects this isn't so much of a problem, but for small projects (or highly optimized projects), the impact to network transfer/parse/eval is measurable. This RFC is also proposing that @ember/reactive be the place for all our ecosystem's reactivity utilities will end up once they've been proven out, tested, and desire for standardization is seen.

For example, other future exports from @ember/reactive (in future RFCs), may include:

• Resource
• AsyncResource
• TrackedPromise
• localCopy
• certain window properties
• …and more

without the static analysis guarantees of type=module, every consumer of @ember/reactive would always have all of these exports in their build. For some utilities, we can place them under sub-path-exports, such as @ember/reactive/window, for window-specific reactive properties, but the exact specifics of each of these can be hashed out in their individual RFCs.

Consumption

@ember/reactive would be another virtual package built in to ember-source, much like @ember/service, etc.

For polyfilling, a library could be created that uses ember-addon.renamed-modules to instruct our build tooling about a virtual package -- the same technique that ember-source uses.

Once a project updates to a sufficiently new enough ember-source, the polyfilling library should be deleted.

How we teach this

API Docs

Most of the API docs are already written in tracked-built-ins, so we can re-use those.

We would need to adapt all examples as well as the new template-oriented helpers to not use the new-able APIs, as those would not be exposed.

trackedArray

A utility for creating tracked arrays, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedArray can be used in templates and in JavaScript via import

See MDN for more information

import { trackedArray } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedArray = [1, 2, 3];
const addTo = (arr) => arr.push(Math.random());

<template>
    {{#let (trackedArray nonTrackedArray) as |arr|}}
        {{#each arr as |datum|}}
            {{datum}}
        {{/each}}

        <button {{on 'click' (fn addTo arr)}}>Add Item</button> 
    {{/let}}
</template>

trackedObject

A utility for creating tracked objects, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedObject can be used in templates and in JavaScript via import

See MDN for more information

import { trackedObject } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedObject = { a: 1 };
const addTo = (obj) => obj[Math.random()] = Math.random();

<template>
    {{#let (trackedObject nonTrackedObject) as |obj|}}
        {{#each-in obj as |key value|}}
            {{key}} => {{value}}</br>
        {{/each-in}}

        <button {{on 'click' (fn addTo obj)}}>Add Pair</button>
    {{/let}}
</template>

trackedMap

A utility for creating tracked maps, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedMap can be used in templates and in JavaScript via import

See MDN for more information

import { trackedMap } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedMap = new Map();
nonTrackedMap.set('a', 1);
const addTo = (map) => map.set(Math.random(), Math.random());

<template>
    {{#let (trackedMap nonTrackedMap) as |map|}}
        {{#each-in map as |key value|}}
          {{key}} => {{value}}</br>
        {{/each-in}}

        <button {{on 'click' (fn addTo map)}}>Add Pair</button>
    {{/let}}
</template>

trackedWeakMap

A utility for creating tracked weak maps, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedWeakMap can be used in templates and in JavaScript via import

See MDN for more information

import { trackedWeakMap } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedWeakMap = new WeakMap();

<template>
    {{#let (trackedWeakMap nonTrackedWeakMap) as |weakMap|}}

        {{log weakMap}}

    {{/let}}
</template>

trackedSet

A utility for creating tracked maps, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedMap can be used in templates and in JavaScript via import

See MDN for more information

import { trackedSet } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedSet = new Set();
nonTrackedSet.add(1);
const addTo = (set) => set.add(Math.random());

<template>
    {{#let (trackedMap nonTrackedMap) as |set|}}
        {{#each set as |value|}}
          {{value}}</br>
        {{/each}}

        <button {{on 'click' (fn addTo set)}}>Add</button>
    {{/let}}
</template>

trackedWeakSet

A utility for creating tracked weak sets, copying the original data so that mutations to the tracked data don't mutate the original untracked data.

trackedWeakSet can be used in templates and in JavaScript via import

See MDN for more information

import { trackedWeakSet } from '@ember/reactive';
import { on } from '@ember/modifier';
import { fn } from '@ember/helper';

const nonTrackedWeakSet = new WeakSet();

<template>
    {{#let (trackedWeakSet nonTrackedWeakSet) as |weakSet|}}

        {{log weakSet}}

    {{/let}}
</template>

Guides

Existing places that import from tracked-built-ins would update to the new imports -- no other changes would be needed.

• This page needs to be updated as @glimmer/tracking doesn't have TrackedArray today.

Something that could be used today, and definitely should be added is a page on how to handle referential integrity. Most of the "tracked" guides only touch on tracking references (via @tracked). For example, each of trackedArray, trackedMap, trackedSet, etc can be used in these ways:

Static reference

class Demo {
    collection = trackedMap();
}

Changes to this.collection can only happen via Map methods.

Double static reference

class Demo {
    @tracked collection = trackedMap();
}

Changes to this.collection can happen via Map methods, as well as replacing the entirely collection can occur via re-assigning this.collection to a brand new TrackedMap. This also has a potential performance hazard, of re-assigning this.collection to a clone of the TrackedMap.

Based on Args

class Demo extends Component {
    @cached
    get collection() {
        return trackedMap(this.args.otherData);
    }
}

Changes to the collection can happen via Map methods, as well as changes to @otherData will cause the entirety of this.collection to be re-created, with the previous instance being garbage collected. Usage of @cached is important here, because repeat-accesses to this.collection would otherwise create completely unrelated trackedMaps -- i.e.: Updating a TrackedMap would have no effect on a TrackedMap read elsewhere as they are different instances.

Migration

We should do a codemod to convert the newable constructors from tracked-built-ins to the direct-callable variants proposed in this RFC.

Using Vite or Webpack (Embroider 3+), we can alias tracked-built-ins to point at the new modules, using a shim -- for example:

// app/built-ins-shim.js
import { trackedArray } from '@ember/reactive';

export class TrackedArray {
    constructor(arr) {
        return trackedArray(arr);
    }
}

// etc

Drawbacks

• A migration
• however, the migration is completely optional as tracked-built-ins would still exist. The benefit to this RFC is for new projects, and apps that care more about performance.

Alternatives

• reclaim the ember package and export under ember/reactive, add ember to the package.json.
• doing this would require a polyfill, as ember is already available in all versions of projects, but it does not have sub-path-exports that folks use.
• use /reactivity instead of /reactive
• re-use @glimmer/tracking
• would require that @glimmer/tracking move in to the ember-source repo
• would also require a polyfill, as prior versions of @glimmer/tracking would not have the new behaviors
• there is an existing typo in the guides that hints at using this already for TrackedArray

Unresolved questions

none (yet)