What We Do In The Background (Shadows)

TLDR

We introduce a new background syncing API, improve webhooks based on community feedback, and add an experimental cache that can run on client and server.

TriplitClient.subscribeBackground

This week we're introducing a new way to subscribe to queries from the server: TriplitClient.subscribeBackground. This function is similar to TriplitClient.subscribe, but it doesn't provide an onResults callback -- it is strictly responsible for listening to changes to a query from the server and updating the local database. Some of our users have been employing a pattern of one large subscription to the server to keep their local database up to date, and then defining many local-only subscriptions to cut down on traffic to the server. This was possible before, but the original TriplitClient.subscribe method would do the additional work of preparing query results for an onResults handler. subscribeBackground skips this work, which can make a noticeable difference for large databases.

Here's how you might use it:

import { TriplitClient } from '@triplit/client';

const client = new TriplitClient({
  endpoint: 'https://api.example.com',
  token: 'my-secret-token',
});

const unsubscribeBackground = client.subscribeBackground(
  client.query('todos').build(),
  {
    onFulfilled: () => {
      console.log('Received initial results from the server');
    },
    onError: (e: Error) => {
      console.error(e);
    },
  }
);

const unsubscribeLocal = client.subscribe(
  client.query('todos').where('completed', '=', false).build(),
  (results) => {
    console.log('Got latest query results', results);
  },
  (error) => {
    console.error('Error in query', error);
  },
  { localOnly: true }
);

More complete webhooks

Two weeks ago we previewed webhooks for Triplit servers. With a couple more more weeks of development under our belts, and some useful feedback from the community, we've made a key improvement. The payloads of webhooks will now include the new and old versions of an entity, where applicable. So for an update event, the payload will have the entity before and after the change. For a delete event, the payload of the entity before it was deleted.

Here's an example of a webhook payload for a single transaction with an insert, update and delete:

{
  "collectionName": "todos",
  "inserts": [
    {
      "id": "3",
      "entity": {
        "id": "3",
        "text": "Do the dishes",
        "completed": false
      },
      "oldEntity": null
    }
  ],
  "updates": [
    {
      "id": "2",
      "entity": {
        "id": "2",
        "text": "Walk the dogs",
        "completed": true // has been changed
      },
      "oldEntity": {
        "id": "2",
        "text": "Walk the dogs",
        "completed": false
      }
    }
  ],
  "deletes": [
    {
      "id": "1",
      "entity": null,
      "oldEntity": {
        "id": "1",
        "text": "Make breakfast",
        "completed": true
      }
    }
  ]
}

Webhooks remain available in preview in development (i.e. using triplit dev) and for self-hosted instances of Triplit.

Experimental caching

We've added some additional in-memory caching capabilities to the client and server. This is an experimental feature that we've developed with the goal of cutting down on both the frequency with which a client/server reads from durable storage and the amount of processing required to prepare query results. In our internal tests, use of this cache dramatically speeds up the database reads on subsequent queries on the same data.

This feature is experimental and disabled behind an option. On the client, you can enable it like so:

import { TriplitClient } from '@triplit/client';

const client = new TriplitClient({
  experimental: { entityCache: { capacity: 1000 } },
});

And on a self-hosted server or when running a local development server with npx triplit dev, you can enable it by setting environment variables:

ENTITY_CACHE_ENABLED=true
ENTITY_CACHE_CAPACITY=1000000

We will be making this configurable for Triplit Cloud instances in the future.

Other improvements

• We've made changes to our hosted Triplit Cloud instances which should improve stability.
• We've released a fix for Expo SQLite adapter on React Native.