Safely updating schemas in production

Getting setup

Let start with a simple schema, defined at ./triplit/schema.ts in your project directory.

import { Schema as S, ClientSchema } from '@triplit/client';
 
export const schema = {
  todos: {
    schema: S.Schema({
      id: S.Id(),
      text: S.String(),
      completed: S.Boolean({ default: false }),
    }),
  },
} satisfies ClientSchema;

You can start the development server with the schema pre-loaded.

triplit dev

By default, the server will use in-memory storage, meaning that if you shut down the server, all of your data will be lost. This can be useful when you're early in development and frequently iterating on your schema. If you like this quick feedback loop but don't want to repeatedly re-insert test data by hand, you can use Triplit's seed commands. You can use seed command on its own:

triplit seed run my-seed

Or use the --seed flag with triplit dev:

triplit dev --seed=my-seed

If you want a development environment that's more constrained and closer to production, consider using the SQLite (opens in a new tab) persistent storage option for the development server:

triplit dev --storage=sqlite

Your database will be saved to triplit/.data. You can delete this folder to clear your database.

Updating your schema

Let's assume you've run some version of triplit dev shown above and have a server up and running with a schema. You've also properly configured your .env such that Triplit CLI commands will be pointing at it. Let's also assume you've added some initial todos:

import { TriplitClient } from '@triplit/client';
import { schema } from '../triplit';
 
const client = new TriplitClient({
  schema,
  serverUrl: import.meta.env.VITE_TRIPLIT_SERVER_URL,
  token: import.meta.env.VITE_TRIPLIT_TOKEN,
});
 
client.insert('todos', { text: 'Get groceries' });
client.insert('todos', { text: 'Do laundry' });
client.insert('todos', { text: 'Work on project' });

Adding an attribute

Now let's edit our schema by adding a new tagId attribute to todos, in anticipation of letting users group their todos by tag.

export const schema = {
  todos: {
    schema: S.Schema({
      id: S.Id(),
      text: S.String(),
      completed: S.Boolean({ default: false }),
      tagId: S.String(),
    }),
  },
} satisfies ClientSchema;

Pushing the schema

We're trying to mimic production patterns as much as possible, so we're not going to restart the server to apply this change (and in fact, that would cause problems, as we'll soon see). Instead let's use a new command:

triplit schema push

This will look at the schema defined at ./triplit/schema.ts and attempt to apply it to the server while it's still running. In our case, it fails, and we get an error like this:

✖ Failed to push schema to server
Found 1 backwards incompatible schema changes.
Schema update failed. Please resolve the following issues:

Collection: 'todos'
        'tagIds'
                Issue: added an attribute where optional is not set
                Fix:   make 'tagIds' optional or delete all entities in 'todos' to allow this edit

What's at issue here is that we tried to change the shape/schema of a todo to one that no longer matches those in the database. All attributes in Triplit are required by default, and by adding a new attribute without updating the existing todos, we would be violating the contract between the schema and the data.

Thankfully, the error gives us some instructions. We can either

1. Make tagId optional e.g. tagIds: S.Optional(S.String()) and permit existing todos to have a tagId that's undefined.
2. Delete all of the todos in the collection so that there isn't any conflicting data.

While 2. might be acceptable in development, 1. is the obvious choice in production. In production, we would first add the attribute as optional, backfill it for existing entities with calls to client.update, as well as upgrade any clients to start creating new todos with tagId defined. Only when you're confident that all clients have been updated to handle the new schema and all existing data has been updated to reflect the target schema, should you proceed with a backwards incompatible change.

Whenever you try to triplit schema push, the receiving database will run a diff between the current schema and the one attempting to be applied and surface issues like these. Here are all possible conflicts that may arise.

Fixing the issues

Let's make tagId optional:

export const schema = {
  todos: {
    schema: S.Schema({
      id: S.Id(),
      text: S.String(),
      completed: S.Boolean({ default: false }),
      tagId: S.Optional(S.String()),
    }),
  },
} satisfies ClientSchema;

Now we can run triplit schema push again, and it should succeed. For completeness, let's also backfill the tagId for existing todos:

await client.transact(async (tx) => {
  const allTodos = await tx.fetch(client.query('todos').build());
  for (const [_id, todo] of allTodos) {
    await tx.update('todos', todo.id, (entity) => {
      entity.tagId = 'chores';
    });
  }
});

We've now successfully updated our schema in a backwards compatible way. If you're confident that all clients have been updated to handle the new schema and all existing data has been updated to reflect the target schema, you can then choose to make tagId required.

Handling backwards incompatible changes

Adding an attribute where optional is not set

Like in the example above, these changes will be backwards incompatible if you have existing entities in that collection. In production, only add optional attributes, and backfill that attribute for existing entities.

Removing a non-optional attribute

This is a backwards incompatible change, as it would leave existing entities in the collection with a missing attribute. In production, deprecate the attribute by making it optional, delete the attribute from all existing entities (set it to undefined), and then you be allowed to remove it from the schema.

Removing an optional attribute

While not technically a backwards incompatible change, it would lead to data loss. In production, delete the attribute from all existing entities first (set it to undefined) and then it will be possible to remove it from the schema.

Changing an attribute from optional to required

This is a backwards incompatible change, as existing entities with this attribute set to undefined will violate the schema. In production, update all existing entities to have a non-null value for the attribute, and then you will be able to make it required.

Changing the type of an attribute

Triplit will prevent you from changing the type of an attribute if there are existing entities in the collection. In production, create a new optional attribute with the desired type, backfill it for existing entities, and then remove the old attribute following the procedure described above ("Removing an optional attribute").

Changing the type of a set's items

This is similar to changing the type of an attribute, but for sets. In production, create a new optional set attribute with the desired type, backfill it for existing entities, and then remove the old set following the procedure described above ("Removing an optional attribute").

Changing an attribute from nullable to non-nullable

Triplit will prevent you from changing an attribute from nullable to non-nullable if there are existing entities in the collection for which the attribute is null. In production, update all of the existing entities to have a non-null value for the attribute and take care that no clients will continue writing null values to the attribute. Then you will be able to make it non-nullable.

Changing a string to an enum string or updating an enum

Triplit will prevent you from changing a string to an enum string or updating an enum if there are existing entities in the collection with values that are not in the new enum. In production, update all of the existing entities to have a value that is in the new enum and then you will be able to make the change.

Removing a relation

Because relations in Triplit are just views on data in other collections, removing a relation will not corrupt data but can still lead to backward-incompatible behavior between client and server. For instance, if the server's schema is updated to remove a relation, but an out-of-date client continues to issues queries with clauses that reference that relation, such as include, a relational where filter, or an exists filter, the server will reject those queries. In production, you may need to deprecate clients that are still using the old relation and force them to update the app with the new schema bundled in.