In Moving to a server-aware approach for collaboration, @chriszarate explains why the server needs to be at the heart of real-time collaboration, and introduces the engines we’ve been investigating to power the future of RTC. In this post, we’ll explore each of the sync engines built for RTC and the different ways they achieve collaboration.

All engines currently live in an experimental plugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party., gutenberg-sync-engines, which you can run right now to evaluate the engines below. Our goal is to assess engine choices with your help, and choose the next engine foundation for server-side RTC in WordPress.

This post is a guide for testers and an introduction to the concepts of each engine. We’d love for you to try each engine and help us evaluate which best matches an ideal future for RTC in WordPress.

Terms

Here are some common technical terms used in this post worth explaining:

• A sync engine is logic for combining changes when multiple people edit the same post. The three sync engines available in gutenberg-sync-engines are yjs-server, DE-RTC, and intent-log.
• A transport is the way changes travel between a browser and the server. These include a combination of HTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. polling, WebRTC, WebSockets, and Server-sent events (SSE). Any engine works with any transport.
• Any change that is set aside for review is held somewhere but not applied to the post. This can apply to merge conflicts and changes blocked for security reasons. When a change is set aside for review, a person is required to decide whether to accept it.

Setup

Sync engines and transports are chosen under Settings → Collaboration:

The requirements for running the plugin are WordPress 7.0+ and the Presence API plugin, which is used to keep track of which users are on which post. The plugin ships an internal (lightly modified) copy of Gutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ in order to make the sync engine swappable, so it will not work with another active Gutenberg plugin.

To try out the engines, follow the setup steps in the README. Select a sync engine and transport, and start editing a post in multiple browsers.

Engines: What do they have in common?

In all engines, every change is processed through WordPress. When an edit arrives, the server confirms who sent it, checks they are allowed to make it, validates it for unsafe code, determines how to merge it with everyone else’s changes, and stores the result. The differences between engines are the implementations of these steps. Every engine allows multiple users to edit at the same time, works with every transport, and shows who is editing a post in varying levels of detail.

Engine 1: yjs-server

Summary: Built on the conflict A conflict occurs when a patch changes code that was modified after the patch was created. These patches are considered stale, and will require a refresh of the changes before it can be applied, or the conflicts will need to be resolved.-free replicated data type (CRDT) library Yjs, the same technology used for client-side RTC in the current Gutenberg experiment. Every browser uses the Yjs library to merge changes into a document, sending the result to the server. The server verifies changes before they are shared with other users.

Yjs is an open-source and widely used CRDT implementation. In Yjs, two copies of a document edited separately can always be combined, in any order, and both end up identical. There are no merge conflicts. The server implementation runs a PHP version of Yjs to verify changes before passing them on to other users.

Engine: yjs-server
Transport: Polling with a WebRTC advisory channel

How your change travels to peers: You type a word. Your browser applies it locally to a copy of the editor document and sends a binary update to the server. The server opens the shared copy of the post, verifies the update, and saves it. Everyone else receives the update over the engine transport and applies it to their local copy of the document.

Strengths

• The most battle-tested library for collaboration. Yjs is used by many products, and has also received the most testing in Gutenberg.
• No merge conflicts. Yjs has a deterministic algorithm for combining edits, which means there’s never an edit that conflicts with the document.
• The most “real time” collaboration. Yjs updates are fast and small (no matter the size of a document), communicate cursor-level presence, and are designed for a real-time feel.

Weaknesses

• Yjs “hides” conflicts by choosing a winner itself. If two collaborators change a block Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. setting at the same time, one user silently wins. If two people rewrite the same sentence at the same time, it’s possible for the result to be a mix that nobody wrote individually. Nobody is warned about conflicts, because the engine does not treat them as conflicts.
• It communicates in compressed binary messages. Instead of “User A added ‘Lorem ipsum’ to paragraph 1”, Yjs messages look like { "type": "update", "data": "AAAG+/m3og8CA4IERAE0A6gAx…" }. This is efficient for update size, but can make debugging and understanding history more difficult.
• Server agents cannot easily participate in collaboration. Outside tools that alter post content directly can’t easily interact with live collaboration, which can make it possible for an editor’s working copy and post content in the database to diverge.
• Two implementations. The yjs-server engine requires matching JS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. and PHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 7.4 or higher implementations because the browser is in charge of merging changes, but the server must also validate. The implementations are kept matching by shared tests, but a divergence between them would cause issues. Protocol updates must be made on server and client at the same time.

Engine 2: DE-RTC

Summary: Most similar to single-user WordPress. People share and merge changes only when they save. The server combines incoming saves with everyone else’s, and produces merge conflicts when there is an overlap. This engine is not designed for Google Docs-style real-time editing, preferring a slower update cadence. Because changes are made via save, external plugins, scripts, or WP-CLI WP-CLI is the Command Line Interface for WordPress, used to do administrative and development tasks in a programmatic way. The project page is http://wp-cli.org/ https://make.wordpress.org/cli/ commands that change post content can take part almost as easily as human edits.

The “DE” in DE-RTC stands for Distributed Editing. @dmsnell proposed the design in WordPress/wordpress-develop#12334 and WordPress/gutenberg#79602, and we’ve ported it into the sync engines plugin. It follows a more traditional WordPress model: if saving is how people share work, then anything that can save a post can take part, including tools that are not browsers. Although DE-RTC can send advisory data over every transport, post data updates are sent via regular saves on the configured save cadence.

Engine: DE-RTC with 10 second save cadence

How your change travels to peers: You make some changes. Every ten seconds (default), your browser sends the whole post to the server through WordPress’s normal autosave. The server merges based on three copies of the post: the version you started from, your updated version, and the current shared version on the server. Changes that do not overlap are combined automatically. When two people change a block in a way that conflicts, that block is set aside for review.

Engine: DE-RTC with 10 second save cadence. Merge dialog improvements are in progress. Strengths

• Outside tools can easily take part. A plugin, script, or agent that changes post content can collaborate with a live editor instead of replacing their work.
• One implementation for merging, owned by the server. Browser logic is simple and sends the whole post to the server, and the server is in charge of making merges. Unlike yjs-server and intent-log, there is no required duplication of logic between browser and server.
• Great for small, low-cost hosting. A few larger requests over a long interval are usually more friendly than constant polling on resource-constrained hosts.

Weaknesses

• It is not real-time. You see other people’s changes on the save interval, not author cursors with live updates. The save interval can be adjusted, but DE-RTC scales best with infrequent updates.
• Each save sends the whole post, so updates get bigger as the post grows.
• Outside tools must say which version they started from. For outside post edits to merge with folks in the editor, tools are required to say which post version they started from (e.g. Base-version: 123). This is simpler than the other engines, but requires tools to be RTC-aware.
• The server does not know what you meant. DE-RTC only sees content before and after a save, not “I split this paragraph.” Large post changes made together, like splitting or rearranging blocks, are more often set aside for review.

Engine 3: intent-log

Summary: The browser sends messages describing what a person did in Gutenberg terms, and the server is in charge of keeping a log. For example “Split block 1 at index 20”, then “Add text ‘Lorem Ipsum’ to block 2”. Messages explain changes in human terms. Simultaneous changes are combined via operational transform. Incompatible changes cause conflicts like DE-RTC.

Engine: intent-log
Transport: Polling (with 10 second cadence to allow time to edit)

How your change travels to peers: You type some text. Your browser describes it: “insert ‘Lorem ipsum’ at position 0 in block 1”, and sends the message to the server. The server adjusts your change to fit anything that arrived since (if a peer added three letters before yours, your insert moves three places over), adds it to the log, and sends back an answer. Conflicts are set aside for human merge review. Everyone else receives the official server-side log of actions after your changes are processed.

Example intent-log messages for the demo above:

// User B adds content to block 02f5e6c3
{
    "intentId": "...",
    "baseSeq": 585,
    "type": "insert_text",
    "payload":{
        "syncId": "02f5e6c3-2876-43a6-b041-6f3d0a217095",
        "field": "content",
        "offset": 42,
        "text": " in the editor."
    }
}

// At the same time, User A merges block 02f5e6c3 into block 8d979605
{
    "intentId": "...",
    "baseSeq": 585,
    "type": "merge_blocks",
    "payload":{
        "survivorId": "8d979605-e1d7-4cec-bc6e-4a3eaa6f30bc",
        "absorbedId": "02f5e6c3-2876-43a6-b041-6f3d0a217095",
        "field": "content",
        "joinOffset": 48
    }
}

Strengths

• The server understands what happened in human terms. In yjs-server, moving a block in a post can look like a rewrite of the post, because the engine has no context when diffing a change. In intent-log, a block move is a simple message (“Moved block 1 to position 3”). This allows for context-aware merge strategies and server-side understanding of changes.
• Intent-log keeps a human-readable record of the session. Every post has a built-in readable history of recent actions and who made them.
• Outside tools can join by saying which version they started from, similar to DE-RTC. External saves can be turned into intent-log edits and combined with others.

Weaknesses

• Intent-log can only process one update at a time per post. Intent-log uses a lock to manage incoming messages one at a time. Under heavy use within a single post, lock contention may be an issue and cause updates to sync more slowly.
• Operational transforms are difficult. Intent-log is built on operational transforms, which involve realigning concurrent edits so that they can be combined without conflict. This difficult logic still needs a lot of work in intent-log.
• Two implementations. The intent-log engine requires both a JS and PHP implementation because the browser must show your edits instantly as well as keep them in the right place as other changes arrive. This means intent-log runs the same logic in your browser (JS) and on the server (PHP) in order to avoid separation between the two logs. The implementations are kept matching by shared tests, but a divergence between them would cause issues. Protocol updates must be made on server and client at the same time.

Side by side

	yjs-server	DE-RTC	intent-log
Updates arrive quickly?	Yes	No	Yes
Live cursors in the text	Yes, Google Docs-style	No	Yes (not implemented yet)
What the browser sends	Binary updates	The whole post	Short descriptions of edits
Where does the merge logic live?	Server and browser	Server	Server (with browser preview)
Two people make a conflicting change	Combined silently, one person wins	Set aside for review	Set aside for review
Outside tools	Not merged with live session	Join by naming a version	Join by naming a version
Server work per change	Most, grows with post size	Middle, upload grows with post size	Least

Transports: How changes travel

We’re mostly concerned with engine selection in this post, but we have several transports available that all engines can support. We expect transport options to depend on hosting availability. Here’s a quick rundown of those transports:

Transport	How it works	What it needs
Polling	The browser sends local updates and asks the server for updates on a timer, every N seconds. Can be adjusted anywhere from one to 25 seconds (default 5).	No special infrastructure. Works on every host, and can be scaled at the cost of PHP workers.
Polling with WebRTC advisory channel	The browser sends local updates and asks the server for updates on a relaxed polling timer. However, when any peer makes a change, other peers are notified via WebRTC to immediately request an update from the server instead of waiting until the next poll cycle.	Browsers that can reach each other over WebRTC. When not possible, falls back to regular polling.
Server-sent events	The browser keeps one connection open to WordPress and receives updates through it continuously. Local updates are sent to the polling endpoint.	Hosting that allows long-open connections. One server process per open window, can consume PHP workers.
Server-sent events (daemon)	The same one-way connection as server-sent events, but run on a stable, separate process instead of a general PHP worker. Local updates are sent to the polling endpoint.	A server that can run a separate process, and hosting that allows long-open connections.
WebSocket	A connection open in both directions, run on a separate process. Changes arrive almost instantly.	A server that can run a separate WebSocket process.

Higher-fidelity transports are designed to be downgradable and fall back to polling. That means that if you choose a transport that relies on a separate service (such as the WebSocket daemon) and it stops responding, polling will always be available as a backup.

Call to action

Try it out and tell us what you think! When you test, please keep in mind that all three engines need polish and may have bugs. We want to hear about bugs, but care most about the major design differences between the engines and how they might affect your workflow. Are fast updates and live cursors a must for you? Do you prefer the simpler model of DE-RTC? How do you feel about the merge conflicts you encountered?

Report what you find on GitHub issues or in #feature-realtime-collaboration on WordPress Slack. For more in-depth information, see documentation in the repository:

• data-flow.md shows technical engine data flows on one page.
• engine-comparison.md compares engines in even more ways.

Please let us know if there’s anything we can do to improve your testing.