Skip to content

Surface Overlay

A surface is the texture of a mesh. That could be the screen of a TV, a monitor, a billboard, a painting. A surface overlay replaces that texture with live content rendered by CEF, the same web engine that hosts Web UIs. Instead of a static image, the mesh shows a web page, which could be a video, a dashboard, an embedded web app or anything else a browser can render.

A television in the game world playing a video via a surface overlay

The API is split into two parts:

  • A surface selects where the content goes. It is either an entity in the world plus the name of the mesh component on it, or a mesh node of the static world, like a billboard
  • A provider is the browser that produces the content. It keeps track of the URL, resolution, frame rate and input

You bind a provider to a surface to make the surface show the provider's page. One provider can supply several surfaces at once, so the same page can appear on a row of monitors from a single browser.

Finding a surface

An entity surface is addressed by the identity of the entity in the client's game world and the name of the mesh component on it. A world mesh node is addressed by its node key. All of them come from the Surface Picker, a client-side developer tool reached through the console's Tools → Rendering -> Surface Picker menu.

The Surface Picker devtool, listing the surfaces under the crosshair, with the selected billboard highlighted and registered as an overlay target

Point the crosshair at the object you want to target and press Pick under crosshair. The picker lists every surface along that line, closest exact hits first:

  • Distance to the surface and Hit: mesh when the line hits the mesh itself, box when it only passes through its bounding box
  • Input tells whether mouse input can be mapped onto the surface, see Input. ui collision and flat surfaces take mouse input, no input surfaces only keyboard input
  • Surface is the class and identity of the entity or mesh node, and the mesh component

The selected row is highlighted in the world. Register selected turns it into an overlay target so you can preview it, Remove selected drops it again.

The Overlay targets list at the bottom shows every surface that is currently registered, with its identity, component and canvas. Apply colour fills it with the picked colour, Remove unregisters it, and Copy API call creates and stores the required API call in your clipboard for convenient pasting into your code editor. Targets are marked [CEF] once a provider is attached to them, and those cannot be changed or removed from the picker.

Info

Not all identities are "stable" across restarts. Static entities like the computer monitor in the example are, but if you replace a surface on a vehicle for example, make sure to resolve the dynamic ent.EntityId every time you apply the overlay

Node keys are stable across restarts, but change if a game patch moves the node or swaps its mesh

Creating and binding

Create the surface first, then the provider, then bind them:

const ulong identity = 0x7F314E8F10B6BCB2; // from the Surface Picker

var surface = re.Ui.CreateSurfaceOverlay(identity, new CName("tv"));
var provider = re.Ui.CreateOverlayProvider(
    "https://example.com",
    1280,
    720,
    true
);

if (surface == 0 || provider == 0)
{
    Console.WriteLine("Surface overlay could not be created");
    return;
}

re.Ui.BindOverlayProviderToSurface(provider, surface);

Both creation calls return a handle, or 0 on failure. Failures are logged, so check the client log when a surface doesn't change.

Surfaces

re.Ui.CreateSurfaceOverlay(identity, component) registers a surface on an entity and returns its handle. The canvas starts as a placeholder until a provider is bound to it, so a registered surface with no provider shows a debug pattern rather than the original texture.

  • Passing an empty component name targets every mesh component on the entity
  • A surface belongs to the bundle that created it. Asking for the same entity and component again from the same bundle returns the same handle. Another bundle gets 0, and so does a surface that overlaps an existing one (the same entity with an empty component on either side)
  • re.Ui.ReleaseSurfaceOverlay(surface) unregisters it. The mesh goes back to its original texture

re.Ui.CreateNodeSurfaceOverlay(node) does the same for a mesh node of the static world, using the node key from the picker:

var billboard = re.Ui.CreateNodeSurfaceOverlay(0xF6A8B3FCBE07B53D); // from the Surface Picker

The overlay is applied whenever the node streams in, so the surface can be created while the node is far away. Instanced meshes, where many copies of a mesh share one render proxy, are not supported.

re.Ui.HasSurfaceOverlay(identity, component) checks whether a surface is already taken, by any bundle. Use it to pick a free screen before creating one:

if (re.Ui.HasSurfaceOverlay(identity, new CName("tv")))
{
    return; // another bundle already shows something on this screen
}

A surface registered without a component covers every component of the entity, so it is reported for any component name.

Providers

re.Ui.CreateOverlayProvider(url, width, height, sixty_fps) starts a browser and returns its handle.

  • url must be an https:// URL
  • width and height are the canvas resolution. The surface is rendered at this size, up to 8192 pixels per axis. A 1280×720 or 1920×1080 canvas is a good default for a screen, oversizing costs memory for nothing
  • sixty_fps picks the frame rate. true renders at 60 fps, false at 30 fps. Use 30 for static pages and 60 for video
  • Audio the page plays is spatialized at the surface, so a video's sound comes out of the object you bound it to rather than "the player's head"

re.Ui.ReleaseOverlayProvider(provider) unbinds it from every surface it was supplying and closes the browser.

Binding

re.Ui.BindOverlayProviderToSurface(provider, surface) connects the two and makes the surface show the provider's page. It resizes the surface canvas to the provider's resolution.

  • A surface can have one provider at a time. Binding a second provider to an already bound surface fails
  • A provider can supply several surfaces at once. Every surface bound to it shows the same page at the same resolution
  • re.Ui.UnbindOverlayProviderFromSurface(provider, surface) detaches one surface and returns it to its placeholder. The provider keeps running

Changing the page

re.Ui.NavigateOverlayProvider(provider, url) points an existing provider at a different page.

re.Ui.NavigateOverlayProvider(provider, "https://example.com");

The same https:// restriction applies. The call returns false if the provider is unknown or the URL is rejected. Navigating does not resize the canvas, so the new page is fitted into the existing surface resolution.

Input

A provider doesn't receive input until you route it there with re.Ui.SetOverlayProviderInputPolicy. The flags work like the ones of re.Ui.SetInputPolicy for ReUi:

// The player looks at the screen and uses it with mouse and keyboard
re.Ui.SetOverlayProviderInputPolicy(
    provider,
    accept_input: true,     // forward keyboard and mouse input to the page
    exclusive_input: true,  // keep that input from reaching the game
    cursor_active: true     // show the cursor and aim with it
);

// Hand control back to the game
re.Ui.SetOverlayProviderInputPolicy(provider, false, false, false);
  • Mouse input is projected into the world, from the cursor when cursor_active is set, otherwise from the center of the screen (the crosshair). It goes to the page of the surface it hits, at the point it hits. A button held down keeps sending to the same page until it is released
  • Keyboard input goes to every provider that accepts input, without projection. Only enable accept_input on the provider the player is using
  • With exclusive_input, the game doesn't receive the input at all, also while the player looks away from the surface. Bind it to a hotkey or an interaction so the player can always get out again
  • The call returns false if the provider is unknown or belongs to another bundle

Mouse input can only be mapped onto surfaces the picker lists as ui collision (game screens, which carry their own touch area) or flat (thin meshes like billboards and posters). Other meshes still show the page and receive keyboard input, but no mouse input. There is no occlusion test, so a surface also receives mouse input through walls in front of it.

Ownership and lifetime

Surface overlays belong to the bundle that created them. That has three consequences:

  • Only the owner can interact with them. Release, bind, unbind, navigate and the input policy all verify that the calling bundle created the handle. Another bundle's handle returns false, so handles cannot be hijacked or released by accident
  • Unloading the bundle cleans up after it. When your client-side component unloads, every provider and surface it still owns is released automatically. Surfaces return to their original texture and browsers are closed. You do not need to undo them by hand on unload, though releasing explicitly as soon as you are done is still better. A "leaked" provider holds one of the limited slots

Limits

Each client enforces the following:

Limit Value Behaviour when exceeded
Overlay providers 16 at a time Creating another returns 0
Provider dimension 8192 pixels per axis Creating another returns 0
Registered surfaces Shared canvas pool, 32 entries Registering or binding another returns 0

Sixteen browsers on one client is a lot. Each one has its own renderer process, memory footprint and CPU cost, so try to keep it to a minimum. Prefer one provider bound to many surfaces over one provider per surface, and drop providers you no longer need.

Because surface canvases come out of a shared pool, a bundle that registers many surfaces can leave later registrations from any bundle failing. The picker's Overlay targets list is the quickest way to find out what is currently registered.

Content and caching

The page runs in the same sandboxed browser environment as ReUi. It can load remote content and use the usual web platform features, and its audio is routed into the game world.

Overlay content is cached per server, separately from your bundle's ReUi and separately from other servers. Cookies, localStorage and the HTTP cache therefore persist between sessions on the same server and do not leak into another.