Introduction

You can now give Unreal NPC teammates spoken orders and have an LLM plan what they do, using Qualcomm's own open-source code. Snapdragon Game Studios published MindSand v1.0.0 on GitHub on November 10, and

the same day with footage from Starship Troopers: Extermination. MindSand is a behavior-engine plugin. It collects what your squad can see and do, sends that to a language model, and turns the reply into ordinary Unreal gameplay tasks. The code is BSD-3-Clause-Clear, and you can drop it into a UE 5.6+ project today.

What shipped

  • Product: MindSand Behavior Engine v1.0.0, an Unreal Engine plugin, plus a MindsandPlayground.zip sample project
  • Vendor: Qualcomm, through its Snapdragon Game Studios team
  • Modality: LLM-driven NPC behavior: natural-language commands in, multi-step action plans out, with optional speech-to-text and text-to-speech
  • Open vs API: open source under BSD-3-Clause-Clear. It has no model of its own, so you bring the LLM
  • Where to run it today: any Unreal Engine 5.6+ project, alongside the Snapdragon Game AI SDK plugins from snapdragon-game-plugins-for-unreal-engine
  • Concrete fact: the LLM layer has two backends. One is an HTTP provider for a cloud model (the quickstart's example config uses gemini-2.0-flash). The other runs on-device through Qualcomm's Genie SDK on Snapdragon NPUs, and VoiceAI ASR/TTS handles voice the same way

How it works

The plugin's README spells out the loop. On every command, MindSand bundles four things into a prompt:

  • Teammate state: each NPC's name, squad, personality and allowed actions
  • Perception: what Unreal's AIPerception sight sense reports
  • World context: tagged actors from an actor registry, each with a description and live state such as "injured" or "door open"
  • Plan state: what the squad is already doing

The model returns a plan, either as buffered JSON or as a streaming DSL. MindSand parses that plan into an ordered list of actions per teammate, and each action runs as a normal UGameplayTask. Your existing movement, combat and animation code still does the work. The LLM only decides what happens next and what the NPCs say. This lines up with Qualcomm's AI and Games Conference talk on Wednesday, November 11, about adding LLM squadmates to a shipped game without rewriting it.

Actions are data assets. You write a task in C++ or Blueprint, then create an Action Description Asset with a name, a plain-English description, conditions, effects and parameters. MindSand fills in the parameters from the task's UPROPERTY fields.

Try it

  1. Clone both repos with submodules: mindsand-engine, and snapdragon-game-plugins-for-unreal-engine on its engine/5.6 branch.
  2. Copy the MindSand plugin folder and Plugins/SGAI/ into your project's Plugins/ directory, then build.
  3. Point the [HttpLLMProvider] block in DefaultEngine.ini at your model endpoint, or switch to the Genie provider on Snapdragon hardware.
  4. Add TeammateCharacterAgentComponent and AIPerception to a teammate controller, TeammateCharacterAgentDescriptor to its pawn, and LeaderTeammateCharacterAgentComponent to the player controller.
  5. Open the Plan Injector editor tab to send hand-written plans during Play In Editor before you connect a live model.

If you'd rather start from something that already runs, the release's MindsandPlayground.zip sample project is about 2 GB.

Why it matters for game devs

Most "AI NPC" demos swap out the game's brain and hope for the best. MindSand works the other way round. Designers decide exactly which actions exist, the LLM picks from that list, and the engine carries the actions out. That keeps NPCs from inventing abilities, and it means a broken plan fails as an unknown action name instead of as weird behavior. The Plan Injector helps too: you can debug squad behavior without paying for tokens or waiting on a model.

The catch

This is v1.0.0 with no stars or forks yet, so expect rough edges. It's Unreal-only, it requires UE 5.6 or newer, and you still choose, host and pay for the LLM yourself. The on-device ASR, LLM and TTS paths target Snapdragon hardware. On a PC dev box you'll probably start with the HTTP cloud provider, which adds latency to every squad order.

—Titus