i-spent-three-months-learning-unity-sentis-and-here-is-what-it-can-actually-do-inside-a-real-game-build

Unity Sentis is the neural network inference engine built into Unity 6 and it is the most technically powerful and least-talked-about AI feature in the Unity ecosystem. I spent three months getting it working on real game AI behaviors — not tutorial demos, but actual in-game systems. This is the honest breakdown of what Sentis can do, what it cannot, the setup that took two weeks to get right, and whether a solo developer should bother.

🔧 Tools mentioned in this article

Unity Sentis

Neural network inference engine for Unity — free, included with Unity 6 via Package Manager

unity.com

Visit

Unity

Game engine — Personal plan free under $100k revenue, Pro $185/month (€170 / £146 / ₹15,370)

unity.com

Visit

Hugging Face

Model repository used to source ONNX models for Sentis — free, large library of exportable models

huggingface.co

Visit

ONNX

Open Neural Network Exchange format — the model format Sentis uses, free and open standard

onnx.ai

Visit

Marcus Webb

June 23, 2026

What Unity Sentis Actually Is

Unity Sentis is not an AI assistant. It is not a code generator. It is an inference engine — a system that lets you run pre-trained neural network models inside a Unity build at runtime. You bring an ONNX model (a trained neural network exported in the standard ONNX format), load it into Unity with Sentis, and run inference on it in your game loop. The model processes inputs — sensor data, image data, game state values — and returns outputs that your game logic acts on. This is how you get genuinely intelligent game AI behaviors: the kind that cannot be scripted with decision trees because the decision space is too complex.

Three Real Systems I Built With Sentis

• Vision-based enemy detection: Trained a small binary classifier on synthetic data (player visible / not visible from enemy perspective given scene geometry). Ran inference every 0.1 seconds in the enemy update loop. The enemy detected the player through partial occlusion correctly — something a simple raycast check misses when the player is half behind a wall. Training time on Google Colab free tier: 40 minutes. Integration time: 6 hours. In-game performance: inference runs in under 2ms per call on GPU backend.
• Adaptive difficulty classifier: Input features — player death count, average time to complete level sections, accuracy rating — fed into a three-class classifier (struggling / on target / breezing through). Output drove difficulty parameter adjustments in real time. More responsive than a manual threshold system and handles edge cases more gracefully. Training data: 200 simulated player session logs I generated with a simple script. This was the most immediately useful Sentis system I built.
• Gesture recognition for mobile: A sequence classifier that takes accelerometer input from the phone and classifies it as one of six gestures. Trained with 50 examples per gesture recorded from my own phone. Worked well for five of six gestures with over 90% accuracy on test data. The sixth (a circular motion) conflated with a figure-eight in testing. Useful proof of concept, needed more training data to be production-ready.

The Setup That Finally Worked

// Unity Sentis — Minimal Working Setup
// Package: com.unity.sentis (install via Package Manager > Add by name)
// Unity 6 LTS required

using Unity.Sentis;
using UnityEngine;

public class SentisInferenceExample : MonoBehaviour {
    [Header("Model")]
    // Drag your .sentis or .onnx model file here in the Inspector
    public ModelAsset modelAsset;

    [Header("Backend")]
    // GPU is faster; CPU is more compatible across target platforms
    public BackendType backend = BackendType.GPUCompute;

    private Model runtimeModel;
    private IWorker worker;

    void Start() {
        // Load and compile the model
        runtimeModel = ModelLoader.Load(modelAsset);

        // Create a worker (the inference engine instance)
        worker = WorkerFactory.CreateWorker(backend, runtimeModel);

        Debug.Log("Sentis model loaded. Input count: " + runtimeModel.inputs.Count);
    }

    // Call this to run inference with your feature data
    public float[] RunInference(float[] inputFeatures) {
        // Create input tensor from your feature array
        // Shape must match your model's expected input shape
        using var inputTensor = new TensorFloat(
            new TensorShape(1, inputFeatures.Length),
            inputFeatures
        );

        // Set the input and run inference
        worker.SetInput("input", inputTensor); // "input" = your model's input layer name
        worker.Schedule();

        // Read output tensor
        // "output" = your model's output layer name (check in Netron or training code)
        var outputTensor = worker.PeekOutput("output") as TensorFloat;
        outputTensor.CompleteOperationsAndDownload();

        float[] results = outputTensor.ToReadOnlyArray().ToArray();
        return results;
    }

    void OnDestroy() {
        // Always dispose worker and tensors to avoid memory leaks
        worker?.Dispose();
    }
}

// IMPORTANT NOTES:
// 1. Convert .onnx to .sentis format for better Unity integration:
//    Assets > Create > Sentis > Model Asset
//    Then drag your .onnx file into the Source Model field
//
// 2. Find your model's layer names:
//    Open your .onnx file in Netron (netron.app) — free browser tool
//    The first and last node names are your input/output names
//
// 3. BackendType options:
//    GPUCompute: fastest on GPU, requires compute shaders
//    GPUCommandBuffer: alternative GPU path, better iOS/Android compatibility
//    CPU: slowest, works everywhere
//
// 4. Where to get ONNX models for games:
//    - Train your own with PyTorch/TensorFlow and export to ONNX
//    - Hugging Face model hub: huggingface.co (filter by ONNX format)
//    - Unity ML-Agents: train and export agents directly as ONNX

What Sentis Cannot Do That I Expected It To

• Run large language models in-game: The tempting idea of an LLM running inside a Unity build for NPC dialogue is not practical with Sentis. Even quantised small LLMs are too large and too slow for real-time game use on consumer hardware. Sentis is for small, fast, purpose-specific models — not general language intelligence.
• Generate training data for you: You bring the trained model. Sentis only handles inference. All model training happens outside Unity in Python using PyTorch, TensorFlow, or similar. If you do not want to train your own models, Sentis is only as useful as the pre-trained ONNX models you can find on Hugging Face or elsewhere.
• Replace Unity ML-Agents: Unity ML-Agents is the training framework for reinforcement learning agents in Unity. Sentis is the inference engine that runs the trained result. They are complementary. ML-Agents trains inside the Unity environment; the result is an ONNX model that Sentis runs in a shipped build.
• Handle model sizes above roughly 50MB efficiently: I tested a 120MB vision transformer for object recognition. Frame rate dropped to unplayable on mobile hardware. Sentis works best with small models — under 20MB for mobile, under 50MB for PC — purpose-trained for a specific narrow task.

Mistakes That Cost Two Weeks

• Mistake 1: Using the wrong backend for the target platform — built and tested on GPU backend, deployed to an Android device that did not support compute shaders. The model ran on CPU fallback at unacceptable speed. Always test on target hardware with the correct BackendType before designing systems around performance assumptions.
• Mistake 2: Not checking model layer names before writing inference code — wrote the inference code with assumed layer names 'input' and 'output'. The model used 'input_1' and 'dense_1'. Spent half a day debugging a NullReferenceException that was a two-character fix. Use Netron (netron.app) to inspect your model before writing any integration code.
• Mistake 3: Not disposing tensors in a loop — memory leaked steadily during the enemy detection inference loop because I forgot to wrap tensor creation in using blocks. Frame rate dropped over 10 minutes of play. The using pattern shown in the code above prevents this.
• Mistake 4: Training on too little data — the gesture recognition model trained on 50 examples per gesture worked in my own testing and failed for three other people whose motion style differed. Machine learning models require diverse training data. 50 examples from one person is not a dataset.
• Mistake 5: Expecting Sentis to work identically on all Unity versions — I started on Unity 6 preview and some API names changed by the time Unity 6 LTS shipped. If you are learning Sentis from older tutorials, check the Package Manager for the version you have installed and read the changelog for breaking changes.

Should a Solo Developer Learn Unity Sentis

• Learn Sentis if: you want AI behaviors in your game that genuinely cannot be scripted — adaptive systems, sensor-based perception, gesture input, or player behavior modeling. The learning curve is real but the capability ceiling is higher than any scripted system.
• Skip Sentis for now if: you are still learning Unity fundamentals, your game AI needs are satisfied by NavMesh and state machines, or you do not have Python experience to train and export models.
• The time investment estimate: functional setup and first working inference took me about 8 hours. Building the first useful game system on top of that took another 20 hours. Total: roughly 28 hours from zero to a Sentis-powered system in a real project.

Final Thoughts

Unity Sentis is the most technically significant AI feature in Unity 6 and the one that takes the most effort to actually use. The free cost is not the real barrier — the real barrier is that using it requires machine learning knowledge outside of Unity. For developers who have or want to develop that knowledge, Sentis unlocks game AI behaviors that simply are not achievable with conventional scripted systems. For developers who want AI in their games without machine learning knowledge, Unity Muse Behavior and NPC platforms like Inworld are more accessible starting points.