my unity sentis experiment running a real neural network inside a game build and whether it was worth the learning curve
Unity Sentis lets you run actual trained neural networks inside a Unity build at runtime. I spent six weeks trying to understand this well enough to use it for something real in my game. Not a demo. An actual in game system. This is what I built, how long it took to understand the tooling, where I got completely stuck, and my honest opinion on whether a solo developer should spend time learning it.
Marcus Webb
June 24, 2026
Six weeks, one completed system. What I built: an adaptive difficulty classifier that reads five player behavior signals every 30 seconds and adjusts enemy health, damage, and patrol radius accordingly without any manual threshold tuning. The model is a three class classifier I trained in 40 minutes on Google Colab. The integration into Unity took two weeks to get right. Sentis costs nothing, the Colab training was free, and the in game inference runs at under 2 milliseconds per call on GPU backend.
What Made Me Want to Try This
I had been building adaptive difficulty with if else logic for two projects and the result always felt mechanical. When the player dies three times in a row the enemies get weaker. When they clear a section without taking damage the enemies get stronger. It worked but it was not subtle. A player who was struggling on defence but thriving on offence would get the wrong adjustment because the system was looking at one signal at a time. I wanted something that could look at five signals simultaneously and make a more nuanced decision. That is a classification problem. Sentis is how you run a classifier inside Unity.
What Sentis Actually Is and What I Did Not Understand at First
I spent the first week thinking Sentis was some kind of Unity AI system I could configure with settings. It is not. Sentis is an inference engine. It runs a neural network model that you bring from outside Unity. The model is trained separately in Python, exported to the ONNX format, and then loaded into Unity where Sentis runs inference on it at runtime. If you do not know what any of that means you have two to three days of learning before you can meaningfully use Sentis. That is not a criticism. It is information I wish I had on day one rather than day eight.
The System I Built Step by Step
- Step 1, defining the input signals: I chose five features to track per player session. Deaths in last 5 minutes. Average time to complete each room. Accuracy on ranged attacks. Damage taken per enemy encounter. Boss attempts before success. Each of these is a float value I already track in my GameManager.
- Step 2, creating training data: I wrote a script that simulated 300 player sessions across three skill levels. Struggling players had high death counts, low accuracy, and high damage taken. Skilled players had the inverse. I generated this data as a CSV in Python in about an hour.
- Step 3, training the model in Google Colab: A simple three layer neural network trained on the 300 simulated sessions in about 40 minutes on Colab's free GPU. Exported to ONNX format. The model file is 18 kilobytes.
- Step 4, importing into Unity via Sentis: Dragged the ONNX file into Unity, created a Model Asset, wrote the inference controller script. This step took two weeks because I kept hitting the same three problems that are documented in detail below.
- Step 5, connecting inference output to difficulty parameters: The model outputs three probability values. Struggling, On Track, Breezing. I take the highest probability class and apply a corresponding difficulty preset. Smooth and responsive with no arbitrary thresholds.
The Three Problems That Took Two Weeks to Solve
- Problem 1, wrong backend for my target platform: I built and tested everything on my desktop PC using the GPU Compute backend. The first time I tried to run a build on a secondary test laptop that had an older GPU, Sentis silently fell back to CPU inference and the inference time jumped from under 2 milliseconds to 180 milliseconds. The fix was adding a backend capability check at startup and falling back to CPU backend deliberately rather than letting Sentis do it silently. Now I test the build on target hardware before calling a feature done.
- Problem 2, tensor memory leaks: My inference ran once every 30 seconds so the memory leak was slow. After 20 minutes of play the frame rate was noticeably lower. The cause was not wrapping my TensorFloat in a using block. Every inference call was creating a tensor and never disposing it. The fix is one line of code. The leak took three days to diagnose because it was slow enough that I thought it was something else.
- Problem 3, layer name mismatch: My C# code referenced the model output layer as output. The actual layer name in the exported ONNX model was dense_2. This produced a NullReferenceException that pointed to my inference code rather than to the layer name problem. I found the correct layer name using a free browser tool called Netron that lets you inspect ONNX models. Took 10 minutes to fix once I knew what to look for. Took four hours to find.
Is It Worth Learning for a Solo Developer
- Worth it if your game has a system that makes decisions based on multiple inputs simultaneously and you want that decision making to be more nuanced than if else chains. Adaptive difficulty, NPC behavior that learns from player patterns, and gesture recognition are all genuine use cases.
- Worth it if you are comfortable with or willing to learn basic Python for model training. The Unity integration is C# and manageable. The model training side requires Python and some familiarity with PyTorch or scikit learn.
- Not worth it if you want results quickly. Six weeks to a working system is not a fast path. If your game needs to ship in two months this is not the right investment right now.
- Not worth it as a replacement for scripted AI. NavMesh, state machines, and behavior trees handle most game AI needs efficiently. Sentis is for the cases where those tools genuinely fall short.
Final Thoughts
Six weeks was a long time to spend on one system. The result is a difficulty adjustment mechanism that genuinely feels more natural than anything I built with explicit thresholds. Players in my current playtest sessions are not noticing the difficulty changing, which means it is working. The learning curve was real and the three problems I documented above cost most of the time. Sentis is worth learning if you have a specific problem that needs it. It is not worth learning as a general skill to have before you know what you would use it for.