Godot LLM

Isn't it cool to utilize large language model (LLM) to generate contents for your game? LLM has great potential in NPC models, game mechanics and design assisting. Thanks for technology like llama.cpp, "small" LLM, such as llama-3-8B, run reasonably well locally on lower-end machine without a good GPU. I want to experiment LLM in Godot but I couldn't find any good library, so I decided to create one here.

⚠ While LLM is less controversial than image generation models, there can still be legal issues when LLM contents are integrated in games, I have created another page to document some relevant information

Table of Contents

1. Quick start
   • Install
   • Text Generation: GDLlama node
   • Text Embedding: GDEmbedding node
   • Multimodal Text Generation: GDLlava node
   • Vector Database: LlmDB node
   • Template/Demo
2. Retrieval Augmented Generation (RAG)
3. Roadmap
4. Documentation
   • Inspector Properties: GDLlama, GDEmbedding, and GDLlava
   • GDLlama Functions and Signals
   • GDEmbedding Functions and Signals
   • GDLlava Functions and Signals
   • Text generation with Json schema
   • LlmDB Functions and Signals
   • LlmDBMetaData
5. FAQ
6. Compile from Source

Quick Start

Install

1. Get Godot LLM directly from the asset library, or download the vulkan or cpu zip file from the release page, and unzip it to place it in the addons folder in your godot project
2. Now you should be able to see GdLlama, GdEmbedding, GDLlava, and LlmDB nodes in your godot editor. You can add them to a scene in Godot editor, or initialize themm directly by .new().

Text Generation: GDLlama node

1. Download a supported LLM model in GGUF format (recommendation: Meta-Llama-3-8B-Instruct-Q5_K_M.gguf), move the file to somewhere in your godot project
2. Set up your model with GDScript, point model_path to your GGUF file. The default n_predict = -1 generates an infinite sequence, we want it to be shorter here

func _ready():
    var gdllama = GDLlama.new()
    gdllama.model_path = "./models/Meta-Llama-3-8B-Instruct.Q5_K_M.gguf" ##Your model path
    gdllama.n_predict = 20

1. Generate text starting from "Hello"

    var generated_text = gdllama.generate_text_simple("Hello")
    print(generated_text)

1. Text generation is slow, you may want to call gdllama.run_generate_text("Hello", "", "") to run the generation in background, then handle the generate_text_updated or generate_text_finished signals

    gdllama.generate_text_updated.connect(_on_gdllama_updated)
    gdllama.run_generate_text("Hello", "", "")

func _on_gdllama_updated(new_text: String):
    print(new_text)

Text Embedding: GDEmbedding node

1. Download a supported embedding model in GGUF format (recommendation: mxbai-embed-large-v1.Q5_K_M.gguf), move the file to somewhere in your godot project
2. Set up your model with GDScript, point model_path to your GGUF file

func _ready():
    var gdembedding= GDEmbedding.new()
    gdembedding.model_path = "./models/mxbai-embed-large-v1.Q5_K_M.gguf"

1. Compute the embedded vector of "Hello world" in PackedFloat32Array

    var array: PackedFloat32Array = gdembedding.compute_embedding("Hello world")
    print(array)

1. Compute the similarity between "Hello" and "World"

    var similarity: float = gdembedding.similarity_cos_string("Hello", "World")
    print(similarity)

1. Embedding computation can be slow, you may want to call gdembedding.run_compute_embedding("Hello world") or gdembedding.run_similarity_cos_string("Hello", "Worlld") to run the computation in background, then handle the compute_embedding_finished and similarity_cos_string_finished signals

    gdembedding.compute_embedding_finished.connect(_on_embedding_finished)
    gdembedding.run_compute_embedding("Hello world")

func _on_embedding_finished(embedding: PackedFloat32Array):
    print(embedding)

    gdembedding.similarity_cos_string_finished.connect(_on_embedding_finished)
    gdembedding.run_similarity_cos_string("Hello", "Worlld")

func _on_similarity_finished(similarity: float):
    print(similarity)

Note that the current implementation only allows one thread running per node, avoid calling 2 run_* methods consecutively:

    ## Don't do this, this will hang your UI
    gdembedding.run_compute_embedding("Hello world")
    gdembedding.run_similarity_cos_string("Hello", "Worlld")

Instead, always wait for the finished signal or check gdembedding.is_running() before calling a run_* function.

Multimodal Text Generation: GDLlava node

1. Download a supported multimodal model in GGUF format (recommendation: llava-phi-3-mini-int4.gguf), be aware that there are two files needed - a gguf language model and a mmproj model (typical name *mmproj*.gguf), move the files to somewhere in your godot project
2. Set up your model with GDScript, point model_path and mmproj_path to your corresponding GGUF files

func _ready():
    var gdllava = GDLlava.new()
    gdllava.model_path = "./models/llava-phi-3-mini-int4.gguf"
    gdllava.mmproj_path = "./models/llava-phi-3-mini-mmproj-f16.gguf"

1. Load an image (svg, png, or jpg, other format may also works as long as it is supported by Godot), or use your game screen (viewport) as a image

    var image = Image.new()
    image.load("icon.svg")

    ## Or load the game screen instead
    #var image = get_viewport().get_texture().get_image()

1. Generate text to provide "Provide a full description" for the image

    var generated_text = gdllava.generate_text_image("Provide a full description", image)
    print(generated_text)

1. Text generation is slow, you may want to call gdllama.run_generate_text("Hello", "", "") to run the generation in background, then handle the generate_text_updated or generate_text_finished signals

    gdllava.generate_text_updated.connect(_on_gdllava_updated)
    gdllava.run_generate_text_image("Provide a full description", image)

func _on_gdllava_updated(new_text: String):
    print(new_text)

Vector database: LlmDB node

1. LlmDB node extends GDEmbedding node, follow the previous section to download a model and set up the model_path

func _ready():
    var db = LlmDB.new()
    db.model_path = "./models/mxbai-embed-large-v1.Q5_K_M.gguf"

1. Open a database, which creates a llm.db file and connect to it by default

    db.open_db()

1. Set up the structure of the metadata of your textual data, the first metadata should always be an id field with String as the data type, here we use the LlmDBMetaData.create_text, LlmDBMetaData.create_int, and LlmDBMetaData.create_real functions to define the structure of metadata with the corresponding data type.

    db.meta = [
        LlmDBMetaData.create_text("id"),
        LlmDBMetaData.create_int("year"),
        LlmDBMetaData.create_real("attack")
    ]

1. Different models create embedding vectors of different sizes, calibrate the embedding_size property before creating tables

    db.calibrate_embedding_size()

1. Create tables based on the metadata, By default, these table are created:

2. llm_table_meta: which store the metadata for a particular id

3. llm_table: store texts with metadata and embedding
4. Some tables with names containing llm_table_virtual: tables for embedding similarity computation

Note that your .meta property should always match the metadata columns in the database before any storing or retrieving operation, consider setting your .meta property within the _ready() function or within the inspector.

    db.create_llm_tables()

1. Store a piece of text with metadata dictionary specifying the year, note that you can leave out some of the metadata if it is not relevant to the text. If the input text is longer than chunk_size, the function will automatically break it down into smaller pieces to fit in the chunk_size.

    var text = "Godot is financially supported by the Godot Foundation, a non-profit organization formed on August 23rd, 2022 via the KVK (number 87351919) in the Netherlands. The Godot Foundation is responsible for managing donations made to Godot and ensuring that such donations are used to enhance Godot. The Godot Foundation is a legally independent organization and does not own Godot. In the past, the Godot existed as a member project of the Software Freedom Conservancy."

    db.store_text_by_meta({"year": 2024}, text)

1. Retrieve 3 of the most similar text chunks to godot where the year is 2024:

    print(db.retrieve_similar_texts("godot", "year=2024", 3))

1. Depending on the embedding model, storing and retrieving can be slow, consider using the run_store_text_by_meta function, run_retrieve_similar_texts function, and the retrieve_similar_text_finished signal to store and retrieve texts in background. Also, call close_db() when the database is no longer in use.

Template/Demo

The godot-llm-template provides a rather complete demonstration on different functionalities of this plugin

Retrieval-Augmented Generation (RAG)

This plugin now has all the essentaial components for simple Retrieval-Augmented Generation (RAG). You can store information about your game world or your character into the vector database, retrieve relevant texts to enrich your prompt, then generate text for your game, the generated text can be stored back to the vector database to enrich future prompt. RAG complement the shortcoming of LLM - the limited context size force the model to forget earlier information, and with RAG, information can be stored in a database to become long-term memory, and only relevant information are retrieve to enrich the prompt to keep the prompt within the context size.

To get started, you may try the following format for your prompt input:

Document:
{retrieved text}

Question:
{your prompt}

Roadmap

Features

• Platform (backend): windows (cpu, vulkan), macOS(cpu, metal), Linux (cpu, vulkan), Android (cpu)
  • macOS support is on best effort basis since I don't have a mac myself
• llama.cpp-based features
  • Text generation
  • Embedding
  • Multimodal
• Vector database integration based on sqlite and sqlite-vec
  • Split text to chunks
  • Store text embedding
  • Associate metadata with text
  • Retrieve text by embedding similarity and sql constraints on metadata

TODO

• iOS: build should be trivial, but an apple developer ID is needed to run thhe build
• Add in-editor documentation, waiting for proper support in Godot 4.3
• Add utility functions to generate useful prompts, such as llama guard 2
• Download models directly from huggingface
• Automatically generate json schema from data classes in GDSCript
• More llama.cpp features
• mlc-llm integration
• Any suggestion?

Documentation

Inspector Properties: GDLlama, GDEmbedding, and GDLlava

There are 3 base nodes added by this plugin: GdLlama, GdEmbedding, and GdLlava. Each type of node owns a set of properties which affect the computational performance and the generated output. Some of the properties belong to more than one node, and they generally have similar meaning for all types of node.

• Model Path: location of your GGUF model
• Mmproj Path location of your mmproj GGUF file, for GdLlava only
• Instruct: question and answer interactive mode
• Interactive: custom interactive mode, you should set your reverse_prompt, input_prefix, and input_suffix to set up a smooth interaction
• Reverse Prompt: AI stops to wait for user input after seeing this prompt being generated, a good example is "User:"
• Input Prefix: append before every user input
• Input Suffix: append after every user input
• Should Output prompt: whether the input prompt should be included in the output
• Should Output Special: whether the special (e.g., beginning of sequence and ending of sequence) token should be included in the output
• Context Size: number of tokens the model can process at a time
• N Predict: number of new tokens to generate, generate infinite sequence if -1
• N Keep: when the model run out of context size, it starts to forget about earlier context, set this variable to force the model to keep a number of the earliest tokens to keep the conversation relevant
• Temperature: the higher the temperature, the more random the generated text
• Penalty Repeat: penalize repeated sequenc