Llama 3.1 Nuts and Bolts

A holistic way of understanding how Llama and its components run in practice, with code and detailed documentation (GitHub Pages | GitHub). "The nuts and bolts" (practical side instead of theoretical facts, pure implementation details) of required components, infrastructure, and mathematical operations without using external dependencies or libraries.

This project intentionally doesn't have support for GPGPU (such as nVidia CUDA, OpenCL) as well as SIMD because it doesn't aim to be a production application, for now. Instead, the project relies on CPU cores to perform all mathematical operations, including linear algebraic computations. To increase performance, the code has been optimized as much as necessary, utilizing parallelization via goroutines.

^{Llama Nuts and Bolts Screen Recording GIF, captured while the application was running on the Apple MacBook Pro M1 Chip. Predefined prompts within the application were executed. The GIF is 20x faster.}

:thought_balloon: WHY THIS PROJECT?

This project was developed for only educational purposes, and has not been tested for production or commercial usage. The goal is to make an experimental project that can perform inference on the Llama 3.1 8B-Instruct model completely outside of the Python ecosystem. Throughout this journey, the aim is to acquire knowledge and shed light on the abstracted internal layers of this technology.

This journey is an intentional journey of literally reinventing the wheel. While reading this journey in the documentation directory (GitHub Pages | GitHub), you will navigate toward the target with a deductive flow. You will encounter the same stops and obstacles I encountered during this journey.

If you are curious like me about how the LLMs (Large Language Models) and transformers work and have delved into conceptual explanations and schematic drawings in the sources but hunger for deeper understanding, then this project is perfect for you too!

:blue_book: DOCUMENTATION

The journey can be found documented step by step at Llama Nuts and Bolts - GitHub Pages website with a visually better experience, or at docs directory.

:dart: COVERAGE

Due to any of the existing libraries (except the built-in packages and a few helpers) wasn't used, all of the required functions were implemented by this project in the style of Go. However, the main goal of this project is to do inference only on the Llama 3.1 8B-Instruct model, the functionality fulfills only the requirements of this specific model. Not much, not less, because the goal of our project is not to be a production-level tensor framework.

The project provides a CLI (command line interface) application allowing users to choose from predefined prompts or write custom prompts. It then performs inference on the model and displays the generated text on the console. The application supports "streaming," enabling immediate display of generated tokens on the screen without waiting for the entire process to complete.

As you can see in the chapters in the documentation directory (GitHub Pages | GitHub), covered things are:

• All diagrams about the model and the flow are listed in Chapter 20,
• Parallelization and concurrency in Go, see Chapter 13,
• Implementing Memory Mapping that allows us to map a large file content to memory address space in Go for both Linux/MacOS and Windows platforms, see Chapter 2,
• Implementing BFloat16 (Brain Floating Point) data type which isn't supported by the Go language, from scratch, see Chapter 7,
• Implementing support for "streaming" output via Go channels, see Chapter 13,
• Loading a PyTorch model weights file ("consolidated.00.pth") which was saved as Pickle (.pkl) format, from scratch, see Chapter 2 and Chapter 3,
• Loading the model arguments JSON file ("params.json"), see Chapter 4,
• Loading a Byte-Pair Encoding (BPE) tokenizer model which was saved as Tiktoken tokenizer format file ("tokenizer.model"), from scratch, see Chapter 5,
• Implementing a Tensor type, tensor aritmetic and machine learning mathematical operation functions, see Chapter 8,
• Working with C contiguous arrays in multi-dimensional form, see Chapter 8,
• Building the blocks of Llama 3 model architecture, see Chapter 9,
• Implementing RoPE (Rotary Positional Embeddings) and precomputing frequency tensor, see Chapter 10 and Chapter 10.BONUS,
• Understanding tokens, vocabulary, and tokenization, see Chapter 12,
• Generating the next token, internals of transformer block, being an auto-regressive model, multi-head self-attention, and much more, see Chapter 13, Chapter 14, Chapter 15, Chapter 16,
• Understanding internals of the Unicode Standard, UTF-8 encoding, and how emojis are represented and rendered, see Chapter 17,
• And, so much more!

:triangular_ruler: MODEL DIAGRAM

This diagram has extensive and comprehensive content and tries to reveal all of the details as much as possible. These reasons made it to have a considerable length. To use the space here efficiently, it was enclosed within an expandable section as follows:

Click to expand to see the Complete Model Diagram

The whole flow of Llama 3.1 8B-Instruct model without abstraction:

:package: INSTALLATION and BUILDING

Downloading the Official Model Files

Llama model weights files can be found in several formats on the Internet. Meta's official format, HuggingFace format, GGUF format, etc... But our project uses only the official format.

Note: Download chapter of original Llama 3.1 package repository and this How to Install Llama 2 Locally article (disclaimer: there are some differences between this article and Llama 3.1) may help you too.

Click to expand the details of downloading instructions

• Request access from Meta Website https://llama.meta.com/llama-downloads/ by filling the form. The email address must be valid and it's enough to mark "Llama 3.1" checkbox in the model list,
• You will receive an email from Meta including the instructions for downloading the files,
• Download the download.sh file from the official Llama repository,

• Create a parent directory, and copy the download.sh script there:

  sh $ mkdir llama-models $ cd llama-models

• Give executable permission to the download.sh file:

  sh llama-models$ chmod +x download.sh

• Pre-requisites: Make sure you have wget and md5sum installed,

• Run the script:

  sh llama-models$ ./download.sh

• The download script asks for the URL was come with the email:

  sh llama-models$ ./download.sh Enter the URL from email:

• Copy the URL written after When asked for your unique custom URL, please insert the following text in the email, and press ENTER,

• It asks for the model names that you want to download:

  ```sh llama-models$ ./download.sh Enter the URL from email: # URL was entered and ENTER was pressed

  **** Model list *** - meta-llama-3.1-405b - meta-llama-3.1-70b - meta-llama-3.1-8b - meta-llama-guard-3-8b - prompt-guard Choose the model to download: ```

• Write meta-llama-3.1-8b and press ENTER, be careful on writing as it is because it is case-sensitive,

• It asks for the sub-model name:

  ```sh Choose the model to download: meta-llama-3.1-8b

  Selected model: meta-llama-3.1-8b

  **** Available models to download: *** - meta-llama-3.1-8b-instruct - meta-llama-3.1-8b Enter the list of models to download without spaces or press Enter for all: ```

• Write meta-llama-3.1-8b-instruct and press ENTER, be careful on writing as it is because it is case-sensitive,

• If everything goes well, you will see a progress like:

  sh ... Enter the list of models to download without spaces or press Enter for all: meta-llama-3.1-8b-instruct Downloading LICENSE and Acceptable Usage Policy ...

• If you get HTTP 403 error:

  • Check if you wrote the model name correctly as meta-llama-3.1-8b and then meta-llama-3.1-8b-instruct
  • Check if the download link you copied ends with Select text, it shouldn't contain it, as described in the llama 2 github issue

• If you did everything correct, wait for the progress finishes. It will download ~16GB of files.

• If you have issues with downloading process, check out the links above and click on the collapsible paragraph above to expand.

• If everything went well, you will have a directory (in our example it's named llama-models)

  sh llama-models (parent directory) |-download.sh |-LICENSE |-USE_POLICY.md |-Meta-Llama-3.1-8B-Instruct | |-consolidated.00.pth # our model weights file | |-params.json # our model arguments file | |-tokenizer.model # our tokenizer model file

Cloning the repository and maintaining proper directory structure

• Clone this repo and enter into this directory,

• You will see the empty directory named models-original,

  sh llama-nuts-and-bolts (parent directory) |-... |-models-original | |-.gitkeep |-...

• Move the directory named llama-models/Meta-Llama-3.1-8B-Instruct into llama-nuts-and-bolts/models-original/ directory (case sensitive).

• You should have the following directory structure:

  sh llama-nuts-and-bolts (parent directory) |-... |-models-original | |-Meta-Llama-3.1-8B-Instruct | | |-consolidated.00.pth | | |-tokenizer.model | | |-params.json | |-.gitkeep |-...

Production Mode - Native Environment

Alongide you can run this project in a dockerized or virtualized environment, it's more suggested that to run this project without virtualization. You will see the performance difference between them.

The most performant option is to build the project and to run the executable.

You can build the project for your system's operating system, the executable will be named a