LlamaEdge-RAG API server provides a group of OpenAI-compatible web APIs for the Retrieval-Augmented Generation (RAG) applications. The server is implemented in WebAssembly (Wasm) and runs on WasmEdge Runtime.
rag-api-server provides a POST API /v1/models to list currently available models.
Example
You can use curl to test it on a new terminal:
curl -X POST http://localhost:8080/v1/models -H 'accept:application/json'
If the command runs successfully, you should see the similar output as below in your terminal:
{
"object":"list",
"data":[
{
"id":"llama-2-chat",
"created":1697084821,
"object":"model",
"owned_by":"Not specified"
}
]
}
Ask a question using OpenAI's JSON message format.
Example
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a helpful assistant."}, {"role":"user", "content": "Who is Robert Oppenheimer?"}], "model":"llama-2-chat"}'
Here is the response.
{
"id":"",
"object":"chat.completion",
"created":1697092593,
"model":"llama-2-chat",
"choices":[
{
"index":0,
"message":{
"role":"assistant",
"content":"Robert Oppenheimer was an American theoretical physicist and director of the Manhattan Project, which developed the atomic bomb during World War II. He is widely regarded as one of the most important physicists of the 20th century and is known for his contributions to the development of quantum mechanics and the theory of the atomic nucleus. Oppenheimer was also a prominent figure in the post-war nuclear weapons debate, advocating for international control and regulation of nuclear weapons."
},
"finish_reason":"stop"
}
],
"usage":{
"prompt_tokens":9,
"completion_tokens":12,
"total_tokens":21
}
}
In RAG applications, uploading files is a necessary step.
Example: Upload a file
The following command upload a text file paris.txt to the API server via the /v1/files endpoint:
curl -X POST http://127.0.0.1:8080/v1/files -F "file=@paris.txt"
If the command is successful, you should see the similar output as below in your terminal:
{
"id": "file_4bc24593-2a57-4646-af16-028855e7802e",
"bytes": 2161,
"created_at": 1711611801,
"filename": "paris.txt",
"object": "file",
"purpose": "assistants"
}
The id and filename fields are important for the next step, for example, to segment the uploaded file to chunks for computing embeddings.
GET /v1/files endpoint is used for listing all files on the server.
Example: List files
The following command lists all files on the server via the /v1/files endpoint:
curl -X GET http://127.0.0.1:8080/v1/files
If the command is successful, you should see the similar output as below in your terminal:
{
"object": "list",
"data": [
{
"id": "file_33d9188d-5060-4141-8c52-ae148fd15f6a",
"bytes": 17039,
"created_at": 1718296362,
"filename": "test-123.m4a",
"object": "file",
"purpose": "assistants"
},
{
"id": "file_8c6439da-df59-4b9a-bb5e-dba4b2f23c04",
"bytes": 17039,
"created_at": 1718294169,
"filename": "test-123.m4a",
"object": "file",
"purpose": "assistants"
}
]
}
GET /v1/files/{file_id} endpoint is used for retrieving information about a specific file on the server.
Example: Retrieve information about a specific file
The following command retrieves information about a specific file on the server via the /v1/files/{file_id} endpoint:
curl -X GET http://localhost:10086/v1/files/file_b892bc81-35e9-44a6-8c01-ae915c1d3832
If the command is successful, you should see the similar output as below in your terminal:
{
"id": "file_b892bc81-35e9-44a6-8c01-ae915c1d3832",
"bytes": 2161,
"created_at": 1715832065,
"filename": "paris.txt",
"object": "file",
"purpose": "assistants"
}
GET /v1/files/{file_id}/content endpoint is used for retrieving the content of a specific file on the server.
Example: Retrieve the content of a specific file
The following command retrieves the content of a specific file on the server via the /v1/files/{file_id}/content endpoint:
curl -X GET http://localhost:10086/v1/files/file_b892bc81-35e9-44a6-8c01-ae915c1d3832/content
GET /v1/files/download/{file_id} endpoint is used for downloading a specific file on the server.
Example: Download a specific file
The following command downloads a specific file on the server via the /v1/files/download/{file_id} endpoint:
curl -X GET http://localhost:10086/v1/files/download/file_b892bc81-35e9-44a6-8c01-ae915c1d3832
DELETE /v1/files/{file_id} endpoint is used for deleting a specific file on the server.
Example: Delete a specific file
The following command deletes a specific file on the server via the /v1/files/{file_id} endpoint:
curl -X DELETE http://localhost:10086/v1/files/file_6a6d8046-fd98-410a-b70e-0a0142ec9a39
If the command is successful, you should see the similar output as below in your terminal:
{
"id": "file_6a6d8046-fd98-410a-b70e-0a0142ec9a39",
"object": "file",
"deleted": true
}
To segment the uploaded file to chunks for computing embeddings, use the /v1/chunks API.
Example
The following command sends the uploaded file ID and filename to the API server and gets the chunks:
curl -X POST http://localhost:8080/v1/chunks \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"id":"file_4bc24593-2a57-4646-af16-028855e7802e", "filename":"paris.txt"}'
The following is an example return with the generated chunks:
{
"id": "file_4bc24593-2a57-4646-af16-028855e7802e",
"filename": "paris.txt",
"chunks": [
"Paris, city and capital of France, ..., for Paris has retained its importance as a centre for education and intellectual pursuits.",
"Paris’s site at a crossroads ..., drawing to itself much of the talent and vitality of the provinces."
]
}
To compute embeddings for user query or file chunks, use the /v1/embeddings API.
Example
The following command sends a query to the API server and gets the embeddings as return:
curl -X POST http://localhost:8080/v1/embeddings \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"model": "e5-mistral-7b-instruct-Q5_K_M", "input":["Paris, city and capital of France, ..., for Paris has retained its importance as a centre for education and intellectual pursuits.", "Paris’s site at a crossroads ..., drawing to itself much of the talent and vitality of the provinces."]}'
The embeddings returned are like below:
{
"object": "list",
"data": [
{
"index": 0,
"object": "embedding",
"embedding": [
0.1428378969,
-0.0447309874,
0.007660218049,
...
-0.0128974719,
-0.03543198109,
0.03974733502,
0.00946635101,
-0.01531364303
]
},
{
"index": 1,
"object": "embedding",
"embedding": [
0.0697753951,
-0.0001159032545,
0.02073983476,
...
0.03565846011,
-0.04550019652,
0.02691745944,
0.02498772368,
-0.003226313973
]
}
],
"model": "e5-mistral-7b-instruct-Q5_K_M",
"usage": {
"prompt_tokens": 491,
"completion_tokens": 0,
"total_tokens": 491
}
}
/v1/create/rag endpoint provides users a one-click way to convert a text or markdown file to embeddings directly. The effect of the endpoint is equivalent to running /v1/files + /v1/chunks + /v1/embeddings sequently. Note that the --chunk-capacity CLI option is required for the endpoint. The default value of the option is 100. You can set it to different values while starting LlamaEdge-RAG API server.
Example
The following command uploads a text file paris.txt to the API server via the /v1/create/rag endpoint:
curl -X POST http://127.0.0.1:8080/v1/create/rag -F "file=@paris.txt"
The embeddings returned are like below:
{
"object": "list",
"data": [
{
"index": 0,
"object": "embedding",
"embedding": [
0.1428378969,
-0.0447309874,
0.007660218049,
...
-0.0128974719,
-0.03543198109,
0.03974733502,
0.00946635101,
-0.01531364303
]
},
{
"index": 1,
"object": "embedding",
"embedding": [
0.0697753951,
-0.0001159032545,
0.02073983476,
...
0.03565846011,
-0.04550019652,
0.02691745944,
0.02498772368,
-0.003226313973
]
}
],
"model": "e5-mistral-7b-instruct-Q5_K_M",
"usage": {
"prompt_tokens": 491,
"completion_tokens": 0,
"total_tokens": 491
}
}
/v1/info endpoint provides the information of the API server, including the version of the server, the parameters of models, and etc.
Example
You can use curl to test it on a new terminal:
curl -X POST http://localhost:8080/v1/info -H 'accept:application/json'
If the command runs successfully, you should see the similar output as below in your terminal:
{
"version": "0.3.4",
"plugin_version": "b2694 (commit 0d56246f)",
"port": "8080",
"models": [
{
"name": "Llama-2-7b-chat-hf-Q5_K_M",
"type": "chat",
"prompt_template": "Llama2Chat",
"n_predict": 1024,
"n_gpu_layers": 100,
"ctx_size": 4096,
"batch_size": 512,
"temperature": 1.0,
"top_p": 1.0,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"frequency_penalty": 0.0
},
{
"name": "all-MiniLM-L6-v2-ggml-model-f16",
"type": "embedding",
"prompt_template": "Llama2Chat",
"n_predict": 1024,
"n_gpu_layers": 100,
"ctx_size": 384,
"batch_size": 512,
"temperature": 1.0,
"top_p": 1.0,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"frequency_penalty": 0.0
}
],
"qdrant_config": {
"url": "http://localhost:6333",
"collection_name": "default",
"limit": 5,
"score_threshold": 0.4
}
}
/v1/retrieve endpoint sends a query and gets the retrieval results.
Example
You can use curl to test it on a new terminal:
```bash curl -X POST http://localhost:8080/v1/retrieve \ -H 'accept:application/json' \ -H 'Content-Type: application/json' \
$ claude mcp add rag-api-server \
-- python -m otcore.mcp_server <graph>