How to setup Local AI and make it integrated into HighlightX

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures

1. Setup Local AI

If you don't have docker installed, please install docker on your machine. Docker installation guide

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-aio-cpu --cors true
# Do you have a Nvidia GPUs? Use this instead
# CUDA 11
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-nvidia-cuda-11 --cors true
# CUDA 12
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-nvidia-cuda-12 --cors true

Note that we added the --cors true parameter to the command to make sure the local server is accessible from the browser.

Please refer to the Getting Started local AI for more information

2. Wait until the server is ready

3. Set up HighlightX with Local AI

Go to Local AI setting screen

Enter the base URL of the Local AI server. Please make sure the blue dot is shown

If you have configured an API Key on your local AI server, please provide it. Otherwise, the API Key can be left blank by default.

Click on Add Models and then enter Model ID, Modle Name, Model compatibility

- Model ID: The list of Model IDs is available at: Available AI Models

- Model Name: This is the name that will be displayed in model selection menu

- Model compatibility: Supporting vision entails that this model can be provided with images, while generating images denotes the model's ability to create images.

Click on Test and the message "the model gpt-4 is ready to use" should be shown

Click on Save to save the model

4. Chat with Local AI

Select Local AI model from AI models list

Say hi to the model