Not Sure How to Apply This to Docker ollama/ollama:rocm

[amd1890]

I've been trying to get this to work in docker which is using ollama and is painfully slow

It doesn't recognize the GPU and I don't know how to implement the commands inside the docker image or get the image to recognize the GPU.

Anyone have suggestions?

[alexhegit]

1. Where is the URL of this docker image?
2. What is the full command to run the docker image you used? something like docker run -it ...

[amd1890]

I am trying to get Perplexica, a local open-source version of Perplexity.ai, running and using an ollama that accesses rocm. I have gotten perplexica (https://github.com/ItzCrazyKns/Perplexica) working using a regular ollama in a docker but it's painfully slow. My system has rocm and supports Vulkan and I have Ryzen 780M. I am trying to get ollama set up in a docker to use rocm.

I think when I wrote this issue before, I didn't understand that you were validating programs as working correctly with rocm. Perhaps I should have titled the issue "Please help me."

https://github.com/avnigashi/ollama-rocm-docker is one of the repos that isn't seeming to work for me.

https://github.com/hqnicolas/OllamaDockerCasaOs is another repo I am not entirely understanding. I don't understand why casaos is part of this equation. This version runs, but generates errors in the docker terminal "Error: unknown flag: --device" and I'm not sure if it's running correctly.

my experience level is very low, so it's harder for me to understand things than a more experienced user

[alexhegit]

I am trying to get Perplexica, a local open-source version of Perplexity.ai, running and using an ollama that accesses rocm. I have gotten perplexica (https://github.com/ItzCrazyKns/Perplexica) working using a regular ollama in a docker but it's painfully slow. My system has rocm and supports Vulkan and I have Ryzen 780M. I am trying to get ollama set up in a docker to use rocm.

I think when I wrote this issue before, I didn't understand that you were validating programs as working correctly with rocm. Perhaps I should have titled the issue "Please help me."

https://github.com/avnigashi/ollama-rocm-docker is one of the repos that isn't seeming to work for me.

https://github.com/hqnicolas/OllamaDockerCasaOs is another repo I am not entirely understanding. I don't understand why casaos is part of this equation. This version runs, but generates errors in the docker terminal "Error: unknown flag: --device" and I'm not sure if it's running correctly.

my experience level is very low, so it's harder for me to understand things than a more experienced user

Hi @amd1890 , This repo is my practice to using ROCm to power other AI projects and want to back to OSS comunity to sharing my experience.

For your question, I never use Perplexica with ROCm before. Base on your situation, you many try set up Ollama with AMD 780M first refer to https://github.com/alexhegit/Playing-with-ROCm/blob/main/inference/LLM/Run_Ollama_with_AMD_iGPU780M-QuickStart.md and then to access Preplexica with Ollama+ROCm.

I also add it as my wishlist and will find time to try it later.

GOOD LUCK!

[amd1890]

I followed those instructions and could get ollama to run with an extremely high token/s count, indicating using the GPU.

I am still not sure how to integrate that into perplexica. https://github.com/ItzCrazyKns/Perplexica

Step 4 for the docker installation is:

Rename the sample.config.toml file to config.toml. For Docker setups, you need only fill in the following fields:

OPENAI: Your OpenAI API key. You only need to fill this if you wish to use OpenAI's models.

OLLAMA: Your Ollama API URL. You should enter it as http://host.docker.internal:PORT_NUMBER. If you installed Ollama on port 11434, use http://host.docker.internal:11434. For other ports, adjust accordingly. You need to fill this if you wish to use Ollama's models instead of OpenAI's.

GROQ: Your Groq API key. You only need to fill this if you wish to use Groq's hosted models.

ANTHROPIC: Your Anthropic API key. You only need to fill this if you wish to use Anthropic models.

Note: You can change these after starting Perplexica from the settings dialog.

SIMILARITY_MEASURE: The similarity measure to use (This is filled by default; you can leave it as is if you are unsure about it.)

I do not understand if this has to be a docker ollama. I think it does have to be?

This is so hard I may end up giving up. I may not be smart enough to figure this out.

[alexhegit]

I followed those instructions and could get ollama to run with an extremely high token/s count, indicating using the GPU.

I believe that you have set up the ollama over AMD GPU by following my instructions and you can run it like ollama run llama3.1. It means the ollama serve at the http://localhost:11434 as default (if you do not change the port).

Then you can follow the instructions of https://github.com/ItzCrazyKns/Perplexica to run Perplexica.

Here is my trying run Pefplexica with docker at the same machine which ollama serving.

git clone https://github.com/ItzCrazyKns/Perplexica.git

cd Perplexcia
cp sample.config.toml config.toml

Then modify the config.toml for setting of OLLAMA as bellow,

OLLAMA = "http://localhost:11434" # Ollama API URL - http://host.docker.internal:11434

Start the Perplexica docker (pull, build, start...)

docker compose up -d

Wait a few minutes for the setup to complete. You can access Perplexica at http://localhost:3000/ in your web browser.

I could open it by the web browser but it seems that it not running successfully. You can get my issue report from
ItzCrazyKns/Vane#501