Using AMD ROCm™ vLLM Docker Image with AMD Instinct™ MI300X
… Set up the vLLM backend server export MODEL=amd/Llama-3.1-405B-Instruct-FP8-KV export DOCKER IMG=rocm/vllm:rocm6.2 mi300 ubuntu20.04 py3.9 vllm 0.6.4 export HOST PORT=8011 export HF TOKEN="your personal Hugging Face token to access gated models" docker run --rm --device=/dev/kfd --device=/dev/dri -… …