Serve Llama 3.1 405B on Kubernetes on Multi Host GPUs
Serving a really large language model comes with its own challenges. Getting the right amount of resources is hard. But properly sharding the model across hosts and gpus add even more complexity.
Leader Worker Set is a new API from the folks at Kubernetes that aim to simplify that process. In this video we show how you can serve a really large language model like Llama 3.1 405b on GKE using 2 nodes and 16 total H100 GPU for a total of 1280 GB of GPU vRAM.
LWS: https://github.com/kubernetes-sigs/lws
Tutorial: https://cloud.google.com/kubernetes-e...