[Enhancement] GPU selection for multi-GPU setups

[MrDrMcCoy]

Users with multiple GPUs at present do not appear to have a way to choose a GPU to run on with this library. This would be very nice to have, along with an example of creating a pool of workers for each and queuing generation tasks across them. Would this be difficult to implement?

[william-murray1204]

Hi @MrDrMcCoy, there are a few scenarios where multi-GPU setups work and are limited, depending on your hardware. The simplest and currently supported scenario is using multiple GPUs running the same backend, for example two CUDA GPUs. In that case, you can force which GPU is visible via environment variables, as described in this issue. I have an example of how you might go about implementing something like this multi-AMD GPU inference (using a HIP backend) in my Python tests.

If, on the other hand, you want to run multi-backend GPU hardware simultaneously (e.g. a CUDA and an AMD GPU), there is no easy way to do this built into the library, as each build can only support a single backend. It is not impossible, but it would require running separate Python environments with separate installs of stable-diffusion-cpp-python each targeting a different backend. You could then coordinate inference externally, for example, via a task queue that dispatches jobs to each environment.

[abhinandan-chakraborty]

Hi @william-murray1204 , in my case, I am running a single GRPC server which requires access to both GPU's at the same time. I can sure launch a new process just for stable-diffusion-cpp-python with CUDA_VISIBLE_DEVICES or I can use Ray Serve to distribute my GPU processes, but I don't want to overcomplicate this yet. I wish there was something like llama-cpp-python has tensor_split to split GPU usage. I know it's a lot to ask from you but if it is possible then please consider adding this feature, if not, then also it's no big deal.

Thank you for this awesome library anyway! Cheers!

[william-murray1204]

Hi @abhinandan-chakraborty, you don't need to launch a seperate process to use CUDA_VISIBLE_DEVICES as you can set the environment variable in code as shown in the example I linked to above.

Note that this repo is merely a Python interface (bindings) for the original stable-diffusion.cpp internal C API. The llama-cpp-python like functionality you are describing would first need to be requested/implemented in the upstream stable-diffusion.cpp repo for me to be able to support it here.

[ayttop]

https://github.com/william-murray1204/stable-diffusion-cpp-python/blob/main/tests/test_multi_gpu.py
i ask it run with 2gpu rtx4060ti
or not??????????