Text Generation Web UI and APIs – RTX A4000

Description

Text Generation Web UI: A Gradio web UI for Large Language Models. Supports transformers, GPTQ, llama.cpp (GGUF), Llama models

This is a custom packaged template for The Text Generation Web UI.

I do not maintain the code for this repo, I just package everything together so that it is easier for you to use.

If you need help with settings, etc. You can feel free to ask me, but just keep in mind that I am not an expert at the Text Generation Web UI! I’ll try my best to help, but the RunPod community may be better at helping you.

Version 1.12.5

• The blocking and non-blocking APIs have been removed in favour of the Open AI compatible API.
• The Open AI compatible API is now on port 5000.

Included in this Template

• Ubuntu 22.04 LTS
• CUDA 12.1.1
• Python 3.10.12
• Text Generation Web UI
• Legacy API Extension
• Torch 2.1.2
• xformers 0.0.23post1
• Jupyter Lab
• runpodctl
• OhMyRunPod
• RunPod File Uploader
• croc
• rclone
• speedtest-cli
• screen
• tmux

NOTE: The legacy APIs no longer work with the latest version of the Text Generation Web UI. They were deprecated since November 2023 and have now been completely removed. If you want to use the LEGACY APIs, please set the image tag to 1.9.5. You will also have to add port 6000 for the legacy REST API and/or port 6005 for the legacy Websockets API.

#####################################################################

To run on your own system or on google colab download the Docker image
here: ashleykza/oobabooga:1.12.5