How to Set Up PrivateGPT on Azure: A Complete Guide

PrivateGPT is a powerful tool for running private, local AI models with document ingestion capabilities. In this tutorial, we’ll walk through setting up PrivateGPT on an Azure Linux VM using Docker Compose.

---

Prerequisites

• An Azure account (Free tier available)
• A Linux VM (Ubuntu 20.04/22.04 recommended)
• Docker & Docker Compose (Installed in this guide)
• Minimum 8GB RAM (16GB+ recommended for better performance)

---

Step 1: Create an Azure Linux VM

1. Log in to the Azure Portal.
2. Navigate to Virtual Machines → Create VM.
3. Select:

• OS: Ubuntu Server 20.04 LTS or 22.04 LTS
• Size: Standard B4ms (4 vCPUs, 16GB RAM) or better
• Authentication: SSH key (recommended) or password

4. Allow inbound ports: 22 (SSH), 8001 (PrivateGPT), 8080 (Traefik)
5. Click Review + Create, then deploy.

---

Step 2: Connect to the VM via SSH

ssh -i ~/.ssh/your_key.pem azureuser@server_ip

---

Step 3: Install Docker & Docker Compose

Install Docker Engine

sudo apt update && sudo apt upgrade -y
sudo apt install -y docker.io
sudo systemctl enable --now docker

Install Docker Compose Plugin (v2)

sudo mkdir -p /usr/local/lib/docker/cli-plugins
sudo curl -SL https://github.com/docker/compose/releases/download/v2.24.5/docker-compose-linux-x86_64 -o /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose

Verify Installation

docker --version
docker compose version # Should show v2.X

Add User to Docker Group (Avoid Sudo)

sudo usermod -aG docker $USER
newgrp docker # Reload group permissions

---

Step 4: Clone & Run PrivateGPT

Clone the Repository

git clone https://github.com/zylon-ai/private-gpt.git
cd private-gpt

Start PrivateGPT (Ollama CPU Mode)

docker compose --profile ollama-cpu up -d

Check Running Services

docker ps # Should show PrivateGPT and Ollama containers

---

Step 5: Access PrivateGPT

Web UI

• Open in your browser:

http://[VM_PUBLIC_IP]:8001

Traefik Dashboard (Optional)

If using the Ollama profile:

http://[VM_PUBLIC_IP]:8080

---

Troubleshooting

1. Port Conflicts

If you see:

Error: Port 8001 already in use

Run:

sudo lsof -i :8001 # Find the process
sudo kill -9 # Terminate it

2. Low RAM Issues

Increase swap memory:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

3. Docker Permission Denied

Ensure your user is in the docker group:

groups # Check if "docker" is listed

---

Conclusion

You’ve successfully set up PrivateGPT on an Azure VM with Docker Compose! Now you can:

1. Run local AI models securely
2. Ingest and query private documents
3. Scale resources as needed

For advanced setups, consider:

• GPU acceleration (Use --profile ollama-cuda)
• Custom models (Edit docker-compose.yml)

Happy private AI experimenting! 🚀

---

Further Reading

• PrivateGPT GitHub
• Azure VM Docs
• Docker Compose Guide