Self-Hosting RustDesk: Step-by-Step Guide and Best PracticesRemote desktop tools are indispensable for IT admins, developers, and support teams. RustDesk is an open-source remote desktop solution designed as a privacy-respecting, lightweight alternative to commercial products. It supports self-hosting, which gives you full control over your connection relay and rendezvous servers, removes third-party dependency, and improves privacy and potentially performance. This guide leads you through preparing, deploying, securing, and maintaining your own RustDesk server infrastructure, with practical best practices and troubleshooting tips.
Overview: components and architecture
RustDesk’s self-hosted architecture typically uses two components:
- hbbs (Rendezvous/Relay server) — handles peer discovery (rendezvous) and optionally relays traffic when direct peer-to-peer (P2P) connections aren’t possible.
- hbbr (Optional TURN-like relay) — provides a relay service for connections (in some docs hbbr is the relay part; in many deployments only hbbs and a separate relay are needed). Recent RustDesk releases often package these roles; check release notes.
You’ll deploy at least one hbbs instance (and a relay if necessary). Clients (desktop/mobile) connect to your hbbs to find peers and then connect P2P using NAT traversal (UDP hole punching / TCP fallback). Relay is used when P2P fails.
Requirements and planning
- A VPS or server with a public IPv4 address (recommended dedicated IP). A dual-stack (IPv6) environment is useful but not required.
- Minimum: 1 vCPU, 512 MB–1 GB RAM for small teams; scale up for more simultaneous sessions.
- OS: Debian/Ubuntu, CentOS/RHEL, or any Linux with Docker support.
- Domain name (optional but recommended) for TLS certificates and human-friendly addresses.
- Firewall control and ability to open ports (UDP/TCP 21115 by default, plus HTTP/HTTPS for web admin if used).
- Basic knowledge of systemd, Docker, or container orchestration.
Installation approaches
You can self-host RustDesk using several methods. Choose one based on familiarity and environment:
- Docker Compose (recommended for simplicity and reproducibility)
- System packages / precompiled binaries (manual setup)
- Kubernetes (for large-scale/high-availability deployments)
This guide uses Docker Compose for clarity and portability.
Step 1 — Prepare your server
- Provision a server (e.g., a small Linux VPS).
- Update the system:
sudo apt update && sudo apt upgrade -y - Install Docker and Docker Compose (example for Ubuntu):
sudo apt install -y docker.io docker-compose sudo systemctl enable --now docker
Step 2 — Obtain RustDesk server binaries or Docker image
RustDesk provides Docker images and GitHub releases. The Docker registry image tags change over time — use the version matching the clients you’ll run. For Docker Hub / GitHub Container Registry, prefer the official repo.
Example Docker Compose will reference images like rustdesk/rustdesk-server:latest or a specific tag (e.g., v1.2.0). Replace with the latest stable tag you verify from the RustDesk releases.
Step 3 — Create Docker Compose configuration
Create a directory for your deployment and a docker-compose.yml file. Example minimal compose (adjust images/tags as needed):
version: "3.7" services: hbbs: image: rustdesk/rustdesk-server:latest container_name: rustdesk_hbbs restart: unless-stopped environment: - RUSTDESK_HBBS_HOST=0.0.0.0 # Set additional env vars if required by the image ports: - "21115:21115/tcp" - "21115:21115/udp" volumes: - ./data:/data Notes:
- Many community images require specific env vars to enable web UI or TLS; consult the image README.
- Persist data to host volume to keep keys and config across restarts.
Step 4 — Configure DNS and TLS (recommended)
- Assign a domain or subdomain (e.g., rustdesk.example.com) to your server’s public IP via an A record.
- Use Let’s Encrypt to provide TLS for any web/admin endpoints or to secure clients that support TLS. If your rustdesk image supports automatic certs (via ACME), enable it. Otherwise, use a reverse proxy (Caddy or Nginx) to terminate TLS and proxy to the container.
Example minimal Caddyfile (automatic HTTPS):
rustdesk.example.com { reverse_proxy localhost:21116 } Adjust ports to match any HTTP management interface.
TLS isn’t mandatory for RustDesk’s native traffic (it uses its protocol), but securing web interfaces and ensuring certificates for clients that support TLS is a best practice.
Step 5 — Start the services
From your docker-compose directory:
docker-compose up -d Check logs:
docker-compose logs -f Confirm the hbbs service is listening on 21115/tcp+udp and clients can reach it.
Step 6 — Configure RustDesk clients
- In the RustDesk client, open Settings → General (or Settings → Connection).
- Set the “Server” / “Rendezvous” address to your domain or server IP and port (e.g., rustdesk.example.com:21115).
- Optionally configure the relay server address if your deployment separates relay; otherwise the hbbs will coordinate relay use.
- Test a connection between two clients behind different NATs to confirm P2P works and relay fallback functions.
Security best practices
- Use a firewall: Allow only necessary ports (21115/tcp+udp; SSH 22 restricted to admin IPs; HTTP/HTTPS if used).
- Run services least-privileged: Avoid running containers as root where possible; use Docker user namespaces or compose user settings.
- Enable TLS for web/admin interfaces via reverse proxy (Caddy/Nginx) with Let’s Encrypt.
- Network segmentation: Place relay/hbbs behind private network if running additional app services.
- Access control: Use strong SSH keys for server access, disable password SSH login.
- Monitoring & logs: Forward Docker/container logs to centralized logging (ELK, Prometheus + Grafana) for visibility.
- Keep software updated: Regularly update RustDesk server images and client versions to get security patches.
- Back up config and keys: Persist /data volumes and snapshot them regularly; loss of key material may break client registrations.
High-availability and scaling
For organizations with many clients, consider:
- Running multiple hbbs instances behind a TCP/UDP load balancer with session affinity.
- Using multiple relay nodes (hbbr or dedicated relay containers) in different regions for performance.
- Automating deployments with Terraform + Kubernetes for scaling and self-healing.
- Monitoring connection counts and resource usage; scale CPU/RAM based on concurrency.
Troubleshooting common issues
- Connection fails P2P but works via relay: Check UDP/TCP ports on clients and NAT behavior. Some restrictive NATs or corporate firewalls block UDP; enable TCP relay.
- Clients can’t reach server: Verify DNS, open ports, and that hbbs is listening on expected interfaces.
- High latency/poor video: Check relay usage; relayed traffic consumes more bandwidth and CPU. Place relays closer to client locations.
- Persistent client ID changes after server migration: Ensure you preserved /data or key files used by hbbs so client pairings remain valid.
Example advanced Docker Compose (with relay and persistence)
version: "3.8" services: hbbs: image: rustdesk/hbbs:latest restart: unless-stopped ports: - "21115:21115/tcp" - "21115:21115/udp" volumes: - ./hbbs-data:/data hbbr: image: rustdesk/hbbr:latest restart: unless-stopped ports: - "21116:21116/tcp" - "21116:21116/udp" volumes: - ./hbbr-data:/data Adjust image names/tags to match the official release you choose.
Maintenance checklist
- Weekly: Check logs for errors, confirm backups ran.
- Monthly: Update container images, test client compatibility.
- Quarterly: Review firewall rules, rotate keys if needed, run disaster recovery drill.
Final notes and recommendations
Self-hosting RustDesk gives you privacy and control but requires operational responsibility: secure the server, maintain updates, and monitor usage. For small teams, a single VPS with a properly configured hbbs and optional relay is usually sufficient. For larger organizations, plan for redundancy, load balancing, and centralized logging.
If you want, tell me your server OS, number of expected simultaneous remote sessions, and whether you prefer Docker or a package install — I’ll produce a tailored docker-compose or systemd-based setup with exact commands and recommended instance sizing.
Leave a Reply