idea: model-in-Colab as a DSH inference engine via tunnel (horizon; likely ToS-problematic) #9

Open
opened 2026-11-11 20:19:00 +00:00 by dsh · 0 comments
Owner

Operator floated the fun version: run an open-weight model (e.g. Gemma 26A4B) in a Colab session and expose it to DSH as an inference engine via a tunnel — "free DSH for every Google AI customer".

Why it is parked (not just hard: likely against ToS)

  • Colab FAQ disallows on free/no-balance managed runtimes: connecting to remote proxies, file hosting/media serving/web-service offerings unrelated to interactive compute, and using multiple accounts to work around limits. A persistent tunnel exposing inference to DSH is exactly a web service on a managed runtime.
  • paid plans relax some restrictions, but "service offerings" language still applies; commercial/shared use is clearly out.
  • practical: Colab sessions are ephemeral (idle cull), driver latency ~tens of ms + tunnel overhead, no stable networking, and GPU is shared/T4, so it is a poor serving target regardless.

If we ever pursue something like it

  • use a GCP VM / Colab Enterprise / local runtime we control (Colab Enterprise explicitly supports APIs), not consumer Colab;
  • scope it as a research demo with explicit operator consent and no external users;
  • never use it to serve users without a Google Cloud contract.

Decision

Keep as an exploration idea for a future "dsh-inference" plugin where the backend is an owned endpoint (vLLM/LM Studio/Ollama/Colab Enterprise), with Colab as a dev environment only. Revisit if Google ships stable networking/API access for managed runtimes.

Operator floated the fun version: run an open-weight model (e.g. Gemma 26A4B) in a Colab session and expose it to DSH as an inference engine via a tunnel — "free DSH for every Google AI customer". ## Why it is parked (not just hard: likely against ToS) - Colab FAQ disallows on free/no-balance managed runtimes: **connecting to remote proxies**, file hosting/media serving/web-service offerings unrelated to interactive compute, and using multiple accounts to work around limits. A persistent tunnel exposing inference to DSH is exactly a web service on a managed runtime. - paid plans relax some restrictions, but "service offerings" language still applies; commercial/shared use is clearly out. - practical: Colab sessions are ephemeral (idle cull), driver latency ~tens of ms + tunnel overhead, no stable networking, and GPU is shared/T4, so it is a poor serving target regardless. ## If we ever pursue something like it - use a **GCP VM / Colab Enterprise / local runtime** we control (Colab Enterprise explicitly supports APIs), not consumer Colab; - scope it as a *research demo* with explicit operator consent and no external users; - never use it to serve users without a Google Cloud contract. ## Decision Keep as an exploration idea for a future "dsh-inference" plugin where the backend is an owned endpoint (vLLM/LM Studio/Ollama/Colab Enterprise), with Colab as a *dev* environment only. Revisit if Google ships stable networking/API access for managed runtimes.
Sign in to join this conversation.
No description provided.