Multi-tenant Orchestrator: managing multiple isolated MySQL/MariaDB environments from one deployment

Polyglot Projects ProxySQL
1

Hi everyone,

I’ve been working on a multi-tenant implementation for Orchestrator and I’m looking for feedback from people managing multiple independent MySQL/MariaDB environments.

The problem I wanted to solve is simple:

If you manage multiple customers, environments or independent database platforms, running a completely separate Orchestrator deployment for every environment can become operationally expensive.

My implementation allows a single Orchestrator deployment to manage multiple isolated tenants while keeping:

  • independent discovery
  • independent topology state
  • independent analysis and recovery
  • tenant-aware API and CLI
  • tenant-aware hooks and metrics
  • ProxySQL integration per tenant
  • one Raft group, one leader, one HTTP listener and one dashboard

Repository:

The repository also includes documentation for the multi-tenant architecture, configuration, deployment, upgrades and testing.

I’m particularly interested in feedback from people managing many clusters, multiple customers, MSP environments or SaaS platforms.

Would this solve a real operational problem in your environment?

And if anyone wants to test it, I’d really appreciate feedback about edge cases or scenarios I may not have considered.

2

@Maximo_Martinez

My implementation allows a single Orchestrator deployment to manage multiple isolated tenants while keeping:

Indeed, this is possible when managing multiple MySQL/MariaDB clusters with a single orchestrator service.

You can define individual clusters as follows.

mysql> INSERT INTO meta.cluster (anchor, cluster_name, cluster_domain) VALUES (1, 'testcluster', 'example.com');
mysql> INSERT INTO meta.cluster (anchor, cluster_name, cluster_domain) VALUES (1, 'testcluster2', 'example.com');

Orchestrator configurations (/etc/orchestrator.conf.json) can identify the topology details using the following query.

"DetectClusterAliasQuery": "SELECT ifnull(max(cluster_name), '''') as cluster_alias from meta.cluster where anchor=1;"

Cluster discovery and further status can be fetched using separate HOST:PORT details for each cluster node.

orchestrator-client -c discover -i <FIRST_CLUSTER_NODE_HOST:PORT>
orchestrator-client -c discover -i <SECOND_CLUSTER_NODE_HOST:PORT>

E.g,

orchestrator-client -c topology -i testcluster
orchestrator-client -c topology -i testcluster2

You may further refine or modify the DetectClusterAliasQuery query as per your requirement. Have you tested this approach already? If “not”, I would suggest testing this across all use cases (failovers, workload, etc.) to see whether it meets your requirements.

Having Raft further ensures HA(availability) of the orchestrator and ensures it avoids a single point of failure.

3

Thanks, Anil. To be clear, the core is unchanged: each tenant runs the same Orchestrator discovery, analysis and recovery logic, and it can discover many clusters. The multi-cluster part works as you describe.

What I added is a layer above it. One deployment can run several independent groups of clusters (tenants), each with its own configuration. A tenant can hold many clusters, and a group can even be PostgreSQL. The reason is that, in a single Orchestrator, almost everything is global:

  • Config is global. Credentials, polling, hooks, ProxySQL, KV store, lag queries and the engine (ProviderType) are one setting for the whole process. Only the recovery filters can be scoped per cluster alias (regex), and hooks can branch on the alias inside the script. So different groups of clusters can’t have specific configuration, such as one group with its own credentials, hooks and hostgroups and another with different ones. A MySQL/MariaDB group and a PostgreSQL group can’t coexist either.
  • Namespace and backend are shared. One backend and one InstanceKey space, so overlapping host:port across groups (e.g. different customers) collide.
  • The alias approach is intrusive. DetectClusterAliasQuery runs on the managed instances, so meta.cluster must exist in every customer’s database.

With tenants, each group gets its own configuration, backend, ProxySQL hostgroups, KV prefix, hooks and recovery policy. They share one process, one Raft group and one dashboard, with a tenant-scoped API (/api/tenants/{id}/…), CLI (-T) and Prometheus label (tenant). A failover in one tenant doesn’t touch the others, and the config loader rejects tenants that would overlap on ProxySQL hostgroups or KV prefix.

Tested with real containers: three tenants (two MariaDB, one PostgreSQL), failover isolation, and Raft scenarios (leader partition/pause, kill -9 mid-failover, rolling restart, snapshot/restore, mismatched tenant sets), plus a rolling-upgrade suite against a single-tenant node.

Known limits: authentication and ReadOnly are global (the same users can act on every tenant), SIGHUP reloads only the global config (changing a tenant needs a node restart), agents and seeds are out of scope in this version, and hooks inherit the process environment, so tenants that don’t trust each other should use credentials files instead of ${ENV}.

Docs: docs/multi-tenant.md in GitHub - maximart27/orchestrator · GitHub, and the upstream discussion is Feature request: multi-tenant mode (N independent topologies in one process, one Raft group) · Issue #132 · ProxySQL/orchestrator · GitHub. If the alias approach covers a case I’m missing, I’d like to hear it