On-prem MLOps in a hospital: advice needed for production monitoring of self-built and vendor models? [D]

TL;DR

Hospital, fully on-prem OpenShift cluster. Multiple teams building prediction models, so we’re setting up a self-service platform with boundary policies. Evaluating ClearML vs OpenShift AI for the full MLOps lifecycle. Both look fine for development/deployment, but neither seems to give us production monitoring at the level we need (drift, bias, live dashboards per model). Extra twist: we also need to monitor models that run at our vendors, where all we get is an input/output data feed. Looking for real-world experience.

Our situation

We’re a hospital running an on-prem OpenShift cluster. No cloud, patient data stays inside the building. We have multiple teams across the organisation working on prediction models, at quite different levels of maturity.

So what we’re building is a self-service platform with boundary policies: teams get their own project/namespace, and can work independently - but within guardrails we define centrally (access control, resource limits, what can be deployed to production, what has to be logged and monitored). We don’t want to be the bottleneck for every team, but we also can’t have twelve teams each inventing their own way of putting a model into clinical use.

That means we’re looking for a full MLOps lifecycle platform on our on-prem cluster - data prep, notebooks, training, pipelines, model registry, serving, and monitoring - and we want to pick the right stack now.

What we’re evaluating

We’re currently evaluating:

• Red Hat OpenShift AI (we already run OpenShift)
• ClearML (also self-hosted)

For development - notebooks, pipelines, training, model registry, serving - both look reasonable. That’s not really where our doubt is.

The actual problem: production monitoring

Our models make predictions that hospital staff act on. That means we fall under MDR (EU 2017/745) and the EU AI Act, so post-market monitoring and logging aren’t nice-to-haves - they’re legal requirements.

What we need in production, live:

• Usage monitoring - who/what is calling the model, how often, is it actually being used or ignored
• Drift detection - data drift and prediction drift, per model
• Bias / fairness monitoring - and specifically subgroup performance (sensitivity/specificity/calibration per group), not just statistical parity, because in a clinical setting unequal miss rates are the actual harm
• Model-specific custom metrics - every clinical model has its own definition of “is this still working”
• Per-project dashboards - a model owner should be able to open one screen and see the state of their model, and in a self-service setup this has to work without central IT building it for them each time
• Alerting with a named owner - monitoring nobody responds to is worthless
• Immutable inference logging for audit/traceability

So I’ve been looking at running Evidently AI alongside it, self-hosted, computing metrics in a pipeline and pushing to Grafana. That seems like the pragmatic answer, but I’d like a sanity check before we commit.

The hard requirement: third-party vendor models

This is the part that seems to break most platform-native monitoring. A growing share of our AI is bought from vendors and runs on their infrastructure. We don’t control the serving runtime, we can’t attach a sidecar, we can’t instrument anything.

What we can do - and what we’re putting into procurement contracts - is require the vendor to deliver us the input/output data of every inference, which we then ingest and run our own monitoring pipeline on.

Legally the vendor is the manufacturer/provider and owns post-market surveillance, but as the deployer we still have our own obligations, and frankly we want independent evidence rather than just trusting their reporting. At the same time we’re doing more and more in-house model development, which is exactly why we want one platform covering the full lifecycle rather than only a monitoring tool bolted on afterwards.

Whatever we choose has to handle “model running on our own cluster” and “model running at a vendor” in one consistent way.

What I’m hoping to learn

Given all of the above, what would you say is the best solution for our environment, and does anyone have real-world advice on platforms from running something like this in a regulated environment?