Deploying Kubeflow as an Azure ML Alternative

Azure Machine Learning is Microsoft's cloud-native machine learning platform that provides experiment tracking, managed compute, pipelines, a model registry, and model serving as hosted services within the Azure ecosystem, but it ties teams to Azure-specific APIs, managed compute pricing, and Microsoft's tooling. Kubeflow is an open-source machine learning platform built on Kubernetes that provides self-hosted alternatives to Azure ML capabilities through modular, portable components, letting organizations retain complete control over infrastructure, data residency, scalability, and operational costs. This guide deploys Kubeflow on a Kubernetes cluster as a self-managed replacement for Azure Machine Learning, covering installation with Kustomize manifests, notebook setup, ML pipeline orchestration, distributed training with the Trainer v2 API, model serving through KServe, hyperparameter optimization using Katib, RBAC and access management, object storage integration, and migration considerations for existing Azure ML workflows. By the end, you'll have a self-hosted ML platform running notebooks, pipelines, distributed training, model serving, and automated hyperparameter tuning. Understanding Azure Machine Learning vs Kubeflow Azure ML and Kubeflow provide comparable ML platform capabilities, but they differ in deployment and operational models. Azure ML delivers these as fully managed services within the Azure ecosystem, while Kubeflow provides equivalent open-source components that run on any Kubernetes cluster. The following table maps each Azure ML feature to its Kubeflow counterpart. | Azure Machine Learning | Kubeflow Equivalent | Description | |---|---|---| | Azure ML Notebooks | Kubeflow Notebooks | Interactive development environments with JupyterLab, VS Code, and RStudio | | Azure ML Jobs | Kubeflow Trainer | Distributed training for PyTorch, DeepSpeed, MLX, JAX, and XGBoost workloads | | Azure ML Pipelines | Kubeflow Pipelines (KFP) | Directed Acyclic Graph (DAG) based ML workflow orchestration | | Azure ML Model Registry | Kubeflow Model Registry | Versioned model artifact management with metadata tracking | | Azure ML Endpoints | KServe | Serverless model serving with autoscaling and canary deployments | | Azure ML Experiments | Katib | Automated hyperparameter tuning with multiple search algorithms | Self-hosting with Kubeflow eliminates per-minute compute charges, keeps all data within your own cluster, runs on any cloud provider or on-premises hardware, and allows complete customization of every component. Prerequisites Before you begin, you need to: - Have access to a multi-node Kubernetes cluster that runs Kubernetes 1.31 or later with at least 4 CPU cores and 16 GB of RAM per node (minimum 3 nodes recommended). - Install kubectl and configure it to connect to your cluster. - Install Kustomize version 5.4.3 or later. - Have a default StorageClass that is configured in your cluster for provisioning persistent volumes. Install Kubeflow Kubeflow uses Kustomize to deploy its components as Kubernetes resources. The official kubeflow/manifests repository contains all component manifests that are organized under common/ for shared infrastructure services such as Istio, cert-manager, and Dex, and under applications/ for Kubeflow-specific applications such as Pipelines, Notebooks, and KServe. Deploy Kubeflow via Manifests The following steps clone the Kubeflow manifests repository and deploy all components to the cluster. 1. Verify the Kubernetes cluster connection: $ kubectl cluster-info 2. Check the Kubernetes server version: $ kubectl version Verify that the Server Version field shows version 1.31 or later. 3. Clone the official Kubeflow manifests repository: $ git clone https://github.com/kubeflow/manifests.git 4. Switch to the manifests directory: $ cd manifests 5. Check out the latest stable release tag: $ git checkout 26.03 6. Deploy all Kubeflow components: The command uses a bounded retry loop that attempts the installation up to 5 times, which accommodates the time that Kubernetes CRDs and webhooks need to register before dependent resources apply. The loop exits automatically after a successful apply or after reaching the retry limit. $ for i in 1 2 3 4 5; do kustomize build example | kubectl apply --server-side --force-conflicts -f - && break || { echo "Attempt $i failed, retrying in 30s..."; sleep 30; }; done The first one or two attempts may output errors about CRDs or webhooks not being established. These errors are expected and resolve on subsequent attempts after the CRDs register. The loop exits automatically when the apply succeeds, which typically happens on the second or third attempt. The full installation takes approximately 10 to 15 minutes after the final successful apply for all pods to reach a Running state. The --server-side --force-conflicts flags are required because some Kubeflow CRDs exceed the annotation size limit that standard kubectl apply supports. The default installation uses the email u***@example.com and password12341234 . Change these credentials before exposing Kubeflow to any network. See the Set Up Access Control section later in this article for instructions. Verify Installation After the deployment completes, verify that all Kubeflow components are running and the CRDs are registered. 1. Check that all pods in the kubeflow namespace reach a Running state: $ kubectl get pods -n kubeflow --field-selector=status.phase!=Succeeded Verify that all listed pods display a Running status with all containers ready. If any pods show CrashLoopBackOff or Pending , check their logs with kubectl logs -n kubeflow POD-NAME and verify that the cluster meets the minimum resource requirements. 2. Check that the Istio ingress gateway service is running: $ kubectl get svc istio-ingressgateway -n istio-system Verify that the service appears in the output. 3. Check that Kubeflow and its component CRDs are registered: $ kubectl get crd | grep -E "kubeflow|kserve|katib|istio|knative|trainer" | wc -l The output displays the count of registered CRDs across Kubeflow and its components. A count of 40 or more indicates a complete installation. Create Default User Profile Kubeflow uses profiles to provide namespace-level isolation for each user. The default installation does not automatically provision a user namespace, so you need to create one manually. 1. Create a new file called user-profile.yaml : $ nano user-profile.yaml 2. Add the following configuration: apiVersion: kubeflow.org/v1 kind: Profile metadata: name: kubeflow-user-example-com spec: owner: kind: User name: u***@example.com Save and close the file. 3. Apply the profile manifest: $ kubectl apply -f user-profile.yaml This command creates an isolated namespace called kubeflow-user-example-com with default Role-Based Access Control (RBAC) policies and a service account for the default user. 4. Verify that the namespace exists: $ kubectl get namespace kubeflow-user-example-com 5. Verify that the default service account exists: The service account takes a few seconds to provision after the profile is created. Wait 10 seconds before running this command. $ kubectl get serviceaccount default-editor -n kubeflow-user-example-com Configure Storage Kubeflow components such as Notebooks, Pipelines, and the Model Registry require persistent storage. The cluster needs a default StorageClass to dynamically provision Persistent Volume Claims (PVCs). 1. Verify that a default StorageClass exists: $ kubectl get storageclass The default StorageClass shows (default) next to its name. If no default exists, set one by annotating an existing StorageClass . Replace STORAGE-CLASS-NAME with the name of an existing StorageClass from the output above. $ kubectl patch storageclass STORAGE-CLASS-NAME -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}' Configure Kubeflow Notebooks Kubeflow Notebooks provides managed JupyterLab, VS Code, and RStudio environments that run as Kubernetes pods with direct access to cluster resources, GPUs, and persistent storage. This component serves as a self-hosted alternative to Azure Machine Learning notebooks and compute instances. Access Dashboard Set up port forwarding and log in to the Kubeflow Central Dashboard. 1. Set up port forwarding to access the Kubeflow Central Dashboard: $ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80 - Open http://localhost:8080 in a web browser. The Kubeflow login screen appears. Click Sign in with Dex. - Enter the default credentials on the Dex login form and click Login. - Email: u***@example.com - Password: 12341234 - The Kubeflow Central Dashboard loads with links to Notebooks, Pipelines, Katib Experiments, KServe Endpoints, and other components. Select kubeflow-user-example-com from the namespace dropdown at the top. Create Notebook Server Launch a new notebook server from the Kubeflow dashboard. - Navigate to Notebooks in the left sidebar and click New Notebook. - Enter ml-workspace in the Name field. - Select the notebook environment from the image cards. Choose JupyterLab for a general-purpose data science environment. Select VisualStudio Code for a code editor interface, or RStudio for R-based statistical computing. To use a specific image version, select Custom Notebook from the dropdown below the cards. - Set Minimum CPU to 0.5 and Minimum Memory Gi to1 . - Leave the Workspace Volume at the default 5Gi . This volume persists data across notebook restarts. - Click Launch and wait for the notebook pod to reach a Running state. The status indicator turns green when the notebook is ready. Connect to Notebook Open the JupyterLab interface and verify that ML libraries are accessible. Click Connect next to the notebook server name. A new tab opens with the JupyterLab interface. Click Python 3 (ipykernel) under the Notebook section in the launcher to create a new notebook. Paste the following code into a cell and press S