Hyperparameter tuning approach question [R]

Hyperparameter Tuning Approach Question [R]

I am doing some work with cell type classification, where I have 4.3 million cells and 512 features (condensed embeddings from the encoder of a transformer). The broader goal is to implement a contextual bandit for augmenting the training set of the dataset, as it is currently imbalanced, and rare cell type classification is poor when I tried a baseline logistic regression classifier.

Dataset:

• Feature matrix shape: (4290471, 512)
• Labels shape: (4290471,)
• Class distribution:
  • T cell: 1966941
  • DC: 858451
  • NK cell: 561904
  • Monocyte: 411170
  • B cell: 375882
  • Platelet: 54576
  • Progenitor cell: 24689
  • ILC: 24254
  • Erythrocyte: 12604

I didn't do any hyperparameter tuning for the LR classifier, but I want to try other ML models (LightGBM, XGBoost, SVM). However, I face a bottleneck with hyperparameter tuning.

The Problem

I want to do 80/10/10 train/validate/test split, but the training set is so large and takes a long time even on H100. What are some solutions to this?

What I've Tried

I tried Optuna but still very long for each hyperparameter trial. I then tried Optuna but instead of using the full 80% for training each time, only 15% of the 80% is used (subsampling from the training set). I'm not sure if this is robust or not. I also couldn't really find anything in the literature.

Anyone been in a similar situation?