Using Callbacks
Instrument the model building process using built-in and custom callbacks
Plexe provides a callback system that allows you to hook into various stages of the model.build()
process.
This is useful for logging, monitoring, custom artifact handling, or triggering external processes.
The Callback System
Callbacks are classes that inherit from plexe.Callback
and implement one or more of the following methods:
on_build_start(info: BuildStateInfo)
: Called once at the beginning of thebuild
process.on_build_end(info: BuildStateInfo)
: Called once at the end of thebuild
process (after success or error).on_iteration_start(info: BuildStateInfo)
: Called at the start of each model building iteration (solution attempt).on_iteration_end(info: BuildStateInfo)
: Called at the end of each model building iteration.
The BuildStateInfo
object passed to these methods contains contextual information like the model intent, provider used, schemas, datasets, current iteration number, and the current solution Node
being evaluated (especially relevant in on_iteration_end
).
Built-in Callbacks
Plexe includes some useful built-in callbacks:
MLFlowCallback
This callback logs parameters, metrics, and artifacts from the build process to an MLflow Tracking server.
Prerequisites:
- Install MLflow:
pip install mlflow
- Have an MLflow tracking server running or use local file logging.
Usage:
The MLFlowCallback
logs:
- Parameters: Intent, provider, schemas, timeouts, iteration number.
- Metrics: Performance metrics (e.g., accuracy, RMSE) reported by the agent for each iteration, execution time.
- Artifacts: Training code (
trainer_source.py
), model artifacts saved by the training script. - Tags: Provider used, whether an exception occurred during the iteration.
Creating Custom Callbacks
You can create your own callbacks by subclassing plexe.Callback
.
Example: A simple callback to print iteration progress.
By implementing custom callbacks, you can integrate Plexe’s model building process seamlessly into your existing workflows and monitoring systems.