model.build()
process.
This is useful for logging, monitoring, custom artifact handling, or triggering external processes.
The Callback System
Callbacks are classes that inherit fromplexe.Callback
and implement one or more of the following methods:
on_build_start(info: BuildStateInfo)
: Called once at the beginning of thebuild
process.on_build_end(info: BuildStateInfo)
: Called once at the end of thebuild
process (after success or error).on_iteration_start(info: BuildStateInfo)
: Called at the start of each model building iteration (solution attempt).on_iteration_end(info: BuildStateInfo)
: Called at the end of each model building iteration.
BuildStateInfo
object passed to these methods contains contextual information like the model intent, provider used, schemas, datasets, current iteration number, and the current solution Node
being evaluated (especially relevant in on_iteration_end
).
Built-in Callbacks
Plexe includes some useful built-in callbacks:MLFlowCallback
This callback logs parameters, metrics, and artifacts from the build process to an MLflow Tracking server.
Prerequisites:
- Install MLflow:
pip install mlflow
- Have an MLflow tracking server running or use local file logging.
The
MLFlowCallback
logs:- Parameters: Intent, provider, schemas, timeouts, iteration number.
- Metrics: Performance metrics (e.g., accuracy, RMSE) reported by the agent for each iteration, execution time.
- Artifacts: Training code (
trainer_source.py
), model artifacts saved by the training script. - Tags: Provider used, whether an exception occurred during the iteration.
Creating Custom Callbacks
You can create your own callbacks by subclassingplexe.Callback
.
Example: A simple callback to print iteration progress.