Currently, all related running Indicium instances need to be stopped and/or restarted when there is an IAM or Software Factory upgrade (as mentioned in the Deployer and in the documentation).
From a system administrator perspective, this is quite some extra work, especially if there are multiple instances looking at the same database.
It would be amazing if one or two of the following points can be implemented;
- Make Indicium aware of upgrades in progress, maybe by adding a table per database containing whether an upgrade is being done or not, to stop scheduled tasks and other things while the upgrade is running, if necessary, mark the instance as unhealthy.
- Make Indicium aware of an outdated model, if Indicium sees the model is outdated (which needs a restart), mark the instance as unhealthy.
This gives the ability to have more self-healing actions through health checks, for example on Azure, AWS, Google Cloud or in containers
