Model updating¶
As seen when using river, a sentralised modelling and prediction system can be updated realtime based on incomming data.
In practice this comes with some caveats:
Faulty or biased data can give an undesired shift in the model (e.g., Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day (2016))
Some control is lost with regard to verification and validation.
Continuous Integration (CI) typically involves pushing updates often to a version control system where testing and building is performed once a day (nightly build).
Model deployment¶
Traditional software (and model) development cycles have typically involved alpha and beta testing phases for internal or limited user groups before full deployment with long upgrade cycles.
Continuous Delivery (CD) is public release counterpart of CI, often performed in an integrated loop with short cycles.
For software/models with large user groups, canary deployment can be employed to reduce possible damage from faulty updates.
A copy of the production system is made and the new features/properties are applied to the copy.
Small user groups, either randomly chosen or selected through developer programs, are given access (unknowing or willingly).
Feedback and adoption is used for approval for full deployment or adjustment before a new canary cycle.
Companies like Spotify perform testing on user groups, setting up statistical experiments where the effect of changes to user interfaces and music recommendation algorithms can be measured real-time.