Version Control and Rollback Strategies for ML Models with Modelbit

Machine learning projects are iterative by nature. How do you keep track of all the moving pieces and evolving components like large datasets, code, models, pipelines, and production services? In traditional software engineering, version control has largely been solved with Git. But, because ML requires concurrent versioning of all those ever-evolving components, using Git alone can slow your workflow.

Here’s the problem: Standard Git isn’t designed for the kind of concurrent versioning that ML projects expect. It does not handle the versioning of large binary files efficiently (model and data), which slows down iteration. Most ML teams need a tool that takes the core functionalities of Git and makes it better for ML projects.

On the other hand, using systems and practices for version control specific to ML workflows, without discarding Git (which your product engineering team uses for every other area of your product), is vital to how you rapidly experiment and deploy models. 

So in this post, we’ll take a look at the need for version control designed for ML and walk through the steps of how to implement it for your production model code.

Version Control for ML Models with Modelbit

Modelbit is a lightweight platform that makes deploying and managing any ML model to a production endpoint fast and simple. It can track and manage model iterations with Git repositories. It stores the binary files in S3 without size limitations and with encryption and compression using Git’s smudge and clean filters. When you run "git add", the filter encrypts and uploads the file to S3, then stores a pointer file in Git.

Likewise, when you "git pull", the filter sees the pointer file and replaces it with the file downloaded and decrypted from S3. These filters only run on your current commit, so historical versions don’t slow your pull or take up storage space. 

Modelbit’s Git repos also consolidate model updates and deployment into continuous integration and continuous delivery (CI/CD) pipelines like GitHub Actions and GitLab CI/CD. This helps your team collaboratively address issues during deployment and post-deployment as they can create different branches via git and work on different project parts without impacting the main branch. You can merge the branch with the main when changes are complete.

Step-by-Step Guide on Using Modelbit for Version Control and Rollback 

You will start by syncing your Modelbit Git repository to your GitHub account (or your team’s). For this, you'll need a Modelbit account. If you don't have a Modelbit account, you can sign up for a free account to complete this walkthrough.

Sync the Modelbit Git repository with your GitHub account

Once your account is set up, the next step is syncing your Modelbit Git repository with a GitHub repository. This is essential to using GitHub's features like code review, CI/CD, and pull request workflows for your Modelbit deployments.

Create a new, empty GitHub repository dedicated to Modelbit. After setup, copy its SSH URL. This URL will link your Modelbit and GitHub repositories, integrating your ML model deployment workflows.

After setting up your GitHub repository, you would have to integrate it with Modelbit.

Go to the Modelbit dashboard and find the "Git Settings" section. Connect your Modelbit account to the GitHub repository by entering the SSH URL (e.g., git@github.com:your-username/modelbit-vcs):

Proceed by clicking "Next." This will prompt you to copy a Deploy Key, which acts as Modelbit's public key. Similar to a user's public key, you use it for pushing and pulling from your Git repository. It's important to note that this “Deploy Key” parameter is not confidential.

Successfully running this step ensures your Modelbit environment is synced with your GitHub repository for version control and collaboration with other Users.

Next, integrate this Deploy Key into your newly created GitHub repository. To do this, ensure you have the Deploy Key copied, and then navigate back to GitHub. 

Add the key to your repository with write access, establishing a secure connection for Modelbit to interact with your GitHub repository. This step enables Modelbit to securely push updates and pull changes from your repository:

Go back to the Modelbit interface and proceed by clicking "Next." Modelbit will perform a connection test with your GitHub repository at this stage. This test ensures that the Deploy Key you provided has been correctly set up with write access and that there are no branch protection rules in place that could hinder Modelbit's synchronization with the repository.

After clicking "Next," select the "Sync Now" option. This action will complete the synchronization process.

Once these checks are complete, your Modelbit workspace will be fully synchronized with your GitHub repository. This integration is crucial for streamlining your ML model development and deployment processes. Click “Finish” to proceed.

Build and deploy a simple model to Modelbit

To see how Modelbit helps with git-based workflows for rapid deployment and rollback, you’ll build a simple enough model to illustrate the workflow.

Find the entire code for this walkthrough in this repository.

Install ModelBit via pip. Run this command in your notebook:

!pip install modelbit

After installing, login to your Modelbit account directly from your notebook:

import modelbit
mb = modelbit.login()

Build a simple logistic regression model that trains on the Iris dataset. This workflow works irrespective of the model type you are deploying. Find all our model deployment tutorials on our blog.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression


# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target


# Create and train the logistic regression model
model = LogisticRegression()
model.fit(X, y)

Define an inference function. The inference function is what you'll deploy. It will take input data and use the fitted model to make predictions. Here's an example based on the logistic regression model you trained:

# Function for deployment
def predict_iris_species(sepal_length, sepal_width, petal_length, petal_width):
   """
   Predict the species of an iris flower using logistic regression.


   Args:
   sepal_length (float): Sepal length in cm.
   sepal_width (float): Sepal width in cm.
   petal_length (float): Petal length in cm.
   petal_width (float): Petal width in cm.


   Returns:
   str: Predicted species of the iris.
   """
   prediction = model.predict([[sepal_length, sepal_width, petal_length, petal_width]])
   return iris.target_names[prediction][0]

Deploy the inference function with “mb.deploy()”:

mb.deploy(predict_iris_species)

After deploying the model, click “View in Modelbit” to access and review the details from the Modelbit web application. This centralized dashboard view provides the tools and information to use, manage, and monitor the model.

The information includes:

• API Endpoints: Detailed information about the API endpoints generated for your deployed model to understand how to interact with your model programmatically.
• Logs: Logs that provide insights into the operational aspects of your model, including runtime behavior and potential issues.
• Tests: Information on unit tests, offering an overview of the model's performance and reliability.
• Training Jobs: Details about the training jobs associated with your model, including parameters used, duration, and outcomes.
• Source Code: View the source code of your deployed model, allowing for review and potential modifications.
• Environment: Detailed information about the environment in which your model is running, including the software and hardware configurations.‍
• Usage: Insights into the usage statistics of your model, such as the number of requests it has processed, peak usage times, and more.

Establish secure SSH with your Modelbit Git repository

To establish a connection with your Modelbit Git repository, uploading an SSH key is essential. This key allows you to securely clone the repository onto your local machine. Modelbit's Git repositories function similarly to those of GitHub and GitLab regarding SSH key usage.

Here's a step-by-step guide to setting this up:

1. Generate an SSH key:

• Run the following code in your CLI, replacing `your-email@gmail.com` with the email associated with your GitHub account:

ssh-keygen -t ed25519-sk -C your-email@gmail.com

This command generates a new SSH key using your email as a label. Navigate to the path where the ssh key was saved to retrieve its content.

You can refer to GitHub's comprehensive guide for more detailed information on generating and managing SSH keys.

2. Add the SSH key to Modelbit:

• Navigate to the Settings in your Modelbit account.
• Under “User Settings”, select 'SSH Keys'.
• Click on “Add Key” in Modelbit.
• Open the generated SSH key file (usually found at `~/.ssh/id_ed25519.pub`) and copy its contents.
• Paste the key into the 'Key' field in Modelbit and assign a nickname like "Modelbit Connect" for easy identification.

Click “Add Key” to save your SSH key in Modelbit.

With these steps, your SSH key will be securely stored in Modelbit, enabling you to use the “modelbit clone” command to clone repositories to your local machine.

Next, you have the option to clone your Modelbit deployment repository. An advantageous feature of the Modelbit workspace is its support for large binary files, such as “.pkl” and “.joblib”. These files, commonly used for pickle machine learning artifacts, are typically not supported by standard Git operations. Cloning your Modelbit deployment repository is straightforward. Run the following in your CLI or directly within your notebook cell:

modelbit clone predict_iris_species

This code will clone into the Modelbit git repository and create a folder named “predict_iris_species”, 

Output:

Here’s the directory structure:

├── README.md
├── bin
│   ├── format.sh
│   └── setupHooks.sh
├── common
├── datasets
├── deployments
│   └── predict_iris_species
├── endpoints
└── packages

After cloning the remote repository, you can make local changes and push them back to the repository. See how this process works by updating your source code to the “predict_iris_species” deployment folder with the following file structure:

└── predict_iris_species
    ├── data
    ├── metadata.yaml
    ├── requirements.txt
    └── source.py

Here's how to do it:

1. First, check out of the main branch to make changes to our “source_file”:

git checkout -b new_branch

2. Navigate to the “source.py”  file. Simply update the source code.

3. Next, stage and commit the changes to the source file:

git add source.py
git commit -m "Update source code"

4. Next, push the changes to the new branch of the remote repository hosted on Modelbit as follows:

git push –set-upstream origin new_branch

5. After committing your changes locally, push them to the remote repository hosted on Modelbit:

git push origin master

Output:

This command will update your branch on the remote repository with your latest changes. If you return to our GitHub repository, you will see a new pull request with the changes on the “new_branch,” ready for review and merging to the main branch.

Perfect! Now you can easily manage and synchronize changes between your local and remote repositories in Modelbit, allowing for a flexible and efficient workflow in managing your ML model artifacts. 

With this workflow, you can retrain models by updating the inference function in “source.py” for deployment.

Go to the “Git Settings” on the Modelbit UI and find the new branch created. To safeguard against unintended modifications in Modelbit, use the feature of protected branches.

Branching in Modelbit

Working with separate branches in the remote Modelbit repository is a seamless process, allowing for isolated development and experimentation without affecting the main codebase.

Creating a New Branch via the Modelbit Dashboard:

• Navigate to the Modelbit dashboard.
• Create a new branch:

Switching to the New Branch in Your Notebook:

If you want to enhance or modify your model in this newly created branch, switch to the branch directly from your notebook. Here's how you can do it:

1. Switch to the new branch. In your Jupyter notebook (or any Python environment), use the “mb.switch_branch()” API to switch branches:

import modelbit as mb

# Replace "another_branch" with the name of the branch you created.
mb.switch_branch("another_branch") 

Any new deployments or commits you make from your notebook will be directed to the branch “another_branch” in your remote Modelbit Git repository. This ensures that your main codebase remains unaffected while you work on the improvements or changes in a separate branch.

This branching functionality in Modelbit improves the flexibility of your ML model development. You can run new project iterations, make commits that trigger your CI pipeline, and develop and deploy different model versions simultaneously without impacting your production service or, subsequently, your users.

Rollback & Experiment Strategies for ML Models

Implementing effective rollback strategies is essential for maintaining stable production services. You can leverage any of these strategies when a new model version you deployed shows unexpected behavior or performance issues. They enable you to quickly “roll back” to a previous, more reliable version. 

Below are the various rollback methodologies you implement with Modelbit:

1. Version control rollback: Using version control, you can efficiently revert to a stable model version. In Modelbit, you can switch to a previous commit (or the model branch) to restore the last state of your model, including the version name and related configuration files.
2. Rolling deployments:  Modelbit runs an n+1 deployment (version increments) where multiple instances of the model service run on independent nodes. Here, you grade each version for every new deployment. After upgrading each version, you may test it for stability. If you find issues, roll back to the previous API version. The service stays online during deployment as the other nodes share the load.‍
3. A/B Testing: A/B testing involves deploying different model versions to distinct segments of users. Their performances are compared critically, and the new model is rolled back if it doesn't meet expectations. We created a comprehensive tutorial on simulating this strategy using Modelbit.‍
4. Shadow deployment: The new model version “shadow” can operate parallel with the “champion” model (main version in production) but does not return results but is monitored to validate stability and performance. On successful validation, the new version is merged into the main branch. In Modelbit, you could deploy one version of your model on the main production branch, deploy another on a different “shadow_branch” branch, and use request mirroring to receive production traffic.

Each strategy offers a unique approach to managing model deployments, ensuring that ML systems remain robust and perform optimally even when new model versions are unstable or perform poorly.

Conclusion

Alright, so we talked about the significance of robust version control and strategic rollback planning for your machine learning projects. Integrating these strategies with Modelbit can enable a streamlined and efficient workflow for those wishing to improve their ML model development and deployment. 

Modelbit's version control feature is ideal for adequately managing ML projects and handling large binary files because it encrypts and decodes the assets upon every “push” or “pull”. The next steps are to explore Modelbit's features further, see how to integrate them with a CI tool like GitLab CI, and use these best practices in your ML deployment processes for optimal results.