Automating Breaking Schema Changes? (Or alternative best practice)

[stack_engineering]

Hi all!

We developed quite a few solid pipelines over the past few months. We can rebuild the entire db and app automagically, or push incremental data changes up with little to no user intervention or decision making. This works for us because our application is read only/configuration, and not taking writes or storing irreplaceable user data.

Recently, we tried to push up a schema change, and this resulted in a breaking change warning from the cli:

push failed: error validating server: cannot make breaking change: update is missing property.

Doing some quick research, we found this excellent post on how to resolve this issue, which worked perfectly. It involves turning off sync, logging into the UI to complete the schema update, then re-enabling sync.

Realm CLI push with breaking Changes - MongoDB Atlas App Services & Realm - MongoDB Developer Community Forums

After that, I wanted to see if this step could be automated from the CLI. It appears it cannot be, and this is by design:

Because breaking or destructive schema changes require special handling, App Services does not support making these changes via App Services CLI or automated deploy with GitHub. Instead, you should make breaking changes in the App Services UI.

Update Your Data Model — Atlas App Services (mongodb.com)

I guess my question is, what’s the best way to handle this type of operation in a pipeline? Do most people just pause the deployment at that point, exit out of the automation (maybe send an email to the change team), and then resume once the manual step is completed? Also I am guessing most of the time you want to avoid Breaking Changes, but sometimes it may be necessary.

Would be curious to see how others are handling this scenario and if there is any quality of life type of automation that’s possible.

[Tyler_Kaye]

Hi,

First, I wanted to say thank you for your clear and concise feedback.

You are correct that right now we do not allow for destructive changes to be made from the CLI. This is unfortunate, but we view it as important because the CLI previously had no way for the server to warn the CLI user about the dangers of making certain changes (it causes a restart of device sync) in a way similar to how we warn the users in the UI when making these changes.

Fortunately, a new version of the CLI has been released and it should enable us to provide some sort of back-and-forth to warn users of the dangers of making a breaking change and providing some sort of [y/n] input or a --force flag that would automatically force it through. We have a ticket for this and I will bring it up during our next planning cycle.

Its worth noting that we will be focussing some effort on cleaning up the experience of using CI/CD and config-as-code across all of our services. In fact, I believe there is some research being done into this now and if you would have any interest in providing feedback, I can forward this to the team in charge of this and they might reach out to you.

As for the “what you can do now”, the best way forward is to have your CLI perform the following actions:

1. Terminate sync and push (set the “state” field in the sync config to an empty string)
2. Make the changes to the schema
3. Enable sync (set the state field back to “enabled”)

This should let you do what you want and should be effectively the same thing as making a destructive change.

Best,
Tyler

[stack_engineering]

Thank you Tyler! Appreciate the information and always impressed by the level of engagement by the Mongo employees direct on the forums. That’s awesome.

Will incorporate the above procedure into our workflow.

I’d be happy to provide any feedback to the CI/CD team. We’re currently using simple bash scripts for our pipelines, while working out the process flow and different change scenarios, but plan on integrating into a GitHub actions type of CI/CD solution down the road.

Currently we use the following CLI tools for builds:

• mongodump (backups)
• mongoexport/import (moving data)
• appservices (Push and Pull the App)
• mongosh (querying db for meta data or operations like truncating target Collections to import data without causing a Breaking Change)

Referenced Lauren Schaefer’s excellent article as a blueprint:

How to Build CI/CD Pipelines for MongoDB Realm Apps Using GitHub Actions | MongoDB