Can you help me understand the architecture of the new db migration? Why do you attempt to create the databases on each migration run? What is the best practice for running the migration task? Should it be performed in a downtime? I have yet to try stopping ingestion but the create dbs part of the sync task gets a timeout while executing. Would love some insight into how others are managing their clusters during upgrades.

Why do you attempt to create the databases on each migration run?

The command that get executed is

CREATE ... IF NOT EXISTS ...

, so if db already exists it doesn't do anything.

What is the best practice for running the migration task? Should it be performed in a downtime?

The objective of the new schema migrator is zero downtime migrations. Either migration happens and the upgrade is successful without any downtime or no upgrade (with upgrade failure reasons).

but the create dbs part of the sync task gets a timeout while executing.

This indicates that something is blocking the DDL execution. See if you have any mutations pending.

Thanks...there are no pending mutations. Any other troubleshooting tips?

Are there any unfinished DDl tasks? Did it timeout when it tries to establish connection or did it timeout when it's waiting for

CREATE ... IF NOT EXISTS ...

to complete?