Skip to main content
Solved

Automatic data migration between branches when renaming tables / columns

  • 4 July 2024
  • 8 replies
  • 98 views

Our branching strategy is setup in the following manner: we have a “MAIN” branch and each time we release a production version a separate branch is created. The upgrade process basically consists of upgrading from one branch to another branch. This works fine, but i recently stumbled upon a problem.

When renaming a table or column the data migration in the SF does not automatically see this as a change, but as a “removed”  and “new” table / column. This is probably because the upgrade scripts are generated between branches instead of within a single branch. I don't know how the SF determines when table / columns are changed or new. Is automatic data migration not possible between branches or am i perhaps missing something?

For now we fixed it by by creating data migration scripts and manually set the data migration.

This topic has been closed for comments

8 replies

Mark Jongeling
Rank
Userlevel 7
Badge +23

Hi Ling Wu,

It is not intended to work like this I think. Why would you create a new branch everytime a new version is released?

The Software Factory is temporal, meaning all changes are stored in history tables. At anytime you are able to "go back” and create a branch from a specific point in time. You can simply release from MAIN for example and let de SF create a model version for you with the Production version name.

When we release a new platform version, the MAIN is generated with the version name; e.g. 2024.3. This will in turn create a model version with that name. If at any time we need to go back for some reason, we can create a branch from this model version.

"Wake me up before you low-code"
E
Rank
Userlevel 6
Badge +16

Did you use re-apply intermediate mapping? By default it sets the columns to new/removed. After using re-apply intermediate mapping it should fix the data migration. 

P
Rank
Userlevel 1
Badge +2

I can reproduce this with the following situation (in SF 2024.2):

  1. Database is in sync with Main
  1. In Main: rename table/col
  1. Create branch from main
     
  2. Generate upgrade script for the database to the branch. 

    Re-apply intermediate mappings have no effect:

 

Even in the proposed branching strategy (Manage branches with Thinkwise | Thinkwise Community (thinkwisesoftware.com)) this could happen when working with a hotfix branch at production, when creating feature branches and possibly when using a production database to make a new acceptance (release branch) database with fresh data.

This seems very risky for potential data loss, so I hope a solution can be found. At the very least there could be a validation message for objects that have status ‘removed’ by data migration so that a developer has a trigger to manually update the data migration (even though there could be false-positives). 

Mark Jongeling
Rank
Userlevel 7
Badge +23

Hey @PeterKeeris,

This scenario should also work I believe. Which platform version are you running on?

In our latest release I tried your reproduction but I do get a mapping:

Mapping
Mapped

Edit: I worked with a branch and a branch of that branch, that might be why it's different; checking...

"Wake me up before you low-code"
M
Rank

@Ling Wu Reading your issue, I can confirm I also have encountered a few occasions where the renamed table was seen as new/remove after the merge. No idea what exactly causes it to detect it correctly one situation, but not in another.

Seeing you mention you did create manual data migration scripts, I wanted to let you know you can change the “new” table to let it know it has an old “remove” table to get its data from. Use the task “Update table mapping” in the Data migration and let give it the old table as “From table”.
 

Afterwards it will show your table as “Changed”.

Perhaps this is what you meant by manual scripts, but I’d better let you know in case it is now :-) Hope it helps for the times you need it.

Mark Jongeling
Rank
Userlevel 7
Badge +23

I have discussed this with the team and we determined the effort it takes to resolve this would be significantly higher than the value it brings.

Reasons include that:

  1. Working in the Origin branch is not encouraged. Renaming objects such as tables and column require a deployment first before branching.
  2. Creating a branch from an "work in progress” origin is not recommended.
  3. Before upgrading, the data migration should be verified to prevent data loss - such as would happen in this case.

The workaround would simply be to link the old and the new table so the data mapping is in place and data is transferred to the new table.

"Wake me up before you low-code"
L
Rank
Badge

Thanks for the answers.

We rarely change table names or columns. In future upgrades we plan to resolve this by creating conversion scripts for the data migration that are executed during the upgrade to make sure no data is lossed and to rename only if absolutely necessary.

Mark Jongeling
Rank
Userlevel 7
Badge +23

Renaming tables or columns should not be a scary thing to do. The data migration is a vital part of upgrading. Simply verify it before upgrading and link/unlink or adjust default values accordingly.

"Wake me up before you low-code"
Terms & ConditionsCookie settings