Calling: Update data migration source task

Is there some info on this? Would be much appreciated!

Hi Edwin and Alex,

This Task sadly was not made API-friendly, leaving the only possibility to execute a SQL-query on the database to use this Task. 

We have been made aware and we are aiming to resolve these troubles in our current sprint. At this moment, I'm not sure we could resolve this with a platform improvement, but it will certainly be addressed and resolved for 2023.2 which will be released in June 2023. Sorry for the inconvenience.

Ah, that’s too bad. 
Thanks for the reply! We will try and find a way around this.. 

Hi @Edwin van de Peppel,

We have been automating deployments before TWP 2023.1 (see our lengthy replies here). We have started testing with TWP 2023.1 and noticed our Deployment Automation project needs a lot of updates, thanks to the SF Model/Branch changes. So sharing experiences is very much appreciated!

Regarding your question on the update_data_migration_source_version task:

• Up until TWP 2022.2 we used a view on our target Database to retrieve the last sf_product_info version and before initiating the Creation process we update the Previous version in Project versions. I believe that value is then also used for Data conversion source version.
• Would that still work in TWP 2023.1? If so, it might be a workaround for your scenario. If not, a Hotfix for the update_data_migration_source would be very much appreciated for us too @Mark Jongeling!

Hi @Edwin van de Peppel and @Arie V,

Due to looking at the title of the topic, I was looking at the shell of the true Task that updates the source version. The shell works via the GUI side of things, but the task that sets up the process of updating the source version is called: add_job_to_update_data_migration_source_version

Using the Indicium API, a call can be called like this:

POST [indicium]/iam/sf/add_job_to_update_data_migration_source_version

{
  "model_id": "[MODEL]",
  "branch_id": "[BRANCH]",
  "select_data_migration_source_version_option": 0,
  "from_branch_id": "string",
  "from_model_vrs_id": "2023-03-18T12:00:20.859Z",
  "runtime_configuration_id": "default",
  "job_id": null
}

Note that select_data_migration_source_version_option accepts the input 0 and 1.

• 0 = Obtain from database - requires runtime_configuration_id to be filled
• 1 = Specific model version - requires from_branch_id and from_model_vrs_id to be filled

(However, my test is failing due to a 422 error, but as it is Saturday at the moment, I can't ask for help 😅 )

Hi everyone, 

The issue has been found. This add job task has a mandatory hidden field which does not have a default value. We will be providing a hotfix for this as the task is now unavailable to be executed via the Indicium API. We have a User story currently on the sprint to ensure other API tasks are also correct.

@Mark Jongeling great to hear. I’ll check the tcp for the hotfix. 

As for the 2023.2  version, is it an idea to add this as a parameter to the create deployment process flow. As a deployment package is created so that the software will be installable at a customer. And therefor it should be used as the start point for the next version.

@Arie V Thank you for the insights. Now as for our build and deployment strategy for Thinkwise Applications. (As theses are linked).
We use Azure DevOps for both the build, as the deployment of said applications. We have a simple build tool which runs the create deployment package task (start_deployment_package) trough indicium. When the build completes, we move the upgrade database scripts and the model to a git repo, and commit them.
This triggers a new build which takes all database script & the model, and packages them as a build artifact.
For deployment, we first run all the migration database scripts on the target database. We have tooling in place that will skip any script that already has been run. And then we apply the model script to the IAM database, and then create a new application for this version of the software, based upon the last active version of the application. Note this is all done trough DevOps deployments, eliminating the need for manual interaction by a user.

This allows use to migrate from any version of the application, to next lasted version of the software in a mater of minutes. All environments are migrated this way. From test, uat to prod.

If for some reason, we have multiple releases that needs to be migrated, the database script are automatically run in the correct order, migrating the database and iam to the correct version of the model.

@Blommetje When the fix is delivered, I will run it on the sf database, and update the build tool so we can call this method during build. I’ll keep you updated so the tool can be updated on your end as well.

Thanks all

As for the 2023.2  version, is it an idea to add this as a parameter to the create deployment process flow. As a deployment package is created so that the software will be installable at a customer. And therefor it should be used as the start point for the next version.

We did not add this as when creating deployment packages, the Data migration should be set up correctly in advance of creating the package.

In case you would create a deployment package from a previously named version, and with the new upgrade introduced a new mandatory field without default value, it could cause the Deployment package to be created correctly, but upon upgrading the end product causing an error due to missing data migration.

Maybe we can revise this process a little to let the Data migration to play a bigger role in this Deployment package screen/process. I'll discuss this with the team.

@Mark Jongeling We're about to upgrade to TWP 2023.1 and I am trying to wrap my head around a couple of changes.

Deployment Package - Named version: Both here and in this topic you are referring to the requirement that a Named version is needed as Previous version.

• But in Model overview > Branches it seems that I can select any existing ‘Source branch’ + ‘Source model version’, with or without a Version name in Model settings.
• And also in the Generate definition > Specific model version I can select values that become hidden when enabling the ‘Named versions’ pre-filter in the lookup.

If the Deployment Package is not using these set Unnamed versions, then how do we know (and set) which Named version is actually used?

Data migration: I like the changes on the Creation screen, allowing to set the Source and the Next action in the Tasks on Generate definition! Now I am wondering: why not take this a step further and add the Data migration as a tab in Creation itself?

Also, I believe that on the topic of Data Migration there are quite a bit of loopholes, and on top of fixing the known issues, I truly believe significant improvements could be made to ensure that Data Migration changes (including manually Updated table/column mappings and upgrade scripts) are properly remembered from Branch to (MAIN) Branch, as well as between Model versions. 

Data migrations are supposed to be properly handled in the Branches by the Developers, and properly remembered after Merging to the Main, after which we could automate build and deployment of the MAIN after every Merge to the MAIN Branch.

And then once we are ready for another Release to our Production environment, we would actually use the Deployment package, of which it should be a no-brainer to be compiled, since the migration and testing has been dealt with earlier in the process.

So come to think of it, I believe it would actually be quite valuable to have the same Selection methods as on the Creation tasks on the Create deployment package task:

• ‘Obtain from database’ would work for our Deployment package-based Production deployments, as with the introduced functionality to store different Pool user credentials, we could obtain the right version directly.
• ‘Specific model version’ would allow us to select the right version, allowing us to know specifically for the Deployment package which Source model version is used.

Does this make sense or am I missing something here?

@Edwin van de Peppel I'm not sure if this would cover your request, as it seems you are referring to setting the Source model version after creating the Deployment package?

@Arie V Not quite, but i see where you are going to.
You are correct about the source model beeing set after creation, as it marks the start point for the next release. This is done so that at the next release we do not need to figure out what version marks the last release, and we just can start the creation process. Initial testing with the “obtain from database” option did not always have the desired result. I’m not sure if the option did not work because we did something wrong, but alas. The manual setting did give the correct results, so we went this route.

To start with the original question, the release platform improvement now allows users to call the Indicium API and run the add_job_to_update_data_migration_source_version task to initiate a job that will update the source version of a given branch.

To answer @Arie V's questions:

If the Deployment Package is not using these set Unnamed versions, then how do we know (and set) which Named version is actually used?

You are able to give model versions a status, such as Test, Acceptance or Production. We understand that can be difficult and tedious to manage but it does supply a way of noting which model version is in which status. ---> Feel free to create an Idea for how this process could be improved.

Knowing exactly which model version is run in for example Production, the Software Factory can't know that for sure until you obtain the source version from the Production database. We do advise to create a structure like mentioned in this blog: 

• Manage branches with Thinkwise Blog

With this structure, branches have a specific purpose and allow you to have a dedicated Production branch, and Acceptance branch, a Testing branch, A Development branch and several Feature branches. Merging these branches is the intended method of moving model data from one branch to another.

Also, it is possible to roll-out unnamed model versions. This is for Development purposes, making it quick and easy to roll-out new version of your development branches without having to worry about version names. That is why the source version can also be an unnamed version.

For deployment packages, this is not available. The Thinkwise deployment center needs a previous version to have a name. That allows the model version timestamp (model_vrs_id) to be a little different but still have the same name. This could come in handy when a hotfix is released but the name of the model version hasn't changed.

It is intended that the Thinkwise deployment center requires a previous model version to be named.

Data migration: I like the changes on the Creation screen, allowing to set the Source and the Next action in the Tasks on Generate definition! Now I am wondering: why not take this a step further and add the Data migration as a tab in Creation itself?

This could be an option to do of course. Data migration is an important part of successfully upgrading applications. Although during development, is may be less relevant. When we refactored the Creation screen, we did have a conversation internally about placing the Data migration screen in the Creation screens, but we decided to not do that. To my knowledge, we decided to not include it as this screen was to allow the Creation screens to stay similar to how it was in the past (pre 2021.3) to allow for recognizable screens. ---> Feel free to create an Idea for this, I'm interested to see how other Community members look at this. We do have the option to navigate to the Data migration from the Creation (Generate definitions) screen.

So come to think of it, I believe it would actually be quite valuable to have the same Selection methods as on the Creation tasks on the Create deployment package task

We believe that the source version should already have been set before creating a deployment package as the Data migration has to be set up properly before this process. I do feel the process can be made more robust by implementing a way to involve the Data migration in this screen, either by allowing the/a Task to navigate to the Data migration screen or by incorporating the screen inside the already present Deployment package screen. --->  Feel free to create another Idea for this part. Looking forward to see your view on it 😄

I will select Best answer on the earlier reply with the API calls as it answers the original question. This topic is derailing a bit I feel.

@Mark Jongeling Idea raised for ‘Set source for Data migration’ in Deployment package screen:

2023.1: Set source for Data migration in Deployment package screen | Thinkwise Community (thinkwisesoftware.com)