As a side note, I have not looked into the possibility of parallel processing in SQL, my question is purely based on my past experience with .NET (C#)
Best answer by Henk Nicolai
View original
Parallel processes
- Warrior
- 63 replies
In the process flow modeller it is possible to create parallel processes, now I know that these are handled linearly and depth-first. But what I would like to know is, will it ever (in the near future) be possible to actually execute them in parallel. In my specific case I'm calling multiple web api's, which would benefit from parallel processing (even if the results would be processed on the main thread). Now I know parallel processing is quite complex, and would go far beyond the scope of what the average 'modeler' should know, but for my case parallel processing would have been perfect.
As a side note, I have not looked into the possibility of parallel processing in SQL, my question is purely based on my past experience with .NET (C#)
As a side note, I have not looked into the possibility of parallel processing in SQL, my question is purely based on my past experience with .NET (C#)
Userlevel 2
+1
We've done a preliminary research this month as to how asynchronous process flows could be implemented, and have not reached any definitive conclusions yet. Concurrency in particular becomes an issue - even with native C# synchronization primitives (e.g. async and lock keywords) asynchronous C# code is notoriously hard to write and debug. The SF doesn't even offer those kinds of primitives, and I'm not sure what they should be. I only know that access to shared resources such as process flow variables should be synchronized.
As an SF developer, I need to ask myself the question what will happen if a novice developer finds the 'asynchronous' checkbox and decides to check that option for all process actions because he believes that this must be good for performance. And then, once he runs into concurrency problems, how are they going to manifest themselves? Is there a risk for data corruption? And, how are we going to reproduce, analyze and resolve issues with this feature?
The good news is that we plan to move part of the process flow implementation to Indicium so we can introduce server-side process actions, and provide hooks so we can finally add multi-user workflows to the SF. After this project, I expect that asynchronous process flows become much more natural to fit in. So I remain hopeful that we can return to this feature this or next year.
As an SF developer, I need to ask myself the question what will happen if a novice developer finds the 'asynchronous' checkbox and decides to check that option for all process actions because he believes that this must be good for performance. And then, once he runs into concurrency problems, how are they going to manifest themselves? Is there a risk for data corruption? And, how are we going to reproduce, analyze and resolve issues with this feature?
The good news is that we plan to move part of the process flow implementation to Indicium so we can introduce server-side process actions, and provide hooks so we can finally add multi-user workflows to the SF. After this project, I expect that asynchronous process flows become much more natural to fit in. So I remain hopeful that we can return to this feature this or next year.
- Warrior
- 63 replies
-
22 February 2019
Hi Henk,
Thank you for your reply.
It is good to hear Thinkwise also has this issue in view. I know parallel processing is very complex, and have myself experienced the potential problems with it. And I fully understand the risks in allowing users to enable tasks to run asynchronously. But I do think that never allowing parallel processing because of this would be a real shame though, given the potential benefits.
But I am curious to see where Thinkwise will go with this, and if, when and how this will be implemented.
Thank you for your reply.
It is good to hear Thinkwise also has this issue in view. I know parallel processing is very complex, and have myself experienced the potential problems with it. And I fully understand the risks in allowing users to enable tasks to run asynchronously. But I do think that never allowing parallel processing because of this would be a real shame though, given the potential benefits.
But I am curious to see where Thinkwise will go with this, and if, when and how this will be implemented.
Userlevel 5
+5
Currently our Windows User Interface supports asynchronous execution of SQL tasks and reports exported to files, like PDF, Excel or Zip. You may use them within a process flow defined in the SF as well. Notice that the execution of the flow execution will not wait for the completion of the action. The process step will be assumed to be successful and the flow will continue to execute the rest of the flow directly after the start of the asynchronous step.
To configure a Task or a Report to execute asynchronous from the SF, set field "await result" to "No".
As Henk mentioned before, as a SF developer, think about the consequences of using this without waiting for the completion of the action in a flow.
I think, file access and web service calls by a process flow would have the same option and work the same way. These actions communicate with external resources and in most cases the GUI won't have to wait for it to continue executing user processes.
To configure a Task or a Report to execute asynchronous from the SF, set field "await result" to "No".
As Henk mentioned before, as a SF developer, think about the consequences of using this without waiting for the completion of the action in a flow.
I think, file access and web service calls by a process flow would have the same option and work the same way. These actions communicate with external resources and in most cases the GUI won't have to wait for it to continue executing user processes.
- Warrior
- 63 replies
-
27 February 2019
Thanks for your reply Erik,
I know of this option, but must say I have never really played around with it.
In this specific case though, this will not work. I have a process flow with multiple branches that could run asynchronously, but within each branch the separate steps must deliver their result before the next step can happen.
I know of this option, but must say I have never really played around with it.
In this specific case though, this will not work. I have a process flow with multiple branches that could run asynchronously, but within each branch the separate steps must deliver their result before the next step can happen.
Userlevel 5
+5
So in this case we should mark a bundle of branches to be executed in parallel. Thanks for your feedback. We will take this in consideration when we design this feature.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.