+12
Since the new version the generation process is done by Indicium. However, often this message is displayed;
It usually disappears/appears a few times and the generation does not fail, but still, It does not look good to me.
What is the cause of this message?
Thanks!
Best answer by pauloportu
View original
+23
Great to hear that Roland. Enjoy the latest version! 
The issue was finally solved by making sure every application pool contained only one web app. Then I found another rather long error message in the Windows event viewer which after some research turned out to be the result of the application pool’s set user account not having enough rights on the IAM database (login worked but without access to the actual tables in the IAM database). After I gave the account sysadmin rights in SQL Server the tables became accessible, no new errors appeared in event viewer and Indicium started successfully. The license was now also renewed automatically. Everything seems to be working normally now.
+23
Hi Roland,
What said the latest log entry at 07:07? Indicium does needs its own application pool, you may not put two in the same pool. They can be under the same main website. Here more about it: https://docs.thinkwisesoftware.com/docs/deployment/indicium#create-an-application-pool
Could you create a ticket for this in TCP? There you can attach the complete log and explain the infrastructure. This way we can see the probable cause and resolve it.
Hi Mark,
We have two Indicium instances. One for development (SF) and one for production (end-product). The dev version seems to work fine in that it renews the license but still SF fails to connect to it. The logs stopped at 7:07 this morning. Indicium production doesn’t build up a log at all. I don’t understand why because this was setup for us and both use the same application pool and are under the same main website in IIS.
I downloaded the installation package yesterday with all hotfixes included.
+23
Hi
Can you check the Indicium log? Maybe there's an error hindering this process. Did you upgrade to 2022.1 with the latest installation package; when did you download it?
We are getting the exact same message in SF after upgrading from 2021.1 to 2022.1 yesterday. This rendered the SF almost unusable but I also don’t really want to downgrade after putting all the time in the upgrade process. I don’t get why this happens. The deployment tool displayed green arrows after upgrading all databases, indicium and the windows GUI.
Online documentation doesn’t really cover how to deal with this issue. Also how does SF know where to contact Indicium in the first place? I can’t find any setting for this.
Indicium is definitely running. What is going on here?
+23
+12
It should still update the agent check-in datetime every 30 seconds so the 61 or 65 second-check is not per se the issue here.
I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:
select * from last_agent_check_in
2021-12-16T09:06:08.2865241+00:00 [ERR] Error scheduling system flow 'system_flow_generate_definition' for application 11. (0c778fc7)
Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 209) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Indicium.BackgroundServices.SystemFlowSchedulerBase.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 112
at Indicium.BackgroundServices.SystemFlowScheduler.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowScheduler.cs:line 90
at Indicium.BackgroundServices.SystemFlowSchedulerBase.ScheduleSystemFlow(Int32 guiApplID, String systemFlowID, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 84
ClientConnectionId:cebd2f0d-f4be-42c7-b3e5-a33b7b1c34ad
Error Number:1205,State:52,Class:13
+12
Time out is probably the problem:
In IIS, ensure in the application pool's Advanced Settings that the Idle Time-out (minutes) is set to 0 (disabled) instead of 20, so scheduled process flows will continue running even if there is no user activity.
This was already the case in my IIS.
Although I still saw the error, I have a feelings it's less often. I'll keep an eye out and after today check the logs.
Time out is probably the problem:
In IIS, ensure in the application pool's Advanced Settings that the Idle Time-out (minutes) is set to 0 (disabled) instead of 20, so scheduled process flows will continue running even if there is no user activity.
+23
To add to this discussion still. The log files show a log file entry every 30 seconds even with the text that we stopped hearing from indicium. What works is to go to indicium site and login. When I go back to SF the message disappears (after a refresh)…
Again, I recommend creating a ticket for this in TCP so we can investigate your case.
+16
To add to this discussion still. The log files show a log file entry every 30 seconds even with the text that we stopped hearing from indicium. What works is to go to indicium site and login. When I go back to SF the message disappears (after a refresh)…
+12
If you all could create a ticket in TCP, that would be great 
I cannot help much here sadly as each of your cases may need a different solution. That requires deeper investigation and that is something the Community website is not suited for.
Done. Hopefully we can find it.
+23
If you all could create a ticket in TCP, that would be great
I cannot help much here sadly as each of your cases may need a different solution. That requires deeper investigation and that is something the Community website is not suited for.
+12
Sure, sounds good. Just to check, is TW going to look into this now, or do you need more from us? Just so we’re not waiting on each other.
I don’t think there is anything we can do right now to be honest.
+23
But you’re going to fix it anyway right 
Aside from that, happy to see i’m not the only one with these issues. Perhaps there is more at the handje 
then we think!
Internally we haven't heard any of these issues and personally only encounter this problem when (indeed) Indicium is not running. I'm also surprised as the logic behind it is pretty simple. And of course we want to solve the issue as soon as we can 
Every 30 seconds, the System flow will update a datetime field in the Software Factory to determine if Indicium is still picking up and running new system flows. If the difference between current datetime and the logged datetime is greater than 65 seconds, we can safely assume Indicium is not picking up new system flows.
But it seems like your Indicium is picking up system flows selectively, or not updating the datetime field for some reason. ← That is what we need to find out 
+12
But you’re going to fix it anyway right
Aside from that, happy to see i’m not the only one with these issues. Perhaps there is more at the handje then we think!
+23
after checking our logs we found the following:
2021-12-13T14:27:30.0377288+01:00 [ERR] Error scheduling system flow 'system_flow_agent_check_in' for application 14. (34d69cfe)
Microsoft.Data.SqlClient.SqlException (0x80131904): <msg id="system_flow_still_running"></msg>
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Indicium.BackgroundServices.SystemFlowSchedulerBase.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 112
at Indicium.BackgroundServices.SystemFlowScheduler.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowScheduler.cs:line 90
at Indicium.BackgroundServices.SystemFlowSchedulerBase.ScheduleSystemFlow(Int32 guiApplID, String systemFlowID, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 84
ClientConnectionId:5f93a48f-65ed-41ac-a260-7e708c204d01
Error Number:50000,State:10,Class:16
------------
maybe some recognizes this error? Or maybe TW team can work with the info.
thank you,
Paul
+23
probably redundant info, but same here:
After restart indicium, no satisfactory results ubfortunatly.
please advise,
thank you,
Paul
Hi
We do need more information to help you with this. In normal circumstances this functionality should function just fine. Does Indicium show any errors in the error log? Does the system flow system_flow_agent_check_in have an active schedule?
Every case can be different, so I would advise you all to create a TCP ticket for this so we can investigate this.
I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:
select * from last_agent_check_in
probably redundant info, but same here:
After restart indicium, no satisfactory results ubfortunatly.
please advise,
thank you,
Paul
+23
It should still update the agent check-in datetime every 30 seconds so the 61 or 65 second-check is not per se the issue here.
I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:
select * from last_agent_check_in
+23
This message is given when Indicium does not pick up new system flows for at least 61 seconds.
Originally it was 61 seconds but later we changed it to 65 seconds I just remembered, so we did increase it a bit. Looking forward seeing the schedule log 
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.