What causes the message 'We stopped hearing from Inidicum'
Since the new version the generation process is done by Indicium. However, often this message is displayed;
It usually disappears/appears a few times and the generation does not fail, but still, It does not look good to me.
What is the cause of this message?
Thanks!
Page 1 / 2
The issue was finally solved by making sure every application pool contained only one web app. Then I found another rather long error message in the Windows event viewer which after some research turned out to be the result of the application pool’s set user account not having enough rights on the IAM database (login worked but without access to the actual tables in the IAM database). After I gave the account sysadmin rights in SQL Server the tables became accessible, no new errors appeared in event viewer and Indicium started successfully. The license was now also renewed automatically. Everything seems to be working normally now.
Great to hear that Roland. Enjoy the latest version!
after checking our logs we found the following:
2021-12-13T14:27:30.0377288+01:00 ERR] Error scheduling system flow 'system_flow_agent_check_in' for application 14. (34d69cfe) Microsoft.Data.SqlClient.SqlException (0x80131904): <msg id="system_flow_still_running"></msg> at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption) at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod) at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult) at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location --- at Indicium.BackgroundServices.SystemFlowSchedulerBase.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 112 at Indicium.BackgroundServices.SystemFlowScheduler.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowScheduler.cs:line 90 at Indicium.BackgroundServices.SystemFlowSchedulerBase.ScheduleSystemFlow(Int32 guiApplID, String systemFlowID, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 84 ClientConnectionId:5f93a48f-65ed-41ac-a260-7e708c204d01 Error Number:50000,State:10,Class:16 ------------
maybe some recognizes this error? Or maybe TW team can work with the info.
thank you,
Paul
@pauloportu , well it's certainly a piece of the puzzle but I don’t have the solution right away sadly. If a restart of Indicium doesn't fix this then I'm not sure
But you’re going to fix it anyway right
Aside from that, happy to see i’m not the only one with these issues. Perhaps there is more at the handje then we think!
Internally we haven't heard any of these issues and personally only encounter this problem when (indeed) Indicium is not running. I'm also surprised as the logic behind it is pretty simple. And of course we want to solve the issue as soon as we can
Every 30 seconds, the System flow will update a datetime field in the Software Factory to determine if Indicium is still picking up and running new system flows. If the difference between current datetime and the logged datetime is greater than 65 seconds, we can safely assume Indicium is not picking up new system flows.
But it seems like your Indicium is picking up system flows selectively, or not updating the datetime field for some reason. ← That is what we need to find out
If you all could create a ticket in TCP, that would be great
I cannot help much here sadly as each of your cases may need a different solution. That requires deeper investigation and that is something the Community website is not suited for.
Done. Hopefully we can find it.
We are getting the exact same message in SF after upgrading from 2021.1 to 2022.1 yesterday. This rendered the SF almost unusable but I also don’t really want to downgrade after putting all the time in the upgrade process. I don’t get why this happens. The deployment tool displayed green arrows after upgrading all databases, indicium and the windows GUI.
Online documentation doesn’t really cover how to deal with this issue. Also how does SF know where to contact Indicium in the first place? I can’t find any setting for this.
Indicium is definitely running. What is going on here?
Hi @Roland ,
Can you check the Indicium log? Maybe there's an error hindering this process. Did you upgrade to 2022.1 with the latest installation package; when did you download it?
Hi Mark,
We have two Indicium instances. One for development (SF) and one for production (end-product). The dev version seems to work fine in that it renews the license but still SF fails to connect to it. The logs stopped at 7:07 this morning. Indicium production doesn’t build up a log at all. I don’t understand why because this was setup for us and both use the same application pool and are under the same main website in IIS.
I downloaded the installation package yesterday with all hotfixes included.
Could you create a ticket for this in TCP? There you can attach the complete log and explain the infrastructure. This way we can see the probable cause and resolve it.
Hi,
This message is given when Indicium does not pick up new system flows for at least 61 seconds. This usually means that Indicium is not running anymore and should be reactivated.
Hi,
This message is given when Indicium does not pick up new system flows for at least 61 seconds. This usually means that Indicium is not running anymore and should be reactivated.
Hi @Mark Jongeling,
How can we tweak this setting. My partner here in Brazil encounters this message more than I do. How does it exactly work? We have the Indicium running in the cloud. When he gets the message more than I , does this directly relate to internet speeds? And is there a way to tweak with settings?
Because I have been monitoring the Indicium and it hasn't gone offline yet, but still we receive these messages that: 'we stopped hearing from Indicium'.
@Mark Jongeling In addition to what Freddy is saying: our developers also encounter this message infrequently, while Indicium is absolutely not offline at those times.
@Freddy@Arie V, sorry to hear you both are encountering this. I'm curious to see the Schedule log of the System flow that makes this functionality work.
When going into IAM, navigate to Applications, select the SF application, navigate to detail tab Scheduled system flows. In here, the system flow "system_flow_agent_check_in” is the one I'm interested in. It should have a log record every 30 seconds like this; is that also the case for you both or is it different?:
(Ignore the new available Task, that's 2022.1 functionality )
This message is given when Indicium does not pick up new system flows for at least 61 seconds.
Originally it was 61 seconds but later we changed it to 65 seconds I just remembered, so we did increase it a bit. Looking forward seeing the schedule log
@Mark Jongeling Our System Log shows an entry every 30 seconds as expected. All other System Flows related to Creation have an interval of every 5 seconds, could it be that their Indicium inactive check is given after less than 65 seconds?
@Mark Jongeling Our System Log shows an entry every 30 seconds as expected. All other System Flows related to Creation have an interval of every 5 seconds, could it be that their Indicium inactive check is given after less than 65 seconds?
It should still update the agent check-in datetime every 30 seconds so the 61 or 65 second-check is not per se the issue here.
I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:
select * from last_agent_check_in
probably redundant info, but same here:
After restart indicium, no satisfactory results ubfortunatly.
please advise,
thank you,
Paul
probably redundant info, but same here:
After restart indicium, no satisfactory results ubfortunatly.
please advise,
thank you,
Paul
Hi @pauloportu,
We do need more information to help you with this. In normal circumstances this functionality should function just fine. Does Indicium show any errors in the error log? Does the system flow system_flow_agent_check_in have an active schedule?
Every case can be different, so I would advise you all to create a TCP ticket for this so we can investigate this.
I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:
select * from last_agent_check_in
@pauloportu , well it's certainly a piece of the puzzle but I don’t have the solution right away sadly. If a restart of Indicium doesn't fix this then I'm not sure
@pauloportu , well it's certainly a piece of the puzzle but I don’t have the solution right away sadly. If a restart of Indicium doesn't fix this then I'm not sure
But you’re going to fix it anyway right
Aside from that, happy to see i’m not the only one with these issues. Perhaps there is more at the handje then we think!
Sure, sounds good. Just to check, is TW going to look into this now, or do you need more from us? Just so we’re not waiting on each other.
I don’t think there is anything we can do right now to be honest.
If you all could create a ticket in TCP, that would be great
I cannot help much here sadly as each of your cases may need a different solution. That requires deeper investigation and that is something the Community website is not suited for.
To add to this discussion still. The log files show a log file entry every 30 seconds even with the text that we stopped hearing from indicium. What works is to go to indicium site and login. When I go back to SF the message disappears (after a refresh)…
To add to this discussion still. The log files show a log file entry every 30 seconds even with the text that we stopped hearing from indicium. What works is to go to indicium site and login. When I go back to SF the message disappears (after a refresh)…
@Freddy, this sounds like Indicium runs with a time-out and becomes inactive. In the docs we specified some settings that should be altered from the default: docs.
Again, I recommend creating a ticket for this in TCP so we can investigate your case.
In IIS, ensure in the application pool's Advanced Settings that the Idle Time-out (minutes) is set to 0 (disabled) instead of 20, so scheduled process flows will continue running even if there is no user activity.