What causes the message 'We stopped hearing from Inidicum'

Hi,

This message is given when Indicium does not pick up new system flows for at least 61 seconds. This usually means that Indicium is not running anymore and should be reactivated.

Hi @Mark Jongeling, 

How can we tweak this setting. My partner here in Brazil encounters this message more than I do. How does it exactly work? We have the Indicium running in the cloud. When he gets the message more than I , does this directly relate to internet speeds? And is there a way to tweak with settings?

Because I have been monitoring the Indicium and it hasn't gone offline yet, but still we receive these messages that: 'we stopped hearing from Indicium'. 

@Mark Jongeling In addition to what Freddy is saying: our developers also encounter this message infrequently, while Indicium is absolutely not offline at those times.

@Freddy @Arie V, sorry to hear you both are encountering this. I'm curious to see the Schedule log of the System flow that makes this functionality work. 

When going into IAM, navigate to Applications, select the SF application, navigate to detail tab Scheduled system flows. In here, the system flow "system_flow_agent_check_in” is the one I'm interested in. It should have a log record every 30 seconds like this; is that also the case for you both or is it different?:

(Ignore the new available Task, that's 2022.1 functionality  )

Originally it was 61 seconds but later we changed it to 65 seconds I just remembered, so we did increase it a bit. Looking forward seeing the schedule log  

@Mark Jongeling Our System Log shows an entry every 30 seconds as expected. All other System Flows related to Creation have an interval of every 5 seconds, could it be that their Indicium inactive check is given after less than 65 seconds?

It should still update the agent check-in datetime every 30 seconds so the 61 or 65 second-check is not per se the issue here.

I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:

select * from last_agent_check_in

probably redundant info, but same here:

After restart indicium, no satisfactory results ubfortunatly.

please advise,

thank you,

Paul

Hi @pauloportu,

We do need more information to help you with this. In normal circumstances this functionality should function just fine. Does Indicium show any errors in the error log? Does the system flow system_flow_agent_check_in have an active schedule?

Every case can be different, so I would advise you all to create a TCP ticket for this so we can investigate this.

I advise you to create a TCP ticket in which we can take a deeper look at it. What we need is a couple of screenshots of the Schedule log, the “We stopped hearing from Indicium” text appearing, the Indicium error log of the day when the texts appear and the result of the following SQL query run on the SF database:

select * from last_agent_check_in

@pauloportu , well it's certainly a piece of the puzzle but I don’t have the solution right away sadly. If a restart of Indicium doesn't fix this then I'm not sure

But you’re going to fix it anyway right 

Aside from that, happy to see i’m not the only one with these issues. Perhaps there is more at the handje  then we think! 

Internally we haven't heard any of these issues and personally only encounter this problem when (indeed) Indicium is not running. I'm also surprised as the logic behind it is pretty simple. And of course we want to solve the issue as soon as we can 

Every 30 seconds, the System flow will update a datetime field in the Software Factory to determine if Indicium is still picking up and running new system flows. If the difference between current datetime and the logged datetime is greater than 65 seconds, we can safely assume Indicium is not picking up new system flows.

But it seems like your Indicium is picking up system flows selectively, or not updating the datetime field for some reason. ← That is what we need to find out 

Sure, sounds good. Just to check, is TW going to look into this now, or do you need more from us? Just so we’re not waiting on each other.

I don’t think there is anything we can do right now to be honest. 

If you all could create a ticket in TCP, that would be great  

I cannot help much here sadly as each of your cases may need a different solution. That requires deeper investigation and that is something the Community website is not suited for.

Done. Hopefully we can find it. 

To add to this discussion still. The log files show a log file entry every 30 seconds even with the text that we stopped hearing from indicium. What works is to go to indicium site and login. When I go back to SF the message disappears (after a refresh)… 

@Freddy, this sounds like Indicium runs with a time-out and becomes inactive. In the docs we specified some settings that should be altered from the default: docs.

Again, I recommend creating a ticket for this in TCP so we can investigate your case.

Time out is probably the problem:

Idle time-out​

In IIS, ensure in the application pool's Advanced Settings that the Idle Time-out (minutes) is set to 0 (disabled) instead of 20, so scheduled process flows will continue running even if there is no user activity.

 

This was already the case in my IIS.

@Mark Jongeling also told me to do a shrink on the iam DB. I shrunk files and logs. 

Although I still saw the error, I have a feelings it's less often. I'll keep an eye out and after today check the logs. 

select * from last_agent_check_in

@Mark Jongeling TCP 2648S raised. Noticed an interesting error, hope that it is related to this issue:

2021-12-16T09:06:08.2865241+00:00  [ERR] Error scheduling system flow 'system_flow_generate_definition' for application 11. (0c778fc7)
Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 209) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at Indicium.BackgroundServices.SystemFlowSchedulerBase.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 112
   at Indicium.BackgroundServices.SystemFlowScheduler.RunSystemFlow(Int32 guiApplID, TSFApplication application, FullProcessFlow systemFlow, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowScheduler.cs:line 90
   at Indicium.BackgroundServices.SystemFlowSchedulerBase.ScheduleSystemFlow(Int32 guiApplID, String systemFlowID, DateTime scheduledTime) in C:\azp\agent\_work\1\s\src\Indicium\BackgroundServices\SystemFlowSchedulerBase.cs:line 84
ClientConnectionId:cebd2f0d-f4be-42c7-b3e5-a33b7b1c34ad
Error Number:1205,State:52,Class:13

@Arie V  Yes, we have the same Deadlock-error. i’ve seen it in the log file as well - uploaded to the tcp in the ticket. 

@Mark Jongeling  Yesterday we had the issue just as much as normal though, so the shrink did not solve it. 

@Blommetje, can you confirm that in the Ticket and attach the log file? Thanks!

We are getting the exact same message in SF after upgrading from 2021.1 to 2022.1 yesterday. This rendered the SF almost unusable but I also don’t really want to downgrade after putting all the time in the upgrade process. I don’t get why this happens. The deployment tool displayed green arrows after upgrading all databases, indicium and the windows GUI.

Online documentation doesn’t really cover how to deal with this issue. Also how does SF know where to contact Indicium in the first place? I can’t find any setting for this.

Indicium is definitely running. What is going on here?

Hi @Roland ,

Can you check the Indicium log? Maybe there's an error hindering this process. Did you upgrade to 2022.1 with the latest installation package; when did you download it?