Lifetime of idle staged resources

Hello Jeroen,

Staged resources are currently not cleaned up automatically. There are a few reasons that have contributed to this decision and our estimation that this is not an issue:

• Most clients that use resource staging are Universal GUIs and the Universal GUI always cleans up resources when it’s done with them, except, as you stated, when it’s closed improperly, in which case it could leak one staged resource.
• Those clients that are not Universal GUIs almost always use the simpler APIs where the resource is staged and committed within a single request and Indicium itself ensures that the resource is cleaned up. Those clients that operate more like the Universal GUI does (such as yours), should try to ensure that staged resources are always either committed or deleted.
• Staged resources almost never use a noteworthy amount of memory. The typical staged resource probably uses less than a kilobyte of memory. Blob and clob data types such as varbinary(max) and varchar(max) do allow resources to potentially use a lot of memory. We are looking into the possibility of allowing a maximum length setting on these data types as well. It's important to add here that files which are uploaded to a staged resource using /upload_<file_property_id> are never stored in memory in their entirety. Instead they are streamed to a file cache and then streamed to their final location upon committing the resource.
• We recommend recycling Indicium periodically as a policy. A good recycling policy is, for instance, once a week. Recycling Indicium will clean up all staged resources.

There are also some more technical reasons, such as the fact that user sessions are not statefully managed by Indicium as such. Authentication cookies are self-contained tokens and Indicium therefore has no concept of sessions expiring. Although we could in theory tie staged resources to the user sessions that are tracked by IAM and periodically clear them.

With that said, the only relevant risk to consider here is that someone is purposefully creating staged resources and not cleaning them up. First, it is important to realize that only authenticated users would be able to do this. Secondly, having staged resource expire with a user’s session would not mitigate this risk in any way, as this person could simply keep their session alive. A better solution would be some form of rate limiting, which is something that we are looking into to support natively, but it is also something that firewalls (WAFs) will take care of for you.

Another thing to mention here is that we are currently finalizing our implementation to make Indicium fully stateless with regard to application state (i.e. staged resources, process flows, etc.) as well. This is an important goal for us to improve Indicium’s ability to scale horizontally, which removes the need for sticky sessions when load balancing and also allows for seamless fail-over scenarios where users can continue working without noticing a thing when an instance of Indicium goes down. With this, all application state will be stored in a Redis cache which can be hosted anywhere (e.g. Azure, AWS, on-premises). Every Redis cache entry will in fact have an expiry time to ensure that the cache doesn’t grow indefinitely. By default Indicium will keep storing these things in memory, but it does provide an option to achieve what you want, if it’s a worry.

So I would not worry about the memory filling up with staged resources by accident. I would simply recommend to recycle Indicium once a week to mitigate this risk entirely. As for the memory filling up with staged resources through malicious means, I would recommend strong passwords and two-factor authentication and if that feels insufficient, using a competent WAF.

I hope this clarifies our position. Please let me know if you have additional questions.