Hello @martijngaasbeek,
The 502 Bad Gateway response coming from nginx indicates that there is some kind of connectivity issue between the nginx reverse proxy and Indicium. Since the issue appears to be related to how many details are expanded, my first guess is that Indicium takes too long to complete the request, which causes nginx to assume that there will be no response from Indicium and give the 502 Bad Gateway response.
Expanding detail references can be very taxing, depending on the amount of data that's involved. Expanding three detail references at the same time is even more taxing. Assuming that there is quite a bit of data involved, I'm not surprised that a request like that could cause a timeout. The only thing that doesn't make sense is that this timeout only occurs with bearer authentication and not with basic authentication. I think it's likely that the two requests are not authenticated as the same user, even though you might be under the impression that they are. One possible explanation is that one of the requests includes an authentication cookie that overrides either the basic authentication or bearer authentication, because authentication cookies take precedence over all other forms of authentication.
My assumption right now is that the basic authentication request applies prefilters and the bearer authentication request does not, because it is a different user.
I have a few questions to help clear this up:
- Does it take very long for you to get the 502 Bad Gateway response? For example, 30 seconds or 60 seconds?
- Does the basic authentication request that succeeds take much less long?
- Can you confirm that both requests are authenticated by the same user, for example by looking at the Session Log in IAM? Or calling a view that selects dbo.tsf_user() for example.
- Can you give the user in question the Developer Mode administrator role in IAM and compare the query of both requests by looking at Indicium's Database Event Log?
- Is there an obvious difference between the queries, for example is one of them missing prefilters?
- Is there a large difference in duration of the queries?
- What version of Indicium are you on?
@Vincent Doppenberg
Do you have time for a quick call? I think it will be much faster to show you the issues?
It is only since the last upgrade of Indicium. Before that we didn't have this problem.
To answer your questions:
The queries are identical in the dbEventLog for that user
Hello @martijngaasbeek,
What version of Indicium is production on? And does the bearer authentication request work if you skip the nginx proxy and call Indicium directly?
Development: 2024.1
Production: 2023.3
If I make the call directly using the IP address, then the call succeeds in both cases
Hello @martijngaasbeek,
With all of this information, the most logical explanation to me seems to be that the nginx reverse proxy has some kind of maximum response size or buffer size which is being exceeded, causing it to reject the request. The reason that the change in Indicium version might trigger this problem could be because the new version of Indicium returns a little bit more data, such as some extra headers. It would make sense that removing some of the expanded details would then fix the issue, as this would shrink the response size.
I would expect nginx to log an error message somewhere that specifically points to what the exact problem is. If not, you can try simply configuring the proxy buffer size to see if it fixes the problem. You could start with 32k for instance and go lower if it works.
I hope this helps.
@Vincent Doppenberg
One control question before we change configuration.
If I keep the cookie in postman, (the cookie from basic auth) and then run the call also with a bearer token, the correct response is given instead of a 502. Is this also in line with your expectation?
so reproduction:
- First run a succesful call with basic auth
- Run the same call without deleting the cookie and do it with a bearer token.
- Get a normal response.
If I delete the cookie from the basic auth call, I get the 502 again.
Hello @martijngaasbeek
Yes, that would be my expectation, because cookie authentication overrules bearer authentication. So sending the cookie from your basic authentication session causes the bearer token to be ignored completely.
To be a bit more clear regarding what I think the actual problem here is, I think that the authentication cookie returned by Indicium after bearer authentication is (much) larger than the authentication cookie returned by Indicium after basic authentication. The reason for this would be the number of claims contained by the JWT token that is provided for bearer authentication. These claims will be stored in the authentication cookie that is returned by Indicium.
If you call Indicium directly by bypassing your nginx proxy and you compare the responses of your basic authentication request and your bearer authentication request, I would expect that the Set-Cookie response header of the bearer authentication request is larger. Specifically, I would expect the .AspNetCore.Identity.Application cookie to be larger or even be split up into multiple cookies called .AspNetCore.Identity.ApplicationC1, .AspNetCore.Identity.ApplicationC2, etc.
If my assumptions above are correct then there are two ways of looking at this problem.
- Nginx should be able to handle larger response headers - for which the solution is to increase the size of the proxy buffer.
- The cookies should not become this large. In my opinion, this perspective is only reasonable if the .AspNetCore.Identity.Application is actually much larger than its basic authentication equivalent. For example, if the cookie is only 10% larger than is basic authentication equivalent, and just barely crosses nginx's buffer size threshold, then point 1 is the correct way of looking at this problem. However, if the cookie is multitudes larger, then you could argue that the cookie contains an abnormal amount of claims, most likely caused by a very large bearer token that also contains all of these claims. If this seems to be the case, then we can continue the conversation from there, but given all the information you've provided so far, I'm leaning towards the scenario that the response is only marginally larger and coincidentally just passes the proxy buffer size (which is rather small by default).
I hope this helps.
@Vincent Doppenberg
Sorry for the late response:
With basic auth the .AspNetCore.Identity.Application cookie length = 838
With bearer token the .AspNetCore.Identity.Application cookie length = 2203
Hello @martijngaasbeek,
So the cookie is a fair bit larger when using the Bearer token, but not to the extent that I would consider another solution than increasing the nginx proxy buffer size. Because the basic authentication version is at most 1.4K away from hitting the same limit, which is a very small margin. So the safer, future-proof solution would be to increase the proxy buffer size.
I hope this helps.
