Users intermittently received a 304 (Not Modified has no content) response code for some artifact downloads.
We opened an urgent ticket with our CDN provider support
Initial investigation had shown that the issue was caused by our CDN provider when they were working to resolve slow downloads in India by establishing a new caching layer.
Eventually, our CDN provider informed us that the issue was with edge-node caches and an invalid Real User Monitoring (RUM) configuration they had implemented - this was causing some Bintray downloads to fail with a 304 response
Fixing the invalid RUM configuration did not resolve the problem completely.
Our CDN provider responded that due to the intermittent nature of the issue it was not easy to reproduce.
Only when caches were fast-purged and RUM was disabled, the issue was resolved and JFrog log analysis showed only 200 (OK) responses codes
Lessons Learned and Action Items
Together with our CDN provider, we are examining the actual level of support we received vs. their SLA with JFrog (highest level) to make sure that in the future, our CDN provider satisfies the SLA they are committed to, which did not meet for this incident
We are awaiting an RCA from our CDN provider to share directly with the users.
We are examining selectively falling back to other CDN solutions used by Bintray.