Intermittent download failures (304 errors) triaged to CDN provider.
Incident Report for Bintray

Incident Description

Users intermittently received a 304 (Not Modified has no content) response code for some artifact downloads.

Event Chronology

  • March 15, 2018: First complaints were received.

We opened an urgent ticket with our CDN provider support

  • Initial investigation had shown that the issue was caused by our CDN provider when they were working to resolve slow downloads in India by establishing a new caching layer.

  • Eventually, our CDN provider informed us that the issue was with edge-node caches and an invalid Real User Monitoring (RUM) configuration they had implemented - this was causing some Bintray downloads to fail with a 304 response

  • Fixing the invalid RUM configuration did not resolve the problem completely.

  • Our CDN provider responded that due to the intermittent nature of the issue it was not easy to reproduce.

  • Only when caches were fast-purged and RUM was disabled, the issue was resolved and JFrog log analysis showed only 200 (OK) responses codes

Lessons Learned and Action Items

  • Together with our CDN provider, we are examining the actual level of support we received vs. their SLA with JFrog (highest level) to make sure that in the future, our CDN provider satisfies the SLA they are committed to, which did not meet for this incident

  • We are awaiting an RCA from our CDN provider to share directly with the users.

  • We are examining selectively falling back to other CDN solutions used by Bintray.

Posted 20 days ago. Apr 02, 2018 - 20:42 UTC

Resolved
This incident has been resolved.
Posted 24 days ago. Mar 29, 2018 - 08:10 UTC
Monitoring
It **looks like** all the problematic edge servers were identified and their configuration fixed by the CDN provider. Please let us know if you still see those pesky 304-s!
Posted 25 days ago. Mar 28, 2018 - 14:48 UTC
Update
Within the few coming hours, the cache of the CDN provider will be refreshed. Some temporary slowness can be experienced when downloading files that are not in the cache after the cleanup.
Posted 25 days ago. Mar 28, 2018 - 11:18 UTC
Update
Our CDN provider asked for your help in identifying the problematic files. If you are still encountering failures based on 304 errors, please run the following curl command:

`curl -D out.txt -H 'Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id' -sL https://{304 URL}`

Send the results to Bintray support (support@bintray.com)
Posted 26 days ago. Mar 27, 2018 - 20:36 UTC
Update
Our CDN provider asked for your help in identifying the problematic files. If you are still encountering failures based on 304 errors, please run the following curl command:

`curl -D out.txt -H 'Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id' -sL https://{304 URL}`

Send the results to Bintray support (support@bintray.com)

Thank you for your help.
Posted 26 days ago. Mar 27, 2018 - 18:36 UTC
Identified
CDN provider has been notified, they are working on the fix.
Posted 26 days ago. Mar 27, 2018 - 15:48 UTC
Investigating
Temporary slowness within the downloads and the UI
Posted 26 days ago. Mar 27, 2018 - 09:47 UTC