Failing uploads on large files
Incident Report for AARNet Support
Update
The AARNet Cloud Services team are continuing to work with our software partners to analyse the root cause of these issues. The team have been testing new software to help address these issues and expect to implement this as early as the week beginning January 18, if not earlier.

We apologise for the significant inconvenience caused. We will provide regular updates on progress here.
Posted Jan 11, 2021 - 15:30 AEDT
Update
CloudStor is still experiencing errors for large file uploads. This is generally for files > 1GB but excessive waits and timeouts may occur for smaller files as well. Users may see "Upload Failed" errors in their browser, or "400 Bad Request" errors in client based uploads. The AARNet Cloud Services team are working with our software partners to diagnose the root cause of these issues. We apologise for the significant inconvenience caused. We will provide regular updates on progress here.
Posted Dec 17, 2020 - 16:22 AEDT
Update
The Cloud Services team have identified the cause of the file upload failures for large files. File uploads have improved for the majority of files larger than 1 GB, and we are working towards a complete remediation.
Posted Dec 17, 2020 - 08:57 AEDT
Identified
The Cloud Services team are making changes on the file upload services that we expect will significantly reduce the number of upload failures for >1 GB files. The changes will be implemented this Friday afternoon during the current maintenance period. We will then monitor the uploads and identify next steps early next week. We apologise for the inconvenience caused and thank you for your patience as we work through these issues.
Posted Dec 11, 2020 - 16:28 AEDT
Update
The Cloud Services team are still investigating this issue of an increase of error rate for large file uploads on CloudStor, and are treating it as the highest priority. We apologise for the inconvenience caused and we are working towards a resolution as quickly as possible.
Posted Dec 09, 2020 - 14:35 AEDT
Investigating
The Cloud Services team have identified an increase of error rate for large file uploads on CloudStor. Users will see "Upload Failed" errors in their browser, or "400 Bad Request" errors in client based uploads. This may be resolved by retrying, or smaller uploads. The team are investigating as a matter of priority and we will keep you informed as the investigation progresses.

This does not affect S3 services.
Posted Dec 07, 2020 - 14:06 AEDT
This incident affects: CloudStor (CloudStor Service).