Feature: Chunked Uploads (HTTP + Client)
Purpose
Support reliable, resumable uploads of large files over unreliable connections by splitting a payload into chunks and completing with an integrity check (CRC32):
- server stages chunks to disk via
ChunkUploadService - server merges chunks and optionally commits to
IStorage - client computes CRC32 during upload and reports progress
Main Flows
sequenceDiagram
participant C as Client (ManagedCode.Storage.Client)
participant API as StorageControllerBase
participant Ch as ChunkUploadService
participant S as IStorage
loop For each chunk
C->>API: POST /upload-chunks/upload (multipart)
API->>Ch: AppendChunkAsync(...)
Ch-->>API: Result
API-->>C: Result
end
C->>API: POST /upload-chunks/complete (JSON)
API->>Ch: CompleteAsync(...)
Ch->>S: UploadAsync(mergedStream, UploadOptions) [CommitToStorage=true]
S-->>Ch: Result<BlobMetadata>
Ch-->>API: Result<ChunkUploadCompleteResponse> (includes checksum)
API-->>C: Result<ChunkUploadCompleteResponse>
Components
Server-side:
Integraions/ManagedCode.Storage.Server/ChunkUpload/ChunkUploadService.csIntegraions/ManagedCode.Storage.Server/ChunkUpload/ChunkUploadSession.csIntegraions/ManagedCode.Storage.Server/Models/FileUploadPayload.csIntegraions/ManagedCode.Storage.Server/Models/ChunkUploadCompleteRequest.csIntegraions/ManagedCode.Storage.Server/Models/ChunkUploadCompleteResponse.cs- Endpoints:
Client-side:
Integraions/ManagedCode.Storage.Client/StorageClient.cs(UploadLargeFile(...))- CRC helpers:
ManagedCode.Storage.Core/Helpers/Crc32Helper.cs
Current Behavior
- Chunks are staged under the server temp path (controlled by server options).
- A session is admitted only when capacity is available. Retries replace a part after the incoming bytes have been staged successfully.
- Completion checks the declared length when available and computes CRC32 during the merge before committing.
- Completion step merges chunks in order and can:
- keep merged file on disk (
KeepMergedFile) - commit merged file to
IStorage(CommitToStorage)
- keep merged file on disk (
- Client requires
StorageClient.ChunkSizeto be configured before uploading large files.
Tests
Tests/ManagedCode.Storage.Tests/Server/ChunkUploadServiceTests.csTests/ManagedCode.Storage.Tests/Server/ChunkUploadPathIntegrityTests.cs: original outside-file bytes, part-name collision, rejected identity isolation and explicit merged-file retention using the real filesystem provider.Tests/ManagedCode.Storage.Tests/Core/StorageClientChunkTests.cs- Controller-level flows:
Temporary file ownership
Each admitted session gets a new server-generated working directory, including
when a previously completed upload ID is reused. IDs accept ASCII letters,
digits, hyphens and underscores, up to 128 characters. The
service generates the merged staging filename independently of client file
names. Client names cannot overwrite files outside a session or collide with
numbered part files. KeepMergedFile retains only the merged content after the
part files are cleaned; callers requesting retention own its eventual cleanup.
Expired sessions are cleaned up before completion can write to storage.
For a known upload length, append requests check the combined size before staging.
A replacement counts only its new size, and a rejected replacement preserves the accepted chunk.