Large File Uploads Done Right
Modern web applications frequently need to handle large file uploads--videos, high-resolution images, document archives, and user-generated content. When these files exceed standard upload limits or need reliable transfer, AWS S3's multipart upload feature becomes essential. This guide walks through implementing robust multipart uploads using Node.js on the backend and React on the frontend, combining the performance benefits of parallel uploads with the security of presigned URLs.
Our /services/web-development/ expertise covers cloud infrastructure integration, ensuring your applications handle large-scale file operations efficiently and securely.
Why Use Multipart Upload for S3
Standard single-put uploads to S3 work well for small files but become problematic as file sizes grow. Network interruptions can require restarting entire uploads, and the maximum single upload size is limited to 5GB. Multipart upload addresses these limitations by breaking large files into smaller parts that upload independently and in parallel.
Performance Benefits
Parallel uploads leverage multiple network connections simultaneously, dramatically reducing transfer time for large files. A 1GB file split into 100MB chunks and uploaded across 10 concurrent connections can complete in approximately one-tenth the time of a sequential upload, assuming sufficient bandwidth.
Resumability and Recovery
Multipart upload provides the ability to resume interrupted uploads without losing progress. Each part uploaded to S3 is tracked separately, and the upload process can be paused and resumed. This capability is crucial for mobile applications where users may lose connectivity.
AWS S3 Limits: 5MB minimum per part, 5TB maximum total object size, last part can be smaller than 5MB.
According to the AWS multipart upload documentation, each part must be at least 5MB except for the final part, and the total object size can reach up to 5TB.
Parallel Uploads
Upload multiple chunks simultaneously for faster throughput
Resumable Transfers
Resume interrupted uploads without losing progress
Error Isolation
Only retry failed parts, not the entire file
Large File Support
Handle files up to 5TB with proper chunking
The Four-Step Multipart Upload Process
Implementing multipart upload requires coordination between backend and frontend:
Step 1: Create Multipart Upload
The server initiates by calling S3's CreateMultipartUpload command. This establishes the upload session and returns an upload ID used for all subsequent operations.
Step 2: Generate Presigned URLs
With the upload ID, the server generates presigned URLs for each part. These grant temporary, scoped permission for the client to upload specific parts directly to S3.
Step 3: Upload Parts from Client
The client receives presigned URLs and uploads each part directly to S3 using HTTP PUT requests. After each upload, S3 returns an ETag header for the completion step.
Step 4: Complete Multipart Upload
Once all parts are uploaded, the server calls CompleteMultipartUpload with the upload ID and part ETags. S3 validates parts, assembles them, and removes temporary data.
As demonstrated in the LogRocket implementation guide, this four-step pattern provides a clean separation of concerns between server-side security and client-side performance.
Node.js Backend Implementation
The backend handles sensitive operations: initializing uploads, generating presigned URLs, and completing uploads. Using AWS SDK v3 for JavaScript provides modular architecture with full S3 operation access. Our /services/web-development/ team specializes in implementing secure, scalable cloud integrations like this pattern.
Key Implementation Points
Chunk Size Selection: AWS requires parts of at least 5MB, with the final part allowed to be smaller. A 10MB chunk size offers a good balance between upload efficiency and memory usage.
Presigned URL Generation: URLs are generated for each part number, authorizing exactly that part to be uploaded. They include the upload ID, bucket name, key, and part number with signature validation.
1import {2 S3Client,3 CreateMultipartUploadCommand,4 UploadPartCommand,5 CompleteMultipartUploadCommand6} from '@aws-sdk/client-s3';7import { getSignedUrl } from '@aws-sdk/s3-request-presigner';8 9const client = new S3Client({ region: 'your-region' });10 11// Step 1: Create Multipart Upload12const createCommand = new CreateMultipartUploadCommand({13 Bucket: 'your-bucket-name',14 Key: 'your-file-key'15});16const startResponse = await client.send(createCommand);17const uploadId = startResponse.UploadId;18 19// Step 2: Generate Presigned URLs for each part20const CHUNK_SIZE = 10 * 1024 * 1024; // 10MB21const numberOfParts = Math.ceil(fileSize / CHUNK_SIZE);22const presignedUrls = [];23 24for (let i = 0; i < numberOfParts; i++) {25 const url = await getSignedUrl(26 client,27 new UploadPartCommand({28 Bucket: 'your-bucket-name',29 Key: 'your-file-key',30 UploadId: uploadId,31 PartNumber: i + 132 }),33 { expiresIn: 3600 }34 );35 presignedUrls.push(url);36}37 38// Step 4: Complete Multipart Upload39const completeCommand = new CompleteMultipartUploadCommand({40 Bucket: 'your-bucket-name',41 Key: 'your-file-key',42 UploadId: uploadId,43 MultipartUpload: {44 Parts: completedParts.map(part => ({45 ETag: part.ETag,46 PartNumber: part.PartNo47 }))48 }49});50await client.send(completeCommand);React Frontend Implementation
The React client handles file selection, chunking, and parallel uploads using presigned URLs from the server. This keeps sensitive operations on the server while enabling direct-to-S3 uploads.
Client-Side Responsibilities
File Chunking: Uses Blob.slice() to divide files into byte ranges matching the chunk size.
Parallel Uploads: Uploads chunks simultaneously using Promise.all or similar concurrency patterns.
ETag Collection: Captures ETag headers from S3 responses for the completion step.
Progress Tracking: Monitors each in-flight part and aggregates completion status.
1async function uploadPart(2 fileChunk: Blob,3 presignedUrl: string,4 partNo: number5): Promise<{ ETag: string; PartNo: number }> {6 const response = await fetch(presignedUrl, {7 method: 'PUT',8 body: fileChunk9 });10 11 return {12 ETag: response.headers.get('ETag') ?? '',13 PartNo: partNo14 };15}16 17async function uploadFileWithMultipart(18 file: File,19 presignedUrls: string[],20 chunkSize: number = 10 * 1024 * 102421) {22 const numberOfParts = Math.ceil(file.size / chunkSize);23 const uploadPromises: Promise<{ ETag: string; PartNo: number }>[] = [];24 25 for (let i = 0; i < numberOfParts; i++) {26 const start = i * chunkSize;27 const end = Math.min(start + chunkSize, file.size);28 const blob = file.slice(start, end);29 30 uploadPromises.push(uploadPart(blob, presignedUrls[i], i + 1));31 }32 33 const results = await Promise.all(uploadPromises);34 return results;35}CORS Configuration for Client-Side Uploads
S3 requires proper CORS configuration to allow direct browser uploads. Without correct CORS, browsers block PUT requests even with valid presigned URLs.
Critical Configuration
The ExposeHeaders setting is critical because you need access to the ETag header returned after each part upload. Without CORS configuration, JavaScript cannot read these ETags needed for completion.
As detailed in this DEV Community guide on S3 multipart uploads, proper CORS configuration is essential for browser-based uploads.
1[2 {3 "AllowedHeaders": ["*"],4 "AllowedMethods": ["GET", "PUT", "POST"],5 "AllowedOrigins": ["https://your-domain.com"],6 "ExposeHeaders": ["ETag"],7 "MaxAgeSeconds": 30008 }9]Best Practices and Performance Optimization
Chunk Size Selection
- Minimum: 5MB per part (AWS requirement)
- Recommended: 10MB for balanced performance
- Mobile/unstable networks: 5MB for better resumability
- High-bandwidth/stable: 25MB for reduced overhead
Concurrency Management
- Target 3-6 concurrent uploads for optimal throughput
- Too many concurrent uploads can trigger rate limiting
- Implement semaphore pattern for controlled parallelism
Error Handling Strategies
- Distinguish retriable errors (network timeout) from permanent failures
- Implement retry logic with exponential backoff
- Use abortMultipartUpload for cleanup on unrecoverable errors
- Store upload ID for resumability across sessions
Progress Tracking
- Calculate progress as: (sum of uploaded bytes) / total file size
- Update every 100-500ms for responsive UX
- Provide stage indicators: chunk upload → part completion → final assembly
Common Pitfalls and How to Avoid Them
Part Number Ordering
Parts can be uploaded in any order, but CompleteMultipartUpload must list parts in sequential order with correct part numbers. Ensure ETags are associated with their correct part numbers.
ETag Access
The ETag header must be exposed via CORS ExposeHeaders. Without this, your completion step lacks required identifiers. Test CORS early by checking ETag readability.
Upload ID Persistence
For resumability across browser sessions, store the upload ID in localStorage or provide server-side recovery. Without persistence, users lose progress on page refresh.
Memory Management
Loading entire files into memory before chunking can exceed available memory. Use File API slice operations directly without creating full-file copies.
Frequently Asked Questions
What is the minimum chunk size for S3 multipart upload?
AWS requires a minimum of 5MB per part, with the exception of the final part which can be smaller. For optimal performance, most implementations use 10MB chunks.
Can I upload parts in any order?
Yes, parts can be uploaded in any order. However, the CompleteMultipartUpload request must list parts in sequential order with correct part numbers and their ETags.
How do I handle failed part uploads?
Implement retry logic with exponential backoff for transient errors. Use the part's presigned URL to retry specific failed parts without affecting other uploads. For permanent failures, abort the multipart upload to clean up resources.
What CORS settings are required for direct browser uploads?
You need PUT in AllowedMethods, your domain in AllowedOrigins, and ETag in ExposeHeaders. The ExposeHeaders setting is critical for reading ETags needed to complete the upload.
How many concurrent uploads should I use?
Most implementations find 3-6 concurrent uploads provide good throughput without overwhelming resources or triggering rate limiting. The optimal number depends on your network conditions and typical file sizes.
Conclusion
Multipart upload to S3 from Node.js and React combines server-side setup with client-side parallel uploads for efficient large file handling. The four-step process--create, generate URLs, upload parts, complete--provides a clear architectural framework.
With proper CORS configuration, appropriate chunk sizing, and comprehensive error handling, your implementation will handle large file uploads reliably and efficiently. The investment in robust implementation pays dividends through better user experience, reduced server load, and more resilient file handling.
As your application grows, consider extending this foundation with features like upload queues, background uploads, and serverless architecture integration. For enterprise-grade implementations, our /services/web-development/ team can help architect scalable solutions that meet your specific requirements.
Sources: