Uploader

I think understanding the concept of uploaders and how they work is essential for developers, as it helps in choosing the right approach for their applications and ensures that file uploads are handled efficiently and securely. So, reading this blog post might be helpful if you’re interested in learning more about uploaders and how to implement them in your web applications.

Uploader is a powerful tool that allows developers to handle file uploads efficiently. This blog post explores the concept of uploaders and best practices for managing file uploads in web applications.

We have different types of uploaders(which I worked with), such as:

1. Traditional Form Uploaders
2. Stream Uploaders
3. S3 Uploaders:
   • Whole blob Uploaders(PutObject)
   • Chunked Uploaders(Multipart Upload)

Let's see structure of each approach, how they work and useful libraries for each approach in JavaScript.

Traditional Form Uploaders

These uploaders use forms to submit files to the server. They typically involve a file input element and a submit button. When the form is submitted, the file is sent to the server as part of the form data. The files are saved to file system and metadata is stored in the database.

Pros:

• Simple to implement, works well for small files

Cons:

• Inefficient for large files, can cause server load
• Can lead to timeouts and failed uploads for large files
• Can consume a lot of server resources, especially if multiple users are uploading large files
• Can't handle concurrent uploads efficiently, as each upload is processed sequentially by the server

Useful libraries:

• Multer: A middleware for handling multipart/form-data, which is primarily used for uploading files. It is easy to use and integrates well with Express.js applications.

Stream Uploaders

These uploaders allow files to be uploaded in a streaming manner, which can be more efficient for large files. The client can stream the file data directly to the server, and the server can process the data as it is received. This approach can reduce memory usage and improve performance, especially for large files.

Pros:

• Improved performance, reduced memory usage
• Can handle large files without consuming excessive server resources
• Can handle concurrent uploads more efficiently, as the server can process multiple streams simultaneously
• Has more control over the upload process, allowing for features like progress tracking and resumable uploads

Cons:

• More complex implementation, requires additional server logic to handle streaming data

Useful libraries:

• Busboy: A Node.js module for parsing incoming HTML form data, especially file uploads. It is designed to be efficient and can handle large file uploads without consuming excessive memory.

S3 Uploaders

Why S3 Uploaders?

The main reason for using S3 uploaders is that user can upload files directly to Amazon S3 without passing through the server, which can reduce server load and improve performance. Additionally, S3 provides features like durability, scalability, and security, making it a popular choice for file storage in web applications.

Whole blob Uploaders(PutObject)

These uploaders allow files to be uploaded directly to Amazon S3 without passing through the server. The server generates a pre-signed URL for the file upload, which allows client to upload the file directly to S3. The server can then store the file metadata in the database.

Pros:

• Reduced server load, as files are uploaded directly to S3
• Can handle files without consuming server resources

Cons:

• More complex implementation, requires additional server logic to generate pre-signed URLs and handle metadata
• Less control over the upload process, as the client is responsible for uploading the file directly to S3
• It is not efficient for large files(it is not recommended for more than 100MB) as it uploads the whole file in one request, which can lead to timeouts and failed uploads for large files

Useful libraries:

• AWS SDK for JavaScript: A library that provides a convenient interface for interacting

Chunked Uploaders(Multipart Upload)

These uploaders break large files into smaller chunks and upload them separately and directly to Amazon S3. For more control over the upload process also we have concurrency, the server generates pre-signed URLs for array of chunks, allowing the client to upload each chunk directly to S3. In this way, if the upload is interrupted, the client can resume the upload from the last successfully uploaded chunk, rather than starting over from the beginning.

Also it can upload chunks in parallel, which can improve performance for large files. This approach can improve reliability and performance, especially for large files.

The steps are as follows:

1. The client initiates a multipart upload by sending a request to the server(init), which generates a unique upload ID and key which is a filename on s3. The server returns it to the client.
2. The client send a request to the server(parts) with upload Id, key and part numbers(which are numbers of chunks). For example part numbers can be 1-5 for first 5 chunks or 6-10 for the next 5 chunks. the server generates pre-signed URLs for each chunk and returns them to the client.
3. The client uploads each chunk base on part numbers directly to S3 using the pre-signed URLs. The s3 return an ETag for each chunk, which is a unique identifier for the uploaded chunk. The client keeps track of the ETags for each chunk.
4. It repeats steps 2 and 3 until all chunks have been uploaded.
5. Once all chunks have been uploaded, the client sends a request with the ETags to the server(complete) to complete

See bellow picture for better understanding of the steps:

Typescript code for client side implementation of chunked uploader to S3:

export const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB
export const CONCURRENCY = 4;

type UploadedPart = {
  PartNumber: number;
  ETag: string;
};

export async function uploadS3File(file: File) {
  const initRes = await fetch(`${import.meta.env.VITE_API_URL}/upload/init`, {
    method: "POST",
    body: JSON.stringify({
      fileName: file.name,
      contentType: file.type,
    }),
    headers: { "Content-Type": "application/json" },
  });

  const { uploadId, key } = await initRes.json();

  const totalParts = Math.ceil(file.size / CHUNK_SIZE);
  const uploadedParts: UploadedPart[] = [];

  let currentPart = 1;

  async function uploadChunkBatch() {
    if (currentPart > totalParts) return;
    const batch = [];
    for (let i = 0; i < CONCURRENCY && currentPart <= totalParts; i++) {
      batch.push(currentPart++);
    }

    // Lazy request
    const res = await fetch(`${import.meta.env.VITE_API_URL}/upload/parts`, {
      method: "POST",
      body: JSON.stringify({
        uploadId,
        key,
        partNumbers: batch,
      }),
      headers: { "Content-Type": "application/json" },
    });

    const { urls } = await res.json();

    await Promise.all(
      urls.map(({ partNumber, url }: { partNumber: number; url: string }) =>
        uploadPartWithRetry(file, partNumber, url, uploadedParts)
      )
    );

    return uploadChunkBatch();
  }

  await uploadChunkBatch();
  const completeRes = await fetch(`${import.meta.env.VITE_API_URL}/upload/complete`, {
    method: "POST",
    body: JSON.stringify({
      uploadId,
      key,
      parts: uploadedParts,
    }),
    headers: { "Content-Type": "application/json" },
  });
  const completeData = await completeRes.json();
}

async function uploadPartWithRetry(file: File, partNumber: number, url: string, uploadedParts: UploadedPart[]) {
  const start = (partNumber - 1) * CHUNK_SIZE;
  const end = Math.min(start + CHUNK_SIZE, file.size);
  const blob = file.slice(start, end);

  let attempt = 0;

  while (attempt < MAX_RETRIES) {
    try {
      const res = await fetch(url, {
        method: "PUT",
        body: blob,
      });

      if (!res.ok) throw new Error("Upload failed");

      const etag = res.headers.get("ETag");

      uploadedParts.push({
        ETag: etag?etag.replace(/"/g, ""): "",
        PartNumber: partNumber,
      });

      return;
    } catch (err) {
      attempt++;
      if (attempt >= MAX_RETRIES) {
        throw err;
      }
    }
  }
}

Pros:

• Improved reliability, as it allows for resuming uploads if they are interrupted
• Better performance for large files, as it can upload chunks in parallel
• Reduced server load, as files are uploaded directly to S3
• Has more control over the upload process, allowing for features like progress tracking and resumable uploads

Cons:

• More complex implementation, requires additional server logic to generate pre-signed URLs for each chunk and handle metadata

Useful libraries:

• AWS SDK for JavaScript: A library that provides a convenient interface for interacting with Amazon S3, including support for multipart uploads. It allows developers to easily implement chunked uploads to S3, including generating pre-signed URLs for each chunk and handling the upload process.

Recap

Let's recap the pros and cons of each approach:

I’ve provided this cheat sheet to help you choose the right uploader type based on your application requirements. It’s a general guideline, and the actual choice may still depend on the specific use cases and constraints of your application.

Conclusion

In conclusion, understanding the concept of uploaders and their different types is essential for developers to choose the right approach for their applications.

Each uploader type has its own advantages and disadvantages, and the choice depends on the specific requirements of the application, such as file size, performance needs, and server resources.

---

As always, I may miss some points, so if you have any questions or suggestions, please feel free to share them with me. If you know a better approach or library for handling file uploads, I’d love to hear about it. I’m always open to learning and improving my knowledge.

Stay humble and keep learning!