High performance storage for your app: the Storage Foundation API

Thomas Steiner

The web platform increasingly offers developers the tools they need to build fined-tuned high-performance applications for the web. Most notably, WebAssembly (Wasm) has opened the door to fast and powerful web applications, while technologies like Emscripten now allow developers to reuse tried and tested code on the web. To truly leverage this potential, developers must have the same power and flexibility when it comes to storage.

This is where the Storage Foundation API comes in. The Storage Foundation API is a new fast and unopinionated storage API that unlocks new and much-requested use cases for the web, such as implementing performant databases and gracefully managing large temporary files. With this new interface, developers can "bring their own storage" to the web, reducing the feature gap between web and platform-specific code.

The Storage Foundation API is designed to resemble a very basic file system so it gives developers flexibility by providing generic, simple, and performant primitives on which they can build higher-level components. Applications can take advantage of the best tool for their needs, finding the right balance between usability, performance, and reliability.

Why does the web need another storage API?

The web platform offers a number of storage options for developers, each of which is built with specific use-cases in mind.

  • Some of these options clearly do not overlap with this proposal as they only allow very small amounts of data to be stored, like cookies, or the Web Storage API consisting of the sessionStorage and the localStorage mechanisms.
  • Other options are already deprecated for various reasons like the File and Directory Entries API or WebSQL.
  • The File System Access API has a similar API surface, but its use is to interface with the client's file system and provide access to data that may be outside of the origin's or even the browser's ownership. This different focus comes with stricter security considerations and higher performance costs.
  • The IndexedDB API can be used as a backend for some of the Storage Foundation API's use-cases. For example, Emscripten includes IDBFS, an IndexedDB-based persistent file system. However, since IndexedDB is fundamentally a key-value store, it comes with significant performance limitations. Furthermore, directly accessing subsections of a file is even more difficult and slower under IndexedDB.
  • Finally, the CacheStorage interface is widely supported and is tuned for storing large-sized data such as web application resources, but the values are immutable.

The Storage Foundation API is an attempt at closing all the gaps of the previous storage options by allowing for the performant storage of mutable large files defined within the origin of the application.

Suggested use cases for the Storage Foundation API

Examples of sites that may use this API include:

  • Productivity or creativity apps that operate on large amounts of video, audio, or image data. Such apps can offload segments to disk instead of holding them in memory.
  • Apps that rely on a persistent file system accessible from Wasm and that need more performance than what IDBFS can guarantee.

What is the Storage Foundation API?

There are two main parts to the API:

  • File system calls, which provide basic functionality to interact with files and file paths.
  • File handles, which provide read and write access to an existing file.

File system calls

The Storage Foundation API introduces a new object, storageFoundation, that lives on the window object and that includes a number of functions:

  • storageFoundation.open(name): Opens the file with the given name if it exists and otherwise creates a new file. Returns a promise that resolves with the opened file.
  • storageFoundation.delete(name): Removes the file with the given name. Returns a promise that resolves when the file is deleted.
  • storageFoundation.rename(oldName, newName): Renames the file from the old name to the new name atomically. Returns a promise that resolves when the file is renamed.
  • storageFoundation.getAll(): Returns a promise that resolves with an array of all existing file names.
  • storageFoundation.requestCapacity(requestedCapacity): Requests new capacity (in bytes) for usage by the current execution context. Returns a promise that resolved with the remaining amount of capacity available.
  • storageFoundation.releaseCapacity(toBeReleasedCapacity): Releases the specified number of bytes from the current execution context, and returns a promise that resolves with the remaining capacity.
  • storageFoundation.getRemainingCapacity(): Returns a promise that resolves with the capacity available for the current execution context.

File handles

Working with files happens via the following functions:

  • NativeIOFile.close(): Closes a file, and returns a promise that resolves when the operation completes.
  • NativeIOFile.flush(): Synchronizes (that is, flushes) a file's in-memory state with the storage device, and returns a promise that resolves when the operation completes.
  • NativeIOFile.getLength(): Returns a promise that resolves with the length of the file in bytes.
  • NativeIOFile.setLength(length): Sets the length of the file in bytes, and returns a promise that resolves when the operation completes. If the new length is smaller than the current length, bytes are removed starting from the end of the file. Otherwise the file is extended with zero-valued bytes.

  • NativeIOFile.read(buffer, offset): Reads the contents of the file at the given offset through a buffer that is the result of transferring the given buffer, which is then left detached. Returns a NativeIOReadResult with the transferred buffer and the number of bytes that were successfully read.

    A NativeIOReadResult is an object that consists of two entries:

    • buffer: An ArrayBufferView, which is the result of transferring the buffer passed to read(). It is of the same type and length as source buffer.
    • readBytes: The number of bytes that were successfully read into buffer. This may be less than the buffer size, if an error occurs or if the read range spans beyond the end of the file. It is set to zero if the read range is beyond the end of the file.

  • NativeIOFile.write(buffer, offset): Writes the contents of the given buffer into the file at the given offset. The buffer is transferred before any data is written and is therefore left detached. Returns a NativeIOWriteResult with the transferred buffer and the number of bytes that were successfully written. The file will be extended if the write range exceeds its length.

    A NativeIOWriteResult is an object that consists of two entries:

    • buffer: An ArrayBufferView which is the result of transferring the buffer passed to write(). It is of the same type and length as the source buffer.
    • writtenBytes: The number of bytes that were successfully written into buffer. This may be less than the buffer size if an error occurs.

Complete examples

To make the concepts introduced above clearer, here are two complete examples that walk you through the different stages in the lifecycle of Storage Foundation files.

Opening, writing, reading, closing

// Open a file (creating it if needed).
const file = await storageFoundation.open('test_file');
try {
  // Request 100 bytes of capacity for this context.
  await storageFoundation.requestCapacity(100);

  const writeBuffer = new Uint8Array([64, 65, 66]);
  // Write the buffer at offset 0. After this operation, `result.buffer`
  // contains the transferred buffer and `result.writtenBytes` is 3,
  // the number of bytes written. `writeBuffer` is left detached.
  let result = await file.write(writeBuffer, 0);

  const readBuffer = new Uint8Array(3);
  // Read at offset 1. `result.buffer` contains the transferred buffer,
  // `result.readBytes` is 2, the number of bytes read. `readBuffer` is left
  // detached.
  result = await file.read(readBuffer, 1);
  // `Uint8Array(3) [65, 66, 0]`
  console.log(result.buffer);
} finally {
  file.close();
}

Opening, listing, deleting

// Open three different files (creating them if needed).
await storageFoundation.open('sunrise');
await storageFoundation.open('noon');
await storageFoundation.open('sunset');
// List all existing files.
// `["sunset", "sunrise", "noon"]`
await storageFoundation.getAll();
// Delete one of the three files.
await storageFoundation.delete('noon');
// List all remaining existing files.
// `["sunrise", "noon"]`
await storageFoundation.getAll();

Demo

You can play with the Storage Foundation API demo in the embed below. Create, rename, write into, and read from files, and see the available capacity you have requested update as you make changes. You can find the source code of the demo on Glitch.

Security and permissions

The Chromium team designed and implemented the Storage Foundation API using the core principles defined in Controlling Access to Powerful Web Platform Features, including user control, transparency, and ergonomics.

Following the same pattern as other modern storage APIs on the web, access to the Storage Foundation API is origin-bound, meaning that an origin may only access self-created data. It is also limited to secure contexts.

User control

Storage quota will be used to distribute access to disk space and to prevent abuse. Memory you want to occupy needs to be requested first. Like other storage APIs, users can clear the space taken by Storage Foundation API through their browser.

Helpful links

Acknowledgements

The Storage Foundation API was specified and implemented by Emanuel Krivoy and Richard Stotz. This article was reviewed by Pete LePage and Joe Medley.

Hero image via Markus Spiske on Unsplash.