How to store data using web3.storage

In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client libraries in JavaScript and Go. This includes making your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin — all for free.

If you just want to quickly store a few files using web3.storage rather than include upload functionality in an app or service you're building, you may want to skip this guide for now and simply use the Files page on the web3.storage site.

For developers, web3.storage provides a simple interface for storing data using syntax inspired by familiar web APIs such as fetch and File. This guide focuses on JavaScript client library and Go client library, which are the simplest way to use web3.storage programmatically.

If you're using another language, see the HTTP API reference for details on working with the underlying HTTP API.

Uploading data to web3.storage using a client library requires a free API token, which in turn requires a web3.storage account. If you already have an account and a token, read on. If not, have a look at the quickstart guide to get up and running in just a few minutes.

CAUTION

All data uploaded to web3.storage is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using web3.storage.

Installing the client

JavaScript
Go

In your JavaScript project, add the web3.storage package to your dependencies:

npm install web3.storage

In your Go project, add the client package to your dependencies using go get:

go get github.com/web3-storage/go-w3s-client

Creating a client instance

JavaScript
Go

First, create a web3.storage client object, passing in an API token to its constructor:

import { Web3Storage } from 'web3.storage'

function getAccessToken () {
  // If you're just testing, you can paste in a token
  // and uncomment the following line:
  // return 'paste-your-token-here'

  // In a real app, it's better to read an access token from an
  // environement variable or other configuration that's kept outside of
  // your code base. For this to work, you need to set the
  // WEB3STORAGE_TOKEN environment variable before you run your code.
  return process.env.WEB3STORAGE_TOKEN
}

function makeStorageClient () {
  return new Web3Storage({ token: getAccessToken() })
}

First, make sure to import the client w3s package:

import "github.com/web3-storage/go-w3s-client"

You can create a client instance with the NewClient function, passing in an API token using the WithToken option:

token := "<AUTH_TOKEN_GOES_HERE>"
client, err := w3s.NewClient(w3s.WithToken(token))

Preparing files for upload

JavaScript
Go

The web3.storage client's put method accepts an array of File objects.

There are a few different ways of creating File objects available, depending on your platform.

Browser
Node.js

In the browser, you can use a file input element to allow the user to select files for upload:

function getFiles () {
  const fileInput = document.querySelector('input[type="file"]')
  return fileInput.files
}

You can also manually create File objects using the native File constructor provided by the browser runtime. This is useful when you want to store data created by your application, instead of files from the user's computer.

function makeFileObjects () {
  // You can create File objects from a Blob of binary data
  // see: https://developer.mozilla.org/en-US/docs/Web/API/Blob
  // Here we're just storing a JSON object, but you can store images,
  // audio, or whatever you want!
  const obj = { hello: 'world' }
  const blob = new Blob([JSON.stringify(obj)], { type: 'application/json' })

  const files = [
    new File(['contents-of-file-1'], 'plain-utf8.txt'),
    new File([blob], 'hello.json')
  ]
  return files
}

In Node.js, the web3.storage package exports some helpful utility functions from the files-from-path module that allow you to easily read File objects from the local file system. The getFilesFromPath helper asynchronously returns an array of Files that you can use directly with the put client method:

import { getFilesFromPath } from 'web3.storage'

If you expect to be loading a lot of large files, you may be better served by the filesFromPath helper. It reduces memory pressure by yielding File objects one by one as they're loaded from disk, instead of loading everything into memory. You can then issue multiple put requests to send each file to web3.storage.

You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer.

import { File } from 'web3.storage'

async function getFiles (path) {
  const files = await getFilesFromPath(path)
  console.log(`read ${files.length} file(s) from ${path}`)
  return files
}

function makeFileObjects () {
  // You can create File objects from a Buffer of binary data
  // see: https://nodejs.org/api/buffer.html
  // Here we're just storing a JSON object, but you can store images,
  // audio, or whatever you want!
  const obj = { hello: 'world' }
  const buffer = Buffer.from(JSON.stringify(obj))

  const files = [
    new File(['contents-of-file-1'], 'plain-utf8.txt'),
    new File([buffer], 'hello.json')
  ]
  return files
}

The Go client accepts standard os.File pointers as returned by os.Open, so there's no preparation needed for uploading single files.

import "os"

func openSingleFile() {
	file, err := os.Open("path/to/my/file.jpg")
	if err != nil {
		// do something with error
	}
	// pass file into web3.storage client...
}

To upload a directory, you'll need to import the w3fs package from github.com/web3-storage/go-w3s-client/fs and use the NewDir function to specify the directory contents.

import (
	"io/fs"
	"os"

	"github.com/web3-storage/w3s-go-client/fs"
)

func opendir() {
	file1, err := os.Open("path/to/my/file.jpg")
	if err != nil {
		// do something with error
	}
	file2, err := os.Open("path/to/another/file.jpg")
	if err != nil {
		// do something with error
	}
	dir := w3fs.NewDir("images", []fs.File{file1, file2})

	// dir can now be passed into the client's Put method
}

Tip

When uploading multiple files, try to give each file a unique name. All the files in a put request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name.

Uploading to web3.storage

Once your files are ready, uploading is a simple method call on the client object.

IMPORTANT

Deleting files from the web3.storage site's Files page will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. Do not use web3.storage for data that may need to be permanently deleted in the future.

JavaScript
Go

Once you have an array of Files, you can upload them with the client's put method:

async function storeFiles (files) {
  const client = makeStorageClient()
  const cid = await client.put(files)
  console.log('stored files with cid:', cid)
  return cid
}

Showing progress to the user

The put method has some options that can be passed in to get progress on the upload as it happens in the background. There are two callback parameters you can use: onRootCidReady, and onStoredChunk.

The onRootCidReady callback is invoked as soon as the client has calculated the content identifier (CID) that identifies the data being uploaded. Because this calculation happens locally on the client, the callback is invoked before the upload begins.

As each chunk of data is uploaded, the onStoredChunk callback gets invoked with the size of the chunk in bytes passed in as a parameter.

Here's a simple example of using the callbacks to print the progress of an upload to the console:

async function storeWithProgress (files) {
  // show the root cid as soon as it's ready
  const onRootCidReady = cid => {
    console.log('uploading files with cid:', cid)
  }

  // when each chunk is stored, update the percentage complete and display
  const totalSize = files.map(f => f.size).reduce((a, b) => a + b, 0)
  let uploaded = 0

  const onStoredChunk = size => {
    uploaded += size
    const pct = 100 * (uploaded / totalSize)
    console.log(`Uploading... ${pct.toFixed(2)}% complete`)
  }

  // makeStorageClient returns an authorized web3.storage client instance
  const client = makeStorageClient()

  // client.put will invoke our callbacks during the upload
  // and return the root cid when the upload completes
  return client.put(files, { onRootCidReady, onStoredChunk })
}

The Go client's Client interface defines a Put method that accepts a context.Context and an fs.File, which can be either a single file or a directory.

The example below reads in an API token and the path to a file from the command line arguments, then uploads the file to web3.storage using the client's Put method.

A more in-depth example can be found in the client's GitHub repository, including uploads of multiple files and directories.

package main

import (
	"context"
	"fmt"
	"os"
	"path"

	"github.com/web3-storage/go-w3s-client"
)

func main() {
	token, ok := os.LookupEnv("WEB3_STORAGE_TOKEN")
	if !ok {
		fmt.Fprintln(os.Stderr, "No API token - set the WEB3_STORAGE_TOKEN environment var and try again.")
		os.Exit(1)
	}

	if len(os.Args) != 2 {
		fmt.Fprintf(os.Stderr, "usage: %s <filename>\n", os.Args[0])
		os.Exit(1)
	}
	filename := os.Args[1]

	// Create a new web3.storage client using the token
	client, err := w3s.NewClient(w3s.WithToken(token))
	if err != nil {
		panic(err)
	}

	// Open the file for reading
	file, err := os.Open(filename)
	if err != nil {
		panic(err)
	}

	basename := path.Base(filename)
	// Upload to web3.storage
	fmt.Printf("Storing %s ...\n", basename)
	cid, err := client.Put(context.Background(), file)
	if err != nil {
		panic(err)
	}

	gatewayURL := fmt.Sprintf("https://%s.ipfs.dweb.link/%s\n", cid.String(), basename)
	fmt.Printf("Stored %s with web3.storage! View it at: %s\n", basename, gatewayURL)
}

Directory wrapping

By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish.

The CID you get back from the client when uploading is the CID of the directory, not the file itself! To link to the file itself using an IPFS URI, just add the filename to the CID, separated by a / like this: ipfs://<cid>/<filename>.

To make a gateway link, use https://<cid>.ipfs.<gateway-host>/<filename> or https://<gateway-host>/ipfs/<cid>/<filename>, where <gateway-host> is the address of an HTTP gateway like dweb.link.

Once uploaded, you can retrieve the directory or list the contents without downloading it.

To avoid having your files wrapped in a directory listing, set the wrapWithDirectory: option to false when uploading using the JavaScript client.

Storing IPFS Content Archives

So far we've focused on using the put method, which accepts regular files and packs them into an IPFS Content Archive (CAR) file before uploading to web3.storage. If you're already using IPFS in your application, or if you want more control over the IPLD graph used to structure your data, you can construct your own CAR files and upload them directly.

See Working with CAR files for more information about Content Archives, including how to create and manipulate them with code or command-line tools.

JavaScript
Go
HTTP API (curl)

Once you have a Content Archive, you can use the putCar client method to upload it to web3.storage.

The putCar method accepts a CarReader, which is a type defined by the @ipld/car package.

You can create a CarReader from a Uint8Array using the fromBytes static method:

import { CarReader } from '@ipld/car';

// assume loadCarData returns the contents of a CAR file as a Uint8Array
const carBytes = await loadCarData();
const reader = await CarReader.fromBytes(carBytes);

const client = makeStorageClient();
const cid = await client.putCar(reader);
console.log('Uploaded CAR file to web3.storage! CID:', cid);

See the putCar reference documentation for more information about putCar, including optional parameters.

The Working with CAR files guide has more information about the @ipld/car package, including how to implement the loadCarData function and other ways to construct a CarReader.

The Go client's PutCar method takes a context.Context and an io.Reader, which must supply a binary stream of data in CAR format.

client, err := w3s.NewClient(w3s.WithToken("<AUTH_TOKEN>"))
carfile, err := os.Open("path/to/data.car")
if err != nil {
  // ...
}

cid, err := client.PutCar(context.Background(), carfile)

If you're using the HTTP API directly, you can upload a CAR file by sending a POST request to the /car endpoint.

Your request must include an Authorization header with a value of Bearer $API_TOKEN, where $API_TOKEN is a valid JWT token for the web3.storage service.

The body of the request should be the binary CAR data, with no additional padding or encoding. Please note that the maximum size for a single CAR upload is 100 MiB. See Working with CAR files for information about splitting CARs into smaller chunks for upload.

The example below assumes that you have a CAR file named mydata.car that you want to upload.

When using curl, it's important to set the --data-binary flag. A common mistake is to use the -d flag instead, which sets the Content-Type header to the wrong value.

curl --header "Authorization: Bearer $API_TOKEN" --data-binary @mydata.car

The above example uses the generic --header flag to set the Authorization header, since setting headers is supported by all HTTP clients. If you are using curl, you can optionally use the more consise --oauth2-bearer flag:

curl --oauth2-bearer $API_TOKEN --data-binary @mydata.car

You can also upload CAR data from non-file sources (for example, another network request) by piping it into the curl commmand and setting the input to /dev/stdin:

some-command-that-outputs-car-data | curl --oauth2-bearer $API_TOKEN --data-binary @/dev/stdin

Next steps

The client returns an IPFS content identifier (CID) that can be used to fetch your files over IPFS. Once uploaded, your data is immediately available for retrieval via IPFS and will be stored with Filecoin storage providers within 48 hours. To learn how to fetch your data using the web3.storage client, or directly from IPFS using a gateway or the IPFS command line, see the how-to guide on retrieval.

You can also get more information about the status of your data. See the query how-to guide to learn how to get more details about your data, including the status of any Filecoin storage deals.

Was this information helpful?

Help us improve this site!

Edit this page on GitHub or open an issue for it

Suggest new content

Storage Limit Increase Request

How to store data using web3.storage

CAUTION

Installing the client

Creating a client instance

Preparing files for upload

Tip

Uploading to web3.storage

IMPORTANT

Showing progress to the user

Directory wrapping

Storing IPFS Content Archives

Next steps

Was this information helpful?

Help us improve this site!