GridFS

Overview

In this guide, you can learn how to store and retrieve large files in MongoDB using GridFS. GridFS is a specification implemented by the driver that describes how to split files into chunks when storing them and reassemble them when retrieving them. The driver implementation of GridFS is an abstraction that manages the operations and organization of the file storage.

You should use GridFS if the size of your files exceed the BSON document size limit of 16MB. For more detailed information on whether GridFS is suitable for your use case, see the GridFS server manual page.

See the following sections that describe GridFS operations and how to perform them:

• Create a GridFS bucket

• Store Files

• Retrieve File Information

• Download Files

• Rename Files

• Delete Files

• Delete a GridFS bucket

How GridFS Works

GridFS organizes files in a bucket, a group of MongoDB collections that contain the chunks of files and information describing them. The bucket contains the following collections, named using the convention defined in the GridFS specification:

• The chunks collection stores the binary file chunks.

• The files collection stores the file metadata.

When you create a new GridFS bucket, the driver creates the preceding collections, prefixed with the default bucket name fs, unless you specify a different name. The driver also creates an index on each collection to ensure efficient retrieval of the files and related metadata. The driver only creates the GridFS bucket on the first write operation if it does not already exist. The driver only creates indexes if they do not exist and when the bucket is empty. For more information on GridFS indexes, see the server manual page on GridFS Indexes.

When storing files with GridFS, the driver splits the files into smaller chunks, each represented by a separate document in the chunks collection. It also creates a document in the files collection that contains a file id, file name, and other file metadata. You can upload the file from memory or from a stream. See the following diagram to see how GridFS splits the files when uploaded to a bucket.

When retrieving files, GridFS fetches the metadata from the files collection in the specified bucket and uses the information to reconstruct the file from documents in the chunks collection. You can read the file into memory or output it to a stream.

Create a GridFS Bucket

To store or retrieve files from GridFS, create a bucket or get a reference to an existing one on a MongoDB database. Call the GridFSBuckets.create() helper method with a MongoDatabase instance as the parameter to instantiate a GridFSBucket. You can use the GridFSBucket instance to call read and write operations on the files in your bucket.

MongoDatabase database = mongoClient.getDatabase("mydb");
GridFSBucket gridFSBucket = GridFSBuckets.create(database);

To create or reference a bucket with a custom name other than the default name fs, pass your bucket name as the second parameter to the create() method as shown below:

GridFSBucket gridFSBucket = GridFSBuckets.create(database, "myCustomBucket");

For more information on the classes and methods mentioned in this section, see the following API Documentation:

• GridFSBuckets

• GridFSBucket

Store Files

To store a file in a GridFS bucket, you can either upload it from an instance of InputStream or write its data to a GridFSUploadStream.

For either upload process, you can specify configuration information such as file chunk size and other field/value pairs to store as metadata. Set this information on an instance of GridFSUploadOptions as shown in the following code snippet:

GridFSUploadOptions options = new GridFSUploadOptions()
        .chunkSizeBytes(1048576) // 1MB chunk size
        .metadata(new Document("myField", "myValue"));

See the GridFSUploadOptions API Documentation for more information.

String filePath = "/path/to/project.zip";
try (InputStream streamToUploadFrom = new FileInputStream(filePath) ) {
    // Defines options that specify configuration information for files uploaded to the bucket
    GridFSUploadOptions options = new GridFSUploadOptions()
            .chunkSizeBytes(1048576)
            .metadata(new Document("type", "zip archive"));
    // Uploads a file from an input stream to the GridFS bucket
    ObjectId fileId = gridFSBucket.uploadFromStream("myProject.zip", streamToUploadFrom, options);
    // Prints the "_id" value of the uploaded file
    System.out.println("The file id of the uploaded file is: " + fileId.toHexString());
}

Path filePath = Paths.get("/path/to/project.zip");
byte[] data = Files.readAllBytes(filePath);
// Defines options that specify configuration information for files uploaded to the bucket
GridFSUploadOptions options = new GridFSUploadOptions()
        .chunkSizeBytes(1048576)
        .metadata(new Document("type", "zip archive"));
try (GridFSUploadStream uploadStream = gridFSBucket.openUploadStream("myProject.zip", options)) {
    // Writes file data to the GridFS upload stream
    uploadStream.write(data);
    uploadStream.flush();
    // Prints the "_id" value of the uploaded file
    System.out.println("The file id of the uploaded file is: " + uploadStream.getObjectId().toHexString());
// Prints a message if any exceptions occur during the upload process
} catch (Exception e) {
    System.err.println("The file upload failed: " + e);
}

Retrieve File Information

In this section, you can learn how to retrieve file metadata stored in the files collection of the GridFS bucket. The metadata contains information about the file it refers to, including:

• The id of the file

• The name of the file

• The length/size of the file

• The upload date and time

• A metadata document in which you can store any other information

To retrieve files from a GridFS bucket, call the find() method on the GridFSBucket instance. The method returns a GridFSFindIterable from which you can access the results.

The following code example shows you how to retrieve and print file metadata from all your files in a GridFS bucket. Among the different ways that you can traverse the retrieved results from the GridFSFindIterable, the example uses a Consumer functional interface to print the following results:

gridFSBucket.find().forEach(new Consumer<GridFSFile>() {
    @Override
    public void accept(final GridFSFile gridFSFile) {
        System.out.println(gridFSFile);
    }
});

The next code example shows you how to retrieve and print the file names for all files that match the fields specified in the query filter. The example also calls sort() and limit() on the returned GridFSFindIterable to specify the order and maximum number of results:

Bson query = Filters.eq("metadata.type", "zip archive");
Bson sort = Sorts.ascending("filename");
// Retrieves 5 documents in the bucket that match the filter and prints metadata
gridFSBucket.find(query)
        .sort(sort)
        .limit(5)
        .forEach(new Consumer<GridFSFile>() {
            @Override
            public void accept(final GridFSFile gridFSFile) {
                System.out.println(gridFSFile);
            }
        });

Since metadata is an embedded document, the query filter specifies the type field within the document using dot notation. See the server manual guide on how to Query on Embedded/Nested Documents for more information.

For more information on the classes and methods mentioned in this section, see the following resources:

• GridFSFindIterable API Documentation

• GridFSBucket.find() API Documentation

• Sort Results

• Limit the Number of Returned Results

Download Files

You can download a file from GridFS directly to a stream or you can save it to memory from a stream. You can specify the file to retrieve using either the file id or file name.

GridFSDownloadOptions downloadOptions = new GridFSDownloadOptions().revision(1);

GridFSDownloadOptions downloadOptions = new GridFSDownloadOptions().revision(0);
// Downloads a file to an output stream
try (FileOutputStream streamToDownloadTo = new FileOutputStream("/tmp/myProject.zip")) {
    gridFSBucket.downloadToStream("myProject.zip", streamToDownloadTo, downloadOptions);
    streamToDownloadTo.flush();
}

ObjectId fileId = new ObjectId("60345d38ebfcf47030e81cc9");
// Opens an input stream to read a file containing a specified "_id" value and downloads the file
try (GridFSDownloadStream downloadStream = gridFSBucket.openDownloadStream(fileId)) {
    int fileLength = (int) downloadStream.getGridFSFile().getLength();
    byte[] bytesToWriteTo = new byte[fileLength];
    downloadStream.read(bytesToWriteTo);
    // Prints the downloaded file's contents as a string
    System.out.println(new String(bytesToWriteTo, StandardCharsets.UTF_8));
}

Rename Files

You can update the name of a GridFS file in your bucket by calling the rename() method. You must specify the file to rename by its file id rather than its file name.

The following code example shows you how to update the name of the file referenced by the fileId variable to "mongodbTutorial.zip":

ObjectId fileId = new ObjectId("60345d38ebfcf47030e81cc9");
// Renames the file that has a specified "_id" value to "mongodbTutorial.zip"
gridFSBucket.rename(fileId, "mongodbTutorial.zip");

For more information on this method, see the rename() API Documentation.

Delete Files

You can remove a file from your GridFS bucket by calling the delete() method. You must specify the file by its file id rather than its file name.

The following code example shows you how to delete the file referenced by the fileId variable:

ObjectId fileId = new ObjectId("60345d38ebfcf47030e81cc9");
// Deletes the file that has a specified "_id" value from the GridFS bucket
gridFSBucket.delete(fileId);

For more information on this method, see the delete() API Documentation.

Delete a GridFS Bucket

The following code example shows you how to delete the default GridFS bucket on the database named "mydb". If you need to reference a custom named bucket, see the section of this guide on how to create a custom bucket.

MongoDatabase database = mongoClient.getDatabase("mydb");
GridFSBucket gridFSBucket = GridFSBuckets.create(database);
gridFSBucket.drop();

For more information on this method, see the drop() API Documentation.

Additional Resources

• MongoDB GridFS specification

• Runnable example GridFSTour.java from the MongoDB Java Driver repository.