Resume pagination after specific document

Working with Data Node.js Frameworks
, , , ,
1

I’m loading user comments on my website. Initially, I only load a subset of comments and show a Load more button which loads the next page and appends it to the end of the comments list.

enter image description here

Loading another page works by using the usual skip and limit aggregation steps in MongoDB:

const aggregationResult = await ResourceCommentModel.aggregate()
    .match({
        resourceId: new mongoose.Types.ObjectId(req.params.resourceId),
        parentCommentId: undefined
    })
    .sort({ 'createdAt': -1 })
    .facet({
        comments: [
            { $skip: (page - 1) * pageSize },
            { $limit: pageSize },
            {
                $lookup: {
                    from: 'resourcecomments',
                    localField: '_id',
                    foreignField: 'parentCommentId',
                    as: 'replies',
                }
            }
        ],
        totalCount: [{ $group: { _id: null, count: { $sum: 1 } } }]
    })
    .exec();

Problem: If new comments have been posted in the meantime, comments from page one have been pushed to page 2, and we are loading duplicate comments (and showing them in the UI).

enter image description here

One option would be to not rely on the page number, but instead start after the last loaded timestamp:

comments: [
    { $match: { createdAt: { $gt: new Date(continueAfterTimestamp) } } },
    { $limit: pageSize },
    {
        $lookup: {
            from: 'resourcecomments',
            localField: '_id',
            foreignField: 'parentCommentId',
            as: 'replies',
        }
    }
],

However, there are 2 problems with this approach:

  1. Although unlikely, if two comments were posted at the exact same millisecond, one of them will not be loaded.

  2. The bigger problem: I’m planning to later allow sorting comments by upvotes, and then I don’t have a way to start after the last loaded document anymore.

2

Hi @Florian_Walther ,

Why won’t you pre load comments for first 4-5 pages and name in the facet comments1-5?

Otherwise you can use unique commentId and sort by them and always pass the last commentId to the next query, when you need to use upvotes add them to the beginning of the sort and sort on (upvotes, commentId)

Thanks
Pavel

3

Thank you for your answer!

Wouldn’t this kinda destroy the purpose of pagination (to load less data at once)? Also, it doesn’t solve my sorting problem, does it?

Yea that’s what I’m doing right now after finding out that the _id contains a timestamp.

I don’t think this will work. To start after the last comment id, I have to use match with $lt/$gt: commentId. If I sort by upvotes first, the ids will not be ordered from smallest to largest anymore.

4

Also, how do I know when I reached the last document? At the moment, I’m returning the last document as a separate facet step so I can check client side if we received this document in our dataset yet. Is there a better way to handle this?

.facet({
    comments: [
        ...(continueAfterId ? [{ $match: { _id: { $lt: new mongoose.Types.ObjectId(continueAfterId) } } }] : []),
        { $sort: { _id: -1 } },
        { $limit: pageSize },
    ],
    lastComment: [
        { $sort: { _id: 1 } },
        { $limit: 1 },
    ]
})
5

Returning the last document was buggy in certain situations. I now figure out the end of pagination by returning 1 document more than the page size, and then I check if the length of the results is pageSize + 1, which means that there are more documents to come. What do you think about this approach?

async function fetchComments(pageSize: number, ...) {

    ...

    const aggregationResult = await ResourceCommentModel.aggregate()
        .match({
            ...
        })
        .facet({
            comments: [
                ...
                { $sort: { _id: sort } }, 
                { $limit: pageSize + 1 },
            ],
        })

...

    return {
        resourceComments: resourceCommentsPopulated.slice(0, pageSize),
        endOfPaginationReached: resourceComments.length <= pageSize,
    }
}
1 Like
6

Another option is to hold a cursor with a batch size of your page size and run a “next” command every time the page is being rolled forward…

7

Interesting, thank you. I will try to look into this!