Paginate your results using .NET V4 SDK for Azure Cosmos DB

When developing your API, you often need to return a list of elements. In most of the cases you can’t return all of these at once. This is where the pagination mechanism comes in!

In this tutorial I will show you the important parts of this mechanism using the new Azure Cosmos DB nuget inside an Azure Functions. Please note that this NuGet package is in preview at the moment of writing this article.

This new version implements the best practices and common patterns to use Cosmos DB. The entire source code example is of course available in the associated Github repository.

The workflow

Let’s imagine this scenario: You have to return a list of vegetables from your API. You have a very large number of vegetables in your database, so you need to paginate the results.

The continuation token

Let’s suppose you have 1000 elements in your list and you just want to send 25 elements per page. Your API also needs to send something to help the client to know on which page to resume the rest of the elements. This is where continuation token comes in.

This token is returned by Azure Cosmos DB to help you to get the next elements. When the client asks for the next elements it must send this continuation token to the API. This is like a bookmark in your API.

Here is a little schema to help you understand the principle:

When there is no more elements to retrieve, Azure Cosmos DB will return a value of null for the continuation token.

The Azure.Cosmos nuget package returns a string property called ContinuationToken which is a flat json. Here is the format:

{
   "Version":"1.1",
   "QueryPlan":"{\"partitionedQueryExecutionInfoVersion\":2,\"queryInfo\":{\"distinctType\":\"None\",\"top\":null,\"offset\":null,\"limit\":null,\"orderBy\":[],\"orderByExpressions\":[],\"groupByExpressions\":[],\"groupByAliases\":[],\"aggregates\":[],\"groupByAliasToAggregateType\":{},\"rewrittenQuery\":\"\",\"hasSelectValue\":false},\"queryRanges\":[{\"min\":\"\",\"max\":\"FF\",\"isMinInclusive\":true,\"isMaxInclusive\":false}]}",
   "SourceContinuationToken":"[{\"token\":\"-RID:~OuEWAPVCbA0UAAERTTAAAAA==#RT:1#TRC:20#ISV:2#IEO:65567#QCF:4\",\"range\":{\"min\":\"\",\"max\":\"FF\"}}]"
}

The most important value here is contained inside the SourceContinuationToken property. This is the continuation token itself, the client will need to send it back to the API to get the next piece of data.

To extract this value, we can create a basic object to deserialize this json:

public class ContinuationToken
{
    [JsonProperty("Version")]
    public string Version { get; set; }

    [JsonProperty("QueryPlan")]
    public string QueryPlan { get; set; }

    [JsonProperty("SourceContinuationToken")]
    public string SourceContinuationToken { get; set; }
}

Pagination

Now it’s time to look at the pagination code itself:

public async Task<PagedListResponse<Vegetable>> GetPaginatedVegetablesAsync(int pageSize, string continuationToken)
{
    try
    {
        // Get the container where the elements are stored
        var container = GetContainer();
        // Create your query
        var query = new QueryDefinition("SELECT * FROM v");

        var queryResultSetIterator = container.GetItemQueryIterator<Vegetable>(query, requestOptions: new QueryRequestOptions()
        {
            MaxItemCount = pageSize,
        }, continuationToken: continuationToken).AsPages();

        var result = await queryResultSetIterator.FirstOrDefaultAsync();

        // Deserialize the ContinuationToken propety to access the SourceContinuationToken
        var sourceContinuationToken = result.ContinuationToken != null ? 
                                JsonConvert.DeserializeObject<ContinuationToken>(result.ContinuationToken).SourceContinuationToken : null;
        // Send the data with the continuation token
        return new PagedListResponse<Vegetable>()
        {
            ContinuationToken = sourceContinuationToken,
            Data = result.Values.ToList(),
        };
    }
    catch (CosmosException ex)
    {
        _logger.LogError($"Entities was not retrieved successfully - error details: {ex.Message}");

        if (ex.Status != (int)HttpStatusCode.NotFound)
        {
            throw;
        }

        return null;
    }
}

As you can see above, we start by getting the container where we want to retreive the data. Then we define our query, this is a basic select however you can inject parameters with the WithParameter() method.

The QueryRequestOptions define the maximum number of elements by page. Of course if you ask for 10 and you have only 3 you will receive 3.

The FirstOrDefaultAsync method is from the System.Linq.Async nuget package which is really usefull to manipulate the IAsyncEnumerable objects. This method is accessible inside the using System.Linq; namespace.

Next, we deserialize the ContinuationToken property so we can access the SourceContinuationToken which will be used for the next call.

Finally we return a PagedListResponse object which contains the list of data and the continuation token.

Get the next elements of the list

With this mechanism ready, how to use it in practice?

When you call the end point the first time, you do it directly like this: https://YOUR_HOST/api/demo-az-cosmos-pagination?page_size=25 without specifying the continuation property because you don’t have it at the beginning.

Then, for the next calls the url will be populated with the continuation property like below: https://YOUR_HOST/api/demo-az-cosmos-pagination?page_size=25&continuation=%5B%7B%22token%22%3A%22-RID%3A~OuEWA...

The continuation token value is converted to its escaped representation to pass it to the url. To help the client, your API can send it directly like this:

public string UrlEncodedContinuationToken
{
    get => Uri.EscapeDataString(this.ContinuationToken ?? string.Empty);
}

This is the kind of results that you will have:

{
  "continuationToken": "[{\"token\":\"-RID:~OuEWAPVCbA0PAAAAAAAAAA==#RT:3#TRC:15#ISV:2#IEO:65567#QCF:4\",\"range\":{\"min\":\"\",\"max\":\"FF\"}}]",
  "data": [
    {
      "id": "ef99e80e-b028-4d0c-ac2f-f277800f6886",
      "name": "squash"
    },
    {
      "id": "cf8fc35d-adae-44ae-9a47-010d790de966",
      "name": "spinach"
    },
    {
      "id": "c6986662-78ab-422f-8887-8530621d9da9",
      "name": "bean"
    },
    {
      "id": "3bc0c55d-9f47-4deb-a31b-c0b877605e20",
      "name": "lamb's lettuce"
    },
    {
      "id": "8ecd6b1b-1d61-409f-beb5-91e9f2a7fa21",
      "name": "melon"
    }
  ],
  "urlEncodedContinuationToken": "%5B%7B%22token%22%3A%22-RID%3A~OuEWAPVCbA0PAAAAAAAAAA%3D%3D%23RT%3A3%23TRC%3A15%23ISV%3A2%23IEO%3A65567%23QCF%3A4%22%2C%22range%22%3A%7B%22min%22%3A%22%22%2C%22max%22%3A%22FF%22%7D%7D%5D"
}

Final touch

As you saw it’s really easy to add a pagination with the new Azure.Cosmos nuget. You will find an entire example using an Azure Functions in this Github repository. All the setup is inside the README.md of this project.

Happy coding!

You liked this tutorial? Leave a star in the associated Github repository!

Sources:

• .NET V4 SDK pour Azure Cosmos DB documentation

Do not hesitate to follow me on Twitter to not miss my next tutorial!