[SOLVED] Technical limits for pagination or is it a bug? Big Values for "Skip"

Bugs Solved Bugs
1

Hello everyone,

today me and my colleagues encountered a problem implementing pagination for big collections.
We have a big collection with 22255 entities and tried to implement pagination for it, but unfortunately we got error messages eventually.

The call we executed was:
https://cloud.squidex.io/api/content/XXXXX/COLLECTION?q={“take”: 10, “skip”: 4700}

{
  "message": "Result set is too large to be retrieved. Use $take parameter to reduce the number of items.",
  "traceId": "XXXXXXXXXXXXXXXX",
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "statusCode": 400
}

The result tells us, that we need to adjust the take parameter, but we think a take of 10 should work pretty well.

Our question is if we encountered some technical limitation we are currently not aware of or if we found a bug?

Regards,
Erek

2

Hi,

are you sure that there is further filter or full text in the query?

In general it is a known limitation, because when you use pagination without an index, then MongoDB keeps take+skip items in memory for the sort operation and there is a memory constraint. But for the default queries, indexes are configured.

If you have a filter you have hit the limit, if not, I need to have a look.

3

Hi Sebastian,

Thanks for your reply. I will give you some more information so that you can decide if this is a problem you have to look into or if we are doing something wrong on our side.

By the way. We are using the cloud version of squidex.

We are not using any filter right now at least we are not aware of any.

squidex_issue
squidex_issue.png1238×365 13.8 KB

As you can see in the query we are just using top and skip in our graphQL queries. The goal is to have a paginated view over all entries in the collection. Therefore we increase the skip parameter step by step (skip += top).

By the way: The squidex UI suffers from the same issue.

If you need more information please ask.

Regards,
Erek

4

Okay, thank you. I will investigate it next week

5

Hi Sebastian,

Any new information regarding this issue? Do you need more information from our side? If so please do not hesitate to ask.

Regards,
Erek

6

Hi, can you give me access to your app?

7

Or at least app-name

8

I have added another index, your example works now.

9

Hi Sebastian,

thanks a lot for looking into our issue.

The index helped a little bit, but eventually it runs into the same problem. Now with a skip value of 4960 and top equals to 10.

squidex_issue4
squidex_issue4.png1776×318 20.2 KB

Is there anything else we are missing? The GQL Query we execute has 3 references (arrays in squidex scheme). Is that an issue?

Best regards,
Erek

10

It could be the MongoDB version that I am running. I am still runinng 3.4 in the cloud, but upgrading is a little bit critical.

I can schedule it for wednesday evening.

11

It has been fixed, the latest MongoDB version works better…

12

Hey Sebastian,

I can confirm that it works now. Thanks for your hard work on that issue :slight_smile:

Best Regards,
Erek

1 Like
13