Paged queries with more than 100,000 matching objects

Content Platform Tenant Management

Version

9.7.x

File Size

4269 KB

Audience

anonymous

Part Number

MK-95HCPH002-19

If a large number of objects match the query criteria, a paged query can consume a large amount of memory. If more than 100,000 objects match the query criteria, limit memory use by using multiple paged queries. Each paged query should retrieve results for no more than 100,000 objects. To do this, use the changeTimeMilliseconds as the basis for generating the paged queries, as follows:

Issue a request with a count entry value of zero and a changeTimeMilliseconds criterion with a range from zero to some time in the past, such as this:
```
<queryRequest>
    <object>
        <query>+changeTimeMilliseconds:[0 TO 1262304000000.00]
            +retentionClass:hlthReg-107</query>
        <count>0</count>
    </object>
</queryRequest>
```
If the count property in the response is greater than 100,000, repeat this step with an earlier changeTimeMilliseconds end time until the count property in the response is no more than 100,000.
Use a paged query with:
- A changeTimeMilliseconds criterion that specifies the same time as you used in the last request in step 1
- A count entry value that specifies the number of objects you want per page
- An offset entry that you increment by the count value in each request
For example, the request body for the third iteration of the paged query might look like this:
```
<queryRequest>
    <object>
        <query>+changeTimeMilliseconds:[0 TO 1150000000000.00]
            +retentionClass:hlthReg-107</query>
        <sort>changeTimeMilliseconds</sort>
        <count>50</count>
        <offset>150</offset>
    </object>
</queryRequest>
```
Stop when the code property of the status entry in the response is COMPLETE.
Repeat step 1 above using a changeTimeMilliseconds entry that specifies a range with start value equal to the end value of the changeTimeMilliseconds range you used in step 2.
Use a curly opening brace for the range so that the last entry in the previous result set is not included in the new results.
For example, use a changeTimeMilliseconds value like this:
```
changeTimeMilliseconds:{1150000000000.00 TO 1341000000000.00]
```
Then repeat step 2 using the new query criteria.
Repeat step 3 until you retrieve the last matching object.
Use a value of * (for an unlimited range) as the end of the changeTimeMilliseconds range in the last paged query to ensure that you retrieve all objects including those that were most recently added.