python / expert
Snippet
Batch Processing QuerySet Generator for Large Array Fields
Keyset pagination via primary key evaluation avoids the exponential query slowdown of traditional OFFSET/LIMIT pagination when processing massive Django querysets. Using Python generators, large datasets are loaded into memory in controlled chunk arrays.
snippet.py
python
1
2
3
4
5
6
7
8
9
10
11
12
from typing import Generator, Anyfrom django.db.models import QuerySetdef chunk_queryset(qs: QuerySet, chunk_size: int = 500) -> Generator[list[Any], None, None]:pk = 0base_qs = qs.order_by('pk')while True:chunk = list(base_qs.filter(pk__gt=pk)[:chunk_size])if not chunk:breakpk = chunk[-1].pkyield chunk
django
Breakdown
1
from typing import Generator, Any
Imports type hints for generator functions and generic model instances.
2
from django.db.models import QuerySet
Imports Django QuerySet type definition for database operations.
3
def chunk_queryset(qs: QuerySet, chunk_size: int = 500) -> Generator[list[Any], None, None]:
Defines generator function splitting querysets into paginated memory-safe batches.
4
pk = 0
Initializes primary key cursor reference for keyset pagination.
5
base_qs = qs.order_by('pk')
Ensures deterministic query ordering by primary key attribute.
6
while True:
Starts infinite chunking loop terminated when query results are exhausted.
7
chunk = list(base_qs.filter(pk__gt=pk)[:chunk_size])
Fetches slice of models evaluated via database keyset filtering greater than last pk.
8
if not chunk:
Checks if slice operation returned empty list signaling end of dataset.
9
break
Exits while loop when no remaining records exist.
10
pk = chunk[-1].pk
Updates tracking cursor to primary key of last element in current batch.
11
yield chunk
Yields populated list batch to caller before continuing next pagination cycle.