python / intermediate
Snippet
Memory-Efficient Queryset Processing via QuerySet Iterator Chunking
When dealing with hundreds of thousands of Django ORM records, evaluating a standard QuerySet caches every model instance in Python memory. Using the iterator() method with a specified chunk_size streams database results in manageable batches, keeping RAM consumption minimal during large data processing tasks.
snippet.py
python
1
2
3
4
5
6
from myapp.models import AuditLogdef process_large_audit_logs():logs = AuditLog.objects.filter(processed=False)for log in logs.iterator(chunk_size=1000):log.mark_as_processed()
django
Breakdown
1
logs = AuditLog.objects.filter(processed=False)
Creates an un-evaluated QuerySet for unprocessed audit log entries.
2
for log in logs.iterator(chunk_size=1000):
Fetches records in database batches of 1,000 without loading the entire QuerySet cache into memory.
3
log.mark_as_processed()
Executes processing logic on each individual model instance.