python / intermediate
Snippet
Chunking Large Database Results with QuerySet.iterator()
Using `iterator()` evaluates a QuerySet by streaming results from the database in chunks, avoiding loading huge datasets entirely into server memory.
snippet.py
python
1
2
3
4
5
6
from myapp.models import AuditLogdef process_audit_logs():large_dataset = AuditLog.objects.filter(processed=False).iterator(chunk_size=1000)for log_entry in large_dataset:log_entry.mark_as_processed()
django
Breakdown
1
from myapp.models import AuditLog
Imports the AuditLog model class.
2
def process_audit_logs():
Defines a function to iteratively process unprocessed log records.
3
large_dataset = AuditLog.objects.filter(processed=False).iterator(chunk_size=1000)
Fetches records in batches of 1000 using database server-side cursors.
4
for log_entry in large_dataset:
Loops through each individual log entry as it is streamed from memory.
5
log_entry.mark_as_processed()
Calls a domain method on each log model instance.