python / intermediate
Snippet
Efficient Data Batching with Bulk Create and Upsert Handling
Inserting or updating large collections of database records individually degrades performance due to excessive roundtrips. Using bulk_create with conflict update parameters performs batch upserts in single SQL queries.
snippet.py
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from myapp.models import MetricEntrydef batch_upsert_metrics(metric_data_list):instances = [MetricEntry(device_id=item['id'], value=item['val'], timestamp=item['ts'])for item in metric_data_list]MetricEntry.objects.bulk_create(instances,batch_size=500,update_conflicts=True,update_fields=['value', 'timestamp'],unique_fields=['device_id'])
django
Breakdown
1
instances = [MetricEntry(device_id=item['id'], value=item['val'], timestamp=item['ts']) for item in metric_data_list]
Constructs uncommitted model instances in memory from raw dictionary collections.
2
batch_size=500,
Splits the database operation into chunks of 500 records per query to manage query length limits.
3
update_conflicts=True, update_fields=['value', 'timestamp'], unique_fields=['device_id']
Configures database-level upsert logic to update specific fields when a unique constraint collides.