Unable to import domain data in new instance

akashkj · December 1, 2022, 4:49am

Issue:

Imported data for the first time using django manage command django-manage load_domain_data. It failed with the following error:

(cchq) azureuser@control1:~$ cchq dns django-manage load_domain_data data-dump-india-nutrition-project-2022-11-18T210639Z.zip
ssh azureuser@10.0.0.9 -t -o UserKnownHostsFile=/home/azureuser/environments/dns/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/dns/current; python_env/bin/python manage.py load_domain_data data-dump-india-nutrition-project-2022-11-18T210639Z.zip'"'"''
Ubuntu 18.04.6 LTS
Loading data from data-dump-india-nutrition-project-2022-11-18T210639Z.zip.

Loading _tmp_load__data-dump-india-nutrition-project-2022-11-18T210639Z.zip/domain.gz using 'domain' data loader.
Started at 2022-11-18 22:43:02
Processing [.] 1/1 100% 0:00:00.000569 elapsed
Finished at 2022-11-18 22:43:02
Elapsed time: 0:00:00
Loaded Domain

Loading _tmp_load__data-dump-india-nutrition-project-2022-11-18T210639Z.zip/sql.gz using 'sql' data loader.
Started at 2022-11-18 22:43:02
2022-11-18 22:45:44,627 ERROR [load_sql] Error saving data    ] 76470/207481 37% 0:04:31 remaining
Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "saved_reports_scheduledreportscheckpoint_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 176, in load_data_for_db
    obj.save(using=db_alias, force_insert=True)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "saved_reports_scheduledreportscheckpoint_pkey"
DETAIL:  Key (id)=(1) already exists.

Loading DB 'p1' complete
Loading DB 'p2' complete
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "saved_reports_scheduledreportscheckpoint_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 176, in load_data_for_db
    obj.save(using=db_alias, force_insert=True)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "saved_reports_scheduledreportscheckpoint_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 136, in worker
    coro.send(obj)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 183, in load_data_for_db
    raise type(err)(
django.db.utils.IntegrityError: Could not load saved_reports.ScheduledReportsCheckpoint(pk=1) in DB 'default'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/manage.py", line 188, in <module>
    main()
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/manage.py", line 46, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/management/commands/load_domain_data.py", line 93, in handle
    loaded_meta.update(self._load_data(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/management/commands/load_domain_data.py", line 128, in _load_data
    return loader.load_from_path(extracted_dump_path, dump_meta, force=self.force, dry_run=self.dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/interface.py", line 63, in load_from_path
    counts = self.load_from_file(path, dump_meta, force, dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/interface.py", line 76, in load_from_file
    loaded_object_count = self.load_objects(object_strings, force, dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 90, in load_objects
    raise errors[0] if len(errors) == 1 else Exception(errors)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 53, in collect_results
    load_stats.append(worker.result())
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
django.db.utils.IntegrityError: Problem loading data '_tmp_load__data-dump-india-nutrition-project-2022-11-18T210639Z.zip': ("Could not load saved_reports.ScheduledReportsCheckpoint(pk=1) in DB 'default'", "Error in worker 'default'")
Processing [...................                               ] 77843/207481 38% 0:02:53.578701 elapsed

Deleted domain using delete_domain command that was successful.
Imported newer version of domain data again with load_domain_data command and it failed.

$ cchq dns django-manage load_domain_data data-dump-india-nutrition-project-2022-11-24T113904Z.zip --use-extracted
ssh azureuser@10.0.0.9 -t -o UserKnownHostsFile=/home/azureuser/environments/dns/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/dns/current; python_env/bin/python manage.py load_domain_data data-dump-india-nutrition-project-2022-11-24T113904Z.zip --use-extracted'"'"''
Ubuntu 18.04.6 LTS
Loading data from data-dump-india-nutrition-project-2022-11-24T113904Z.zip.

Loading _tmp_load__data-dump-india-nutrition-project-2022-11-24T113904Z.zip/domain.gz using 'domain' data loader.
Started at 2022-12-01 04:48:22
Processing [.] 1/1 100% 0:00:00.004072 elapsed
Finished at 2022-12-01 04:48:22
Elapsed time: 0:00:00
Loaded Domain

Loading _tmp_load__data-dump-india-nutrition-project-2022-11-24T113904Z.zip/sql.gz using 'sql' data loader.
Started at 2022-12-01 04:48:22
2022-12-01 04:48:22,938 ERROR [load_sql] Error saving data    ] 0/208341 0% -:--:-- remaining
Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "locations_locationtype_pkey"
DETAIL:  Key (id)=(1441) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 176, in load_data_for_db
    obj.save(using=db_alias, force_insert=True)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "locations_locationtype_pkey"
DETAIL:  Key (id)=(1441) already exists.

2022-12-01 04:48:22,978 ERROR [load_sql] Error saving data
Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "blobs_blobmeta_key_a9ed5760_uniq"
DETAIL:  Key (key)=(8f2ae9737c634a7b8eacbbddafcc2fbc) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 176, in load_data_for_db
    obj.save(using=db_alias, force_insert=True)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "blobs_blobmeta_key_a9ed5760_uniq"
DETAIL:  Key (key)=(8f2ae9737c634a7b8eacbbddafcc2fbc) already exists.

Traceback (most recent call last):
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/manage.py", line 188, in <module>
    main()
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/manage.py", line 46, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/management/commands/load_domain_data.py", line 93, in handle
    loaded_meta.update(self._load_data(
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/management/commands/load_domain_data.py", line 128, in _load_data
    return loader.load_from_path(extracted_dump_path, dump_meta, force=self.force, dry_run=self.dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/interface.py", line 63, in load_from_path
    counts = self.load_from_file(path, dump_meta, force, dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/interface.py", line 76, in load_from_file
    loaded_object_count = self.load_objects(object_strings, force, dry_run)
  File "/home/cchq/www/dns/releases/2022-11-17_08.45/corehq/apps/dump_reload/sql/load.py", line 90, in load_objects
    raise errors[0] if len(errors) == 1 else Exception(errors)
Exception: Problem loading data '_tmp_load__data-dump-india-nutrition-project-2022-11-24T113904Z.zip': [IntegrityError("Could not load locations.LocationType(pk=1441) in DB 'default'", "Error in worker 'default'"), IntegrityError("Could not load blobs.BlobMeta(key=8f2ae9737c634a7b8eacbbddafcc2fbc) in DB 'p1'", "Error in worker 'p1'")]
Processing [                                                  ] 334/208341 0% 0:00:11.655727 elapsed

How to successfully import domain data to new instance? Anyway to skip parts of data that are already imported and present in the new sytem?

Simon_Kelly · December 1, 2022, 6:47am

Hi Akash

It is possible that the 'delete-domain' command isn't fully deleting the data. I should also point out that the dump / load process is intended to be used with a clean environment (i.e. no other data present at all).

You should check your DB to see what data is present in the DB prior to importing.