Syncing Elastic Search index Error: run_ptop --all or --pillow-key=core causing Error

Dear All,
After installing Commcare HQ with the dockerized services I’m was not able to solve the following problem: I can’t run the following command: ./manage.py run_ptop --pillow-key=core
neither ./manage.py run_ptop --all in order to sync the elastic search index. I’m receiving this error by running the command (see bellow)

I tried these way to solve the problem:
1- Deleting the the duplicated values in the DB table: pillowtop_kafkacheckpoint :
2- Resetting the Elasticseach index: with curl -X DELETE http://localhost:9200/*. * and
./manage.py ptop_preindex && ./manage.py ptop_es_manage --flip_all_aliases.
3- Using a non dockerized postgresql server with version 9.6 and 10

I’m receiving the same error in Postgres with randomly changed checkpoint_id i.e (UserPillow-hqusers_2017-09-07, ApplicationToElasticsearchPillow-hqapps_2017-05-22_1426, …)

I’m stuck at this step. any body give me a hand getting past this one?
Thanks

Error Log by running
./manage.py run_ptop --pillow-key=core
corehq.util.log.HqAdminEmailHandler logger is being changed to logging.StreamHandler
corehq.util.log.NotifyExceptionEmailer logger is being changed to logging.StreamHandler
/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/kombu/transport/django/models.py:29: RemovedInDjango20Warning: on_delete will be a required arg for ForeignKey in Django 2.0. Set it to models.CASCADE on models and in existing migrations if you want to maintain the current default behavior. See https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.ForeignKey.on_delete
queue = models.ForeignKey(Queue, related_name=‘messages’)
/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/captcha/fields.py:4: RemovedInDjango20Warning: Importing from django.core.urlresolvers is deprecated in favor of django.urls.
from django.core.urlresolvers import reverse, NoReverseMatch
[pillowtop] Starting pillow processes
2018-01-30 07:55:10,904 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,910 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,917 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,930 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,959 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,970 INFO Configuring consumer to fetch topic ‘case’, partition 0
2018-01-30 07:55:10,970 INFO Configuring consumer to fetch topic ‘case-sql’, partition 0
2018-01-30 07:55:10,971 INFO Consumer fetching stored offsets
2018-01-30 07:55:10,971 INFO Configuring consumer to fetch topic ‘form’, partition 0
2018-01-30 07:55:10,972 INFO Configuring consumer to fetch topic ‘form-sql’, partition 0
2018-01-30 07:55:10,972 INFO Consumer fetching stored offsets
2018-01-30 07:55:10,977 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:10,999 INFO Configuring consumer to fetch topic ‘form-sql’, partition 0
2018-01-30 07:55:11,000 INFO Configuring consumer to fetch topic ‘form’, partition 0
2018-01-30 07:55:11,000 INFO Consumer fetching stored offsets
2018-01-30 07:55:11,002 INFO Configuring consumer to fetch topic ‘case’, partition 0
2018-01-30 07:55:11,002 INFO Configuring consumer to fetch topic ‘case-sql’, partition 0
2018-01-30 07:55:11,002 INFO Consumer fetching stored offsets
2018-01-30 07:55:11,003 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:11,020 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
File “./corehq/ex-submodules/pillowtop/models.py”, line 92, in get_or_create_for_checkpoint_id
.values_list(‘topic’, ‘partition’)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 122, in iter
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 844, in results_iter
for rows in results:
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 1289, in cursor_iter
sentinel):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 1288, in
for rows in iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/utils.py”, line 101, in inner
return func(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/utils.py”, line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/utils.py”, line 101, in inner
return func(*args, **kwargs)
ProgrammingError: no results to fetch
2018-01-30 07:55:11,040 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
[pillowtop] all processes started, pids: [31512, 31513, 31514, 31515, 31516, 31517, 31518, 31519, 31520, 31521, 31522]
2018-01-30 07:55:11,049 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
Process Process-6:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
Process Process-3:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
File “./corehq/ex-submodules/pillowtop/models.py”, line 103, in get_or_create_for_checkpoint_id
cls.objects.bulk_create(to_create)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 443, in bulk_create
ids = self._batched_insert(objs_without_pk, fields, batch_size)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1093, in _batched_insert
inserted_id = self._insert(item, fields=fields, using=self.db, return_id=True)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1076, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 1112, in execute_sql
cursor.execute(sql, params)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/utils.py”, line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/raven/contrib/django/client.py”, line 114, in execute
return real_execute(self, sql, params)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/utils.py”, line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
IntegrityError: duplicate key value violates unique constraint "pillowtop_kafkacheckpoin_checkpoint_id_topic_part_c9d04dbb_uniq"
DETAIL: Key (checkpoint_id, topic, partition)=(UserPillow-hqusers_2017-09-07, web-user, 0) already exists.

2018-01-30 07:55:11,061 INFO Configuring consumer to fetch topic ‘app’, partition 0
2018-01-30 07:55:11,062 INFO Consumer fetching stored offsets
Process Process-7:
File “./corehq/ex-submodules/pillowtop/models.py”, line 92, in get_or_create_for_checkpoint_id
.values_list(‘topic’, ‘partition’)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 122, in iter
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 841, in results_iter
results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
File “./corehq/ex-submodules/pillowtop/models.py”, line 92, in get_or_create_for_checkpoint_id
.values_list(‘topic’, ‘partition’)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 122, in iter
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 841, in results_iter
results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

2018-01-30 07:55:11,073 INFO interface Starting pillow <class ‘pillowtop.pillow.interface.ConstructedPillow’>
2018-01-30 07:55:11,078 INFO Configuring consumer to fetch topic ‘app’, partition 0
2018-01-30 07:55:11,079 INFO Consumer fetching stored offsets
Process Process-9:
Process Process-11:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 54, in get_or_create_wrapped
checkpoint = get_or_create_checkpoint(self.checkpoint_id, self.sequence_format)
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 31, in get_or_create_checkpoint
’timestamp’: datetime.utcnow()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 464, in get_or_create
return self.get(**lookup), False
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 374, in get
num = len(clone)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 232, in len
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
File “./corehq/ex-submodules/pillowtop/models.py”, line 92, in get_or_create_for_checkpoint_id
.values_list(‘topic’, ‘partition’)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 122, in iter
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 841, in results_iter
results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Process Process-8:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 54, in get_or_create_wrapped
checkpoint = get_or_create_checkpoint(self.checkpoint_id, self.sequence_format)
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 31, in get_or_create_checkpoint
’timestamp’: datetime.utcnow()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 464, in get_or_create
return self.get(**lookup), False
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 374, in get
num = len(clone)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 232, in len
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Process Process-10:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 83, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 177, in get_or_create_wrapped
checkpoints = self._get_checkpoints()
File “./corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 174, in _get_checkpoints
return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
File “./corehq/ex-submodules/pillowtop/models.py”, line 92, in get_or_create_for_checkpoint_id
.values_list(‘topic’, ‘partition’)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 122, in iter
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 841, in results_iter
results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

2018-01-30 07:55:11,170 ERROR interface [XFormToElasticsearchPillow] Error on change: fe0fb041-a145-4adb-8552-a976f9d7570a, connection already closed. Logged as: PillowError.get_or_create failed
Traceback (most recent call last):
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 275, in handle_pillow_error
error = PillowError.get_or_create(change, pillow)
File “./corehq/ex-submodules/pillow_retry/models.py”, line 87, in get_or_create
error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 374, in get
num = len(clone)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 232, in len
self._fetch_all()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 887, in execute_sql
cursor = self.connection.cursor()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 254, in cursor
return self._cursor()
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 231, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/utils.py”, line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 231, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File “/home/ubuntu/.virtualenvs/endehq/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py”, line 220, in create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed
2018-01-30 07:55:11,174 ERROR Notify Exception: processor error in pillow XFormToElasticsearchPillow connection already closed
Traceback (most recent call last):
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 113, in process_changes
self.process_with_error_handling(change)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 137, in process_with_error_handling
handle_pillow_error(self, change, ex)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 293, in handle_pillow_error
raise e
InterfaceError: connection already closed
Process Process-2:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 99, in run
self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 113, in process_changes
self.process_with_error_handling(change)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 137, in process_with_error_handling
handle_pillow_error(self, change, ex)
File “./corehq/ex-submodules/pillowtop/pillow/interface.py”, line 293, in handle_pillow_error
raise e
InterfaceError: connection already closed

Postgres Error Log 1

postgres_1 | ERROR: duplicate key value violates unique constraint "pillowtop_kafkacheckpoin_checkpoint_id_topic_part_c9d04dbb_uniq"
postgres_1 | DETAIL: Key (checkpoint_id, topic, partition)=(UserPillow-hqusers_2017-09-07, web-user, 0) already exists.
postgres_1 | STATEMENT: INSERT INTO “pillowtop_kafkacheckpoint” (“checkpoint_id”, “topic”, “partition”, “offset”, “last_modified”) VALUES (‘UserPillow-hqusers_2017-09-07’, ‘web-user’, 0, 0, ‘2018-01-30T07:55:11.001346’::timestamp), (‘UserPillow-hqusers_2017-09-07’, ‘commcare-user’, 0, 0, ‘2018-01-30T07:55:11.001391’::timestamp) RETURNING “pillowtop_kafkacheckpoint”."id"
postgres_1 | ERROR: current transaction is aborted, commands ignored until end of transaction block
postgres_1 | STATEMENT: SELECT DISTINCT ON (“pillowtop_kafkacheckpoint”.“topic”, “pillowtop_kafkacheckpoint”.“partition”) “pillowtop_kafkacheckpoint”.“topic”, “pillowtop_kafkacheckpoint”.“partition” FROM “pillowtop_kafkacheckpoint” WHERE (“pillowtop_kafkacheckpoint”.“topic” IN (‘group’) AND “pillowtop_kafkacheckpoint”.“checkpoint_id” = ‘GroupToUserPillow-hqusers_2017-09-07’)

Postgres Error Log 2

postgres_1 | ERROR: duplicate key value violates unique constraint "pillowtop_kafkacheckpoin_checkpoint_id_topic_part_c9d04dbb_uniq"
postgres_1 | DETAIL: Key (checkpoint_id, topic, partition)=(ApplicationToElasticsearchPillow-hqapps_2017-05-22_1426, app, 0) already exists.
postgres_1 | STATEMENT: INSERT INTO “pillowtop_kafkacheckpoint” (“checkpoint_id”, “topic”, “partition”, “offset”, “last_modified”) VALUES (‘ApplicationToElasticsearchPillow-hqapps_2017-05-22_1426’, ‘app’, 0, 0, ‘2018-01-30T08:18:21.238477’::timestamp) RETURNING “pillowtop_kafkacheckpoint”."id"
postgres_1 | ERROR: current transaction is aborted, commands ignored until end of transaction block
postgres_1 | STATEMENT: SELECT DISTINCT ON (“pillowtop_kafkacheckpoint”.“topic”, “pillowtop_kafkacheckpoint”.“partition”) “pillowtop_kafkacheckpoint”.“topic”, “pillowtop_kafkacheckpoint”.“partition” FROM “pillowtop_kafkacheckpoint” WHERE (“pillowtop_kafkacheckpoint”.“topic” IN (‘group’) AND “pillowtop_kafkacheckpoint”.“checkpoint_id” = ‘GroupToUserPillow-hqusers_2017-09-07’)