Dear Cory, please which first error are you reffering to? Thnks again.
"connection already closed" might also imply that something went wrong
earlier in the process leaving the connection in a broken state. so another
avenue to pursue would be looking in the logs for different traces. though
maybe the first error you sent is the root problem.Thanks Cory for the support and encouragement. Seems to be something
around multiprocessing and database:ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify
Exception: processor error in pillow CaseToElasticsearchPillow connection
already closed
Traceback (most recent call last):
File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 94, in
process_changes
self.process_with_error_handling(change)
File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 111,
in process_with_error_handling
handle_pillow_error(self, change, ex)
File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 179,
in handle_pillow_error
error = PillowError.get_or_create(change, pillow)
File "corehq/ex-submodules/pillow_retry/models.py", line 88, in
get_or_create
error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get
num = len(clone)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len
self._fetch_all()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all
self._result_cache = list(self.iterator())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator
results = compiler.execute_sql()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 838, in execute_sql
cursor = self.connection.cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 162, in cursor
cursor = self.make_debug_cursor(self._cursor())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor
return self.create_cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor
return self.create_cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py",
line 212, in create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closedI will keep looking.
Euc
Hey,
It's pretty weird to me that your DB connection is working in other
places but not in pillowtop. I would also have a look in your postgres logs
if the psycopg thing doesn't work.Cory
Update: psycopg 2.6.2 addresses inconsistent states in externally
closed connections bug in 2.6.1.I will try this tomorrow.
The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned "pillows".Someone help a brother .... self hosted Commcare is not recommended.
I need a proof of concept box to show my colleagues. thanks.Seems to be something with multiprocessing and database connections
- process -1 is successful whoever, subsequent processes fail. here is the
log -
Process Process-5:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258,
in _bootstrap
checkpoint = get_or_create_checkpoint(self.checkpoint_id)
File "corehq/ex-submodules/pillowtop/checkpoints/manager.py",
line 17, in get_or_create_checkpoint
'timestamp': datetime.utcnow()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114,
in run
self._target(*self._args, **self._kwargs)
File "corehq/ex-submodules/pillowtop/pillow/interface.py", line
82, in run
self.process_changes(since=self.get_last_checkpoint_sequence(),
forever=True)
File "corehq/ex-submodules/pillowtop/pillow/interface.py", line
66, in get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File "corehq/ex-submodules/pillowtop/checkpoints/manager.py",
line 37, in get_or_create_wrapped
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 405, in get_or_create
return self.get(**lookup), False
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get
num = len(clone)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len
self._fetch_all()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all
self._result_cache = list(self.iterator())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator
results = compiler.execute_sql()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 840, in execute_sql
cursor.execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute
return self.cursor.execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute
return self.cursor.execute(sql, params)
DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.Dear Cory, Dear all,
Thanks for the guidance. I modified logging for pillowtop in
setting.py to:PILLOWTOP_LOG_FILE = "%s/%s" % (FILEPATH, "pillowtop.log")
'handlers': {
'pillowtop': {
'level': 'INFO',
'class': 'logging.handlers.RotatingFileHandler',
'filename': PILLOWTOP_LOG_FILE,
'maxBytes': 10 * 1024 * 1024, # 10 MB
'backupCount': 20, # Backup 200 MB of logs
'formatter': 'pillowtop'ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --alland had these errors:
OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detectedLog files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.Not tried this yet. Please assist. thanks.
Euc
Hi Eucalyptus,
That error is actually coming from the error logging/retry
framework (so there is the logging error and then whatever real error is
causing the error).I would try to see what the query it is trying to run is, and
then paste that output here.You could also try reraising the exception here instead of
logging it and see if that gives you more information about the root cause.Cory
Thanks Cory,
I have attached three files for your attention. EOF Error has
the trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.I noticed that /tmp has some xml files (I have attached a
screenshot of tmp directory listing in tmp folder.jpg file) which has the
data submitted by the phone. I have attached one of the xml files. I am
happy that the server receives the data, just worrying that the data does
not show up in reports.Kindly help resolve this. I have learnt and I am continuing to
learn with your guidance. Thanks for the push.regards,
Euc
can you send a larger stack trace?
Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.I am able to complete and successfully submit a form to the
server from my test phones. Case sharing works and another user can see
cases and update/close and sync with the server.However, reports show 0 or are empty. I have run manage.py
run_ptop --all/core/core_ext and received this error:OperationalError: SSL SYSCALL error: EOF detected
Kindly provide leads/solutions, thanks.
Euc
--
You received this message because you are subscribed to the
Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the
Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to a topic in
the Google Groups "CommCare Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to a topic in
the Google Groups "CommCare Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to a topic in the
Google Groups "CommCare Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.