Empty reports/no data in reports

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the server from my
test phones. Case sharing works and another user can see cases and
update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop
–all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc

can you send a larger stack trace?

··· On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the server from
my test phones. Case sharing works and another user can see cases and
update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop
–all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Cory,

I have attached three files for your attention. EOF Error has the trace
when I run /home/commcarehq/commcare-hq/bin/python manage.py run_ptop
–pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a screenshot of tmp
directory listing in tmp folder.jpg file) which has the data submitted by
the phone. I have attached one of the xml files. I am happy that the server
receives the data, just worrying that the data does not show up in reports.

Kindly help resolve this. I have learnt and I am continuing to learn with
your guidance. Thanks for the push.

regards,

Euc

EOF Error.txt (5.31 KB)

cd98a968068144c5950a79ead7ca2701-body-xml.txt (1.53 KB)

··· On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: > > can you send a larger stack trace? > > On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D <eucal...@me.net.ng > wrote: > >> Thanks everyone. I have a running install of Commcare. >> Pillowtop/Elasticsearch and other components appear to be running normally. >> >> I am able to complete and successfully submit a form to the server from >> my test phones. Case sharing works and another user can see cases and >> update/close and sync with the server. >> >> However, reports show 0 or are empty. I have run manage.py run_ptop >> --all/core/core_ext and received this error: >> >> OperationalError: SSL SYSCALL error: EOF detected >> >> Kindly provide leads/solutions, thanks. >> >> Euc >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "CommCare Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to commcare-developers+unsubscribe@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > >

Hi Eucalyptus,

That error is actually coming from the error logging/retry framework (so
there is the logging error and then whatever real error is causing the
error).

I would try to see what the query it is trying to run is, and then paste
that output here.

You could also try reraising the exception here
https://github.com/dimagi/commcare-hq/blob/master/corehq/ex-submodules/pillowtop/pillow/interface.py#L111
instead of logging it and see if that gives you more information about the
root cause.

Cory

··· On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D wrote:

Thanks Cory,

I have attached three files for your attention. EOF Error has the trace
when I run /home/commcarehq/commcare-hq/bin/python manage.py run_ptop
–pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a screenshot of
tmp directory listing in tmp folder.jpg file) which has the data submitted
by the phone. I have attached one of the xml files. I am happy that the
server receives the data, just worrying that the data does not show up in
reports.

Kindly help resolve this. I have learnt and I am continuing to learn with
your guidance. Thanks for the push.

regards,

Euc

On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote:

can you send a larger stack trace?

On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the server from
my test phones. Case sharing works and another user can see cases and
update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop
–all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
‘backupCount’: 20, # Backup 200 MB of logs
‘formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

pillowtop-error.txt (6.7 KB)

pillowtop.log (9.34 KB)

··· On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote: > > Hi Eucalyptus, > > That error is actually coming from the error logging/retry framework (so > there is the logging error and then whatever real error is causing the > error). > > I would try to see what the query it is trying to run is, and then paste > that output here. > > You could also try reraising the exception here > > instead of logging it and see if that gives you more information about the > root cause. > > Cory > > On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D <eucal...@me.net.ng > wrote: > >> Thanks Cory, >> >> I have attached three files for your attention. EOF Error has the trace >> when I run /home/commcarehq/commcare-hq/bin/python manage.py run_ptop >> --pillow-key=core command. >> >> I noticed that /tmp has some xml files (I have attached a screenshot of >> tmp directory listing in tmp folder.jpg file) which has the data submitted >> by the phone. I have attached one of the xml files. I am happy that the >> server receives the data, just worrying that the data does not show up in >> reports. >> >> Kindly help resolve this. I have learnt and I am continuing to learn with >> your guidance. Thanks for the push. >> >> regards, >> >> Euc >> >> >> On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: >>> >>> can you send a larger stack trace? >>> >>> On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D wrote: >>> >>>> Thanks everyone. I have a running install of Commcare. >>>> Pillowtop/Elasticsearch and other components appear to be running normally. >>>> >>>> I am able to complete and successfully submit a form to the server >>>> from my test phones. Case sharing works and another user can see cases and >>>> update/close and sync with the server. >>>> >>>> However, reports show 0 or are empty. I have run manage.py run_ptop >>>> --all/core/core_ext and received this error: >>>> >>>> OperationalError: SSL SYSCALL error: EOF detected >>>> >>>> Kindly provide leads/solutions, thanks. >>>> >>>> Euc >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "CommCare Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to commcare-developers+unsubscribe@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "CommCare Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to commcare-developers+unsubscribe@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > >

Seems to be something with multiprocessing and database connections -
process -1 is successful whoever, subsequent processes fail. here is the
log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in
_bootstrap
checkpoint = get_or_create_checkpoint(self.checkpoint_id)
File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 17, in
get_or_create_checkpoint
’timestamp’: datetime.utcnow()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method
self.run()
File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 82, in run
self.process_changes(since=self.get_last_checkpoint_sequence(),
forever=True)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 66, in
get_last_checkpoint_sequence
return self.checkpoint.get_or_create_wrapped().wrapped_sequence
File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 37, in
get_or_create_wrapped
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 405, in get_or_create
return self.get(**lookup), False
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get
num = len(clone)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len
self._fetch_all()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all
self._result_cache = list(self.iterator())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator
results = compiler.execute_sql()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 840, in execute_sql
cursor.execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute
return self.cursor.execute(sql, params)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute
return self.cursor.execute(sql, params)
DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

··· On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D wrote:

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in setting.py
to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote:

Hi Eucalyptus,

That error is actually coming from the error logging/retry framework (so
there is the logging error and then whatever real error is causing the
error).

I would try to see what the query it is trying to run is, and then paste
that output here.

You could also try reraising the exception here
https://github.com/dimagi/commcare-hq/blob/master/corehq/ex-submodules/pillowtop/pillow/interface.py#L111
instead of logging it and see if that gives you more information about the
root cause.

Cory

On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks Cory,

I have attached three files for your attention. EOF Error has the trace
when I run /home/commcarehq/commcare-hq/bin/python manage.py run_ptop
–pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a screenshot of
tmp directory listing in tmp folder.jpg file) which has the data submitted
by the phone. I have attached one of the xml files. I am happy that the
server receives the data, just worrying that the data does not show up in
reports.

Kindly help resolve this. I have learnt and I am continuing to learn
with your guidance. Thanks for the push.

regards,

Euc

On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote:

can you send a larger stack trace?

On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the server
from my test phones. Case sharing works and another user can see cases and
update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop
–all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/commcare-developers/EWLcDCLU4VY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not recommended. I need
a proof of concept box to show my colleagues. thanks.

··· On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote: > > Seems to be something with multiprocessing and database connections - > process -1 is successful whoever, subsequent processes fail. here is the > log - > > Process Process-5: > Traceback (most recent call last): > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in > _bootstrap > checkpoint = get_or_create_checkpoint(self.checkpoint_id) > File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 17, > in get_or_create_checkpoint > 'timestamp': datetime.utcnow() > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py", > line 127, in manager_method > self.run() > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 82, in > run > self.process_changes(since=self.get_last_checkpoint_sequence(), > forever=True) > File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 66, in > get_last_checkpoint_sequence > return self.checkpoint.get_or_create_wrapped().wrapped_sequence > File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 37, > in get_or_create_wrapped > return getattr(self.get_queryset(), name)(*args, **kwargs) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", > line 405, in get_or_create > return self.get(**lookup), False > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", > line 328, in get > num = len(clone) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", > line 144, in __len__ > self._fetch_all() > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", > line 965, in _fetch_all > self._result_cache = list(self.iterator()) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", > line 238, in iterator > results = compiler.execute_sql() > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", > line 840, in execute_sql > cursor.execute(sql, params) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", > line 79, in execute > return super(CursorDebugWrapper, self).execute(sql, params) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", > line 64, in execute > return self.cursor.execute(sql, params) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py", > line 98, in __exit__ > six.reraise(dj_exc_type, dj_exc_value, traceback) > File > "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", > line 64, in execute > return self.cursor.execute(sql, params) > DatabaseError: server closed the connection unexpectedly > This probably means the server terminated abnormally > before or while processing the request. > > > On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D wrote: > >> >> Dear Cory, Dear all, >> >> Thanks for the guidance. I modified logging for pillowtop in setting.py >> to: >> >> PILLOWTOP_LOG_FILE = "%s/%s" % (FILEPATH, "pillowtop.log") >> >> 'handlers': { >> 'pillowtop': { >> 'level': 'INFO', >> 'class': 'logging.handlers.RotatingFileHandler', >> 'filename': PILLOWTOP_LOG_FILE, >> 'maxBytes': 10 * 1024 * 1024, # 10 MB >> 'backupCount': 20, # Backup 200 MB of logs >> 'formatter': 'pillowtop' >> >> ran: >> /home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all >> >> and had these errors: >> >> OperationalError: SSL error: decryption failed or bad record mac >> OperationalError: SSL SYSCALL error: EOF detected >> >> Log files are attached. I saw some posts about concurrency and >> Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option >> with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in >> celery. >> >> Not tried this yet. Please assist. thanks. >> >> Euc >> >> On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote: >>> >>> Hi Eucalyptus, >>> >>> That error is actually coming from the error logging/retry framework (so >>> there is the logging error and then whatever real error is causing the >>> error). >>> >>> I would try to see what the query it is trying to run is, and then paste >>> that output here. >>> >>> You could also try reraising the exception here >>> >>> instead of logging it and see if that gives you more information about the >>> root cause. >>> >>> Cory >>> >>> On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D wrote: >>> >>>> Thanks Cory, >>>> >>>> I have attached three files for your attention. EOF Error has the trace >>>> when I run /home/commcarehq/commcare-hq/bin/python manage.py run_ptop >>>> --pillow-key=core command. >>>> >>>> I noticed that /tmp has some xml files (I have attached a screenshot of >>>> tmp directory listing in tmp folder.jpg file) which has the data submitted >>>> by the phone. I have attached one of the xml files. I am happy that the >>>> server receives the data, just worrying that the data does not show up in >>>> reports. >>>> >>>> Kindly help resolve this. I have learnt and I am continuing to learn >>>> with your guidance. Thanks for the push. >>>> >>>> regards, >>>> >>>> Euc >>>> >>>> >>>> On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: >>>>> >>>>> can you send a larger stack trace? >>>>> >>>>> On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D wrote: >>>>> >>>>>> Thanks everyone. I have a running install of Commcare. >>>>>> Pillowtop/Elasticsearch and other components appear to be running normally. >>>>>> >>>>>> I am able to complete and successfully submit a form to the server >>>>>> from my test phones. Case sharing works and another user can see cases and >>>>>> update/close and sync with the server. >>>>>> >>>>>> However, reports show 0 or are empty. I have run manage.py run_ptop >>>>>> --all/core/core_ext and received this error: >>>>>> >>>>>> OperationalError: SSL SYSCALL error: EOF detected >>>>>> >>>>>> Kindly provide leads/solutions, thanks. >>>>>> >>>>>> Euc >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "CommCare Developers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to commcare-developers+unsubscribe@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "CommCare Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to commcare-developers+unsubscribe@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "CommCare Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> commcare-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > >

Update: psycopg 2.6.2 addresses inconsistent states in externally closed
connections bug in 2.6.1.

I will try this tomorrow.

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not recommended. I
need a proof of concept box to show my colleagues. thanks.

Seems to be something with multiprocessing and database connections -
process -1 is successful whoever, subsequent processes fail. here is the
log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in
_bootstrap

checkpoint = get_or_create_checkpoint(self.checkpoint_id)

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 17,
in get_or_create_checkpoint

'timestamp': datetime.utcnow()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method

self.run()

File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in run
self._target(*self._args, **self._kwargs)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 82, in
run

self.process_changes(since=self.get_last_checkpoint_sequence(),

forever=True)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 66, in
get_last_checkpoint_sequence

return self.checkpoint.get_or_create_wrapped().wrapped_sequence

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line 37,
in get_or_create_wrapped

return getattr(self.get_queryset(), name)(*args, **kwargs)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 405, in get_or_create

return self.get(**lookup), False

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get

num = len(clone)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len

self._fetch_all()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all

self._result_cache = list(self.iterator())

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator

results = compiler.execute_sql()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 840, in execute_sql

cursor.execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 79, in execute

return super(CursorDebugWrapper, self).execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute

return self.cursor.execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute

return self.cursor.execute(sql, params)

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in
setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

Hi Eucalyptus,

That error is actually coming from the error logging/retry framework
(so there is the logging error and then whatever real error is causing the
error).

I would try to see what the query it is trying to run is, and then
paste that output here.

You could also try reraising the exception here instead of logging it
and see if that gives you more information about the root cause.

Cory

Thanks Cory,

I have attached three files for your attention. EOF Error has the
trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a screenshot
of tmp directory listing in tmp folder.jpg file) which has the data
submitted by the phone. I have attached one of the xml files. I am happy
that the server receives the data, just worrying that the data does not
show up in reports.

Kindly help resolve this. I have learnt and I am continuing to learn
with your guidance. Thanks for the push.

regards,

Euc

can you send a larger stack trace?

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the server
update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop
–all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

··· On Nov 10, 2016 4:28 PM, "Eucalyptus D" wrote: > On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote: >> On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D wrote: >>> On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote: >>>> On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D wrote: >>>>> On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: >>>>>> On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D wrote: from my test phones. Case sharing works and another user can see cases and > For more options, visit https://groups.google.com/d/optout.

Hey,

It’s pretty weird to me that your DB connection is working in other places
but not in pillowtop. I would also have a look in your postgres logs if the
psycopg thing doesn’t work.

Cory

··· On Thu, Nov 10, 2016 at 11:15 PM, Eucalyptus Deglupta wrote:

Update: psycopg 2.6.2 addresses inconsistent states in externally closed
connections bug in 2.6.1.

I will try this tomorrow.

On Nov 10, 2016 4:28 PM, “Eucalyptus D” eucalyptus@me.net.ng wrote:

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not recommended. I
need a proof of concept box to show my colleagues. thanks.

On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote:

Seems to be something with multiprocessing and database connections -
process -1 is successful whoever, subsequent processes fail. here is the
log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in
_bootstrap

checkpoint = get_or_create_checkpoint(self.checkpoint_id)

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line
17, in get_or_create_checkpoint

'timestamp': datetime.utcnow()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/manager.py”, line 127, in manager_method

self.run()

File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in
run

self._target(*self._args, **self._kwargs)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 82,
in run

self.process_changes(since=self.get_last_checkpoint_sequence(),

forever=True)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 66,
in get_last_checkpoint_sequence

return self.checkpoint.get_or_create_wrapped().wrapped_sequence

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line
37, in get_or_create_wrapped

return getattr(self.get_queryset(), name)(*args, **kwargs)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 405, in get_or_create

return self.get(**lookup), False

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 328, in get

num = len(clone)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 144, in len

self._fetch_all()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 965, in _fetch_all

self._result_cache = list(self.iterator())

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 238, in iterator

results = compiler.execute_sql()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py”, line 840, in execute_sql

cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 79, in execute

return super(CursorDebugWrapper, self).execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py”,
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D eucalyptus@me.net.ng wrote:

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in
setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote:

Hi Eucalyptus,

That error is actually coming from the error logging/retry framework
(so there is the logging error and then whatever real error is causing the
error).

I would try to see what the query it is trying to run is, and then
paste that output here.

You could also try reraising the exception here instead of logging it
and see if that gives you more information about the root cause.

Cory

On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks Cory,

I have attached three files for your attention. EOF Error has the
trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a screenshot
of tmp directory listing in tmp folder.jpg file) which has the data
submitted by the phone. I have attached one of the xml files. I am happy
that the server receives the data, just worrying that the data does not
show up in reports.

Kindly help resolve this. I have learnt and I am continuing to learn
with your guidance. Thanks for the push.

regards,

Euc

On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote:

can you send a larger stack trace?

On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the
server from my test phones. Case sharing works and another user can see
cases and update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py
run_ptop --all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit https://groups.google.com/d/
topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/commcare-developers/EWLcDCLU4VY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Cory for the support and encouragement. Seems to be something around
multiprocessing and database:

ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify
Exception: processor error in pillow CaseToElasticsearchPillow connection
already closed
Traceback (most recent call last):
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 94, in
process_changes
self.process_with_error_handling(change)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 111, in
process_with_error_handling
handle_pillow_error(self, change, ex)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 179, in
handle_pillow_error
error = PillowError.get_or_create(change, pillow)
File “corehq/ex-submodules/pillow_retry/models.py”, line 88, in
get_or_create
error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get
num = len(clone)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len
self._fetch_all()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all
self._result_cache = list(self.iterator())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator
results = compiler.execute_sql()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 838, in execute_sql
cursor = self.connection.cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 162, in cursor
cursor = self.make_debug_cursor(self._cursor())
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor
return self.create_cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor
return self.create_cursor()
File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py",
line 212, in create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed

I will keep looking.

Euc

··· On Friday, November 11, 2016 at 8:27:56 AM UTC+1, Cory Zue wrote: > > Hey, > > It's pretty weird to me that your DB connection is working in other places > but not in pillowtop. I would also have a look in your postgres logs if the > psycopg thing doesn't work. > > Cory > > On Thu, Nov 10, 2016 at 11:15 PM, Eucalyptus Deglupta <eucal...@me.net.ng > wrote: > >> Update: psycopg 2.6.2 addresses inconsistent states in externally closed >> connections bug in 2.6.1. >> >> I will try this tomorrow. >> >> On Nov 10, 2016 4:28 PM, "Eucalyptus D" <eucal...@me.net.ng > wrote: >> > >> > The issue is somewhere within run_pillowtop.py having issues with >> multiprocessing and database connection. Seems a single database connection >> is being reused by spawned "pillows". >> > >> > Someone help a brother .... self hosted Commcare is not recommended. I >> need a proof of concept box to show my colleagues. thanks. >> > >> > >> > On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote: >> >> >> >> Seems to be something with multiprocessing and database connections - >> process -1 is successful whoever, subsequent processes fail. here is the >> log - >> >> >> >> Process Process-5: >> >> Traceback (most recent call last): >> >> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in >> _bootstrap >> >> checkpoint = get_or_create_checkpoint(self.checkpoint_id) >> >> File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line >> 17, in get_or_create_checkpoint >> >> 'timestamp': datetime.utcnow() >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py", >> line 127, in manager_method >> >> self.run() >> >> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in >> run >> >> self._target(*self._args, **self._kwargs) >> >> File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 82, >> in run >> >> self.process_changes(since=self.get_last_checkpoint_sequence(), >> forever=True) >> >> File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 66, >> in get_last_checkpoint_sequence >> >> return self.checkpoint.get_or_create_wrapped().wrapped_sequence >> >> File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line >> 37, in get_or_create_wrapped >> >> return getattr(self.get_queryset(), name)(*args, **kwargs) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", >> line 405, in get_or_create >> >> return self.get(**lookup), False >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", >> line 328, in get >> >> num = len(clone) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", >> line 144, in __len__ >> >> self._fetch_all() >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", >> line 965, in _fetch_all >> >> self._result_cache = list(self.iterator()) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", >> line 238, in iterator >> >> results = compiler.execute_sql() >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", >> line 840, in execute_sql >> >> cursor.execute(sql, params) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", >> line 79, in execute >> >> return super(CursorDebugWrapper, self).execute(sql, params) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", >> line 64, in execute >> >> return self.cursor.execute(sql, params) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py", >> line 98, in __exit__ >> >> six.reraise(dj_exc_type, dj_exc_value, traceback) >> >> File >> "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", >> line 64, in execute >> >> return self.cursor.execute(sql, params) >> >> DatabaseError: server closed the connection unexpectedly >> >> This probably means the server terminated abnormally >> >> before or while processing the request. >> >> >> >> >> >> On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D <eucal...@me.net.ng > wrote: >> >>> >> >>> >> >>> Dear Cory, Dear all, >> >>> >> >>> Thanks for the guidance. I modified logging for pillowtop in >> setting.py to: >> >>> >> >>> PILLOWTOP_LOG_FILE = "%s/%s" % (FILEPATH, "pillowtop.log") >> >>> >> >>> 'handlers': { >> >>> 'pillowtop': { >> >>> 'level': 'INFO', >> >>> 'class': 'logging.handlers.RotatingFileHandler', >> >>> 'filename': PILLOWTOP_LOG_FILE, >> >>> 'maxBytes': 10 * 1024 * 1024, # 10 MB >> >>> 'backupCount': 20, # Backup 200 MB of logs >> >>> 'formatter': 'pillowtop' >> >>> >> >>> ran: >> >>> /home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all >> >>> >> >>> and had these errors: >> >>> >> >>> OperationalError: SSL error: decryption failed or bad record mac >> >>> OperationalError: SSL SYSCALL error: EOF detected >> >>> >> >>> Log files are attached. I saw some posts about concurrency and >> Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option >> with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in >> celery. >> >>> >> >>> Not tried this yet. Please assist. thanks. >> >>> >> >>> Euc >> >>> >> >>> On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote: >> >>>> >> >>>> Hi Eucalyptus, >> >>>> >> >>>> That error is actually coming from the error logging/retry framework >> (so there is the logging error and then whatever real error is causing the >> error). >> >>>> >> >>>> I would try to see what the query it is trying to run is, and then >> paste that output here. >> >>>> >> >>>> You could also try reraising the exception here instead of logging >> it and see if that gives you more information about the root cause. >> >>>> >> >>>> Cory >> >>>> >> >>>> On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D wrote: >> >>>>> >> >>>>> Thanks Cory, >> >>>>> >> >>>>> I have attached three files for your attention. EOF Error has the >> trace when I run /home/commcarehq/commcare-hq/bin/python manage.py >> run_ptop --pillow-key=core command. >> >>>>> >> >>>>> I noticed that /tmp has some xml files (I have attached a >> screenshot of tmp directory listing in tmp folder.jpg file) which has the >> data submitted by the phone. I have attached one of the xml files. I am >> happy that the server receives the data, just worrying that the data does >> not show up in reports. >> >>>>> >> >>>>> Kindly help resolve this. I have learnt and I am continuing to >> learn with your guidance. Thanks for the push. >> >>>>> >> >>>>> regards, >> >>>>> >> >>>>> Euc >> >>>>> >> >>>>> >> >>>>> On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: >> >>>>>> >> >>>>>> can you send a larger stack trace? >> >>>>>> >> >>>>>> On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D wrote: >> >>>>>>> >> >>>>>>> Thanks everyone. I have a running install of Commcare. >> Pillowtop/Elasticsearch and other components appear to be running normally. >> >>>>>>> >> >>>>>>> I am able to complete and successfully submit a form to the >> server from my test phones. Case sharing works and another user can see >> cases and update/close and sync with the server. >> >>>>>>> >> >>>>>>> However, reports show 0 or are empty. I have run manage.py >> run_ptop --all/core/core_ext and received this error: >> >>>>>>> >> >>>>>>> OperationalError: SSL SYSCALL error: EOF detected >> >>>>>>> >> >>>>>>> Kindly provide leads/solutions, thanks. >> >>>>>>> >> >>>>>>> Euc >> >>>>>>> >> >>>>>>> -- >> >>>>>>> >> >>>>>>> --- >> >>>>>>> You received this message because you are subscribed to the >> Google Groups "CommCare Developers" group. >> >>>>>>> To unsubscribe from this group and stop receiving emails from it, >> send an email to commcare-developers+unsubscribe@googlegroups.com >> . >> >>>>>>> For more options, visit https://groups.google.com/d/optout. >> >>>>>> >> >>>>>> >> >>>>> -- >> >>>>> >> >>>>> --- >> >>>>> You received this message because you are subscribed to the Google >> Groups "CommCare Developers" group. >> >>>>> To unsubscribe from this group and stop receiving emails from it, >> send an email to commcare-developers+unsubscribe@googlegroups.com >> . >> >>>>> For more options, visit https://groups.google.com/d/optout. >> >>>> >> >>>> >> >>> -- >> >>> >> >>> --- >> >>> You received this message because you are subscribed to a topic in >> the Google Groups "CommCare Developers" group. >> >>> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe >> . >> >>> To unsubscribe from this group and all its topics, send an email to >> commcare-developers+unsubscribe@googlegroups.com . >> >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> > -- >> > >> > --- >> > You received this message because you are subscribed to a topic in the >> Google Groups "CommCare Developers" group. >> > To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe >> . >> > To unsubscribe from this group and all its topics, send an email to >> commcare-developers+unsubscribe@googlegroups.com . >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "CommCare Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to commcare-developers+unsubscribe@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > >

“connection already closed” might also imply that something went wrong
earlier in the process leaving the connection in a broken state. so another
avenue to pursue would be looking in the logs for different traces. though
maybe the first error you sent is the root problem.

··· On Fri, Nov 11, 2016 at 9:59 AM, Eucalyptus D wrote:

Thanks Cory for the support and encouragement. Seems to be something
around multiprocessing and database:

ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify
Exception: processor error in pillow CaseToElasticsearchPillow connection
already closed
Traceback (most recent call last):
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 94, in
process_changes
self.process_with_error_handling(change)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 111, in
process_with_error_handling
handle_pillow_error(self, change, ex)
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 179, in
handle_pillow_error
error = PillowError.get_or_create(change, pillow)
File “corehq/ex-submodules/pillow_retry/models.py”, line 88, in
get_or_create
error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/manager.py”, line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 328, in get
num = len(clone)
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 144, in len
self._fetch_all()
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 965, in _fetch_all
self._result_cache = list(self.iterator())
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 238, in iterator
results = compiler.execute_sql()
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py”, line 838, in execute_sql
cursor = self.connection.cursor()
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 162, in cursor
cursor = self.make_debug_cursor(self._cursor())
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 137, in _cursor
return self.create_cursor()
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py”,
line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 137, in _cursor
return self.create_cursor()
File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/postgresql_psycopg2/base.py”, line 212, in
create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed

I will keep looking.

Euc

On Friday, November 11, 2016 at 8:27:56 AM UTC+1, Cory Zue wrote:

Hey,

It’s pretty weird to me that your DB connection is working in other
places but not in pillowtop. I would also have a look in your postgres logs
if the psycopg thing doesn’t work.

Cory

On Thu, Nov 10, 2016 at 11:15 PM, Eucalyptus Deglupta <eucal...@me.net.ng wrote:

Update: psycopg 2.6.2 addresses inconsistent states in externally closed
connections bug in 2.6.1.

I will try this tomorrow.

On Nov 10, 2016 4:28 PM, “Eucalyptus D” eucal...@me.net.ng wrote:

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not recommended. I
need a proof of concept box to show my colleagues. thanks.

On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote:

Seems to be something with multiprocessing and database connections -
process -1 is successful whoever, subsequent processes fail. here is the
log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258, in
_bootstrap

checkpoint = get_or_create_checkpoint(self.checkpoint_id)

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line
17, in get_or_create_checkpoint

'timestamp': datetime.utcnow()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/manager.py”, line 127, in manager_method

self.run()

File “/usr/lib/python2.7/multiprocessing/process.py”, line 114, in
run

self._target(*self._args, **self._kwargs)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
82, in run

self.process_changes(since=self.get_last_checkpoint_sequence(),

forever=True)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
66, in get_last_checkpoint_sequence

return self.checkpoint.get_or_create_wrapped().wrapped_sequence

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”, line
37, in get_or_create_wrapped

return getattr(self.get_queryset(), name)(*args, **kwargs)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/query.py”, line 405, in get_or_create

return self.get(**lookup), False

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/query.py”, line 328, in get

num = len(clone)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/query.py”, line 144, in len

self._fetch_all()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/query.py”, line 965, in _fetch_all

self._result_cache = list(self.iterator())

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/query.py”, line 238, in iterator

results = compiler.execute_sql()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/models/sql/compiler.py”, line 840, in execute_sql

cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/backends/utils.py”, line 79, in execute

return super(CursorDebugWrapper, self).execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py”,
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packa
ges/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D eucal...@me.net.ng wrote:

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in
setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote:

Hi Eucalyptus,

That error is actually coming from the error logging/retry
framework (so there is the logging error and then whatever real error is
causing the error).

I would try to see what the query it is trying to run is, and then
paste that output here.

You could also try reraising the exception here instead of logging
it and see if that gives you more information about the root cause.

Cory

On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks Cory,

I have attached three files for your attention. EOF Error has the
trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a
screenshot of tmp directory listing in tmp folder.jpg file) which has the
data submitted by the phone. I have attached one of the xml files. I am
happy that the server receives the data, just worrying that the data does
not show up in reports.

Kindly help resolve this. I have learnt and I am continuing to
learn with your guidance. Thanks for the push.

regards,

Euc

On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote:

can you send a larger stack trace?

On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the
server from my test phones. Case sharing works and another user can see
cases and update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py
run_ptop --all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the
Google Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in
the Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/commcare-developers/EWLcDCLU4VY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/commcare-developers/EWLcDCLU4VY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dear Cory, please which first error are you reffering to? Thnks again.

“connection already closed” might also imply that something went wrong
earlier in the process leaving the connection in a broken state. so another
avenue to pursue would be looking in the logs for different traces. though
maybe the first error you sent is the root problem.

Thanks Cory for the support and encouragement. Seems to be something
around multiprocessing and database:

ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify
Exception: processor error in pillow CaseToElasticsearchPillow connection
already closed

Traceback (most recent call last):
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 94, in
process_changes

self.process_with_error_handling(change)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 111,
in process_with_error_handling

handle_pillow_error(self, change, ex)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 179,
in handle_pillow_error

error = PillowError.get_or_create(change, pillow)

File “corehq/ex-submodules/pillow_retry/models.py”, line 88, in
get_or_create

error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method

return getattr(self.get_queryset(), name)(*args, **kwargs)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get

num = len(clone)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len

self._fetch_all()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all

self._result_cache = list(self.iterator())

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator

results = compiler.execute_sql()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 838, in execute_sql

cursor = self.connection.cursor()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 162, in cursor

cursor = self.make_debug_cursor(self._cursor())

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor

return self.create_cursor()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py",
line 137, in _cursor

return self.create_cursor()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py",
line 212, in create_cursor

cursor = self.connection.cursor()

InterfaceError: connection already closed

I will keep looking.

Euc

Hey,

It’s pretty weird to me that your DB connection is working in other
places but not in pillowtop. I would also have a look in your postgres logs
if the psycopg thing doesn’t work.

Cory

Update: psycopg 2.6.2 addresses inconsistent states in externally
closed connections bug in 2.6.1.

I will try this tomorrow.

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not recommended.
I need a proof of concept box to show my colleagues. thanks.

Seems to be something with multiprocessing and database connections

  • process -1 is successful whoever, subsequent processes fail. here is the
    log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258,
in _bootstrap

checkpoint = get_or_create_checkpoint(self.checkpoint_id)

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”,
line 17, in get_or_create_checkpoint

'timestamp': datetime.utcnow()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 127, in manager_method

self.run()

File “/usr/lib/python2.7/multiprocessing/process.py”, line 114,
in run

self._target(*self._args, **self._kwargs)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
82, in run

self.process_changes(since=self.get_last_checkpoint_sequence(),

forever=True)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
66, in get_last_checkpoint_sequence

return self.checkpoint.get_or_create_wrapped().wrapped_sequence

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”,
line 37, in get_or_create_wrapped

return getattr(self.get_queryset(), name)(*args, **kwargs)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 405, in get_or_create

return self.get(**lookup), False

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 328, in get

num = len(clone)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 144, in len

self._fetch_all()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 965, in _fetch_all

self._result_cache = list(self.iterator())

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py",
line 238, in iterator

results = compiler.execute_sql()

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 840, in execute_sql

cursor.execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 79, in execute

return super(CursorDebugWrapper, self).execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute

return self.cursor.execute(sql, params)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py",
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File
"/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute

return self.cursor.execute(sql, params)

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in
setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

Hi Eucalyptus,

That error is actually coming from the error logging/retry
framework (so there is the logging error and then whatever real error is
causing the error).

I would try to see what the query it is trying to run is, and
then paste that output here.

You could also try reraising the exception here instead of
logging it and see if that gives you more information about the root cause.

Cory

Thanks Cory,

I have attached three files for your attention. EOF Error has
the trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a
screenshot of tmp directory listing in tmp folder.jpg file) which has the
data submitted by the phone. I have attached one of the xml files. I am
happy that the server receives the data, just worrying that the data does
not show up in reports.

Kindly help resolve this. I have learnt and I am continuing to
learn with your guidance. Thanks for the push.

regards,

Euc

can you send a larger stack trace?

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the
server from my test phones. Case sharing works and another user can see
cases and update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py
run_ptop --all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the
Google Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the
Google Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in
the Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.

To unsubscribe from this group and all its topics, send an email
to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in
the Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

··· On Nov 11, 2016 9:03 AM, "Cory Zue" wrote: > On Fri, Nov 11, 2016 at 9:59 AM, Eucalyptus D wrote: >> On Friday, November 11, 2016 at 8:27:56 AM UTC+1, Cory Zue wrote: >>> On Thu, Nov 10, 2016 at 11:15 PM, Eucalyptus Deglupta < eucal...@me.net.ng> wrote: >>>> On Nov 10, 2016 4:28 PM, "Eucalyptus D" wrote: >>>> > On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote: >>>> >> On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D wrote: >>>> >>> On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote: >>>> >>>> On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D wrote: >>>> >>>>> On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote: >>>> >>>>>> On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D < eucal...@me.net.ng> wrote: > For more options, visit https://groups.google.com/d/optout.

i just meant the other trace you sent on get_or_create_checkpoint

··· On Fri, Nov 11, 2016 at 10:41 AM, Eucalyptus Deglupta wrote:

Dear Cory, please which first error are you reffering to? Thnks again.

On Nov 11, 2016 9:03 AM, “Cory Zue” czue@dimagi.com wrote:

“connection already closed” might also imply that something went wrong
earlier in the process leaving the connection in a broken state. so another
avenue to pursue would be looking in the logs for different traces. though
maybe the first error you sent is the root problem.

On Fri, Nov 11, 2016 at 9:59 AM, Eucalyptus D eucalyptus@me.net.ng wrote:

Thanks Cory for the support and encouragement. Seems to be something
around multiprocessing and database:

ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify
Exception: processor error in pillow CaseToElasticsearchPillow connection
already closed

Traceback (most recent call last):
File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 94,
in process_changes

self.process_with_error_handling(change)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 111,
in process_with_error_handling

handle_pillow_error(self, change, ex)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line 179,
in handle_pillow_error

error = PillowError.get_or_create(change, pillow)

File “corehq/ex-submodules/pillow_retry/models.py”, line 88, in
get_or_create

error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/manager.py”, line 127, in manager_method

return getattr(self.get_queryset(), name)(*args, **kwargs)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 328, in get

num = len(clone)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 144, in len

self._fetch_all()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 965, in _fetch_all

self._result_cache = list(self.iterator())

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 238, in iterator

results = compiler.execute_sql()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py”, line 838, in execute_sql

cursor = self.connection.cursor()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 162, in cursor

cursor = self.make_debug_cursor(self._cursor())

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 137, in _cursor

return self.create_cursor()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py”,
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/base/base.py”, line 137, in _cursor

return self.create_cursor()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/postgresql_psycopg2/base.py”, line 212, in
create_cursor

cursor = self.connection.cursor()

InterfaceError: connection already closed

I will keep looking.

Euc

On Friday, November 11, 2016 at 8:27:56 AM UTC+1, Cory Zue wrote:

Hey,

It’s pretty weird to me that your DB connection is working in other
places but not in pillowtop. I would also have a look in your postgres logs
if the psycopg thing doesn’t work.

Cory

On Thu, Nov 10, 2016 at 11:15 PM, Eucalyptus Deglupta < eucal...@me.net.ng> wrote:

Update: psycopg 2.6.2 addresses inconsistent states in externally
closed connections bug in 2.6.1.

I will try this tomorrow.

On Nov 10, 2016 4:28 PM, “Eucalyptus D” eucal...@me.net.ng wrote:

The issue is somewhere within run_pillowtop.py having issues with
multiprocessing and database connection. Seems a single database connection
is being reused by spawned “pillows”.

Someone help a brother … self hosted Commcare is not
recommended. I need a proof of concept box to show my colleagues. thanks.

On Thursday, November 10, 2016 at 4:21:08 PM UTC+1, Eucalyptus D wrote:

Seems to be something with multiprocessing and database
connections - process -1 is successful whoever, subsequent processes fail.
here is the log -

Process Process-5:
Traceback (most recent call last):
File “/usr/lib/python2.7/multiprocessing/process.py”, line 258,
in _bootstrap

checkpoint = get_or_create_checkpoint(self.checkpoint_id)

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”,
line 17, in get_or_create_checkpoint

'timestamp': datetime.utcnow()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/manager.py”, line 127, in manager_method

self.run()

File “/usr/lib/python2.7/multiprocessing/process.py”, line 114,
in run

self._target(*self._args, **self._kwargs)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
82, in run

self.process_changes(since=self.get_last_checkpoint_sequence(),

forever=True)

File “corehq/ex-submodules/pillowtop/pillow/interface.py”, line
66, in get_last_checkpoint_sequence

return self.checkpoint.get_or_create_

wrapped().wrapped_sequence

File “corehq/ex-submodules/pillowtop/checkpoints/manager.py”,
line 37, in get_or_create_wrapped

return getattr(self.get_queryset(), name)(*args, **kwargs)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 405, in get_or_create

return self.get(**lookup), False

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 328, in get

num = len(clone)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 144, in len

self._fetch_all()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 965, in _fetch_all

self._result_cache = list(self.iterator())

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/query.py”, line 238, in iterator

results = compiler.execute_sql()

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py”, line 840, in execute_sql

cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 79, in execute

return super(CursorDebugWrapper, self).execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py”,
line 98, in exit

six.reraise(dj_exc_type, dj_exc_value, traceback)

File “/home/commcarehq/commcare-hq/local/lib/python2.7/site-
packages/django/db/backends/utils.py”, line 64, in execute

return self.cursor.execute(sql, params)

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

On Thu, Nov 10, 2016 at 2:47 PM, Eucalyptus D eucal...@me.net.ng wrote:

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in
setting.py to:

PILLOWTOP_LOG_FILE = “%s/%s” % (FILEPATH, “pillowtop.log”)

‘handlers’: {
‘pillowtop’: {
‘level’: ‘INFO’,
‘class’: ‘logging.handlers.RotatingFileHandler’,
‘filename’: PILLOWTOP_LOG_FILE,
‘maxBytes’: 10 * 1024 * 1024, # 10 MB
’backupCount’: 20, # Backup 200 MB of logs
’formatter’: ‘pillowtop’

ran:
/home/commcarehq/commcare-hq/bin/python manage.py run_ptop
–all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac
OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and
Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option
with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in
celery.

Not tried this yet. Please assist. thanks.

Euc

On Wednesday, November 9, 2016 at 4:19:05 PM UTC+1, Cory Zue wrote:

Hi Eucalyptus,

That error is actually coming from the error logging/retry
framework (so there is the logging error and then whatever real error is
causing the error).

I would try to see what the query it is trying to run is, and
then paste that output here.

You could also try reraising the exception here instead of
logging it and see if that gives you more information about the root cause.

Cory

On Wed, Nov 9, 2016 at 1:17 PM, Eucalyptus D eucal...@me.net.ng wrote:

Thanks Cory,

I have attached three files for your attention. EOF Error has
the trace when I run /home/commcarehq/commcare-hq/bin/python manage.py
run_ptop --pillow-key=core command.

I noticed that /tmp has some xml files (I have attached a
screenshot of tmp directory listing in tmp folder.jpg file) which has the
data submitted by the phone. I have attached one of the xml files. I am
happy that the server receives the data, just worrying that the data does
not show up in reports.

Kindly help resolve this. I have learnt and I am continuing to
learn with your guidance. Thanks for the push.

regards,

Euc

On Wednesday, November 9, 2016 at 8:51:28 AM UTC+1, Cory Zue wrote:

can you send a larger stack trace?

On Wed, Nov 9, 2016 at 12:08 AM, Eucalyptus D < eucal...@me.net.ng> wrote:

Thanks everyone. I have a running install of Commcare.
Pillowtop/Elasticsearch and other components appear to be running normally.

I am able to complete and successfully submit a form to the
server from my test phones. Case sharing works and another user can see
cases and update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py
run_ptop --all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc


You received this message because you are subscribed to the
Google Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the
Google Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from
it, send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic
in the Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-developers/
EWLcDCLU4VY/unsubscribe.

To unsubscribe from this group and all its topics, send an email
to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in
the Google Groups “CommCare Developers” group.

To unsubscribe from this topic, visit https://groups.google.com/d/
topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “CommCare Developers” group.

To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to a topic in the
Google Groups “CommCare Developers” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/commcare-developers/EWLcDCLU4VY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.