Docker : pillowtop errors

Developers
1

Hi everyone,

Following up on the docker setup, I'm now trying to run celery / rabbitmq properly
to allow the import / export of data. However even before getting there, pillowtop
worries me a little bit.

I'm running these services in their own containers :

  • postgres
  • couchdb
  • redis
  • elasticsearch
  • rabbitmq

Following README.md, I then initialize the DB :

  • create postgres DB
  • create couchdb admin and DB
  • manage.py syncdb
  • manage.py migrate
  • manage.py collectstatic
  • manage.py bootstrap example example@example.com example

So far so good : running an HQ instance as that point is a success, at least for
creation application and managing users.

Still following the README.md, I then run ptop on its own container connected to
the databases :
$ docker run --rm --name run_ptop --link postgres:postgres --link
couchdb:couchdb --link redis:redis --link elasticsearch:elasticsearch --link
rabbitmq:rabbitmq -v pwd:/usr/src/commcare-hq charlesfleche/commcare-hq
python manage.py run_ptop --all

A few weird things happen:

ResourceNotFound: missing

run_ptop.txt (23.9 KB)

··· =========================

pillowtop seems to fail to get CouchDB documents, but these documents are not
always the same between runs :

First run (accessing these documents raise ResourceNotFound: missing in
"couchdbkit/resource.py", line 129, in request)
pillowtop_fluff.TDHInfantClassificationFluffPillow.870ceaecb4ba
pillowtop_corehq.pillows.reportcase.ReportCasePillow.8c10a7564b6af5052f8b8669
3bf6ac07.870ceaecb4ba
pillowtop_fluff.TDHChildClassificationFluffPillow.870ceaecb4ba
pillowtop_fluff.TauxDeRuptureFluffPillow.870ceaecb4ba
pillowtop_fluff.McctMonthlyAggregateFormFluffPillow.870ceaecb4ba
pillowtop_fluff.MalariaConsortiumFluffPillow.870ceaecb4ba
pillowtop_fluff.IntraHealthFluffPillow.870ceaecb4ba
pillowtop_fluff.TDHNewbornTreatmentFluffPillow.870ceaecb4ba
pillowtop_corehq.pillows.domain.DomainPillow.hqdomains_a3dbc2012ed3ba384bed
595735ef5100.870ceaecb4ba
pillowtop_corehq.apps.userreports.pillow.CustomDataSourcePillow.870ceaecb4ba
pillowtop_fluff.OPMHierarchyFluffPillow.870ceaecb4ba
pillowtop_fluff.CouvertureFluffPillow.870ceaecb4ba
pillowtop_corehq.pillows.application.AppPillow.fe975a5128b655ba175ac81b035498
d0.870ceaecb4ba
pillowtop_corehq.pillows.user.GroupToUserPillow.870ceaecb4ba
pillowtop_fluff.OpmHealthStatusAllInfoFluffPillow.870ceaecb4ba
pillowtop_corehq.pillows.user.UnknownUsersPillow.870ceaecb4ba
pillowtop_fluff.TDHNewbornClassificationFluffPillow.870ceaecb4ba
pillowtop_corehq.pillows.sofabed.CaseDataPillow.870ceaecb4ba
pillowtop_fluff.TDHEnrollChildFluffPillow.870ceaecb4ba

Second run (accessing these documents raise ResourceNotFound: missing in
"couchdbkit/resource.py", line 129, in request)
pillowtop_fluff.CareSAFluffPillow.59a34724a421
pillowtop_corehq.pillows.sofabed.FormDataPillow.59a34724a421
pillowtop_fluff.UnicefMalawiFluffPillow.59a34724a421
pillowtop_corehq.pillows.sms.SMSPillow.708c77f8e5fe00286fa5791e9fa7d45f.59a3
4724a421
pillowtop_fluff.OPMHierarchyFluffPillow.59a34724a421
pillowtop_corehq.pillows.reportxform.ReportXFormPillow.report_xforms_e26fad18
b95711e4a7bc3c970e7be43b.59a34724a421
pillowtop_corehq.pillows.user.GroupToUserPillow.59a34724a421
pillowtop_fluff.TDHChildClassificationFluffPillow.59a34724a421
pillowtop_fluff.AllHmisCaseFluffPillow.59a34724a421
pillowtop_fluff.TDHInfantClassificationFluffPillow.59a34724a421
pillowtop_corehq.pillows.application.AppPillow.fe975a5128b655ba175ac81b035498
d0.59a34724a421
pillowtop_fluff.VhndAvailabilityFluffPillow.59a34724a421
pillowtop_fluff.FarmerRecordFluffPillow.59a34724a421
pillowtop_fluff.TDHEnrollChildFluffPillow.59a34724a421
pillowtop_fluff.RecouvrementFluffPillow.59a34724a421
pillowtop_corehq.pillows.log.PhoneLogPillow.59a34724a421

ERROR 404

Consistently, a process fails and raise an error 404 accessing
http://commcarehq:commcarehq@couchdb:5984/commcarehq

Process Process-55:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "submodules/pillowtop-src/pillowtop/listener.py", line 172, in run
self.new_changes()
File "submodules/pillowtop-src/pillowtop/listener.py", line 152, in new_changes
since=self.since,
File "submodules/pillowtop-src/pillowtop/listener.py", line 235, in since
checkpoint = self.get_checkpoint()
File "/usr/src/commcare-hq/corehq/pillows/cacheinvalidate.py", line 37, in
get_checkpoint
current_db_seq = self.couch_db.info()['update_seq']
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 280, in info
return self.res.get().json_body
File "/usr/local/lib/python2.7/dist-packages/restkit/resource.py", line 114, in get
params_dict=params_dict, **params)
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/resource.py", line 129, in
request
response=e.response)
ResourceNotFound: {'msg': '', 'status_int': 404, 'response':
<couchdbkit.resource.CouchDBResponse object at 0x77736d0>}

This is weird as running 'curl
http://commcarehq:commcarehq@couchdb:5984/commcarehq' in the same
container succeed.

FILE ALREADY EXISTS

Another consistently raised Exception is 'PreconditionFailed: The database could
not be created, the file already exists.'

Process Process-51:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()

2

Hey Charles,

It's hard to know exactly what's going on without being able to replicate
your environment, but I will give the following guidelines:

  1. It's definitely possible/likely that pillowtop fails in certain
    circumstances, one of which could definitely be no data in an elastic
    index. Another could be that the index itself isn't initialized (see 3).
  2. Pillowtop mostly powers our custom reports. So you can probably getting
    away with just running the "core" subset of pillows. ./manage.py run_ptop
    --pillow_key core (or similar)
  3. I would recommend running the following bootstrap commands (which we run
    on every deploy) to see if that fixes it:

./manage.py preindex_everything
./manage.py ptop_es_manage --flip_all_aliases

  1. The fabfile, and particularly the awesome_deploy command, is
    representative of how we deploy our production infrastructure, so looking
    there for other clues might be helpful.

cheers,
Cory

··· On Wed, Feb 25, 2015 at 1:43 PM, Charles Flèche wrote:

Hi everyone,

Following up on the docker setup, I'm now trying to run celery / rabbitmq
properly to allow the import / export of data. However even before getting
there, pillowtop worries me a little bit.

I'm running these services in their own containers :

  • postgres

  • couchdb

  • redis

  • elasticsearch

  • rabbitmq

Following README.md, I then initialize the DB :

  • create postgres DB

  • create couchdb admin and DB

  • manage.py syncdb

  • manage.py migrate

  • manage.py collectstatic

  • manage.py bootstrap example example@example.com example

So far so good : running an HQ instance as that point is a success, at
least for creation application and managing users.

Still following the README.md, I then run ptop on its own container
connected to the databases :

$ docker run --rm --name run_ptop --link postgres:postgres --link
couchdb:couchdb --link redis:redis --link elasticsearch:elasticsearch
--link rabbitmq:rabbitmq -v pwd:/usr/src/commcare-hq
charlesfleche/commcare-hq python manage.py run_ptop --all

A few weird things happen:

ResourceNotFound: missing

=========================

pillowtop seems to fail to get CouchDB documents, but these documents are
not always the same between runs :

First run (accessing these documents raise ResourceNotFound: missing in
"couchdbkit/resource.py", line 129, in request)

pillowtop_fluff.TDHInfantClassificationFluffPillow.870ceaecb4ba

pillowtop_corehq.pillows.reportcase.ReportCasePillow.8c10a7564b6af5052f8b86693bf6ac07.870ceaecb4ba

pillowtop_fluff.TDHChildClassificationFluffPillow.870ceaecb4ba

pillowtop_fluff.TauxDeRuptureFluffPillow.870ceaecb4ba

pillowtop_fluff.McctMonthlyAggregateFormFluffPillow.870ceaecb4ba

pillowtop_fluff.MalariaConsortiumFluffPillow.870ceaecb4ba

pillowtop_fluff.IntraHealthFluffPillow.870ceaecb4ba

pillowtop_fluff.TDHNewbornTreatmentFluffPillow.870ceaecb4ba

pillowtop_corehq.pillows.domain.DomainPillow.hqdomains_a3dbc2012ed3ba384bed595735ef5100.870ceaecb4ba

pillowtop_corehq.apps.userreports.pillow.CustomDataSourcePillow.870ceaecb4ba

pillowtop_fluff.OPMHierarchyFluffPillow.870ceaecb4ba

pillowtop_fluff.CouvertureFluffPillow.870ceaecb4ba

pillowtop_corehq.pillows.application.AppPillow.fe975a5128b655ba175ac81b035498d0.870ceaecb4ba

pillowtop_corehq.pillows.user.GroupToUserPillow.870ceaecb4ba

pillowtop_fluff.OpmHealthStatusAllInfoFluffPillow.870ceaecb4ba

pillowtop_corehq.pillows.user.UnknownUsersPillow.870ceaecb4ba

pillowtop_fluff.TDHNewbornClassificationFluffPillow.870ceaecb4ba

pillowtop_corehq.pillows.sofabed.CaseDataPillow.870ceaecb4ba

pillowtop_fluff.TDHEnrollChildFluffPillow.870ceaecb4ba

Second run (accessing these documents raise ResourceNotFound: missing in
"couchdbkit/resource.py", line 129, in request)

pillowtop_fluff.CareSAFluffPillow.59a34724a421

pillowtop_corehq.pillows.sofabed.FormDataPillow.59a34724a421

pillowtop_fluff.UnicefMalawiFluffPillow.59a34724a421

pillowtop_corehq.pillows.sms.SMSPillow.708c77f8e5fe00286fa5791e9fa7d45f.59a34724a421

pillowtop_fluff.OPMHierarchyFluffPillow.59a34724a421

pillowtop_corehq.pillows.reportxform.ReportXFormPillow.report_xforms_e26fad18b95711e4a7bc3c970e7be43b.59a34724a421

pillowtop_corehq.pillows.user.GroupToUserPillow.59a34724a421

pillowtop_fluff.TDHChildClassificationFluffPillow.59a34724a421

pillowtop_fluff.AllHmisCaseFluffPillow.59a34724a421

pillowtop_fluff.TDHInfantClassificationFluffPillow.59a34724a421

pillowtop_corehq.pillows.application.AppPillow.fe975a5128b655ba175ac81b035498d0.59a34724a421

pillowtop_fluff.VhndAvailabilityFluffPillow.59a34724a421

pillowtop_fluff.FarmerRecordFluffPillow.59a34724a421

pillowtop_fluff.TDHEnrollChildFluffPillow.59a34724a421

pillowtop_fluff.RecouvrementFluffPillow.59a34724a421

pillowtop_corehq.pillows.log.PhoneLogPillow.59a34724a421

ERROR 404

=========

Consistently, a process fails and raise an error 404 accessing
http://commcarehq:commcarehq@couchdb:5984/commcarehq

Process Process-55:

Traceback (most recent call last):

File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap

self.run()

File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

self._target(*self._args, **self._kwargs)

File "submodules/pillowtop-src/pillowtop/listener.py", line 172, in run

self.new_changes()

File "submodules/pillowtop-src/pillowtop/listener.py", line 152, in
new_changes

since=self.since,

File "submodules/pillowtop-src/pillowtop/listener.py", line 235, in since

checkpoint = self.get_checkpoint()

File "/usr/src/commcare-hq/corehq/pillows/cacheinvalidate.py", line 37, in
get_checkpoint

current_db_seq = self.couch_db.info()['update_seq']

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line
280, in info

return self.res.get().json_body

File "/usr/local/lib/python2.7/dist-packages/restkit/resource.py", line
114, in get

params_dict=params_dict, **params)

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/resource.py", line
129, in request

response=e.response)

ResourceNotFound: {'msg': '', 'status_int': 404, 'response':
<couchdbkit.resource.CouchDBResponse object at 0x77736d0>}

This is weird as running 'curl http://commcarehq:commcarehq@couchdb:5984/commcarehq'
in the same container succeed.

FILE ALREADY EXISTS

===================

Another consistently raised Exception is 'PreconditionFailed: The database
could not be created, the file already exists.'

Process Process-51:

Traceback (most recent call last):

File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap

self.run()

File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

self._target(*self._args, **self._kwargs)

File "/usr/src/commcare-hq/corehq/apps/userreports/pillow.py", line 32, in
run

self.bootstrap()

File "/usr/src/commcare-hq/corehq/apps/userreports/pillow.py", line 40, in
bootstrap

self.tables = [IndicatorSqlAdapter(self.get_sql_engine(), config) for
config in configs]

File "/usr/src/commcare-hq/corehq/apps/userreports/models.py", line 180,
in all

db = cls.get_db()

File
"/usr/local/lib/python2.7/dist-packages/couchdbkit/ext/django/schema.py",
line 153, in get_db

db = get_db(app_label)

File
"/usr/local/lib/python2.7/dist-packages/couchdbkit/ext/django/loading.py",
line 178, in get_db

db = server.get_or_create_db(dbname)

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line
146, in create_db

return self.get_db(dbname, create=True, **params)

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line
131, in get_db

return Database(self._db_uri(dbname), server=self, **params)

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line
267, in init

self.server.res.put('/%s/' % self.dbname, **params).json_body

File "/usr/local/lib/python2.7/dist-packages/restkit/resource.py", line
153, in put

headers=headers, params_dict=params_dict, **params)

File "/usr/local/lib/python2.7/dist-packages/couchdbkit/resource.py", line
136, in request

response=e.response)

PreconditionFailed: The database could not be created, the file already
exists.

PARSING ERROR

=============

I also sometimes get parsing errors in
"/usr/local/lib/python2.7/dist-packages/http_parser/http.py", line 210, in
next :

ParserError: nparsed != recved (0 != 245)
[{"db_name":"commcarehq","doc_count":872,"doc_del_count":0,"update_seq":914,"purge_seq":0,"compact_running":false,"disk_size":2575223,"data_size":396786,"instance_start_time":"1424879167613551","disk_format_version":6,"committed_update_seq":908}

]

I'm not sure how to interpret all this… Are these exception expected to be
raised ? The database has just been created and is pretty empty after all
at that point, maybe some function are trying to access document not
created at the moment ? Can these errors be safely ignored ?

Many thanks,

--

Charles Flèche

mHealth Advisor

Télécoms Sans Frontières http://www.tsfi.org

Première Urgence - Aide Médicale Internationale http://www.pu-ami.org

--

You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Cory,

Many thanks for these ideas. I still try to wrap my head around all the CommCare
components, it should get better after some time digging into the source.

  1. Pillowtop mostly powers our custom reports. So you can probably getting
    away with just running the "core" subset of pillows. ./manage.py run_ptop
    --pillow_key core (or similar)

Good point. I'm following the procedure in README.md to run pillowtop, and there
is only an example for MacOSX and Windows. I'm running the OSX command on my
Linux box, maybe that's the root of all evil ?

That should be enough for the current purpose of the Docker setup (demo /
training). Which by the way proved very useful yesterday when CommCareHQ went
down : this didn't prevent a training I was giving as we could roll a VM on a windows
laptop with boot2docker.

I'd however love to be able to run a production-grade CommCareHQ on a laptop in
the future. Let's make it step by step…

Many thanks Cory, I'll tell you if I make progress on the Docker side.

··· -- Charles Flèche mHealth Advisor Télécoms Sans Frontières http://www.tsfi.org Première Urgence - Aide Médicale Internationale http://www.pu-ami.org
4

By the way it is pretty easy to replicate my environment :

  1. install docker on your workstation
  2. run docker/docker-run.sh in the CommCareHQ source folder

I then usually mount a CommCareHQ source tree into the
charlesfleche/commcare-hq docker container by adding the the option -v
pwd:/usr/src/commcare-hq to the docker run command.

··· On Thursday 26 February 2015 13:59:59 Cory Zue wrote: > It's hard to know exactly what's going on without being able to replicate > your environment, but I will give the following guidelines:

--
Charles Flèche
mHealth Advisor
Télécoms Sans Frontières http://www.tsfi.org
Première Urgence - Aide Médicale Internationale http://www.pu-ami.org

5

In fabfile.py, I absolutely love this !!!

if datetime.datetime.now().isoweekday() == 5:
    print('')
    print('┓┏┓┏┓┃')
    print('┛┗┛┗┛┃＼○／')
    print('┓┏┓┏┓┃  /      Friday')
    print('┛┗┛┗┛┃ノ)')
    print('┓┏┓┏┓┃         deploy,')
    print('┛┗┛┗┛┃')
    print('┓┏┓┏┓┃         good')
    print('┛┗┛┗┛┃')
    print('┓┏┓┏┓┃         luck!')
    print('┃┃┃┃┃┃')
    print('┻┻┻┻┻┻')
··· On Thursday 26 February 2015 13:59:59 Cory Zue wrote: > 4. The fabfile, and particularly the awesome_deploy command, is > representative of how we deploy our production infrastructure, so looking > there for other clues might be helpful.

--
Charles Flèche
mHealth Advisor
Télécoms Sans Frontières http://www.tsfi.org
Première Urgence - Aide Médicale Internationale http://www.pu-ami.org

6

me too :slight_smile:

··· On 27 February 2015 at 12:39, Charles Flèche wrote:

On Thursday 26 February 2015 13:59:59 Cory Zue wrote:

  1. The fabfile, and particularly the awesome_deploy command, is

representative of how we deploy our production infrastructure, so looking

there for other clues might be helpful.

In fabfile.py, I absolutely love this !!!

if datetime.datetime.now().isoweekday() == 5:

print('')

print('┓┏┓┏┓┃')

print('┛┗┛┗┛┃＼○／')

print('┓┏┓┏┓┃ / Friday')

print('┛┗┛┗┛┃ノ)')

print('┓┏┓┏┓┃ deploy,')

print('┛┗┛┗┛┃')

print('┓┏┓┏┓┃ good')

print('┛┗┛┗┛┃')

print('┓┏┓┏┓┃ luck!')

print('┃┃┃┃┃┃')

print('┻┻┻┻┻┻')

--

Charles Flèche

mHealth Advisor

Télécoms Sans Frontières http://www.tsfi.org

Première Urgence - Aide Médicale Internationale http://www.pu-ami.org

--

You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Simon Kelly
Senior Engineer | Dimagi South Africa