After-reboot on a monolith consistently fails on start Zookeeper service

erobinson · June 25, 2022, 8:37pm

When running after-reboot, I consistently get an error on the Zookeeper start play:

PLAY [Zookeeper] ***********************************************************************************************************************************************************************

TASK [zookeeper : start Zookeeper service] *********************************************************************************************************************************************
fatal: [10.1.0.4]: FAILED! => {"changed": false, "msg": "Could not find the requested service zookeeper-server: host"}

At this point, if I do a check_services, there are always 3 failed services:

FAILURE (Took   0.05s) kafka          : Could not connect to Kafka: NoBrokersAvailable
SUCCESS (Took   0.00s) redis          : Redis is up and using 533.52M memory
SUCCESS (Took   0.05s) postgres       : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK Successfully got a user from postgres
SUCCESS (Took   0.06s) couch          : Successfully queried an arbitrary couch view
FAILURE (Took   0.00s) celery         : async_restore_queue has been blocked for 0:24:22.778680 (max allowed is 0:01:00)
background_queue has been blocked for 0:24:22.792954 (max allowed is 0:10:00)
case_import_queue has been blocked for 0:24:22.810978 (max allowed is 0:01:00)
celery has been blocked for 0:24:22.799202 (max allowed is 0:01:00)
celery_periodic has been blocked for 0:24:22.788920 (max allowed is 0:10:00)
email_queue has been blocked for 0:24:22.767711 (max allowed is 0:00:30)
export_download_queue has been blocked for 0:24:22.891070 (max allowed is 0:00:30)
SUCCESS (Took   0.18s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.16s) blobdb         : Successfully saved a file to the blobdb
FAILURE (Took   0.14s) formplayer     : Formplayer returned a 502 status code: https://xxxxxx.xxx/formplayer/serverup
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK

I can get them up OK with

cchq monolith service celery restart
cchq monolith service kafka restart
cchq monolith service formplayer restart

...however, even when they're all up, I get the same zookeeper start error if I run the after-reboot playbook. I'm concerned something's amiss and perhaps it's not the final task in the playbook and other tasks are not getting run?

Any advice is appreciated!

robynton · June 26, 2022, 8:33pm

Agree that problem existed. First error is the same, but after

cchq monolith service kafka restart

I've got

ansible zookeeper -m service -i /home/lamp/environments/monolith/inventory.ini -a 'name=zookeeper state=restarted' --diff -u ansible --become -e @/home/lamp/environments/monolith/public.yml -e @/home/lamp/environments/monolith/.generated.yml -e @/home/lamp/environments/monolith/vault.yml --vault-password-file=/home/lamp/commcare-cloud/src/commcare_cloud/ansible/echo_vault_password.sh '--ssh-common-args=-o UserKnownHostsFile=/home/lamp/environments/monolith/known_hosts'
192.169.233.136 | FAILED! => {
"changed": false,
"msg": "Could not find the requested service zookeeper: host"
}

robynton · June 26, 2022, 8:37pm

It seems that problem is the same as

robynton · July 3, 2022, 4:15pm

Just done fresh monolith install on VM Ubuntu 18.04, got the same error after restarting kafka service

(cchq) lamp@capi:~/environments/monolith$ cchq monolith service kafka restart
Vault Password for 'monolith':
ansible kafka -m service -i /home/lamp/environments/monolith/inventory.ini -a 'name=kafka-server state=restarted' --diff -u ansible --become -e @/home/lamp/environments/monolith/public.yml -e @/home/lamp/environments/monolith/.generated.yml -e @/home/lamp/environments/monolith/vault.yml --vault-password-file=/home/lamp/commcare-cloud/src/commcare_cloud/ansible/echo_vault_password.sh '--ssh-common-args=-o UserKnownHostsFile=/home/lamp/environments/monolith/known_hosts'
Enter passphrase for key '/home/lamp/.ssh/id_rsa':
192.169.233.133 | CHANGED => {
"changed": true,
"name": "kafka-server",
"state": "started",
"status": {
"ActiveEnterTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"ActiveEnterTimestampMonotonic": "3048251794",
"ActiveExitTimestamp": "Sun 2022-07-03 16:06:15 UTC",
"ActiveExitTimestampMonotonic": "3047172732",
"ActiveState": "active",
"After": "systemd-journald.socket network.target sysinit.target basic.target system.slice remote-fs.target kafka-zookeeper.service",
"AllowIsolate": "no",
"AmbientCapabilities": "",
"AssertResult": "yes",
"AssertTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"AssertTimestampMonotonic": "3048250576",
"Before": "multi-user.target shutdown.target",
"BlockIOAccounting": "no",
"BlockIOWeight": "[not set]",
"CPUAccounting": "no",
"CPUQuotaPerSecUSec": "infinity",
"CPUSchedulingPolicy": "0",
"CPUSchedulingPriority": "0",
"CPUSchedulingResetOnFork": "no",
"CPUShares": "[not set]",
"CPUUsageNSec": "[not set]",
"CPUWeight": "[not set]",
"CacheDirectoryMode": "0755",
"CanIsolate": "no",
"CanReload": "no",
"CanStart": "yes",
"CanStop": "yes",
"CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend",
"CollectMode": "inactive",
"ConditionResult": "yes",
"ConditionTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"ConditionTimestampMonotonic": "3048250576",
"ConfigurationDirectoryMode": "0755",
"Conflicts": "shutdown.target",
"ControlGroup": "/system.slice/kafka-server.service",
"ControlPID": "0",
"DefaultDependencies": "yes",
"Delegate": "no",
"Description": "Apache Kafka server (broker)",
"DevicePolicy": "auto",
"Documentation": "Apache Kafka",
"DynamicUser": "no",
"Environment": "JMX_PORT=9999",
"ExecMainCode": "0",
"ExecMainExitTimestampMonotonic": "0",
"ExecMainPID": "69730",
"ExecMainStartTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"ExecMainStartTimestampMonotonic": "3048251738",
"ExecMainStatus": "0",
"ExecStart": "{ path=/opt/kafka/bin/kafka-server-start.sh ; argv[]=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }",
"ExecStop": "{ path=/opt/kafka/bin/kafka-server-stop.sh ; argv[]=/opt/kafka/bin/kafka-server-stop.sh ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }",
"FailureAction": "none",
"FileDescriptorStoreMax": "0",
"FragmentPath": "/etc/systemd/system/kafka-server.service",
"GID": "998",
"Group": "kafka",
"GuessMainPID": "yes",
"IOAccounting": "no",
"IOSchedulingClass": "0",
"IOSchedulingPriority": "0",
"IOWeight": "[not set]",
"IPAccounting": "no",
"IPEgressBytes": "18446744073709551615",
"IPEgressPackets": "18446744073709551615",
"IPIngressBytes": "18446744073709551615",
"IPIngressPackets": "18446744073709551615",
"Id": "kafka-server.service",
"IgnoreOnIsolate": "no",
"IgnoreSIGPIPE": "yes",
"InactiveEnterTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"InactiveEnterTimestampMonotonic": "3048250506",
"InactiveExitTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"InactiveExitTimestampMonotonic": "3048251794",
"InvocationID": "ad6fb02c4fff492e88e0cf021362418f",
"JobRunningTimeoutUSec": "infinity",
"JobTimeoutAction": "none",
"JobTimeoutUSec": "infinity",
"KeyringMode": "private",
"KillMode": "control-group",
"KillSignal": "15",
"LimitAS": "infinity",
"LimitASSoft": "infinity",
"LimitCORE": "infinity",
"LimitCORESoft": "0",
"LimitCPU": "infinity",
"LimitCPUSoft": "infinity",
"LimitDATA": "infinity",
"LimitDATASoft": "infinity",
"LimitFSIZE": "infinity",
"LimitFSIZESoft": "infinity",
"LimitLOCKS": "infinity",
"LimitLOCKSSoft": "infinity",
"LimitMEMLOCK": "67108864",
"LimitMEMLOCKSoft": "67108864",
"LimitMSGQUEUE": "819200",
"LimitMSGQUEUESoft": "819200",
"LimitNICE": "0",
"LimitNICESoft": "0",
"LimitNOFILE": "32000",
"LimitNOFILESoft": "32000",
"LimitNPROC": "63802",
"LimitNPROCSoft": "63802",
"LimitRSS": "infinity",
"LimitRSSSoft": "infinity",
"LimitRTPRIO": "0",
"LimitRTPRIOSoft": "0",
"LimitRTTIME": "infinity",
"LimitRTTIMESoft": "infinity",
"LimitSIGPENDING": "63802",
"LimitSIGPENDINGSoft": "63802",
"LimitSTACK": "infinity",
"LimitSTACKSoft": "8388608",
"LoadState": "loaded",
"LockPersonality": "no",
"LogLevelMax": "-1",
"LogsDirectoryMode": "0755",
"MainPID": "69730",
"MemoryAccounting": "no",
"MemoryCurrent": "[not set]",
"MemoryDenyWriteExecute": "no",
"MemoryHigh": "infinity",
"MemoryLimit": "infinity",
"MemoryLow": "0",
"MemoryMax": "infinity",
"MemorySwapMax": "infinity",
"MountAPIVFS": "no",
"MountFlags": "",
"NFileDescriptorStore": "0",
"NRestarts": "1",
"Names": "kafka-server.service",
"NeedDaemonReload": "no",
"Nice": "0",
"NoNewPrivileges": "no",
"NonBlocking": "no",
"NotifyAccess": "none",
"OOMScoreAdjust": "0",
"OnFailureJobMode": "replace",
"PermissionsStartOnly": "no",
"Perpetual": "no",
"PrivateDevices": "no",
"PrivateNetwork": "no",
"PrivateTmp": "no",
"PrivateUsers": "no",
"ProtectControlGroups": "no",
"ProtectHome": "no",
"ProtectKernelModules": "no",
"ProtectKernelTunables": "no",
"ProtectSystem": "no",
"RefuseManualStart": "no",
"RefuseManualStop": "no",
"RemainAfterExit": "no",
"RemoveIPC": "no",
"Requires": "sysinit.target remote-fs.target system.slice network.target",
"Restart": "always",
"RestartUSec": "1s",
"RestrictNamespaces": "no",
"RestrictRealtime": "no",
"RestrictSUIDSGID": "no",
"Result": "success",
"RootDirectoryStartOnly": "no",
"RuntimeDirectoryMode": "0755",
"RuntimeDirectoryPreserve": "no",
"RuntimeMaxUSec": "infinity",
"SameProcessGroup": "no",
"SecureBits": "0",
"SendSIGHUP": "no",
"SendSIGKILL": "yes",
"Slice": "system.slice",
"StandardError": "inherit",
"StandardInput": "null",
"StandardInputData": "",
"StandardOutput": "journal",
"StartLimitAction": "none",
"StartLimitBurst": "5",
"StartLimitIntervalUSec": "10s",
"StartupBlockIOWeight": "[not set]",
"StartupCPUShares": "[not set]",
"StartupCPUWeight": "[not set]",
"StartupIOWeight": "[not set]",
"StateChangeTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"StateChangeTimestampMonotonic": "3048251794",
"StateDirectoryMode": "0755",
"StatusErrno": "0",
"StopWhenUnneeded": "no",
"SubState": "running",
"SuccessAction": "none",
"SyslogFacility": "3",
"SyslogLevel": "6",
"SyslogLevelPrefix": "yes",
"SyslogPriority": "30",
"SystemCallErrorNumber": "0",
"TTYReset": "no",
"TTYVHangup": "no",
"TTYVTDisallocate": "no",
"TasksAccounting": "yes",
"TasksCurrent": "59",
"TasksMax": "19140",
"TimeoutStartUSec": "1min 30s",
"TimeoutStopUSec": "1min 30s",
"TimerSlackNSec": "50000",
"Transient": "no",
"Type": "simple",
"UID": "997",
"UMask": "0022",
"UnitFilePreset": "enabled",
"UnitFileState": "enabled",
"User": "kafka",
"UtmpMode": "init",
"WantedBy": "multi-user.target",
"WatchdogTimestamp": "Sun 2022-07-03 16:06:16 UTC",
"WatchdogTimestampMonotonic": "3048251793",
"WatchdogUSec": "0"
}
}
ansible zookeeper -m service -i /home/lamp/environments/monolith/inventory.ini -a 'name=zookeeper state=restarted' --diff -u ansible --become -e @/home/lamp/environments/monolith/public.yml -e @/home/lamp/environments/monolith/.generated.yml -e @/home/lamp/environments/monolith/vault.yml --vault-password-file=/home/lamp/commcare-cloud/src/commcare_cloud/ansible/echo_vault_password.sh '--ssh-common-args=-o UserKnownHostsFile=/home/lamp/environments/monolith/known_hosts'
192.169.233.133 | FAILED! => {
"changed": false,
"msg": "Could not find the requested service zookeeper: host"
}

Additional info: when trying cchq monolith service zookeeper status, just got:

usage: cchq {monolith} service [-h] [--limit LIMIT] [--only PROCESS_PATTERN]
{celery,citusdb,commcare,couchdb2,elasticsearch,elasticsearch-classic,formplayer,kafka,nginx,pillowtop,postgresql,rabbitmq,redis,webworker}
[{celery,citusdb,commcare,couchdb2,elasticsearch,elasticsearch-classic,formplayer,kafka,nginx,pillowtop,postgresql,rabbitmq,redis,webworker} ...]
{start,stop,restart,status,logs,help}
cchq {monolith} service: error: argument services: invalid choice: 'zookeeper' (choose from 'celery', 'citusdb', 'commcare', 'couchdb2', 'elasticsearch', 'elasticsearch-classic', 'formplayer', 'kafka', 'nginx', 'pillowtop', 'postgresql', 'rabbitmq', 'redis', 'webworker')

It seems like zookeeper service wasn't even created by the script.

So, how to make zookeper service alive? Any help would be appreciated)

Sravan_Reddy · July 5, 2022, 5:26am

Hello,

Did you check if kafka and zookeeper processes are running without errors by running below?

sudo service kafka-server status
sudo service zookeeper-server status

robynton · July 5, 2022, 7:44pm

Thanks for reply. Here's my results for sudo service kafka-server status:

(cchq) lamp@monolith:~/commcare-cloud$ sudo service kafka-server status
[sudo] password for lamp:
● kafka-server.service - Apache Kafka server (broker)
Loaded: loaded (/etc/systemd/system/kafka-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-05 19:35:35 UTC; 4min 51s ago
Docs: Apache Kafka
Main PID: 70144 (java)
Tasks: 45 (limit: 19660)
CGroup: /system.slice/kafka-server.service
└─70144 java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:
Jul 05 19:35:35 monolith.capi.gstat.gov.ua systemd[1]: Started Apache Kafka server (broker).
Jul 05 19:18:52 monolith.capi.gstat.gov.ua systemd[1]: Started Apache Kafka server (broker).

And for sudo service zookeeper-server status:

(cchq) lamp@capi:~/commcare-cloud$ sudo service zookeeper-server status
[sudo] password for lamp:
● zookeeper-server.service - Apache Zookeeper server
Loaded: loaded (/etc/systemd/system/zookeeper-server.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2022-07-05 19:35:56 UTC; 8min ago
Docs: Index of /doc
Process: 4277 ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/zoo.cfg (code=exited, status=0/SUCCESS)
Process: 4206 ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg (code=exited, status=0/SUCCESS
Main PID: 4065 (code=exited, status=2)
Jul 05 19:35:55 capi.gstat.gov.ua zkServer.sh[4206]: grep: /opt/zookeeper/conf/zoo.cfg: No such file or directory
Jul 05 19:35:55 capi.gstat.gov.ua zkServer.sh[4206]: mkdir: cannot create directory ‘’: No such file or directory
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4206]: Starting zookeeper ... STARTED
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: ZooKeeper JMX enabled by default
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: Using config: /opt/zookeeper/conf/zoo.cfg
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: grep: /opt/zookeeper/conf/zoo.cfg: No such file or directory
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: mkdir: cannot create directory ‘’: No such file or directory
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: Stopping zookeeper ... /opt/zookeeper/bin/zkServer.sh: line 182: kil
Jul 05 19:35:56 capi.gstat.gov.ua zkServer.sh[4277]: STOPPED
Jul 05 19:35:56 capi.gstat.gov.ua systemd[1]: Started Apache Zookeeper server.
lines 1-18/18 (END)

It looks like zookeper config file hasn't been created via script.

Sravan_Reddy · July 7, 2022, 7:26am

This issue has just been fixed.

Can you please do update-code and then deploy zookeeper using commcare-cloud $env_name deploy-stack --skip-check -tags=zookeeper? That should create the missing config file and restart the zookeeper. You can verify that it's fixed by re-looking at the status of the zookeeper process.

robynton · July 7, 2022, 2:22pm

Thanks! It worked well after redeploying on fresh VM. But now at the stage commcare-cloud cchq deploy I've got a warning:

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[192.169.233.128] Executing task '_task'
[192.169.233.128] run: git rev-parse HEAD
[192.169.233.128] Passphrase for private key:
[192.169.233.128] out: fatal: unsafe repository ('/home/cchq/www/cchq/releases/2022-07-07_12.48' is owned by someone else)
[192.169.233.128] out: To add an exception for this directory, call:
[192.169.233.128] out:
[192.169.233.128] out: git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48
[192.169.233.128] out:
Warning: run() received nonzero return code 128 while executing 'git rev-parse HEAD'!
Diff generation skipped. Supply a Github token to see deploy diffs.
New version details:
Branch deployed : commcare: master
Here's the complete diff on github: https://github.com/dimagi/commcare-hq/compare/fatal: unsafe repository ('/home/cchq/www/cchq/releases/2022-07-07_12.48' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48...87ae2f7a767ef8541bbe20a917c3b2ad16b2ebc1
Are you sure you want to preindex and deploy to cchq? [y/N]

But executing git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48 hasn't helped. The same warning.
After ignoring it I've got:

Are you sure you want to preindex and deploy to cchq? [y/N]y
Vault Password for 'cchq':

Sending email: lamp has initiated a CommCare HQ deploy to cchq
Ubuntu 18.04.6 LTS
Enter passphrase for key '/home/lamp/.ssh/id_rsa':
Connection to 192.169.233.128 closed.
commcare-cloud cchq fab deploy_commcare --set code_branch=master --branch master
fab -f /home/lamp/commcare-cloud/src/commcare_cloud/fabfile.py cchq deploy_commcare --set code_branch=master --disable-known-hosts --system-known-hosts /home/lamp/environments/cchq/known_hosts
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
Using commcare-hq branch master
[192.169.233.128] Executing task 'deploy_commcare'
[192.169.233.128] Executing task '_setup_release'
[192.169.233.128] Executing task 'create'
[192.169.233.128] sudo: mkdir -p /home/cchq/www/cchq/releases/2022-07-07_14.18

Fatal error: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.
!!! Parallel execution exception under host '192.169.233.128':
Process 192.169.233.128:
Traceback (most recent call last):
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 449, in connect
sock=sock
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect
passphrase,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth
raise saved_exception
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 735, in _auth
filename, pkey_class, passphrase
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 586, in _key_from_filepath
key = klass.from_private_key_file(key_path, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 235, in from_private_key_file
key = cls(filename=filename, password=password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 55, in init
self._from_private_key_file(filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 175, in _from_private_key_file
data = self._read_private_key_file("RSA", filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 308, in _read_private_key_file
data = self._read_private_key(tag, f, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 334, in _read_private_key
data = self._read_private_key_pem(lines, end, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 386, in _read_private_key_pem
raise PasswordRequiredException("Private key file is encrypted")
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 240, in inner
submit(task.run(*args, **kwargs))
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 180, in inner
return func(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 203, in create
sudo('mkdir -p {}'.format(env.code_root))
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 929, in _run_command
channel=default_channel(), command=wrapped_command, pty=pty,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 397, in default_channel
chan = _open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 389, in _open_session
return connections[env.host_string].get_transport().open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 156, in getitem
self.connect(key)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 148, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 528, in connect
password = prompt_for_password(text)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 605, in prompt_for_password
handle_prompt_abort("a connection or sudo password")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 184, in handle_prompt_abort
abort(reason % "input would be ambiguous in parallel mode")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Fatal error: One or more hosts failed while executing task 'create'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.
[192.169.233.128] Executing task 'send_email'
[192.169.233.128] sudo: /home/cchq/www/cchq/releases/2022-07-07_14.18/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to cchq failed. Try resuming with fab cchq deploy:resume=yes.' 'Traceback:
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'create'"'"'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode'
[192.169.233.128] Passphrase for private key:
[192.169.233.128] out: /bin/bash: line 0: cd: /home/cchq/www/cchq/releases/2022-07-07_14.18: No such file or directory
[192.169.233.128] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: /home/cchq/www/cchq/releases/2022-07-07_14.18/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to cchq failed. Try resuming with fab cchq deploy:resume=yes.' 'Traceback:
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'create'"'"'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/cchq/releases/2022-07-07_14.18 >/dev/null && /home/cchq/www/cchq/releases/2022-07-07_14.18/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to cchq failed. Try resuming with fab cchq deploy:resume=yes.' 'Traceback:
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'create'"'"'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode'"

Aborting.
Traceback (most recent call last):
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['results'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task 'create'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
*args, **kwargs
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 542, in deploy_commcare
_deploy_without_asking(skip_record)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 412, in _deploy_without_asking
traceback_string()
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 242, in send_email
f'{virtualenv_dir}/bin/python manage.py '
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!

Requested: /home/cchq/www/cchq/releases/2022-07-07_14.18/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to cchq failed. Try resuming with fab cchq deploy:resume=yes.' 'Traceback:
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'create'"'"'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/cchq/releases/2022-07-07_14.18 >/dev/null && /home/cchq/www/cchq/releases/2022-07-07_14.18/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to cchq failed. Try resuming with fab cchq deploy:resume=yes.' 'Traceback:
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 319, in _setup_release
execute_with_timing(release.create_code_dir(full_cluster))
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'create'"'"'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode'"
Disconnecting from 192.169.233.128... done.
Sending email: CommCare HQ deploy to cchq failed
Ubuntu 18.04.6 LTS
Enter passphrase for key '/home/lamp/.ssh/id_rsa':
Connection to 192.169.233.128 closed.
Skipping formplayer because commcare failed

It's strange that it talks about [192.169.233.128] out: /bin/bash: line 0: cd: /home/cchq/www/cchq/releases/2022-07-07_14.18: No such file or directory [192.169.233.128] out: Cause there is only one folder

Any help would be appreciated.

Sravan_Reddy · July 8, 2022, 7:59am

Hello,

To run any subsequent cchq commands after quick-install, you need to be SSHed into the VM as either the ansible user or the ssh_username that was set in the install-config.yml with SSH host forwarding (that is ssh -A ansible@VM_IP pr ssh -A ssh_username@VM_IP)

robynton · July 8, 2022, 6:13pm

Thanks for quick reply! Yes, I'm connected to my VM under user lamp that was set in the install-config.yml. And after that I've gov strange error about repository's rights:

(cchq) lamp@monolith:~/commcare-cloud$ commcare-cloud cchq deploy
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[192.169.233.128] Executing task '_task'
[192.169.233.128] run: git rev-parse HEAD
[192.169.233.128] Passphrase for private key:
[192.169.233.128] out: fatal: unsafe repository ('/home/cchq/www/cchq/releases/2022-07-07_12.48' is owned by someone else)
[192.169.233.128] out: To add an exception for this directory, call:
[192.169.233.128] out:
[192.169.233.128] out: git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48
[192.169.233.128] out:
Warning: run() received nonzero return code 128 while executing 'git rev-parse HEAD'!
Diff generation skipped. Supply a Github token to see deploy diffs.
New version details:
Branch deployed : commcare: master
Here's the complete diff on github: https://github.com/dimagi/commcare-hq/compare/fatal: unsafe repository ('/home/cchq/www/cchq/releases/2022-07-07_12.48' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48...f27f74680137509e8db1c7543411bfdfdd5569ee
Are you sure you want to preindex and deploy to cchq? [y/N]y

Executing git config --global --add safe.directory /home/cchq/www/cchq/releases/2022-07-07_12.48 doesn't help.
It is the same on VM installed in manual monolith mode and VM configured by quick install script.

Any clue about it? Thanks in advance!

robynton · July 8, 2022, 7:13pm

The same things when executing via ssh -A lamp@VM_IP. After error from git I've got:

Are you sure you want to preindex and deploy to cchq? [y/N]y
Vault Password for 'cchq':

Sending email: lamp has initiated a CommCare HQ deploy to cchq
Ubuntu 18.04.6 LTS
Enter passphrase for key '/home/lamp/.ssh/id_rsa':
Connection to 192.169.233.128 closed.
commcare-cloud cchq fab deploy_commcare --set code_branch=master --branch master
fab -f /home/lamp/commcare-cloud/src/commcare_cloud/fabfile.py cchq deploy_commcare --set code_branch=master --disable-known-hosts --system-known-hosts /home/lamp/environments/cchq/known_hosts
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
Using commcare-hq branch master
[192.169.233.128] Executing task 'deploy_commcare'
[192.169.233.128] Executing task '_setup_release'
[192.169.233.128] Executing task 'create'
[192.169.233.128] sudo: mkdir -p /home/cchq/www/cchq/releases/2022-07-08_19.10

Fatal error: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.
!!! Parallel execution exception under host '192.169.233.128':
Process 192.169.233.128:
Traceback (most recent call last):
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 449, in connect
sock=sock
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect
passphrase,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth
raise saved_exception
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 735, in _auth
filename, pkey_class, passphrase
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 586, in _key_from_filepath
key = klass.from_private_key_file(key_path, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 235, in from_private_key_file
key = cls(filename=filename, password=password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 55, in init
self._from_private_key_file(filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 175, in _from_private_key_file
data = self._read_private_key_file("RSA", filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 308, in _read_private_key_file
data = self._read_private_key(tag, f, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 334, in _read_private_key
data = self._read_private_key_pem(lines, end, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 386, in _read_private_key_pem
raise PasswordRequiredException("Private key file is encrypted")
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 240, in inner
submit(task.run(*args, **kwargs))
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 180, in inner
return func(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 203, in create
sudo('mkdir -p {}'.format(env.code_root))
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 929, in _run_command
channel=default_channel(), command=wrapped_command, pty=pty,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 397, in default_channel
chan = _open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 389, in _open_session
return connections[env.host_string].get_transport().open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 156, in getitem
self.connect(key)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 148, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 528, in connect
password = prompt_for_password(text)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 605, in prompt_for_password
handle_prompt_abort("a connection or sudo password")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 184, in handle_prompt_abort
abort(reason % "input would be ambiguous in parallel mode")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Fatal error: One or more hosts failed while executing task 'create'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.

Sravan_Reddy · July 11, 2022, 5:23am

Hi,

I am not quite sure what cased that error from occurring.

However, you can first try below command to delete the existing release directories and then run the deploy command after that.

cchq $env fab clean_releases

robynton · July 11, 2022, 5:37pm

The same error when executing the above command. As I've googled problem might be connected with python Fabric parallel mode execution.
As in documentation Parallel execution — Fabric documentation
And the same error guy got out there ssh - Python Fabric Parallel Execution Failure on EC2: Updated - Stack Overflow

May be some ideas what file to fix to undo the parallel mode execution.

(cchq) lamp@monolith:~/commcare-cloud$ commcare-cloud cchq fab clean_releases
fab -f /home/lamp/commcare-cloud/src/commcare_cloud/fabfile.py cchq clean_releases --disable-known-hosts --system-known-hosts /home/lamp/environments/cchq/known_hosts
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
Using commcare-hq branch master
[192.169.233.128] Executing task 'clean_releases'
[192.169.233.128] Executing task 'clean_releases'
[192.169.233.128] sudo: ls /home/cchq/www/cchq/releases

Fatal error: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.
!!! Parallel execution exception under host '192.169.233.128':
Process 192.169.233.128:
Traceback (most recent call last):
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 449, in connect
sock=sock
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect
passphrase,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth
raise saved_exception
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 735, in _auth
filename, pkey_class, passphrase
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/client.py", line 586, in _key_from_filepath
key = klass.from_private_key_file(key_path, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 235, in from_private_key_file
key = cls(filename=filename, password=password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 55, in init
self._from_private_key_file(filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/rsakey.py", line 175, in _from_private_key_file
data = self._read_private_key_file("RSA", filename, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 308, in _read_private_key_file
data = self._read_private_key(tag, f, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 334, in _read_private_key
data = self._read_private_key_pem(lines, end, password)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/paramiko/pkey.py", line 386, in _read_private_key_pem
raise PasswordRequiredException("Private key file is encrypted")
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 240, in inner
submit(task.run(*args, **kwargs))
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 180, in inner
return func(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 255, in clean_releases
releases = sudo('ls {}'.format(env.releases)).split()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 929, in _run_command
channel=default_channel(), command=wrapped_command, pty=pty,
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 397, in default_channel
chan = _open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/state.py", line 389, in _open_session
return connections[env.host_string].get_transport().open_session()
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 156, in getitem
self.connect(key)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 148, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 528, in connect
password = prompt_for_password(text)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 605, in prompt_for_password
handle_prompt_abort("a connection or sudo password")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 184, in handle_prompt_abort
abort(reason % "input would be ambiguous in parallel mode")
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Fatal error: One or more hosts failed while executing task 'clean_releases'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Aborting.
Traceback (most recent call last):
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
*args, **kwargs
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 501, in clean_releases
execute(release.clean_releases, keep)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['results'])
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/lamp/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task 'clean_releases'

Underlying exception:
Needed to prompt for a connection or sudo password (host: 192.169.233.128), but input would be ambiguous in parallel mode

Here's the connected theme Fabric returns exit code 0 on failed parallel tasks · Issue #572 · fabric/fabric · GitHub

robynton · July 11, 2022, 5:43pm

There ara suspicious error line from above output File "/home/lamp/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 255, in clean_releases releases = sudo('ls {}'.format(env.releases)).split()

Sravan_Reddy · August 22, 2022, 8:27am

Hello Roby,

The exact error is here

paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

This means your SSH auth is password protected. Can you try after disabling that?