Fab restart_services: brings an error

Developers
1

Hello,

the following is an error while running the following command: "cchq fab restart_services". the same error appears during redeployment using the command "cchq deploy"

172.19.3.41] out:                                                                                                                                  [62/1977]
                                                                                                             
[172.19.3.41] sudo: nginx -s reload                                                                        
[172.19.3.41] out: nginx: [warn] conflicting server name "www.echisethiopia.org" on 0.0.0.0:80, ignored   
[172.19.3.41] out: nginx: [warn] conflicting server name "www.echisethiopia.org" on 0.0.0.0:443, ignored
[172.19.3.41] out:                                                                                         
                                                                                                                      
[172.19.3.38] Executing task 'restart_webworkers'                                                      
[172.19.3.41] Executing task '_decommission_host'                                                                    
[172.19.3.41] sudo: sed -i.bak -r -e 's/^([ ]*server[ ]+172.19.3.38)/#\1/g' "$(echo /etc/nginx/sites-available/echis_commcare)"
[172.19.3.41] sudo: nginx -t                                                                                              
[172.19.3.41] out: nginx: [emerg] no servers are inside upstream in /etc/nginx/sites-enabled/echis_commcare:29        
[172.19.3.41] out: nginx: configuration file /etc/nginx/nginx.conf test failed                                             
[172.19.3.41] out:


Fatal error: sudo() received nonzero return code 1 while executing!
Requested: nginx -t
Executed: sudo -S -p 'sudo password:' -H  -u "root"  nginx -t

Aborting.
Traceback (most recent call last):
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
    *args, **kwargs
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
    multiprocessing
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
    return task.run(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 542, in deploy_commcare
    _deploy_without_asking(skip_record)
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 420, in _deploy_without_asking
    silent_services_restart()
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 582, in silent_services_restart
execute(supervisor.restart_webworkers)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
    multiprocessing
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
    return task.run(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
    return func(*args, **kwargs)
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/operations/supervisor.py", line 100, in restart_webworkers
    with decommissioned_host(env.host):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/operations/supervisor.py", line 89, in decommissioned_host
    execute(_decommission_host, host)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
    multiprocessing
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
    return func(*args, **kwargs)
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/operations/supervisor.py", line 66, in _decommission_host
    _check_and_reload_nginx()
  File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/operations/supervisor.py", line 80, in _check_and_reload_nginx
    sudo('nginx -t', shell=False, user='root')
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
    stderr=stderr, timeout=timeout, shell_escape=shell_escape,
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
    error(message=msg, stdout=out, stderr=err)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
    return func(message)
  File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!

Requested: nginx -t
Executed: sudo -S -p 'sudo password:' -H  -u "root"  nginx -t
Disconnecting from 172.19.4.35... done.
Disconnecting from 172.19.3.41... done.
Disconnecting from 172.19.4.42... done.
2

Hello Demis

According to the traceback given (highlighted below), it looks like there is an issue with your nginx config. You can manually run sudo nginx -t command on your proxy/nginx machine to see what the issue is.

3

Hi Sravan,

yes, the nginx test failed due to the change made by fab restart service. it comments on the server 172.19.3.38:8181; as shown below

upstream echis_formplayer {
zone echis_formplayer 256k;

hash $cookie_formplayer_session$http_x_formplayer_session consistent;

#server 172.19.3.38:8181;
}

4

If you have manually edited the nginx, please make sure you haven't broken the configuration. If you want to rollback to normal, you can do cchq <env> ap deploy_proxy.yml

5

Hi Sravan,

the Nginx configuration is not edited manually, and cchq <env> ap deploy_proxy.yml task works properly.

The issue every time I run the script fab restart_services , it tries to edit the nginx config and completes with the above error.

6

Does the config get edited, can you please share what gets updated?

7

upstream echis_formplayer {
zone echis_formplayer 256k;

hash $cookie_formplayer_session$http_x_formplayer_session consistent;
#server 172.19.3.38:8181;
}

this line commented #server 172.19.3.38:8181;

8

@demis08 Do you only have one webworker?

9

No, we have 3 web workers as shown below.

[webworkers:children]
echis_server5
echis_server7
echis_server35

10

@demis08 Have you also followed these steps?

1 Like
11

hello @Sravan_Reddy and @CharlSmit

recently found that the error is not related to the fab script. it's related to 'users` on a single host.

I created a new post here: Getting an error on bootstrap-users