Installation Error: Python 3.12 Compatibility (ImportError: formatargspec) during quick_monolith_install

Hi CommCare Cloud Team,

I am attempting a fresh installation using the quick_monolith_install script on a server running Ubuntu with Python 3.12. I am hitting a blocking error during the "Installing Requirements" step.

The Environment:

  • OS: Ubuntu (likely 24.04 based on Python version)

  • Python Version: 3.12.x

  • Command run: bash cchq-install.sh install-config.yml

The Issue:

Initially, the process was halted by the externally-managed-environment (PEP 668) error. To troubleshoot, I modified the script's attempt to run sudo -H pip3 -q install --upgrade pip by removing the sudo -H prefix and trying to run it within an active virtual environment.

While this bypassed the PEP 668 block, the installer now fails with a Python 3.12 compatibility error:

Traceback (most recent call last):
File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/expand.py", line 185, in read_attr
value = getattr(StaticModule(module_name, spec), attr_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/expand.py", line 77, in getattr
raise AttributeError(f"{self.name} has no attribute {attr}") from e
AttributeError: wrapt has no attribute version

  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
      main()
    File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 175, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 520, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
      exec(code, locals())
    File "<string>", line 33, in <module>
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)  # type: ignore[return-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 160, in setup
      dist.parse_config_files()
    File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/_virtualenv.py", line 23, in parse_config_files
      result = old_parse_config_files(self, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 758, in parse_config_files
      setupcfg.parse_configuration(
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 189, in parse_configuration
      meta.parse()
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 504, in parse
      section_parser_method(section_options)
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 479, in parse_section
      self[name] = value
      ~~~~^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 296, in __setitem__
      parsed = self.parsers.get(option_name, lambda x: x)(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 602, in _parse_version
      return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/setupcfg.py", line 421, in _parse_attr
      return expand.read_attr(attr_desc, package_dir, root_dir)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/expand.py", line 190, in read_attr
      module = _load_spec(spec, module_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-npve17de/overlay/lib/python3.12/site-packages/setuptools/config/expand.py", line 211, in _load_spec
      spec.loader.exec_module(module)
    File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/tmp/pip-install-f7_g2sds/wrapt_8065cc175c7a4ad7a9cf04280e5f64ae/src/wrapt/__init__.py", line 10, in <module>
      from .decorators import (adapter_factory, AdapterFactory, decorator,
    File "/tmp/pip-install-f7_g2sds/wrapt_8065cc175c7a4ad7a9cf04280e5f64ae/src/wrapt/decorators.py", line 34, in <module>
      from inspect import ismethod, isclass, formatargspec
  ImportError: cannot import name 'formatargspec' from 'inspect' (/usr/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 25.3 -> 26.0.1
[notice] To update, run: pip install --upgrade pip
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> wrapt

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
File "/home/azureuser/.virtualenvs/cchq/bin/pip-sync", line 7, in 
sys.exit(cli())
^^^^^
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/click/core.py", line 1485, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/piptools/scripts/sync.py", line 148, in cli
sync.sync(
File "/home/azureuser/.virtualenvs/cchq/lib/python3.12/site-packages/piptools/sync.py", line 263, in sync
run(  # nosec
File "/usr/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,

Observations:

Deprecation Conflict: The ImportError confirms that the installer is attempting to build/install an older version of a dependency (likely wrapt) that relies on inspect.formatargspec.

I also attempted to mitigate this by installing and using Python 3.10; however, the cchq-install.sh script continues to encounter friction. It appears to either default back to system paths or trigger sub-processes that ignore the active virtual environment, leading to the same build failures.

Recommended Solution

Upgrade wrapt to a version compatible with Python 3.12.

Inside your virtual environment:

pip install --upgrade pip setuptools wheel
pip install --upgrade wrapt

Then run:

pip-sync


If pip-sync Forces an Old Version

If your requirements.txt pins an old version:

  1. Open your requirements.in

  2. Add:

wrapt>=1.15.0

  1. Recompile and sync:
pip-compile --upgrade
pip-sync


Alternative (If Project Is Legacy)

If this is an older Django or Celery project, it may not support Python 3.12 yet.

In that case, the clean solution is to downgrade Python:

python3.11 -m venv venv
source venv/bin/activate
pip-sync

Python 3.11 is stable and broadly compatible.


Quick Verification

Check your installed version:

pip show wrapt

or

grep wrapt requirements.txt

If the version is lower than 1.15.0That is the issue.

Thanks for the help!

have downgraded the OS to Ubuntu 22.04, following the specifications in the documentation guide

I am attempting to perform a fresh monolithic installation using cchq-install.sh. Because I have made necessary local modifications to the bootstrap-env-playbook.yml and committed them locally, the script fails at Step 4: Setting up users and SSH auth.

The installer detects that my local branch has diverged from the upstream remote and exits with the following error:

#################################################
Step 4: Setting up users and SSH auth 
#################################################
[172.22.0.4]: fetched key
Adding key: 172.22.0.4 ecdsa-sha2-nistp256
Adding key: 172.22.0.4 ssh-ed25519
Adding key: 172.22.0.4 ssh-rsa
Your local 'master' branch has diverged from upstream
local:  fb2733d7de9f5faae1c43171d8a30dc23bcf3a1a
remote: 5dc59c96ad8ac8a58e287d1f91628ff35c9ef995

Run 'git pull' to get the latest code, then try again.
Create a pull request if you have new commits to contribute.
Or add --branch=master to use your local branch (not recommended).

Steps Taken:

  1. Attempted to use --branch=azure and --branch=master to match the intended deployment target.

  2. Attempted to bypass using --force-branch and --ignore-checks, but the script does not recognize these flags or continues to enforce the git sync.

  3. Attempted setting export CCHQ_SKIP_GIT_CHECK=1 before execution.

Request: Is there a supported environment variable or flag to allow the installer to run using local code modifications without forcing a git pull from the upstream commcare-cloud repository? Specifically, I need to keep my changes to the bootstrap-env-playbook.yml intact during the installation process.