Problem: Compatibility issues with Ansible versions in RHEL8-based environments using Semaphore images

[PopC0rnwalk]

Issue

I’ve encountered an issue while using the latest Semaphore image in a RHEL8-based environment that involves Oracle, Rocky and RHEL devices, which may also affect others who rely on semaphore base images. The challenge arises because RHEL8 is stuck on Python 3.6, which is incompatible with newer versions of Ansible. This creates a conflict when attempting to use the Semaphore image in environments that require ansible-core version 2.16.14 or similar older versions.

Here's what happened in my setup and how I resolved it:

Problem Summary:

• RHEL8 systems use Python 3.6, which limits the Ansible versions they can support.

• Semaphore’s latest image uses a more modern Ansible ansible-core version 2.18.1 that is not backward-compatible with Python 3.6.

• This leads to compatibility issues when using Semaphore in RHEL8-based environments.

• Solution I Implemented:

I created a custom Dockerfile to resolve this issue by:

• Removing the existing Ansible version's virtual environment.
• Installing a specific Ansible version (in this case, 9.13.0) compatible with Python 3.6.

Here’s my custom Dockerfile:

# Use the base Semaphore image
FROM semaphoreui/semaphore:v2.11.2

# Switch to root user to install packages and modify user
USER root

# Renew needed environment variables
ARG TARGETARCH="amd64"
ENV ANSIBLE_VERSION=9.13.0
ARG ANSIBLE_VENV_PATH=/opt/semaphore/apps/ansible/${ANSIBLE_VERSION}/venv
ENV CURRENT_ANSIBLE_VERSION=11.1.0

# Remove any current Ansible version (just the venv folder)
RUN if [ -d "/opt/semaphore/apps/ansible/${CURRENT_ANSIBLE_VERSION}" ]; then \
        rm -rf /opt/semaphore/apps/ansible/${CURRENT_ANSIBLE_VERSION}/venv; \
    fi

# Install build dependencies and create the new virtual environment for Ansible version 9.13.0
RUN apk add --no-cache -U python3-dev build-base openssl-dev libffi-dev cargo && \
    mkdir -p ${ANSIBLE_VENV_PATH} && \
    python3 -m venv ${ANSIBLE_VENV_PATH} --system-site-packages && \
    source ${ANSIBLE_VENV_PATH}/bin/activate && \
    # Install Ansible 9.13.0 and required dependencies
    pip3 install --upgrade pip ansible==${ANSIBLE_VERSION} boto3 botocore requests pywinrm && \
    # Clean up build dependencies and temporary files
    apk del python3-dev build-base openssl-dev libffi-dev cargo && \
    rm -rf /var/cache/apk/* && \
    find ${ANSIBLE_VENV_PATH} -iname __pycache__ | xargs rm -rf && \
    chown -R semaphore:0 /opt/semaphore

# Switch back to the semaphore user
USER semaphore

# Configure environment variables for the Ansible virtual environment
ENV VIRTUAL_ENV="$ANSIBLE_VENV_PATH"
ENV PATH="${ANSIBLE_VENV_PATH}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Suggested Improvements:

Would it be possible to implement a configurable environment variable for the Semaphore image to allow users to define the Ansible version at runtime? For instance:

Maintain two Ansible versions in the image (e.g., 11.0.1 and 9.13.0).
Allow users to select the desired version using an environment variable like ANSIBLE_VERSION.
This way, users in environments with legacy Python versions (like RHEL8) can still use Semaphore latest image.

Please let me know if additional details or clarifications are needed.

Thanks for your excellent work!

Impact

Ansible (task execution)

Installation method

Docker

Database

MySQL

Browser

Chrome

Semaphore Version

6b52b698c6a8:~$ semaphore version
v2.11.2-0e9490c-1735214878

Logs & errors

3:39:27 PM
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: future feature annotations is not defined
3:39:27 PM
fatal: [rhel8-test.nw.domain.com]: FAILED! => changed=false 
3:39:27 PM
  module_stderr: |-
3:39:27 PM
    Shared connection to xx.xx.xx.xx closed.
3:39:27 PM
  module_stdout: |-
3:39:27 PM
    Traceback (most recent call last):
3:39:27 PM
      File "<stdin>", line 12, in <module>
3:39:27 PM
      File "<frozen importlib._bootstrap>", line 971, in _find_and_load
3:39:27 PM
      File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
3:39:27 PM
      File "<frozen importlib._bootstrap>", line 894, in _find_spec
3:39:27 PM
      File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
3:39:27 PM
      File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
3:39:27 PM
      File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
3:39:27 PM
      File "<frozen importlib._bootstrap>", line 441, in spec_from_loader
3:39:27 PM
      File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location
3:39:27 PM
      File "/tmp/ansible_ansible.legacy.dnf_payload_jzk6c83a/ansible_ansible.legacy.dnf_payload.zip/ansible/module_utils/basic.py", line 5
3:39:27 PM
    SyntaxError: future feature annotations is not defined
3:39:27 PM
  msg: |-
3:39:27 PM
    MODULE FAILURE: No start of json char found
3:39:27 PM
    See stdout/stderr for the exact error
3:39:27 PM
  rc: 1

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[bbaassssiiee]

Alternatively you could choose the version of ansible-core, and load the python requiremennts.txt with a timer, and only collections required from the projects' collections/requirements.yml, and likewise roles from roles/requirenments.yml.

A SemaphoreUI implementation that caters RHEL8 variant AlmaLinux8 https://github.com/playingfield/controller

[mlanner]

I've run into the same issue after many other problems. I did a ton of work to upgrade/shift my Semaphore+Ansible control server from Rocky Linux 9 to Ubuntu 24.04 in order to get a newer version of Ansible (from PPA repos), which would solve for the issues the outdated legacy version in RL9 was causing. Then I ran into all kinds of Python module and Ansible collections issues. Ultimately I ended up removing the native packages and installing Ansible using pip (using UV's pip compatibility layer). In the process of all of this I also learned about Ansible's new "Execution Environment" implementation and what seems like the path they'll be taking. Maybe needless to say, but I think that running Ansible in a dedicated container(s) that you can modify as/when needed to add all the collections and requirements you need, would be the more sane way of doing it than my somewhat hacky UV/pip way.

All that said -- sorry for the lengthy explanation and backstory -- in my case, once everything was running I then tried to add playbooks for RHEL 8 (derivatives) targets and found out that my new fancy Ansible, ansible [core 2.18.1], can't really work with RHEL 8 due to the old Python version in RHEL 8. (https://www.jeffgeerling.com/blog/2024/newer-versions-ansible-dont-work-rhel-8)

So, along the lines of @PopC0rnwalk's request, it would be great to be able to have environment options for different targets. In my case it appears that the "right" way would be to have a solution like the Execution Environment concept that Ansible is now pushing. I'll wait to see if anyone responds to this here, but I'm inclined to make a feature request issue to support the concept of Execution Environments. (https://docs.ansible.com/ansible/latest/getting_started_ee/index.html)

[fiftin]

Hi @mlanner did you solve the issue?

I agree with you. We need environment variable to choose Ansible version for container. It will be done in next release. But I'm not sure what to do with current version.

Should I rollback Ansible to v2.16?

UPDATE:
Problem with runtime Ansible installation. Ansible requires several huge dev packages for installation. Image will have size 1.6 Gb if we include them to image. It is a reason why Ansible built-in to image.

[fiftin]

You can create custom image with custom Ansible version with using following Dockerfile:

FROM semaphoreui/semaphore:v2.11.2

ENV ANSIBLE_VERSION=9.4.0
ARG ANSIBLE_VENV_PATH=/opt/semaphore/apps/ansible/${ANSIBLE_VERSION}/venv


USER root

RUN apk add --no-cache -U python3-dev build-base openssl-dev libffi-dev cargo


USER 1001

RUN mkdir -p ${ANSIBLE_VENV_PATH} && \
     python3 -m venv ${ANSIBLE_VENV_PATH} --system-site-packages && \
     source ${ANSIBLE_VENV_PATH}/bin/activate && \
     pip3 install --upgrade pip ansible==${ANSIBLE_VERSION} boto3 botocore requests pywinrm

ENV VIRTUAL_ENV="$ANSIBLE_VENV_PATH"
ENV PATH="$ANSIBLE_VENV_PATH/bin:$PATH"

[fiftin]

I think we will have several images for different Ansible versions.

[bbaassssiiee]

Why not change it to a version of ansible-core? Less bloated, more choice. Semaphore can load collecctions/requirements.yml

[mlanner]

Hey @fiftin and thanks for the follow-up. I didn't quite solve it all the way yet, but little-by-little I've been chasing down my own problem of not being able to target RHEL 8 (and derivates) nodes. Thanks for the suggestion around the Docker image. I actually started there a week or ten days ago, but the integration I needed to get it working with Semaphore seemed overwhelming to me at the time. So, in the meantime I decided to build a dedicated test machine to just target the RL8 nodes.

Because my backstory was already long, I left out some details. For my RL8 nodes I also need to be able to target nodes in Azure. That's actually where this story started. That's where everything broke; old this, old that, incompatible version of Ansible, too old Azure collection, etc. And then the upgrade cycle began and took me on the path that I outlined earlier.

So, with all that said, yes, I think being able to have "custom" Ansible containers that can be specifically used for a job in a given environment would be the answer to this long term. I have no idea how I would do that with Semaphore today, though. Basically, I think what I'm describing is really the Ansible Execution Environment idea. Maybe there are a few items that would go into this:

1. Custom built containers for different environments. This is the same idea (I think, if I understand it correctly) as the new Ansible Builder (https://ansible.readthedocs.io/projects/builder/en/latest/index.html) concept.
2. Having a way in Semaphore to call a given Ansible container for various execution environments that have specific requirements (like my Azure + RL8 situtation).

Ultimately, though, I think it would be tedious and inefficient for everyone to build their own containers. Maybe publishing a set of vanilla Ansible containers, by Ansible version, would be a good starting point to make it easier. In my case I'd need a specific version of Ansible to solve for Azure + RL8, but then I'd also need to layer in the Azure collection into that container. Of course, a lot of people won't have the problem I have, so I wouldn't expect there to be a ready-made container like that.

[mlanner]

@bbaassssiiee That's more or less the way I went. Although, I haven't fully finished it yet. That said, I think the current state of things makes it a bit more complex than it needs to be. Granted, that could also be because of my shortcomings when it comes to Ansible and my biased opinion about pip. 😉

[mlanner]

I think we will have several images for different Ansible versions.

@fiftin Just another thought on this ... Would it be possible to separate out the Ansible container(s) from the main Semaphore container? Maybe as an "advanced" deployment option, kind of using the idea behind Ansible Execution Environments? One idea would then be to have a "tab" in the project menu, similar to how Inventory and Repositories* work and then specify the Execution Environment, allowing the Semaphore user/admin to set which Ansible container it should use for the project and/or even have range of options that could be picked on a task template basis. There could be one global environment that would be available as the default if no other Execution Environments have been created and added.

I realize this is now more of a feature request. So, maybe this should be a different issue?