I am trying to use the open source python toolkit, "Aequitas" (GitHub - dssg/aequitas: Bias Auditing & Fair ML Toolkit). One of the functions from the package has a GLIBC-2.29 dependency that my Posit environment won't support (RHEL 8 can only support up to 2.28 (What glibc version is shipped with a particular Red Hat Enterprise Linux? - Red Hat Customer Portal)).
I installed one of the gcc-toolset packages (gcc-toolset-9) available for RHEL 8 to attempt to access a more recent version of GLIBC and was able to download the tar.gz file and create a glibc-2.29/build folder. I have tried a few ways to link it but keep running into errors. Can anyone advise on code that can run in python to use the glibc-2.29 from the toolset for the necessary package? Code for installing a GCC toolset to download, build, and use glibc 2.29 locally follows:
! yum install gcc-toolset-9
import os
import subprocess
import sys
# Define paths
glibc_version = '2.29'
glibc_tarball_url = f'http://ftp.gnu.org/gnu/libc/glibc-{glibc_version}.tar.gz'
glibc_src_dir = f'glibc-{glibc_version}'
build_dir = f'{glibc_src_dir}/build'
install_dir = os.path.expanduser(f'~/glibc-{glibc_version}-install')
# Function to run shell commands
def run_command(command, cwd=None, check=True):
print(f"Running command: {command}")
result = subprocess.run(command, shell=True, cwd=cwd, check=check, text=True, capture_output=True)
print(result.stdout)
if result.stderr:
print(f"Error: {result.stderr}", file=sys.stderr)
return result
# Create a working directory
print(f"Creating working directory...")
run_command(f'mkdir -p ~/glibc-{glibc_version}', check=False)
# Download the glibc source code
print(f"Downloading glibc {glibc_version}...")
run_command(f'wget {glibc_tarball_url}', cwd=os.path.expanduser(f'~/glibc-{glibc_version}'), check=True)
# Extract the source code
print(f"Extracting glibc {glibc_version}...")
run_command(f'tar -xzf glibc-{glibc_version}.tar.gz', cwd=os.path.expanduser(f'~/glibc-{glibc_version}'), check=True)
# Create a build directory
print(f"Creating build directory...")
run_command(f'mkdir -p {build_dir}', cwd=os.path.expanduser(f'~/glibc-{glibc_version}'), check=True)
# Configure the build
print(f"Configuring the build...")
run_command(f'../configure --prefix={install_dir}', cwd=build_dir, check=True)
Upgrading to GLIBC 2.29 looks like the obvious choice here but is not the solution here. GLIBC is so deeply entrenched into an OS that upgrading the same without upgrading the whole OS does typically not end very well...
The main culprit here is fairgbm whose available python wheel has been compiled against GLIBC 2.29. Python wheels are great because they already contain binaries and hence you do not need to recompile from source. But if they are compiled against a too recent GLIBC you're stuck.
You however can force pip to NOT use an existing wheel for certain packages and instead build a wheel itself. In the case of fairgbm / aequitas you can do this via
pip install --no-binary fairgbm aequitas
If you use the approach above I think you no longer need any gcc-toolset either (you however can go up to gcc-toolset-13 (at least this is available in RockyLinux 8).
PS: You will ensure to install cmake to make the above work. fairgbm uses cmake for building the software.
1 Like
Thank you, Michael. This is helpful insight. Are you able to help me figure out where to integrate your pip install recommendation into my current code?:
from aequitas import Audit
import pandas as pd
import seaborn as sns
from aequitas.audit import Audit
import aequitas.plot as ap
import os
import pyodbc
import matplotlib.pyplot as plt
import plotly.express as px
from aequitas.group import Group
from aequitas.bias import Bias
from aequitas.fairness import Fairness
from aequitas.plotting import Plot
from aequitas import plot
from aequitas.flow.utils.logging import clean_handlers
from aequitas.flow.utils.colab import get_examples
version('fairGBM')
from aequitas.flow import Experiment
from aequitas.flow.methods.postprocessing import BalancedGroupThreshold
I am not sure how you installed aequitas in the first place but the easiest IMHO would be to remove aequitas and start the installation with the command provided (maybe create a new virtual env). Alternatively you coud run in your jupyter notebook
!pip install --force-reinstall --no-binary fairgbm fairgbm
which would force reinstall of fairgbm + dependencies. There is a small chance that it may mess up some dependencies for aequitas so the "start fresh" approach feels better to me.
Thank Michael! I am working on this project as well and running into an error. It says I need to install Cmake and dependencies - I have done this but still getting the same thing.
After trying to install it this way, I am getting the following error:
Sorry to hear about your trouble.
- Which OS are you working under ? RHEL 8 as well ?
- How did you install
cmake ?
- What does
cmake --version report on your system ?
- Do you find any additional errors or information in
/hhc_home/lees149/FairGBM_compilation.log ?
My gut is telling me this is a path issue, but I don't know what to do about it
-
Which OS are you working under ? RHEL 8 as well ?
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="Red Hat Enterprise Linux | Red Hat Product Documentation"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"
-
How did you install cmake ?
Pip install cmake
I have also tried to use ! Yum install - y cmake but need admin privileges
-
What does cmake --version report on your system ?
It can't find it
-
Do you find any additional errors or information in /hhc_home/lees149/FairGBM_compilation.log ?
The log looks empty?
Oh, and another in the same output- setup.py install is depreciated?
Ok, thanks for providing all of this data !
I am guessing that you installed cmake via pip outside of any virtual environment, right ? In this case you will need to add the directory (presumably /hhc_home/lees149/.local/bin) to your PATH environment variable. e.g. run
export PATH=/hhc_home/lees149/.local/bin:$PATH
and see if cmake can be found then.
While this may actually work, I would recommend to use a dedicated python venv for your fairGBM/aequitas work. More specifically I would suggest to create a project folder for your aequitas work, and within that folder run
python -m venv .venv
source .venv/bin/activate
pip install cmake
pip install --no-binary fairgbm aequitas
deactivate
The above commands should work . Now, every time you wanrt to work with aequitas you simply need to go back into the project folder, run source .venv/bin/activate and you have all your aequitas/fairGBM stuff available again. Of course you also can register the new virtual environment in jupyter/jupyterlab for better integration - I will leave that up to you as an exercise.
The benefits of using virtual envs are that you really can make them project specific and avoid overwriting or inadvertently updating a package like you would if you would install everything in one user default library path.
Let me know how this goes.
Hi Michael,
Thanks so much for this! We were able to get this up and running 
I ended up needing to run the following in addition to what you sent -
import os
Step 1: Expand the '~' to the full home directory path
home_dir = os.path.expanduser("~")
install_dir = f"{home_dir}/bin"
cmake_dir = f"{install_dir}/cmake"
Step 2: Create the directory if it doesn't exist
!mkdir -p {cmake_dir}
Step 3: Download and install CMake to the chosen directory
Replace 'cmake-install.sh' with the actual path to your CMake installer script if different
!sh cmake-install.sh --prefix={cmake_dir} --skip-license
Step 4: Add CMake to the PATH for this session
os.environ["PATH"] = f"{cmake_dir}/bin:" + os.environ["PATH"]
Step 5: Verify that CMake is correctly installed and accessible
!which cmake
!cmake --version
Optional: Create a symbolic link in the install_dir if necessary
This is in case cmake wasn't installed directly into install_dir
!ln -sf {cmake_dir}/bin/cmake {install_dir}/cmake
Verify again that cmake is accessible
!which cmake
!cmake --version
1 Like
Awesome - I am really glad you figured this out ! Hope you are able to use aequitas and and produce great scientific results out of it !
