skip to primary navigationskip to content
 

CPU affinity and dynamic slots

Update 16-1-2013: the issue of CPU affinity for dynamic slots has apparently been addressed in the development release 7.9.3 by setting the new configuration variable ASSIGN_CPU_AFFINITY to True .

It is useful on a multi-core machine to force jobs to only use the resources allocated to them, be that memory, disk space, etc. Most of these resources can be catered for using a wrapper script (see the section titled "Limiting Resource Usage" in the Implementation and Configuration page), but as of the current stable release (7.8), Condor only enforces CPU affinity, i.e. where jobs are pinned to specific cores, for static slots (see here in the manual and this thread in the Condor-users mailing list). This is a bind for those of us who want to embrace dynamic slots, so we attempt to rectify that here by using a USER_JOB_WRAPPER script to force the required behaviour.

Different universes, different behaviour

The first problem one encounters is that the information available on the execute host to any wrapper script depends heavily on the universe being used. Specifically, the Standard universe does not produce job and machine classad files (given by the environment variables _CONDOR_[JOB|MACHINE]_AD). The one useful environment variable it does load is _CONDOR_SLOT, which allows us to interrogate the Collector for the required information. With that in mind, below is the script that I got to work. It calls the Linux taskset command directly, and will work for dynamic or static slots, and with "any" universe (tested with Vanilla, Java and Standard), though as of writing it could do with more testing, i.e. I'd be grateful for any feedback concerning improvements. I realise that it can be hardened in a number of places, namely for when certain calls may fail, e.g. to lockfile, and time permitting I'll get round to these. Note that we do not use a shared file system anywhere. Finally, do not use Condor's ENFORCE_CPU_AFFINITY option if using this script, even with static slots, as the script doesn't test for the latter's presence.

Caveats

The wrapper will create a file with a list of allocated cores in each scratch directory, and synchronisation for accessing such files between different competing jobs is done via lock files, which necessitates a sticky bit being set on the scratch directories as potentially different jobs will be running as different users. If you consider this aspect at odds with your security model then please walk on by. Furthermore, you'll have to set the variable CONDOR_HOME to point at the location of the Condor installation on the execute host. Finally, you'll obviously need to add any other actions you want such a wrapper script to perform, e.g. limit memory and/or disk usage.

#!/bin/bash

CONDOR_HOME=### CHANGE THIS! ####
CORE_FILE=.cores.reserved
EXECUTE=`$CONDOR_HOME/bin/condor_config_val EXECUTE`

###############################################################################
function get_info {

  # Interrogate the Collector for job and machine characteristics.

  HOSTNAME=`/bin/hostname -f`
  SLOT=$_CONDOR_SLOT
  out=""
  loop_count=0
  max_loops_allowed=5

  while [ "$out" == "" ];
  do
    out=`$CONDOR_HOME/bin/condor_status -constraint Name==\"$SLOT@$HOSTNAME\" -format "%d " Cpus -format "%d " TotalCpus -format "%d\n" JobUniverse`

    if [ "$out" == "" ]; then
      $((loop_count ++))

      if [ $loop_count -gt $max_loops_allowed ]; then
         echo "Failed to get response from Collector. Gah" 
exit -1 fi # Wait for the Collector to get updated sleep 3 fi done echo $out } ############################################################################### function get_dir { here=`pwd` IFS="/" set $here thisDir="${!#}" echo $thisDir } ############################################################################### function get_core_list { cores_requested=$1 total_cores=$2 universe=$3 # Don't actually use this, but was useful in debugging # Construct a mask of available cores. Assume all are at first. for i in $(seq 0 $(($total_cores-1))) do reserved_cores[$i]=0 done array=() dir_stub=$(get_dir) for entry in `/bin/ls $EXECUTE` do if [ ! -d "$EXECUTE/$entry" ]; then # Not a directory (huh?), so ignore. continue fi skip=0 if [ "$dir_stub" != "$entry" ]; then core_file=$EXECUTE/$entry/$CORE_FILE lock_file=$core_file.lock if [ ! -e $core_file ]; then # Deadlock! Break it arbitrarily. if [[ $dir_stub < $entry ]]; then # Skip this directory skip=1 else while [ ! -e $core_file ]; do sleep 1 # Other job may conceviably have finished by now, so check if [ ! -d "$EXECUTE/$entry" ]; then skip=1 break fi done fi fi if [ $skip -ne 0 ]; then continue fi while [ -e $lock_file ]; do sleep 2 done lockfile $lock_file cores=( $( cat $core_file ) ) for i in $(seq 0 $((${#cores[@]} - 1))) do reserved_cores[${cores[$i]}]=1 done # Add this lock file array=( "${array[@]}" $lock_file ) fi done cores_found=0 the_cores=() for i in $(seq 0 $(($total_cores-1))) do if [ ${reserved_cores[$i]} -eq 0 ]; then ((cores_found ++)) the_cores=( "${the_cores[@]}" $i ) if [ $cores_found -eq $cores_requested ]; then set_string="" core_file=`pwd`/$CORE_FILE lock_file=$core_file.lock lockfile $lock_file array=( "${array[@]}" $lock_file ) for j in $(seq 0 $(($cores_found-1))) do echo ${the_cores[$j]} >> $core_file set_string=$set_string"${the_cores[$j]}," done # Remove the last comma set_string="${set_string%?}" break fi fi done for file in "${array[@]}" do rm -f $file done echo $set_string } ############################################################################### # Need this to allow different users to save state info in this scratch dir. chmod 1777 . if [[ $_CONDOR_MACHINE_AD == "" ]] || [[ $_CONDOR_JOB_AD == "" ]]; then # Ouch. Probably a standard universe job, so need to ask the Collector for info. resource_list=$(get_info) else cores_requested=`egrep '^RequestCpus' $_CONDOR_JOB_AD | cut -d ' ' -f 3` total_cores=`egrep '^TotalCpus' $_CONDOR_MACHINE_AD | cut -d ' ' -f 3` universe=`egrep '^JobUniverse' $_CONDOR_JOB_AD | cut -d ' ' -f 3` resource_list="$cores_requested $total_cores $universe" fi core_list=$(get_core_list $resource_list) exec taskset --cpu-list $core_list "$@"