Upgrade Proposal for SLURM 26.5.x

[chriswasser]

Hi everyone,

After we upgraded our cluster to SLURM 26.05.2, I implemented minor changes within pyslurm to make it compatible with the recent SLURM C API changes. Unfortunately, I was not able to use the described upgrade procedure in UPGRADE_C_API.rst due to the AssertionError appended below. Note that I also had to use an older version of autopxd2==2.5.0 as the 3.x versions changed the packages's API (see AttributeError below). However, my scripts already worked with the following small changes:

• many SLURM API functions were changed from taking uint32_t job_id to taking slurm_step_id_t job_id. The old uint32_t job_id are still available with the suffix _jid. Depending on how you want to move forward with pyslurm exposing both functions might make sense. For now, I kept the old behavior and reimplemented the functionality of the _jid-suffixed functions in cython.
• slurm_free_ctl_conf was renamed to slurm_free_conf
• enum constant ESLURM_ERROR_ON_DESC_TO_RECORD_COPY was replaced by ESLURM_MAX_JOB_COUNT
• the job_resources struct received a new member node_ranks

I don't think these changes should immediately be integrated into the main branch as I wasn't able to perform extensive testing, but I hope they can be the basis for a new 26.5.x branch. Thanks for your continued development of this awesome library 🤓👌

Best,

Christian

• The error with autopxd2==2.5.0:

Traceback (most recent call last):
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 298, in <module>
    main()
    ~~~~^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1569, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1490, in main
    rv = self.invoke(ctx)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1353, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 907, in invoke
    return callback(*args, **kwargs)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 293, in main
    translate_slurm_header(slurm_header_dir, "slurm.h")
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 113, in translate_slurm_header
    codegen.visit(
    ~~~~~~~~~~~~~^
        autopxd.parse(
        ^^^^^^^^^^^^^^
    ...<3 lines>...
        )
        ^
    )
    ^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 123, in visit
    rv = super().visit(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 190, in visit
    return visitor(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 197, in generic_visit
    self.visit(c)
    ~~~~~~~~~~^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 123, in visit
    rv = super().visit(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 190, in visit
    return visitor(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 301, in visit_Typedef
    decls = self.collect(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 319, in collect
    self.generic_visit(node)
    ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 197, in generic_visit
    self.visit(c)
    ~~~~~~~~~~^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 123, in visit
    rv = super().visit(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 190, in visit
    return visitor(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 235, in visit_TypeDecl
    decls = self.collect(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 319, in collect
    self.generic_visit(node)
    ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 197, in generic_visit
    self.visit(c)
    ~~~~~~~~~~^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 123, in visit
    rv = super().visit(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/pycparser/c_ast.py", line 190, in visit
    return visitor(node)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 188, in visit_Enum
    value_as_str, maybe_value_as_int = parse_enum_value(item.value, self.constants)
                                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 90, in parse_enum_value
    left_value_as_str, _ = parse_enum_value(node.left, constants)
                           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/autopxd/writer.py", line 107, in parse_enum_value
    assert False, f"Unsuported expression for enum value: {node}"
           ^^^^^
AssertionError: Unsuported expression for enum value: Cast(to_type=Typename(name=None,
                      quals=[
                            ],
                      align=None,
                      type=TypeDecl(declname=None,
                                    quals=[
                                          ],
                                    align=None,
                                    type=IdentifierType(names=['uint64_t'
                                                              ]
                                                        )
                                    )
                      ),
     expr=Constant(type='int',
                   value='1'
                   )
     )

• The error with autopxd2 versions 3.x:

Traceback (most recent call last):
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 298, in <module>
    main()
    ~~~~^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1569, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1490, in main
    rv = self.invoke(ctx)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 1353, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/.venv/lib/python3.13/site-packages/click/core.py", line 907, in invoke
    return callback(*args, **kwargs)
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 292, in main
    translate_slurm_header(slurm_header_dir, "slurm_errno.h")
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rwthfs/rz/cluster/home/cw585968/Desktop/GitHub/pyslurm/scripts/pyslurm_bindgen.py", line 112, in translate_slurm_header
    codegen = autopxd.AutoPxd("slurm/" + hdr)
              ^^^^^^^^^^^^^^^
AttributeError: module 'autopxd' has no attribute 'AutoPxd'

[chriswasser]

Additional info on the autopxd2 issues: The parsing problem of 2.5.0 is probably related to the underlying pycparser backend. It does not seem to like SLURM's usage of the SLURM_BIT macro within the enum value (see here). Since autopxd2 versions 3.x a new libclang backend is supported. Manually invoking autopxd on the command line (and not through the pyslurm_bindgen.py script) worked fine for the most recent version autopxd2==3.2.3 (it worked for both backends though).

[giovtorres]

Thank you for this! I have a PR (#426) that referenced this change and added a few things that were missing. I’m running it through testing to see if I missed anything.

[chriswasser]

Great, thanks 🎉 Then I guess I can close this PR here, right?

[giovtorres]

Great, thanks 🎉 Then I guess I can close this PR here, right?

I think so. You could also test if #426 builds cleanly and works for you on your cluster. So far tests have passed, but i want to test and clean up the PR a bit more before releasing.