tools/lib/python/jobserver.py

Source file repositories/reference/linux-study-clean/tools/lib/python/jobserver.py

File Facts

System
Linux kernel
Corpus path
tools/lib/python/jobserver.py
Extension
.py
Size
6595 bytes
Lines
196
Domain
Support Tooling And Documentation
Bucket
tools
Inferred role
Support Tooling And Documentation: tools
Status
atlas-only

Why This File Exists

Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.

Dependency Surface

Detected Declarations

Annotated Snippet

#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# pylint: disable=C0103,C0209
#
#

"""
Interacts with the POSIX jobserver during the Kernel build time.

A "normal" jobserver task, like the one initiated by a make subprocess would do:

    - open read/write file descriptors to communicate with the job server;
    - ask for one slot by calling::

        claim = os.read(reader, 1)

    - when the job finishes, call::

        os.write(writer, b"+")  # os.write(writer, claim)

Here, the goal is different: This script aims to get the remaining number
of slots available, using all of them to run a command which handle tasks in
parallel. To to that, it has a loop that ends only after there are no
slots left. It then increments the number by one, in order to allow a
call equivalent to ``make -j$((claim+1))``, e.g. having a parent make creating
$claim child to do the actual work.

The end goal here is to keep the total number of build tasks under the
limit established by the initial ``make -j$n_proc`` call.

See:
    https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
"""

import errno
import os
import subprocess
import sys

def warn(text, *args):
    print(f'WARNING: {text}', *args, file = sys.stderr)

class JobserverExec:
    """
    Claim all slots from make using POSIX Jobserver.

    The main methods here are:

    - open(): reserves all slots;
    - close(): method returns all used slots back to make;
    - run(): executes a command setting PARALLELISM=<available slots jobs + 1>.
    """

    def __init__(self):
        """Initialize internal vars."""
        self.claim = 0
        self.jobs = b""
        self.reader = None
        self.writer = None
        self.is_open = False

    def open(self):
        """Reserve all available slots to be claimed later on."""

        if self.is_open:
            return
        self.is_open = True  # We only try once
        self.claim = None
        #

Annotation

Implementation Notes