tools/unittests/test_tokenizer.py

File Facts

System

Linux kernel

Corpus path

tools/unittests/test_tokenizer.py

Extension

.py

Size

14089 bytes

Lines

470

Domain

Support Tooling And Documentation

Bucket

tools

Inferred role

Support Tooling And Documentation: tools

Status

atlas-only

Why This File Exists

Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.

• Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.
• Defines or uses C structs; map object ownership, embedded links, reference counts, and lock ownership.

Dependency Surface

• No C-style include directives detected by the generator.

Detected Declarations

• struct X
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct foo
• struct page_pool_params
• struct page_pool_params
• struct page_pool_params
• struct page_pool_params
• struct page_pool_params
• struct page_pool_params

Annotated Snippet

struct X {
                int arr[10];
                func(a[0], (b + c));
            }
        """,
        "expected": [
            CToken(CToken.STRUCT, "struct"),
            CToken(CToken.NAME, "X"),
            CToken(CToken.BEGIN, "{", brace_level=1),

            CToken(CToken.NAME, "int", brace_level=1),
            CToken(CToken.NAME, "arr", brace_level=1),
            CToken(CToken.BEGIN, "[", brace_level=1, bracket_level=1),
            CToken(CToken.NUMBER, "10", brace_level=1, bracket_level=1),
            CToken(CToken.END, "]", brace_level=1),
            CToken(CToken.ENDSTMT, ";", brace_level=1),
            CToken(CToken.NAME, "func", brace_level=1),
            CToken(CToken.BEGIN, "(", brace_level=1, paren_level=1),
            CToken(CToken.NAME, "a", brace_level=1, paren_level=1),
            CToken(CToken.BEGIN, "[", brace_level=1, paren_level=1, bracket_level=1),
            CToken(CToken.NUMBER, "0", brace_level=1, paren_level=1, bracket_level=1),
            CToken(CToken.END, "]", brace_level=1, paren_level=1),
            CToken(CToken.PUNC, ",", brace_level=1, paren_level=1),
            CToken(CToken.BEGIN, "(", brace_level=1, paren_level=2),
            CToken(CToken.NAME, "b", brace_level=1, paren_level=2),
            CToken(CToken.OP, "+", brace_level=1, paren_level=2),
            CToken(CToken.NAME, "c", brace_level=1, paren_level=2),
            CToken(CToken.END, ")", brace_level=1, paren_level=1),
            CToken(CToken.END, ")", brace_level=1),
            CToken(CToken.ENDSTMT, ";", brace_level=1),
            CToken(CToken.END, "}"),
        ],
    },

    "mismatch_error": {
        "source": "int a$ = 5;",          # $ is illegal
        "log_msg": "Unexpected token",
    },
}

def make_private_test(name, data):
    """
    Create a test named ``name`` using parameters given by ``data`` dict.
    """

    def test(self):
        """In-lined lambda-like function to run the test"""
        tokens = CTokenizer(data["source"])
        result = str(tokens)

        #
        # Avoid whitespace false positives
        #
        result = re.sub(r"\s++", " ", result).strip()
        expected = re.sub(r"\s++", " ", data["trimmed"]).strip()

        msg = f"failed when parsing this source:\n{data['source']}"
        self.assertEqual(result, expected, msg=msg)

    return test

#: Tests to check if CTokenizer is handling properly public/private comments.
TESTS_PRIVATE = {
    #
    # Simplest case: no private. Ensure that trimming won't affect struct
    #
    "__run__": make_private_test,
    "no private": {
        "source": """
            struct foo {

Annotation

• Detected declarations: `struct X`, `struct foo`, `struct foo`, `struct foo`, `struct foo`, `struct foo`, `struct foo`, `struct foo`, `struct foo`, `struct foo`.
• Atlas domain: Support Tooling And Documentation / tools.
• Implementation status: atlas-only.

Implementation Notes

• This generated page is the file-by-file coverage layer; curated subsystem chapters should link here when they synthesize a multi-file control flow.
• Core OS pages should be promoted from atlas-only to deep-reviewed when they explain data structures, invariants, locking, lifecycle, and C implementation snippets.
• Driver-family pages are intentionally pattern-oriented unless they are part of the selected PCIe/NVMe representative device path.