arch/x86/crypto/aes-gcm-vaes-avx2.S

Source file repositories/reference/linux-study-clean/arch/x86/crypto/aes-gcm-vaes-avx2.S

File Facts

System
Linux kernel
Corpus path
arch/x86/crypto/aes-gcm-vaes-avx2.S
Extension
.S
Size
39506 bytes
Lines
1148
Domain
Architecture Layer
Bucket
arch/x86
Inferred role
Architecture Layer: arch/x86
Status
atlas-only

Why This File Exists

CPU and platform-specific kernel glue: boot entry, traps, syscall entry, interrupts, page tables, context switch, and low-level barriers.

Dependency Surface

Detected Declarations

Annotated Snippet

//
// AES-GCM implementation for x86_64 CPUs that support the following CPU
// features: VAES && VPCLMULQDQ && AVX2
//
// Copyright 2025 Google LLC
//
// Author: Eric Biggers <ebiggers@google.com>
//
//------------------------------------------------------------------------------
//
// This file is dual-licensed, meaning that you can use it under your choice of
// either of the following two licenses:
//
// Licensed under the Apache License 2.0 (the "License").  You may obtain a copy
// of the License at
//
//	http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// or
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
//
// 1. Redistributions of source code must retain the above copyright notice,
//    this list of conditions and the following disclaimer.
//
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
// POSSIBILITY OF SUCH DAMAGE.
//
// -----------------------------------------------------------------------------
//
// This is similar to aes-gcm-vaes-avx512.S, but it uses AVX2 instead of AVX512.
// This means it can only use 16 vector registers instead of 32, the maximum
// vector length is 32 bytes, and some instructions such as vpternlogd and
// masked loads/stores are unavailable.  However, it is able to run on CPUs that
// have VAES without AVX512, namely AMD Zen 3 (including "Milan" server CPUs),
// various Intel client CPUs such as Alder Lake, and Intel Sierra Forest.
//
// This implementation also uses Karatsuba multiplication instead of schoolbook
// multiplication for GHASH in its main loop.  This does not help much on Intel,
// but it improves performance by ~5% on AMD Zen 3.  Other factors weighing
// slightly in favor of Karatsuba multiplication in this implementation are the
// lower maximum vector length (which means there are fewer key powers, so we
// can cache the halves of each key power XOR'd together and still use less
// memory than the AVX512 implementation), and the unavailability of the
// vpternlogd instruction (which helped schoolbook a bit more than Karatsuba).

#include <linux/linkage.h>

.section .rodata
.p2align 4

Annotation

Implementation Notes