Object Cache

CV-CUDA includes an internal resource management system that caches allocated objects for efficient reuse. Objects used within CV-CUDA, such as cvcuda.Image, cvcuda.Tensor, cvcuda.ImageBatchVarShape, and cvcuda.TensorBatch, are automatically managed by the CV-CUDA cache.

Note

Only Python objects are cached, there is no C/C++ object caching.

Note

Cache limits and byte accounting are per CUDA device and shared across threads.

Wrapped vs Non-Wrapped Objects

The cache distinguishes between two types of objects based on memory ownership:

Non-wrapped objects are allocated by CV-CUDA and increase the cache size:

import cvcuda
import numpy as np


def main() -> None:
    # Non-wrapped object that increases cache memory
    tensor = cvcuda.Tensor(  # noqa: F841
        (16, 32, 4), np.float32, cvcuda.TensorLayout.HWC
    )

Wrapped objects wrap externally-managed memory and do not increase the cache size:

import cvcuda
import torch


def main() -> None:
    # Wrapped object that does not increase cache memory
    torch_tensor = torch.tensor([1], device="cuda", dtype=torch.uint8)
    cvcuda_tensor = cvcuda.as_tensor(torch_tensor, layout="N")  # noqa: F841

Del and Garbage Collection

Both cvcuda.Image and cvcuda.Tensor are managed by CV-CUDA if they are allocated via CV-CUDA. When a cvcuda.Tensor or cvcuda.Image has been allocated by CV-CUDA and it goes out of scope, the underlying memory is not released. Instead, it is stored and will be reused for future allocations via the CV-CUDA object cache.

For example, when using the del keyword in Python, only the reference to the object is removed. Reuse requires that no application-visible object, view, or exported buffer can still observe the allocation, and that pending GPU work remains correctly ordered. As such, it is best practice to not attempt to manually free memory.

Note

You can manually free all cached memory allocations by calling cvcuda.clear_cache().

Cache Reuse

When a CV-CUDA object goes out of scope, its memory is retained in the cache for efficient reuse. Owned tensors can reuse allocations on the same device with the same data type and layout; owned images require the same device and image format. An exact shape and stride match is preferred. Otherwise, the smallest available idle allocation that satisfies the requested byte size and alignment is reused, with a new descriptor for the requested shape and strides. Pending tensor reuse still requires an exact shape and stride match.

The dimensions may differ, including a wider image with fewer rows, as long as the calculated allocation requirements fit. The cache continues to count the allocation’s full capacity, even when the new object uses fewer bytes:

import cvcuda
import numpy as np


def create_tensor1():
    tensor1 = cvcuda.Tensor(  # noqa: F841
        (16, 32, 4), np.float32, cvcuda.TensorLayout.HWC
    )


def create_tensor2():
    # The wider, shorter tensor fits in the first allocation.
    tensor2 = cvcuda.Tensor(  # noqa: F841
        (4, 64, 4), np.float32, cvcuda.TensorLayout.HWC
    )


def main() -> None:
    create_tensor1()
    # tensor1 runs out of scope after leaving create_tensor1()
    create_tensor2()

In this example, tensor2 has a different shape but fits in tensor1’s allocation. No new GPU memory allocation occurs.

Cache reuse also applies to wrapped objects, improving efficiency even though they don’t consume cache memory.

Controlling Cache Growth

The cache can still grow when requests exceed the capacity of available allocations, use incompatible types or layouts, or keep earlier objects in use. For example, successively larger tensors require new allocations:

import cvcuda
import numpy as np


def create_tensor(h: int, w: int) -> None:
    _ = cvcuda.Tensor((h, w, 3), np.float32, cvcuda.TensorLayout.HWC)


def main(iters: int = 100) -> None:
    for i in range(iters):
        create_tensor(1000 + i, 1000)

CV-CUDA uses the configured cache limit to evict entries that no longer have pending GPU work. A matching tensor can also be reused before its previous GPU work completes, provided no Python wrapper, exported view, reshape view, or retained native handle can still observe it. Below the limit, operators that pass their consumer stream to allocation require the tensor’s synchronization history to permit the new write without any cross-stream wait. Under pressure, reuse may also add CUDA dependencies to order the new access after earlier work. Cache lookup does not wait for the GPU on the host.

The Python bindings supply the consumer stream for tensor Flip, PillowResize, ConvertTo, Normalize, and Reformat. Other tensor allocations retain pressure-only pending reuse. Free-threaded release records may retain Python wrappers themselves; such tensors conservatively remain ineligible while the wrappers are registered, even when no user reference remains.

Variable-shape Flip, PillowResize, and Normalize can also reuse an entire image batch with the same ordered image sizes and formats. The batch and its images must have no observable aliases, and pending accesses must already be ordered on the consumer’s CUDA queue. Reuse leaves the batch descriptors unchanged, even while earlier GPU work is pending. A mismatch falls back to ordinary allocation; generic batch construction never reuses an in-flight descriptor. This early whole-batch reuse is disabled in free-threaded Python builds. In ordinary Python builds, pending child image allocations are retained and indexed individually for cache accounting.

The limit controls retained cache storage, rather than total application GPU memory. Pending work must retain its resources even when a new shape does not fit the cache or the application lowers the limit. Those indexed allocations remain included in the reported cache bytes and can temporarily put the cache above its limit. Retirement during subsequent submissions or stream synchronization removes completed pending entries and evicts excess cache storage. There is no background retirement thread. clear_cache() explicitly waits for pending work before clearing the cache.

An unbounded sequence of distinct shapes or outputs kept alive by the application can still exhaust device memory. The cache limit neither rejects those allocations nor makes live output buffers eligible for reuse. Wrapped external allocations are not included in cache byte accounting.

Configuring the cache limit

import cvcuda


def main() -> None:
    # Get the cache limit (in bytes)
    current_cache_limit = cvcuda.get_cache_limit_inbytes()  # noqa: F841

    # Set the cache limit (in bytes)
    my_new_cache_limit = 12345  # in bytes
    cvcuda.set_cache_limit_inbytes(my_new_cache_limit)

By default, the cache limit is set to half the total GPU memory of the current device when importing cvcuda:

import cvcuda
import torch


def set_default_cache_limit() -> None:
    # Set the cache limit (in bytes)
    total_mem = torch.cuda.mem_get_info()[1]
    cvcuda.set_cache_limit_inbytes(total_mem // 2)

The getter and setter apply to the current CUDA device. A limit larger than that device’s total memory does not provide additional memory and produces a warning.

Setting the limit to zero disables retention of completed allocations. Eligible allocations that must remain alive for pending GPU work can still be reused safely, including allocations larger than the limit. They leave the pending index after retirement.

Querying cache size

def query_cache_size() -> None:
    print(cvcuda.current_cache_size_inbytes())
    img = cvcuda.Image.zeros((1, 1), cvcuda.Format.F32)  # noqa: F841
    print(cvcuda.current_cache_size_inbytes())

Multithreading Considerations

Ordinary reuse searches the calling thread’s cache. When a thread exits, its pending entries transfer to a shared index without losing their byte charge. Eligible tensors in that index can be reused by another thread on an already ordered consumer stream or under pressure; completed entries are removed. Entries in another active thread’s local cache are not searched.

Warning

Cache size and limits are shared between threads. Exercise caution in multithreaded applications.

You can clear the cache for the current thread using cvcuda.clear_cache() with cvcuda.ThreadScope.LOCAL and query the thread-local cache size with cvcuda.cache_size() with cvcuda.ThreadScope.LOCAL:

import threading
import time
import cvcuda
import numpy as np


def create_tensor_and_clear():
    tensor = cvcuda.Tensor(  # noqa: F841
        (16, 32, 4), np.float32, cvcuda.TensorLayout.HWC
    )
    print(cvcuda.cache_size(), cvcuda.cache_size(cvcuda.ThreadScope.LOCAL))  # 2 1
    cvcuda.clear_cache(cvcuda.ThreadScope.LOCAL)
    print(cvcuda.cache_size(), cvcuda.cache_size(cvcuda.ThreadScope.LOCAL))  # 1 0


def main() -> None:
    tensor = cvcuda.Tensor(  # noqa: F841
        (16, 32, 4), np.float32, cvcuda.TensorLayout.HWC
    )
    thread = threading.Thread(target=create_tensor_and_clear)
    thread.start()
    thread.join()

    # WORKAROUND: Race condition between Python thread.join() and C++ thread-local destructors
    #
    # Root Cause:
    # Python's thread.join() returns when the Python thread function completes, but
    # C++ thread-local objects (like the Cache instance) are destroyed asynchronously
    # by the C++ runtime AFTER Python considers the thread "joined". Without this sleep,
    # the main() function exits immediately after join(), triggering the main thread's
    # Cache destructor while the worker thread's Cache destructor is still running.
    # Both destructors then concurrently attempt to clean up CUDA resources (acquire GIL,
    # destroy events, deallocate memory), causing race conditions and segfaults.
    #
    # The sleep keeps the main thread alive longer, ensuring temporal separation between
    # the worker's destructor completing and the main thread's destructor starting.
    #
    # Why This Is Not An Issue In Normal Usage:
    # - Real applications use long-lived thread pools that outlive individual operations
    # - Main threads typically continue running after worker threads complete (serving
    #   requests, processing more data, etc.) providing natural timing separation
    # - This test has the worst-case scenario: main thread exits immediately after
    #   worker joins, with no other work keeping it alive
    #
    # A proper fix would require C++ synchronization primitives (condition variables) to
    # explicitly wait for C++ thread-local destructors, but adds complexity for a rare edge case.
    time.sleep(0.1)  # Keep main thread alive while worker's C++ destructor completes