Skip to content

Utilities

Helpers for stochastic inference, model pruning and Gaussian moment propagation.

sample_with_log_probs returns predictions and matching prior/posterior log densities for Rényi training. See the Rényi guide for usage; predict_with_uncertainty returns predictive mean and standard deviation and does not supply weight densities.

bensemble.utils

enable_dropout

enable_dropout(model: Module) -> None

Enables dropout layers during evaluation for Monte Carlo Dropout.

Parameters:

Name Type Description Default
model Module

Target neural network containing PyTorch dropout layers.

required
Source code in bensemble/utils.py
def enable_dropout(model: nn.Module) -> None:
    """Enables dropout layers during evaluation for Monte Carlo Dropout.

    Args:
        model: Target neural network containing PyTorch dropout layers.
    """
    for module in model.modules():
        if isinstance(module, _DROPOUT_TYPES):
            module.train()

get_total_kl

get_total_kl(model: Module) -> torch.Tensor

Calculates the sum of KL divergences of all Bayesian layers in the model.

Parameters:

Name Type Description Default
model Module

Neural network model containing Bayesian layers.

required

Returns:

Type Description
Tensor

torch.Tensor: Total accumulated KL divergence scalar.

Source code in bensemble/utils.py
def get_total_kl(model: nn.Module) -> torch.Tensor:
    """Calculates the sum of KL divergences of all Bayesian layers in the model.

    Args:
        model: Neural network model containing Bayesian layers.

    Returns:
        torch.Tensor: Total accumulated KL divergence scalar.
    """
    total_kl = 0.0

    for module in model.modules():
        if hasattr(module, "kl_divergence"):
            total_kl += module.kl_divergence()

    return total_kl

predict_with_uncertainty

predict_with_uncertainty(model: Module, x: Tensor, num_samples: int = 100) -> tuple[torch.Tensor, torch.Tensor]

Estimates predictive mean and standard deviation via Monte Carlo sampling.

Restores each layer's original training flag, including on errors.

Parameters:

Name Type Description Default
model Module

Neural network model containing stochastic layers.

required
x Tensor

Input tensor.

required
num_samples int

Number of forward Monte Carlo passes. Defaults to 100.

100

Returns:

Type Description
Tensor

tuple[torch.Tensor, torch.Tensor]: A tuple (mean, std) representing

Tensor

the predictive mean and unbiased sample standard deviation.

Source code in bensemble/utils.py
def predict_with_uncertainty(
    model: nn.Module, x: torch.Tensor, num_samples: int = 100
) -> tuple[torch.Tensor, torch.Tensor]:
    """Estimates predictive mean and standard deviation via Monte Carlo sampling.

    Restores each layer's original training flag, including on errors.

    Args:
        model: Neural network model containing stochastic layers.
        x: Input tensor.
        num_samples: Number of forward Monte Carlo passes. Defaults to 100.

    Returns:
        tuple[torch.Tensor, torch.Tensor]: A tuple (mean, std) representing
        the predictive mean and unbiased sample standard deviation.
    """
    with _preserve_training_modes(model):
        model.eval()

        for module in model.modules():
            if isinstance(module, (BayesianLinear, BayesianConv2d)):
                module.train()

        with torch.no_grad():
            preds = torch.stack([model(x) for _ in range(num_samples)])

        mean = preds.mean(dim=0)
        std = preds.std(dim=0, unbiased=True)
        return mean, std

prune_model

prune_model(model: Module, threshold: float = 0.83) -> float

Applies Graves' SNR-based weight pruning to all Bayesian layers in the model.

Parameters:

Name Type Description Default
model Module

Neural network containing BaseBayesianLayer modules.

required
threshold float

Signal-to-Noise Ratio (SNR) pruning threshold. Defaults to 0.83.

0.83

Returns:

Name Type Description
float float

Overall sparsity ratio of pruned weights across all Bayesian layers (0.0 to 1.0).

Source code in bensemble/utils.py
def prune_model(model: torch.nn.Module, threshold: float = 0.83) -> float:
    """Applies Graves' SNR-based weight pruning to all Bayesian layers in the model.

    Args:
        model: Neural network containing BaseBayesianLayer modules.
        threshold: Signal-to-Noise Ratio (SNR) pruning threshold. Defaults to 0.83.

    Returns:
        float: Overall sparsity ratio of pruned weights across all Bayesian layers (0.0 to 1.0).
    """
    total_weights = 0
    total_pruned = 0

    for module in model.modules():
        if isinstance(module, BaseBayesianLayer):
            masks = module.get_pruning_masks(threshold)
            for mask in masks.values():
                total_weights += mask.numel()
                total_pruned += (mask == 0.0).sum().item()

            module.apply_pruning(threshold)

    overall_sparsity = total_pruned / total_weights if total_weights > 0 else 0.0
    return overall_sparsity

sample_with_log_probs

sample_with_log_probs(model: Module, x: Tensor, num_samples: int = 10) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Draws model weights and returns predictions with their log densities.

Each draw uses one set of weights across the batch and every call to a reused layer. Unlike local reparameterization, these predictions have matching parameter-space prior and posterior densities for a Rényi objective. Model parameters, buffers, and training modes are preserved; BatchNorm running statistics are not updated.

Parameters:

Name Type Description Default
model Module

Model containing BayesianLinear or BayesianConv2d layers. Other layers retain their training modes; set stochastic layers such as dropout to eval if only Bayesian weight uncertainty is wanted.

required
x Tensor

Input batch.

required
num_samples int

Positive number of weight draws.

10

Returns:

Type Description
Tensor

Tuple of predictions shaped (K, batch, ...), log prior densities (K,),

Tensor

and log posterior densities (K,). All three retain gradients.

Raises:

Type Description
ValueError

If the sample count is invalid, no supported layers exist, or distinct layers share posterior means.

TypeError

If the model contains an unsupported Bayesian layer.

Source code in bensemble/utils.py
def sample_with_log_probs(
    model: nn.Module, x: torch.Tensor, num_samples: int = 10
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """Draws model weights and returns predictions with their log densities.

    Each draw uses one set of weights across the batch and every call to a reused
    layer. Unlike local reparameterization, these predictions have matching
    parameter-space prior and posterior densities for a Rényi objective.
    Model parameters, buffers, and training modes are preserved; BatchNorm
    running statistics are not updated.

    Args:
        model: Model containing BayesianLinear or BayesianConv2d layers.
            Other layers retain their training modes; set stochastic layers such
            as dropout to eval if only Bayesian weight uncertainty is wanted.
        x: Input batch.
        num_samples: Positive number of weight draws.

    Returns:
        Tuple of predictions shaped (K, batch, ...), log prior densities (K,),
        and log posterior densities (K,). All three retain gradients.

    Raises:
        ValueError: If the sample count is invalid, no supported layers exist,
            or distinct layers share posterior means.
        TypeError: If the model contains an unsupported Bayesian layer.
    """
    if (
        isinstance(num_samples, bool)
        or not isinstance(num_samples, int)
        or num_samples < 1
    ):
        raise ValueError("num_samples must be a positive integer.")

    layers = []
    parameters = []
    sampled_means = set()
    parameter_uses = {}
    for layer in model.modules():
        for parameter in layer.parameters(recurse=False):
            key = id(parameter)
            parameter_uses[key] = parameter_uses.get(key, 0) + 1
    for name, layer in model.named_modules():
        if not isinstance(layer, BaseBayesianLayer):
            continue
        if not isinstance(layer, (BayesianLinear, BayesianConv2d)):
            raise TypeError(
                "Weight sampling supports BayesianLinear and BayesianConv2d."
            )
        layers.append((layer, layer.training))
        prefix = f"{name}." if name else ""
        for stem in ("w", "b"):
            mu = getattr(layer, f"{stem}_mu")
            if id(mu) in sampled_means or parameter_uses[id(mu)] > 1:
                raise ValueError(
                    "Tied weights across distinct layers are not supported."
                )
            sampled_means.add(id(mu))
            parameters.append(
                (
                    f"{prefix}{stem}_mu",
                    mu,
                    getattr(layer, f"{stem}_rho"),
                    layer.prior_sigma,
                )
            )
    if not layers:
        raise ValueError(
            "The model must contain BayesianLinear or BayesianConv2d layers."
        )

    predictions, log_priors, log_posteriors = [], [], []
    try:
        for layer, _ in layers:
            layer.training = False
        for _ in range(num_samples):
            sampled_parameters = {}
            prior_terms, posterior_terms = [], []
            for name, mu, rho, prior_sigma in parameters:
                posterior = Normal(mu, F.softplus(rho))
                sample = posterior.rsample()
                prior = Normal(mu.new_zeros(()), mu.new_tensor(prior_sigma))
                sampled_parameters[name] = sample
                prior_terms.append(prior.log_prob(sample).sum())
                posterior_terms.append(posterior.log_prob(sample).sum())
            buffers = {name: value.clone() for name, value in model.named_buffers()}
            predictions.append(
                functional_call(
                    model, (sampled_parameters, buffers), (x,), tie_weights=False
                )
            )
            log_priors.append(torch.stack(prior_terms).sum())
            log_posteriors.append(torch.stack(posterior_terms).sum())
    finally:
        for layer, training in layers:
            layer.training = training

    return (
        torch.stack(predictions),
        torch.stack(log_priors),
        torch.stack(log_posteriors),
    )

standard_normal_cdf

standard_normal_cdf(x: Tensor) -> torch.Tensor

Computes the standard normal cumulative distribution function Phi(x).

Parameters:

Name Type Description Default
x Tensor

Input tensor.

required

Returns:

Type Description
Tensor

torch.Tensor: Cumulative distribution value evaluated at x.

Source code in bensemble/utils.py
def standard_normal_cdf(x: torch.Tensor) -> torch.Tensor:
    """Computes the standard normal cumulative distribution function Phi(x).

    Args:
        x: Input tensor.

    Returns:
        torch.Tensor: Cumulative distribution value evaluated at x.
    """
    return 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

standard_normal_pdf

standard_normal_pdf(x: Tensor) -> torch.Tensor

Computes the standard normal probability density function phi(x).

Parameters:

Name Type Description Default
x Tensor

Input tensor.

required

Returns:

Type Description
Tensor

torch.Tensor: Probability density evaluated at x.

Source code in bensemble/utils.py
def standard_normal_pdf(x: torch.Tensor) -> torch.Tensor:
    """Computes the standard normal probability density function phi(x).

    Args:
        x: Input tensor.

    Returns:
        torch.Tensor: Probability density evaluated at x.
    """
    return torch.exp(-0.5 * x * x) * _INV_SQRT_2PI