fenestration.create_pooling_windows

Note

This object is a torch.nn.Module. It therefore has all the methods and attributes from that class, even though they are not documented here (to avoid cluttering this page).

class fenestration.create_pooling_windows(scaling: float | None, img_res: tuple[int, int], min_ecc: float = 0.5, max_ecc: float = 15, radial_to_circumferential_ratio: float = 2, window_type: Literal['cosine', 'gaussian'] = 'gaussian', transition_region_width: float | None = None, std_dev: float | None = 1, device: str | device | None = None)

Bases:

Create two sets of 2d pooling windows that span the visual field.

This creates the pooling windows that we use to average image statistics for metamer generation as done in [6]. This is returned as two 3d torch tensors for further use with a model.

Note that these are returned separately as log-eccentricity and polar angle tensors and if you want the windows used in the paper [6], you’ll need to call torch.einsum (see Examples section) or, better yet, use the PoolingWindows class, which is provided for this purpose.

Parameters:

• scaling (float | None) – The ratio of the eccentricity window’s radial full-width at half-maximum to eccentricity (see the calculate.scaling function).

• img_res (tuple[int, int]) – 2-tuple of ints specifying the resolution of the 2d images to make.

• min_ecc (float) – The minimum eccentricity, the eccentricity below which we do not compute pooling windows (in degrees). Parameter \(e_0\) in equation 11 of the online methods.

• max_ecc (float) – The maximum eccentricity, the outer radius of the image (in degrees). Parameter \(e_r\) in equation 11 of the online methods.

• radial_to_circumferential_ratio (float) – scaling determines the number of log-eccentricity windows we can create; this ratio gives us the number of polar angle ones. Based on scaling, we calculate the width of the windows in log-eccentricity, and then divide that by this number to get their width in polar angle. Because we require an integer number of polar angle windows, we round the resulting number of polar angle windows to the nearest integer, so the ratio in the generated windows approximate this. 2 (the default) is the value used in the paper [6].

• window_type (Literal['cosine', 'gaussian']) – Whether to use the raised cosine function from [6] or a Gaussian that has approximately the same structure. If cosine, transition_region_width must be set; if gaussian, then std_dev must be set.

• transition_region_width (float | None) – The width of the transition region, parameter \(t\) in equation 9 from the online methods.

• std_dev (float | None) – The standard deviation of the Gaussian window. WARNING – if this is too small (say < 3/4), then the windows won’t tile correctly. So we only support std_dev=1 for now.

• device (str | torch.device | None) – the device to create these tensors on

Returns:

• angle_windows – The 3d tensor of 2d polar angle windows. Its shape will be (n_angle_windows, *img_res), where the number of windows is inferred in this function based on the values of scaling and radial_to_circumferential_ratio.

• ecc_windows – The 3d tensor of 2d log-eccentricity windows. Its shape will be (n_eccen_windows, *img_res), where the number of windows is inferred in this function based on the values of scaling, min_ecc, and max_ecc.

See also

PoolingWindows

Generic class to set up and visualize foveated pooling windows

References

[6] (1,2,3,4,5)

Freeman, J., & Simoncelli, E. P. (2011). Metamers of the ventral stream. Nature Neuroscience, 14(9), 1195–1201. http://dx.doi.org/10.1038/nn.2889

[7]

Broderick, W. F., Rufo, G., Winawer, J. & Simoncelli, E. P. (2023). Foveated metamers of the early visual system. eLife, 12:RP90554. http://dx.doi.org/10.7554/eLife.90554.2

Examples

To use, simply call with the desired scaling and image size (for the version seen in the paper, don’t change any of the default arguments; compare this image to the right side of Supplementary Figure 1C).

Although we have hard-coded the standard deviation (to 1, for window_type="gaussian") and transition region width (to 0.5, for window_type="cosine") when creating the PoolingWindows object, it is possible to manually adjust these parameters when using create_pooling_windows. However, only the default values have been tested! It is unclear whether the windows will uniformly tile the images otherwise.

To create gaussian windows (default), you can specify the following arguments:

>>> import fenestration as fen
>>> angle_w, ecc_w = fen.create_pooling_windows(
...     scaling=0.8,
...     img_res=(256, 256),
...     min_ecc=1,
...     max_ecc=10,
...     radial_to_circumferential_ratio=2,
...     window_type="gaussian",
...     transition_region_width=None,
...     std_dev=1,
...     device="cpu",
... )

Similarly for raised cosine windows:

>>> angle_w, ecc_w = fen.create_pooling_windows(
...     scaling=0.8,
...     img_res=(256, 256),
...     min_ecc=1,
...     max_ecc=10,
...     radial_to_circumferential_ratio=2,
...     window_type="cosine",
...     transition_region_width=0.5,
...     std_dev=None,
...     device="cpu",
... )

To create equivalent windows to what is generated by PoolingWindows, you must also normalize resulting windows so they have an L1-norm of 1. This is useful when generating model metamers using the windows’ representation so that each eccentricity contributes equally, facilitating optimization. See [7] for an example.

You can display the various angle and eccentricity windows by plotting a specified index:

>>> import fenestration as fen
>>> import matplotlib.pyplot as plt
>>> angle_w, ecc_w = fen.create_pooling_windows(0.8, (256, 256))
>>> fig, ax = plt.subplots(1, 2, figsize=(8, 4))
>>> ax[0].imshow(ecc_w[0], cmap="Grays_r", interpolation="none")
<matplotlib.image.AxesImage ...>
>>> ax[1].imshow(angle_w[0], cmap="Grays_r", interpolation="none")
<matplotlib.image.AxesImage ...>

(Source code, png, hires.png, pdf)

If you wish to get the windows as shown in Supplementary Figure 1C in the paper [6], use torch.einsum (if you wish to apply these to images, use the PoolingWindows class instead, which has many more features):

>>> import fenestration as fen
>>> import torch
>>> angle_w, ecc_w = fen.create_pooling_windows(0.8, (256, 256))
>>> # we ignore the last ring of eccentricity windows here because
>>> # they're all relatively small, which makes the following plot
>>> # look weird. for how to properly handle them, see the
>>> # PoolingWindows class
>>> windows = torch.einsum("ahw,ehw->eahw", [angle_w, ecc_w[:-1]]).flatten(0, 1)
>>> fig, ax = plt.subplots(1, 1, figsize=(5, 5))
>>> # we use the intersecting amplitude value for gaussian windows, 0.14
>>> for w in windows:
...     ax.contour(w, [0.14], colors="r")
<matplotlib.contour.QuadContourSet ...>

(Source code, png, hires.png, pdf)