fenestration.PoolingWindows#

Note

This object is a torch.nn.Module. It therefore has all the methods and attributes from that class, even though they are not documented here (to avoid cluttering this page).

class fenestration.PoolingWindows(scaling: float, img_res: tuple[int, int], min_ecc: float = 0.5, max_ecc: float = 15, num_scales: int = 1, cache_dir: str | None = None, window_type: Literal['cosine', 'gaussian'] = 'gaussian')#

Bases: Module

Generic class to set up and visualize foveated pooling windows.

This generates foveated pooling windows given a small number of parameters. These windows are organized radially into eccentricity bands with the size, shape, and extent of the windows dependent upon the input parameters. These pooling windows can be used to summarize model statistics across visual space, such that information near the central (or foveal) visual field is pooled over smaller regions whereas information near the outer (or peripheral) visual field is pooled over larger regions.

One tricky thing we do is generate a set of scaling windows for each (appropriately-sized) scale. For example, a V1 model may have 4 scales, so for a 256 x 256 image, the coefficients will have shape (256, 256), (128, 128), (64, 64), and (32, 32). Therefore, we need windows of the same size (could also up-sample the coefficient tensors, but since that would need to happen each iteration of the metamer synthesis, pre-generating appropriately sized windows is more efficient).

We will calculate the minimum eccentricity at which the area of the windows at half-max exceeds one pixel at each scale. For scales beyond the first however, we will not throw an Exception if this value is below min_ecc. We instead print a warning to alert the user and use this value as min_ecc when creating the plots. In order to see what this value was, see calculated_min_eccentricity_pixels.

We can optionally cache the windows tensor we create, if cache_dir is not None. In that case, we’ll also check to see if appropriate cached windows exist before creating them and load them if they do. The path we’ll use is {cache_dir}/scaling-{scaling}_size-{img_res}_e0-{min_ecc}_ em-{max_ecc}_{window_type}.pt. We’ll cache each scale separately, changing the img_res (and potentially min_ecc) values in that save path appropriately.

Parameters:

  • scaling – Scaling parameter that governs the size of the pooling windows.

  • img_res – The resolution of our image (should therefore contains integers). Will use this to generate appropriately sized pooling windows where max_ecc is set to the outer radius of the image.

  • min_ecc – The eccentricity at which the pooling windows start.

  • max_ecc – The eccentricity at which the pooling windows end.

  • num_scales – The number of scales to generate masks for. For the RGC model, this should be 1, otherwise should match the number of scales in the steerable pyramid.

  • cache_dir – The directory to cache the windows tensor in. If set, we’ll look there for cached versions of the windows we create, load them if they exist and create and cache them if they don’t. If None, we don’t check for or cache the windows.

  • window_type – Whether to use the raised cosine function from [1] or a Gaussian that has approximately the same structure.

scaling#

Scaling parameter that governs the size of the pooling windows.

Type:

float

img_res#

The resolution of our image in pixels.

Type:

tuple

min_ecc#

The eccentricity at which the pooling windows start.

Type:

float

max_ecc#

The eccentricity at which the pooling windows end.

Type:

float

angle_windows#

A dict of 3d tensors containing the angular pooling windows in which the model parameters are averaged. Each key corresponds to a different scale and thus is a different size.

Type:

dict

ecc_windows#

A dict of 3d tensors containing the log-eccentricity pooling windows in which the model parameters are averaged. Each key in the dict corresponds to a different scale and thus is a different size.

Type:

dict

norm_factor#

A dict of 3d tensors containing the values used to normalize ecc_windows. Each key corresponds to a different scale. This is stored to undo that normalization for plotting and projection.

Type:

dict

window_width_pixels#

List of dictionaries containing the widths of the windows in pixels; each entry in the list corresponds to the widths for a different scale, as in windows. See above for explanation of the dictionaries. To visualize these, see the plot_window_widths method.

Type:

list

n_polar_windows#

The number of windows we have in the polar angle dimension (within each eccentricity band)

Type:

int

n_eccentricity_bands#

The number of eccentricity bands in our model

Type:

int

calculated_min_eccentricity_pixels#

List of floats (one for each scale) that contain the minimum eccentricity (in pixels) where the area of the window at half-max exceeds one pixel (based on the scaling, size of the image in pixels and in degrees).

Type:

list

central_eccentricity_pixels#

List of 1d arrays (one for each scale), each with shape (self.n_eccentricity_bands,), each value gives the eccentricity of the center of each eccentricity band of windows.

Type:

list

window_approx_area_pixels#

List of dictionaries containing the approximate areas of the windows in pixels; each entry in the list corresponds to the areas for a different scale, as in windows. There are three keys: ‘top’, ‘half’, and ‘full’, corresponding to which width we used to calculate window areas. For cosine windows, top is the width of the flat-top region of each window, where the window’s value is 1; full is the width of the entire window; half is the width at half-max. For gaussian windows, there is no flat-top region, full is 3 standard deviations, and half is the width at half max. To get this approximate area, we multiply the radial and angular widths against each other and then by pi/4 to get the area of the regular ellipse that has those widths (our windows are elongated, so this is probably an under-estimate). To visualize these, see the plot_window_areas method.

Type:

list

deg_to_pix#

List of floats containing the degree-to-pixel conversion factor at each scale

Type:

list

cache_dir#

If str, this is the directory where we cached / looked for cached windows tensors. This directory must already exist, or we raise a FileNotFoundError.

Type:

str or None

cache_paths#

List of strings, one per scale, that we either saved or loaded the cached windows tensors from

Type:

list

num_scales#

Number of scales this object has windows for

Type:

int

window_type#

Whether to use the raised cosine function from [1] or a Gaussian that has approximately the same structure.

Type:

{‘cosine’, ‘gaussian’}

window_max_amplitude#

The max amplitude of an individual window. This will always be 1 for raised-cosine windows. For gaussian windows, this value depends on the standard deviation, which is currently hard-coded at 1. Therefore, for gaussian windows it’s approximately 0.16.

Type:

float

window_intersecting_amplitude#

The amplitude at which two neighboring windows intersect. This will always be .5 for raised-cosine windows, but for gaussian ones, this value depends on the standard deviation. This value is currently hard-coded at 1, therefore it’s half a standard deviation away from the center, approximately 0.14.

Type:

float

Raises:

See also

create_pooling_windows

Create angle and eccentricity window tensors.

Notes

We will calculate the minimum eccentricity at which the area of the windows at half-max exceeds one pixel (based on scaling, img_res and max_ecc) and, if min_ecc is below that, will print a warning.

If you are just interested in the eccentricity and angular filters associated with these pooling windows, this is also possible using a combination of create_pooling_windows and normalize_windows. See Examples section of create_pooling_windows for details on this process.

References

Methods

forward(x[, idx, weights])

Window and pool the input.

load(load_path[, cache_dir])

Load pooling windows parameters and initialize model.

merge(other_PoolingWindows[, scale_offset])

Merge with a second PoolingWindows object.

plot_window_areas([units, scale_num, ...])

Plot the approximate areas of the windows, in degrees or pixels.

plot_window_checks([angle_n, scale])

Make some plots to check whether windows have been normalized properly.

plot_window_values([im, ax, subset, ...])

Plot the windowed average values.

plot_window_widths([units, scale_num, ...])

Plot the widths of the windows, in degrees or pixels.

plot_windows([ax, contour_levels, colors, ...])

Plot the pooling windows on an image.

pool(windowed_x[, idx])

Pool the windowed input.

project(pooled_x[, idx])

Project pooled values back onto an image.

save(save_path)

Save pooling windows model parameters.

summarize_window_sizes([units])

Summarize window sizes.

to(*args, **kwargs)

Move and/or cast the parameters and buffer.

window(x[, idx])

Window the input.
classmethod load(load_path: str, cache_dir: str | None = None, **kwargs: Any) → Module#

Load pooling windows parameters and initialize model.

Helper function that can load the necessary data for model and output model instatiation with those parameters.

Parameters:

  • load_path – The path to the file you wish to load

  • cache_dir – Optional path to a new cache directory to pass the model initialization, overriding the saved value. This allows you to e.g., load from a cache at a different location.

  • kwargs – Any additional kwargs to pass to torch.load

Returns:

pw – A PoolingWindows object created with parameters from loaded dictionary.

See also

save

Method to save pooling windows parameters.

Examples

To use, just input a path to the file saved using save in order to load the parameters needed for initializing the pooling window model.

>>> import fenestration as fen
>>> pw = fen.PoolingWindows(0.5, (256, 256))
>>> pw.save("model_params.pt")
>>> pw_new = fen.PoolingWindows.load("model_params.pt")
>>> pw_new
PoolingWindows()
forward(x: dict | Tensor, idx: int = 0, weights: Tensor | None = None) → dict[Tensor] | Tensor#

Window and pool the input.

We take an input, either a 4d tensor or a dictionary of 4d tensors, and return the pooled window averages. If it’s a 4d tensor, we return a 3d tensor, with windows indexed along the 3rd dimension. If it’s a dictionary, we return a dictionary with the same keys and have changed all the values to 3d tensors, with windows indexed along the 3rd dimension.

If it’s a 4d tensor, we use the idx entry in the windows list. If it’s a dictionary, we assume it’s keys are (scale, orientation) tuples and so use windows[key[0]] to find the appropriately-sized window (this is the case for, e.g., the steerable pyramid). If we want to use differently-structured dictionaries, we’ll need to restructure this.

This is equivalent to calling self.pool(self.window(x, idx), idx), however, we don’t produce the intermediate products and so this is more efficient.

Parameters:

  • x – Either a 4d tensor or a dictionary of 4d tensors.

  • idx – Which entry in the windows list to use. Only used if x is a tensor

  • weights – If not None, should be a tensor of shape (scales, batch, channel, eccentricity, angle), this allows us to reweight the pooled input across scales, eccentricity, and angle. If None, don’t reweight.

Returns:

pooled_x – Same type as x, see above for how it’s created.

See also

window

window the input

pool

pool the windowed input (get the weighted average)

project

the opposite of this, going from pooled values to image

merge(other_PoolingWindows: Module, scale_offset: float = 0.5)#

Merge with a second PoolingWindows object.

This combines the angle_windows, ecc_windows, and window_size dictionaries of two PoolingWindows objects. Since they will both have similarly-indexed keys (0, 1, 2,… based on num_scales), we need some offset to keep them separate, which scale_offset provides. We thus merge the dictionaries like so:

for k, v in other_PoolingWindows.angle_windows.items():
    self.angle_windows[k + scale_offset] = v

and similarly for ecc_windows and norm_factor.

The intended use case for this is to create one PoolingWindows object for a steerable pyramid with some number of scales, and then a second one for a corresponding “half-octave” steerable pyramid, which is built on the original image down-sampled by a factor of \(\sqrt{2}\) in order to sample the frequencies half-way between the scales of the original pyramid. You might want to slightly adjust the shape of the down-sampled image (e.g., to make its size even), so we don’t provide support to automatically create the windows for the half-scales; instead you should create a new PoolingWindows object based on your intended size and merge it into the original.

Note

This method modifies the module in-place.

Parameters:

  • other_PoolingWindows (fenestration.PoolingWindows) – A second instantiated PoolingWindows object

  • scale_offset (float, optional) – The amount to offset all the keys of the second PoolingWindows object by (see above for greater explanation)

plot_window_areas(units: Literal['degrees', 'pixels'] = 'degrees', scale_num: int = 0, figsize: tuple[int, int] = (5, 5), ax: Axes | None = None) → Figure#

Plot the approximate areas of the windows, in degrees or pixels.

We plot the approximate area of the window, calculated using ‘top’, ‘half’, and ‘full’ widths (top is the width of the flat-top region of each window, where the window’s value is 1; full is the width of the entire window; half is the width at the half-max value, which is what corresponds to the scaling value). To get the approximate area, we multiply the radial width against the corresponding angular width, then divide by \(\frac{\pi}{4}\).

The half area shown here is what we use to compare against a threshold value to determine the minimal eccentricity at which windows contain more than 1 pixel.

We plot this as a stem plot against eccentricity, showing the windows at their central eccentricity.

If the unit is ‘pixels’, then we also need to know which scale_num to plot (the windows are created at different scales, and so come in different pixel sizes).

Parameters:

  • units – Whether to show the information in degrees or pixels (both the area and the window location will be presented in the same unit).

  • scale_num – Which scale window we should plot

  • figsize – The size of the figure to create

  • ax – The axis to plot the windows on. If None, will create a new figure with 1 axis

Returns:

fig (Figure) – The figure containing the plot

Raises:

ValueError – If units are not ‘pixels’ or ‘degrees’

plot_window_checks(angle_n: int | list[int] = 0, scale: int = 0) → Figure#

Make some plots to check whether windows have been normalized properly.

This creates a figure with two sets of plots: the first row shows the L1-norm of the windows, the second shows the sum. Each row will have one plot and, if everything worked correctly, they should each look like a sigmoid function that runs from 1 for small eccentricities to 0 for high eccentricities

You can plot multiple angle slices, and each should look more or less the same

Parameters:

  • angle_n (int or list, optional) – Which angle slice(s) to show. Can be a single int or a list of ints, in which case we plot each as a separate color

  • scale (int, optional) – We plot this for one scale at a time. this specifies the scale.

Returns:

fig (Figure) – The figure containing the plot

plot_window_values(im: Tensor | None = None, ax: Axes | None = None, subset: bool = True, windows_scale: int = 0, **kwargs: Any) → Axes#

Plot the windowed average values.

This plots the average values of an image, as computed by these windows, and plots them using contourf as an RGB triple. We plot these within the window contours using window_intersecting_amplitude, so that if you call plot_windows with contour_levels=None, they will outline these regions.

Any additional kwargs are passed to contourf.

Parameters:

  • im – The image whose average values we plot within the windows. If None, we plot random gray values instead.

  • ax – The axis to plot the windows on. If None, will create a new figure with 1 axis.

  • subset – If True, will only plot four of the angle window slices. This is to save time and memory. If False, will plot all of them.

  • windows_scale – Which scale of the windows to use. windows is a list with different scales, so this specifies which one to use.

Returns:

ax (Axes) – The axis with the windows

Raises:

ValueError – If im has more than one batch or channel

plot_window_widths(units: Literal['degrees', 'pixels'] = 'degrees', scale_num: int = 0, figsize: tuple[int, int] = (5, 5), jitter: float | None = 0.25, ax: Axes | None = None) → Figure#

Plot the widths of the windows, in degrees or pixels.

We plot the width of the window in both angular and radial direction, as well as showing the ‘top’, ‘half’, and ‘full’ widths (top is the width of the flat-top region of each window, where the window’s value is 1; full is the width of the entire window; half is the width at the half-max value, which is what corresponds to the scaling value)

We plot this as a stem plot against eccentricity, showing the windows at their central eccentricity

If the unit is ‘pixels’, then we also need to know which scale_num to plot (the windows are created at different scales, and so come in different pixel sizes)

Parameters:

  • units – Whether to show the information in degrees or pixels (both the width and the window location will be presented in the same unit).

  • scale_num – Which scale window we should plot

  • figsize – The size of the figure to create

  • jitter – Whether to add a little bit of jitter to the x-axis to separate the radial and angular widths. There are only two values we separate, so we don’t add actual jitter, just move one up by the value specified by jitter, the other down by that much (we use the same value at each eccentricity).

  • ax – The axis to plot the windows on. If None, will create a new figure with 1 axis.

Returns:

fig (Figure) – The figure containing the plot

Raises:

ValueError – If units are not ‘pixels’ or ‘degrees’

plot_windows(ax: Axes | None = None, contour_levels: ndarray | int | None = None, colors: list[str] | str = 'r', subset: bool = True, windows_scale: int = 0, **kwargs: Any) → Axes#

Plot the pooling windows on an image.

This is just a simple little helper to plot the pooling windows on an axis. The intended use case is overlaying this on top of the image we’re pooling.

Any additional kwargs get passed to contour.

Parameters:

  • ax – The axis to plot the windows on. If None, will create a new figure with 1 axis.

  • contour_levels – The levels argument to pass to contour. From that documentation: “Determines the number and positions of the contour lines / regions. If an int n, use n data intervals; i.e. draw n+1 contour lines. The level heights are automatically chosen. If array-like, draw contour lines at the specified levels. The values must be in increasing order”. If None, will plot the contour that gives the first intersection (.5 for raised-cosine windows, self.window_max_amplitude * np.exp(-.25/2), or half a standard deviation away from max, for gaussian windows), as this is the easiest to see.

  • colors – The colors argument to pass to contour. If a single character, all will have the same color; if a sequence, will cycle through the colors in ascending order (repeating if necessary).

  • subset – If True, will only plot four of the angle window slices. This is to save time and memory. If False, will plot all of them.

  • windows_scale – Which scale of the windows to use. windows is a list with different scales, so this specifies which one to use.

Returns:

ax (Axes) – The axis with the windows

pool(windowed_x: dict[Tensor] | Tensor, idx: int = 0) → dict[Tensor] | Tensor#

Pool the windowed input.

We take the windowed input (as returned by window) and perform a weighted average, dividing each windowed statistic by the sum of the window that generated it.

The input must either be a 5d tensor or a dictionary of 5d tensors and we collapse across the spatial dimensions, returning a 3d tensor or a dictionary of 3d tensors.

Similar to window, if it’s a tensor, we use the idx entry in the windows list. If it’s a dictionary, we assume it’s keys are (scale, orientation) tuples and so use windows[key[0]] to find the appropriately-sized window (this is the case for, e.g., the steerable pyramid). If we want to use differently-structured dictionaries, we’ll need to restructure this

Parameters:

  • windowed_x – Either a 5d tensor or a dictionary of 5d tensors

  • idx – Which entry in the windows list to use. Only used if windowed_x is a tensor

Returns:

pooled_x – Same type as windowed_x, see above for how it’s created.

See also

window

window the input

forward

perform the windowing and pooling simultaneously

project(pooled_x: dict[Tensor] | Tensor, idx: int = 0) → dict[Tensor] | Tensor#

Project pooled values back onto an image.

For visualization purposes, you may want to project the pooled values (or values that have been pooled and then transformed in other ways) back onto an image. This method will do that for you.

It takes a 3d tensor or dictionary of 3d tensors (like the output of forward / pool; the final dimension must have a value for each window) and returns a 4d tensor or dictionary of 4d tensors (like the input of forward / window).

For example, if we have 100 windows, you must pass a i x j x 100 tensor. For each of the i batches and j channels, we’ll then multiply each of the 100 values by the corresponding window to end up with an i x j x 100 x height x width tensor. We then sum across windows to get i x j x heigth x width and return that.

Parameters:

  • pooled_x – 3d Tensor or a dictionary of 3d tensors

  • idx – Which entry in the windows list to use. Only used if pooled_x is a tensor.

Returns:

x – 4d tensor or dictionary of 4d tensors

Raises:

ValueError – If pooled_x is not 3d tensor or dictionary of 3d tensors

See also

forward

the opposite of this, going from image to pooled values

save(save_path: str)#

Save pooling windows model parameters.

This function saves all necessary data for model initialization at the specified path. It does not save the window tensors themselves; these are saved during object initialization if the cache_dir argument was set.

Parameters:

save_path – The file path you wish to save the model parameters to

See also

load

Method to load in the saved pooling windows parameters

Examples

To use, just input a file path in order to save the parameters needed for initializing the pooling window model.

>>> import fenestration as fen
>>> pw = fen.PoolingWindows(0.5, (256, 256))
>>> pw.save("model_params.pt")
>>> pw_new = fen.PoolingWindows.load("model_params.pt")
summarize_window_sizes(units: Literal['pixels', 'degrees'] = 'pixels') → dict#

Summarize window sizes.

This function returns a dictionary summarizing the window sizes at the minimum and maximum eccentricity in the specified units. The "min_window" and "max_window" are those whose centers are closest to min_ecc and max_ecc, respectively. For both of the window sizes, we return a dictionary containing the center, full-width half-max (FWHM, in the radial direction), and approximate area (at half-max). If units="pixels", we calculate these separately for each scale.

Parameters:

units – Which units to return the window size summary in

Returns:

sizes – Dictionary with the keys described above, summarizing window sizes. All values are scalar floats.

Raises:

ValueError – If units are not “pixels” or “degrees”

Examples

In order to display the window size parameters nicely, pprint is recommended:

>>> from pprint import pprint
>>> import fenestration as fen
>>> pw = fen.PoolingWindows(0.5, (256, 256))
>>> summary = pw.summarize_window_sizes()
>>> pprint(summary)
{'max_window_scale_0_area': np.float64(1489.7697961809874),
'max_window_scale_0_center': np.float64(123.18551268877933),
'max_window_scale_0_fwhm': np.float64(61.59275634438966),
'min_window_scale_0_area': np.float64(2.721047914586897),
'min_window_scale_0_center': np.float64(5.26463355455719),
'min_window_scale_0_fwhm': np.float64(2.632316777278595)}
to(*args: Any, **kwargs: Any) → Module#

Move and/or cast the parameters and buffer.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to, but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved to device, if that is given, but with dtype s unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See below for examples.

Note

This method modifies the module in-place.

Parameters:

  • device (torch.device) – The desired device of the parameters and buffers in this module

  • dtype (torch.dtype) – The desired floating point type of the floating point parameters and buffers in this module

  • tensor (torch.Tensor) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module

Returns:

Module

window(x: dict[Tensor] | Tensor, idx: int = 0) → dict[Tensor] | Tensor#

Window the input.

We take an input, either a 4d tensor or a dictionary of 4d tensors, and return a windowed version of it. If it’s a 4d tensor, we return a 5d tensor, with windows indexed along the 3rd dimension. If it’s a dictionary, we return a dictionary with the same keys and have changed all the values to 5d tensors, with windows indexed along the 3rd dimension.

If it’s a 4d tensor, we use the idx entry in the windows list. If it’s a dictionary, we assume it’s keys are (scale, orientation) tuples and so use windows[key[0]] to find the appropriately-sized window (this is the case for, e.g., the steerable pyramid). If we want to use differently-structured dictionaries, we’ll need to restructure this.

Parameters:

  • x – Either a 4d tensor or a dictionary of 4d tensors

  • idx – Which entry in the windows list to use. Only used if x is a tensor

Returns:

windowed_x – Same type as x, see above for how it’s created

Raises:

ValueError – If x is not 4d tensor or dictionary of 4d tensors

See also

pool

pool the windowed input (get the weighted average)

forward

perform the windowing and pooling simultaneously