Operators¶
TOAST workflows (“pipelines”) consist of a toast.Data object that is passed through one or more “operators”:
-
class
toast.Operator[source]¶ Base class for an operator that acts on collections of observations.
An operator takes as input a toast.dist.Data object and modifies it in place.
Parameters: None –
There are very few restrictions on an “operator” class. It can have arbitrary constructor arguments and must define an exec() method which takes a toast.Data instance. TOAST ships with many built-in operators that are detailed in the rest of this section. Operator constructors frequently require many options. Most built-in operators have helper functions in the toast.pipeline_tools module to ease parsing of these options from the command line or argparse input file.
Todo
Document the pipeline_tools functions for each built-in operator.
Pointing Matrices¶
A “pointing matrix” in TOAST terms is the sparse matrix that describes how sky signal is projected to the timestream. In particular, the model we use is
where we write \(s_p\) as a column vector having a number of rows given by the number of pixels in the sky. So the \(\mathcal{A}_{tp}\) matrix has a number of rows given by the number of time samples and a column for every sky pixel. In practice, the pointing matrix is sparse, and we only store the nonzero elements in each row. Also, our sky model often includes multiple terms (e.g. I, Q, and U). This is equivalent to having a set of values at each sky pixel. In TOAST we represent the pointing matrix as a vector of pixel indices (one for each sample) and a 2D array of “weights” whose values are the nonzero values of the matrix for each sample. TOAST includes a generic HEALPix operator to generate a pointing matrix.
Generic HEALPix Representation¶
Each experiment might create other specialized pointing matrices used in solving for instrument-specific signals.
Simulated Detector Signal¶
Every experiment is different, and a realistic simulation will likely require some customized code to model the instrument. However some kinds of simulations are generic, especially when doing trade studies in the early design phase of a project. TOAST comes with several built-in operators for simulating detector signals.
Sky Model¶
“Sky” In this case are sources of signals coming from outside of the Earth. The most simplistic model of the sky is just an input map at the same pixelization that is used for analysis (to avoid pixelization effects). There can be one map per detector or one map for a set of detectors (for example if simulating all detectors in a band without including bandpass variability):
The cosmological and orbital dipole can be simulated with this operator:
TOAST can also use some external packages for more complicated sky simulations. One of these is PySM, which supports generating bandpass-integrated sky maps in memory for each detector from a set of component maps. It can also do basic smoothing of the input signal:
For studies of far side lobes and full 4Pi beams, TOAST can use the external libconviqt package to convolve beam a_lm’s with a sky a_lm at every sample:
Atmosphere Simulation¶
Although many ground experiments have traditionally modelled atmosphere as “correlated noise”, in TOAST we take a different approach. For each observation, a realization of the physical atmosphere slab is created and moved according to the wind speed. Every detector sample is formed by a line of sight integral through this slab. This sort of real-space modelling naturally leads to correlated signal in the time domain that depends on atmospheric parameters and focalplane geometry.
Other Signals¶
Any systematic contaminant that is constant (for an observation) in the scanning reference frame (as opposed to sky coordinates), can be simulated as “Scan Synchronous Signal”:
This next operator can be used to apply random gain errors to the timestreams:
-
class
toast.tod.OpGainScrambler(center=1, sigma=0.001, pattern='.*', name=None, realization=0, component=234567)[source]¶ Apply random gain errors to detector data.
This operator draws random gain errors from a given distribution and applies them to the specified detectors.
Parameters: - center (float) – Gain distribution center.
- sigma (float) – Gain distribution width.
- pattern (str) – Regex pattern to match against detector names. Only detectors that match the pattern are scrambled.
- name (str) – Name of the output signal cache object will be <name_in>_<detector>. If the object exists, it is used as input. Otherwise signal is read using the tod read method.
- realization (int) – if simulating multiple realizations, the realization index.
- component (int) – the component index to use for this noise simulation.
Simulated Detector Noise¶
TOAST includes an operator that simulates basic detector noise (1/f spectrum plus white noise), and also supports generating correlated noise by specifying a mixing matrix to combine individual detector streams with common mode sources.
-
class
toast.tod.OpSimNoise(out='noise', realization=0, component=0, noise='noise', rate=None)[source]¶ Operator which generates noise timestreams.
This passes through each observation and every process generates data for its assigned samples. The dictionary for each observation should include a unique ‘ID’ used in the random number generation. The observation dictionary can optionally include a ‘global_offset’ member that might be useful if you are splitting observations and want to enforce reproducibility of a given sample, even when using different-sized observations.
Parameters: - out (str) – accumulate data to the cache with name <out>_<detector>. If the named cache objects do not exist, then they are created.
- realization (int) – if simulating multiple realizations, the realization index.
- component (int) – the component index to use for this noise simulation.
- noise (str) – PSD key in the observation dictionary.
-
exec(data)[source]¶ Generate noise timestreams.
This iterates over all observations and detectors and generates the noise timestreams based on the noise object for the current observation.
Parameters: data (toast.Data) – The distributed data.
Raises: KeyError– If an observation in data does not have noise object defined under given key.RuntimeError– If observations are not split into chunks.
-
simulate_chunk(*, tod, nse, curchunk, chunk_first, obsindx, times, telescope, global_offset)[source]¶ Simulate one chunk of noise for all detectors.
Parameters: - tod (toast.tod.TOD) – TOD object for the observation.
- nse (toast.tod.Noise) – Noise object for the observation.
- curchunk (int) – The local index of the chunk to simulate.
- chunk_first (int) – First global sample index of the chunk.
- obsindx (int) – Observation index for random number stream.
- times (int) – Timestamps for effective sample rate.
- telescope (int) – Telescope index for random number stream.
- global_offset (int) – Global offset for random number stream.
Returns: Number of simulated samples
Return type: chunk_samp (int)
Timestream Processing¶
Many timestream manipulations done prior to map-making are very specific to the instrument. However there are a few operations that are generically useful.
Filtering¶
This operator is used to build a template in azimuth bins of signal that is fixed in the scanning reference frame. This template is then subtracted from the timestream.
This next operator fits a polynomial to each scan and subtracts it.
-
class
toast.tod.OpPolyFilter(order=1, pattern='.*', name=None, common_flag_name=None, common_flag_mask=255, flag_name=None, flag_mask=255, poly_flag_mask=1, intervals='intervals')[source]¶ Operator which applies polynomial filtering to the TOD.
This applies polynomial filtering to the valid intervals of each TOD.
Parameters: - order (int) – Order of the filtering polynomial.
- pattern (str) – Regex pattern to match against detector names. Only detectors that match the pattern are filtered.
- name (str) – Name of the output signal cache object will be <name_in>_<detector>. If the object exists, it is used as input. Otherwise signal is read using the tod read method.
- common_flag_name (str) – Cache name of the output common flags. If it already exists, it is used. Otherwise flags are read from the tod object and stored in the cache under common_flag_name.
- common_flag_mask (byte) – Bitmask to use when flagging data based on the common flags.
- flag_name (str) – Cache name of the output detector flags will be <flag_name>_<detector>. If the object exists, it is used. Otherwise flags are read from the tod object.
- flag_mask (byte) – Bitmask to use when flagging data based on the detector flags.
- poly_flag_mask (byte) – Bitmask to use when adding flags based on polynomial filter failures.
- intervals (str) – Name of the valid intervals in observation.
Calibration¶
This operator applies a set of gains to the timestreams:
-
class
toast.tod.OpApplyGain(gain, name=None)[source]¶ Operator which applies gains to timelines.
Parameters: - gain (dict) – Dictionary, key “TIME” has the common timestamps, other keys are channel names their values are the gains
- name (str) – Name of the output signal cache object will be <name_in>_<detector>. If the object exists, it is used as input. Otherwise signal is read using the tod read method.
Utilities¶
These operators are used to manipulate cached data or perform other helper functions.
-
class
toast.tod.OpFlagGaps(common_flag_name=None, common_flag_value=1, intervals='intervals')[source]¶ Operator which applies common flags to gaps between valid intervals.
Parameters: - common_flag_name (str) – the name of the cache object to use for the common flags. If None, use the TOD.
- common_flag_value (int) – the integer bit mask (0-255) that should be bitwise ORed with the existing flags.
- intervals (str) – Name of the valid intervals in observation.
-
class
toast.tod.OpFlagsApply(name=None, common_flags=None, flags=None, common_flag_mask=1, flag_mask=1)[source]¶ This operator sets the flagged signal values to zero
-
class
toast.tod.OpMemoryCounter(*other_caching_objects, silent=False)[source]¶ Compute total memory used by TOD objects.
Operator which loops over the TOD objects and computes the total amount of memory allocated.
Parameters: - silent (bool) – Only count and return the memory without printing.
- *other_caching_objects – Additional objects that have a cache member and user wants to include in the total counts (e.q. DistPixels objects).
Map Making Tools¶
CMB map-making codes and algorithms have a long history going back more than 20 years. Early work focused on finding the maximum likelihood solution to the noise-weighted least squares expression for the pixelized map (and other parameters) given the data and an estimate of the time domain noise. Later efforts took other approaches, such as the destriping formalism used in Planck and agressive filtering and binning used in some ground experiments. In terms of algorithms built in to TOAST, there are 2 primary tools. The first is a wrapper around the external libmadam destriping map-maker. The second is a set of native mapmaking tools which are already quite usable for many common cases.
Libmadam Wrapper¶
This operator has some basic options and can also take a dictionary of parameters to pass to the underlying Fortran code for complete control over the options.
Native Tools¶
Todo
Document these more fully once the communication performance bottleneck is fixed.