Tamasis/PACS

Tamasis 4.2

2012-06-07T00:00:00-07:00

What’s new ?

update for pyoperators 0.4 & pyfits 3.x
bring back map-making with distributed maps
comprehensive MPI testing of the naive, ls and rls mapper.
PACS: API Change: change PACS’s get_tod default behaviour, by not applying the flatfield correction and subtracting the mean value. To do so, the appropriate keywords must be set to True
PACS: make the PacsInstrument class be instanciated and used without a PacsObservation
PACS: gather calibration files in PacsInfo class. Implement get_calibration methods, which centralise calibration file handling. Make Responsivity and BadPixelMask calibration files handled from Python. Add access to Absorption and FilterTransmission calibration files. Update calibration files to HCSS 8.
PACS: fix masking of temporal jumps
PACS: flag as ‘other’ all pointings before the first inscan one.
datatypes: add fast median method in Quantity. (Tod takes mask into account)
datatypes: getitem should also handle list of items.
datatypes: Add astype method, fix wrapping of some ndarray’s methods in Tod and Map
datatypes: implement rightward/leftward broadcasting in derived units.
acquisitionmodels: add class PointingMatrix which contain the pointing matrix information and info about the input map. Make ProjectionOperator handle PointingMatrix inputs and the get_pointing_matrix routines return a PointingMatrix object. As a consequence, pointing matrices can be manipulated before being used to initialise a ProjectionOperator. Make get_mask work for MPI-distributed maps and TODs. Make get_ptp work for block column projections. Remove header, npixels_per_sample attributes
acquisitionmodels: new methods intersects, apply_mask, get_pTx_pT1 (can significantly speed up naive maps) in ProjectionOperator. Remove _matrix attribute in projections.
acquisitionmodels: remove 3rd dimension in pointing_matrix_mask/pack. During packing, reject PointingElements with a negative weight.
acquisitionmodels: in InvNttOperator, fix filter_length if the operator has several blocks.
acquisitionmodels: PackOperator and UnpackOperator: add unary and binary rules. Hence remove transpose methods. Rework Fortran routines to differentiate in-place and out-of-place operations. Add comprehensive test cases.
acquisitionmodels: in DdTddOperator, if the the scalar argument is zero, return a ZeroOperator.
acquisitionmodels: make DiscreteDifferenceOperator work even if there are more MPI processes than elements along the first dimension. Fix MPI outplace diffT case.
acquisitionmodels: make the block_diagonal decorator handle implicit partitions, take the usual partitionin and axisin keywords instead of partition and axis.
API change: remove_nonfinite → filter_nonfinite, add outplace operations, with and without a mask, Add processing Fortran interface.
interpolate_linear, filter_median, filter_nonfinite, filter_polynomial: add ‘out’ keyword. Use partition instead of nsamples.
inplace filter_polynomial
implement median function, handling mask and NaN, parallelised along a given axis.
observation.plot_scan: plot detector footprint on the first valid pointing
force numeric locale to ‘POSIX’, to prevent gfortran using comma as fraction separator PR47007.
new functions gather_fitsheader and scatter_fitsheader
move all information messages to Python. Group info messages for block column projection operators.
refactor mapper_ls and mapper_rls, by relying on algebraic rules and by removing the use of ReshapeOperator.
in write_fits, fix MPI case. Do not rely on pyfits private methods and do the padding ourselves.
add argument r0 in airy_disk function to specify first zero, as an alternative to fwhm.
in imshow, discard outliers by using a percentile of the valid values.
general: improve error messages
rename numpyutils.py → utils.py and utils.py → datautils.py.
wscript: preprocess var.py.in file to include configuration options and version name.
wscript: make waf loc display Python/Fortran/Total lines of code. Remove loc-fortran and loc-python.
wscript: run test-mpi test suite with np = 1 even if —enable-mpi has not been set

Tamasis 4.1

2012-01-30T00:00:00-08:00

What’s new ?

Update to PyOperators v0.3
Update to waf 1.6.10
Bring back MPI read/write capabilities for FitsArray/Map/Tod instances
Bring back MPI-distributed discrete difference operator
Pointing class: make constructor more generic and easy to use
API changes: module quantity has been renamed to quantities. Although the old names will be kept for some time, operators now consistently end with 'Operator':
– CompressionAverage → CompressionAverageOperator
– DiscreteDifference → DiscreteDifferenceOperator
– InvNtt → InvNttOperator
– Masking → MaskOperator
– Projection → ProjectionOperator
– ResponseTruncatedExponential → ConvolutionTruncatedExponentialOperator
– Unpacking → UnpackOperator

Tamasis 4.0

2012-01-12T00:00:00-08:00

What’s new ?

first release depending on the PyOperator package, which is the result of the merging of Tamasis’ new acquisition models and of the work by N. Barbey (linear_operators).
the operator class is much more flexible, faster and the memory is better managed. In particular, operators can be combined in blocks. They can also be reduced, following algebraic rules.
as a consequence, are available many more algorithms (FminBFGS, FminCG, FminCOBYLA, FminLBFGSB, FminNCG, FminPowell, FminSLSQP, FminTNC, Lanczos, Double Lookup Inference) and priors through an Operator interface to the wavelet package pywt.
the Instrument and Observation classes are now populated with useful methods, so that Tamasis can now be used for other instruments than PACS.
configuration stage now checks the version of the python modules (waf patch)
dependencies to MPI and WCSLIB have been made optional.
test suite converted to nose.
API changes: in the Projection operator for the PACS instrument, use “downsampling=True”, instead of “oversampling=False”. The Pointing class constructor has now a more flexible single positional argument: Pointing(t,ra,dec,pa) becomes Pointing((ra,dec,pa), time=t).

Tamasis 3.2

2011-06-29T00:00:00-07:00

What’s new ?

parallel write of FITS files using MPI-IO
implement class DistributedArray to ease manipulation of MPI-distributed maps
new general purpose operators: SqrtInvNtt, Offset, Rounding (9 different methods), Clip, Minimum, Maximum
new PACS operators: Jy-to-Volts and Volts-to-ADU converters
linesearch with Armijo rule or strong Wolfe-Powell
command line interface for map-making routines, contributed by Nicolas Barbey

Tamasis 3.1

2011-05-25T00:00:00-07:00

What’s new ?

API change: InvNtt operator now works in direct space so it returns the actual time-time noise correlation matrix ala MADmap.
new PacsObservation method get_detector_stddev, which returns the detector standard deviation given an optional median filtering length.
new plot_tod method useful to display a Tod with the mask information, ex: plot_tod(tod[100]) will plot detector #100’s timeline
parallel (OpenMP+MPI) non-linear conjugate gradient solver with preconditioning, with different descent methods and norms

What is Tamasis?

2000-01-01T00:00:00-08:00

Tamasis is an ASTRONET funded project aiming at enhancing the science that can be produced from the submillimeter surveys currently in preparation in Europe with instruments such as Herschel, ALMA and SCUBA2.

This project is a collaboration between French, Dutch and Austrian institutes: CEA/SAp and IAS in France, Leiden Observatory in the Netherlands, and European Southern observatory in Germany, with support from the FWF, the Austrian Science Fund.

What is a PacsObservation?

2000-01-01T00:00:00-08:00

A PacsObservation or a PacsSimulation stores information about the

instrument (instrument attribute)
observation (slice attribute)
pointing (pointing attribute)

They do not contain the signal or mask data, but provide a means to access them through the method get_tod. They also do not contain the pointing matrix, which is obtained through the method get_pointing_matrix.

How to create a map from a single detector?

2000-01-01T00:00:00-08:00

A PacsObservation gathers information about the instrument settings, which are stored in the instrument attribute. With the exception of the detector channel, the parameters in the instrument attribute can be modified after instantiation. More information can be found on what contains a PacsObservation.

The field detector.removed controls the list of detectors to be included in the timelines and the field detector.masked, the list of detectors to be masked in the timelines. The former is used to distribute the work among processors in an MPI job and the latter to flag the bad detectors. Both control which detectors will contribute to the map.

To speed up the processing and the memory consumption, let’s remove detectors instead of masking them. The number of non-removed detectors can be obtained dynamically through the method get_ndetectors.

>>> obs = PacsObservation('myobs.fits')
>>> obs.get_ndetectors()
2048
>>> obs.get_tod().shape
(2048, 360)

The following snippet removes all detectors but one:

>>> obs.instrument.detector.removed = True
>>> obs.instrument.detector.removed[7,12] = False
>>> obs.get_ndetectors()
1
>>> obs.get_tod().shape
(1, 360)

To restrict the processing to the bolometer matrix 1:

>>> d = obs.instrument.detector
>>> d.removed = d.matrix != 1
>>> obs.get_ndetectors()
256

Once the removed or masked fields are set, you can create a map as usual.

How to create a PACS simulation?

2000-01-01T00:00:00-08:00

The first step is to get a sky map, which will be used for the simulation. You may create a synthetic map or load it if the sky map is in FITS format:

sky = Map('myfile.fits')

Then, let’s setup the spacecraft attitude during the scan

pointing = pacs_create_scan(sky.header['CRVAL1'], # RA of scan center
                            sky.header['CRVAL2'], # decl. of scan center
                            scan_length=10*60,    # scan length in arcsec
                            scan_speed=60,        # scan speed in arcsec/s
                            cross_scan=True)      # include cross-scan

and the instrument accordingly to these pointings

simul = PacsSimulation(pointing, 'blue')

We need an acquisition model to observe the sky map

masking = Masking(simul.pack(simul.instrument.detector.masked))
projection = Projection(simul, header=sky.header)
compression = CompressionAverage(simul.slice.compression_factor)
model = masking * compression * projection

At this stage, we are almost done: getting the simulated timeline is the easy part since the input of the acquisition model is the sky map and its output is the timeline.

tod = model(sky)

Note that this instrument model could be used to create timelines from different sky maps, as long as their shape and header are consistent with those of the original sky map.

It is now possible to add real noise to the simulated timeline. To this effect, we can use a calibration observation which was taken with a still spacecraft, to characterise the noise power spectrum density of the bolometers:

tod += simul.get_random()

This method only works on the CEA/SAp cluster, that’s because I prefer not to increase Tamasis’ package size (and Git index) by several gigabytes.

Finally, the simulated timeline can be saved as a FITS file

simul.save('mysimulatedtod.fits', tod)

for later reuse as a regular PacsObservation (HIPE won’t be able to read it though, and there is no plan to enable it):

simul = PacsObservation('mysimulatedtod.fits')
tod = simul.get_tod()

How to add a point source to a map?

2000-01-01T00:00:00-08:00

TBD

sky = gaussian(...)