User Tools

Site Tools


huibintemaspampipeline

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
huibintemaspampipeline [2017/06/27 10:55] – [Limited bandwidth observations] huibintemahuibintemaspampipeline [2020/10/05 17:46] (current) – [EXPERIMENTAL: Processing uGMRT wideband data] huibintema
Line 23: Line 23:
 Then we derive calibration and flagging information from the primary calibrator(s), pick the best one, transfer those informations to all other source visibilities present in the observation, and export each of these pre-calibrated visibility data sets to UVFITS: Then we derive calibration and flagging information from the primary calibrator(s), pick the best one, transfer those informations to all other source visibilities present in the observation, and export each of these pre-calibrated visibility data sets to UVFITS:
 <code python> <code python>
 +uvfits_file_name = "./fits/<project name>_<observe date>.lta.UVFITS"
 pre_calibrate_targets( uvfits_file_name, flags_file_name = lta_file_name + "##*.FLAGS*" ) pre_calibrate_targets( uvfits_file_name, flags_file_name = lta_file_name + "##*.FLAGS*" )
 </code> </code>
Line 40: Line 41:
 ./fits/<source name>.SP2B.PBCOR.FITS ./fits/<source name>.SP2B.PBCOR.FITS
  
-There'a million options available to make this process work for different / more complicated data sets. Some of the most common situations are discussed below.+The pipeline run can be summarized by typing: 
 +<code python> 
 +summarize_spam_log( "./datfil/spam_<source name>_<observation reference date>_<polarization(s)>_<sideband>*.log"
 +</code> 
 + 
 + 
 +There'many options available to make this process work for different / more complicated data sets. Some of the most common situations are discussed below. Note that more than one of these situations may reflect your data, and therefore multiple of the options discussed below may need to be combined.
  
 ----- -----
Line 123: Line 130:
 channel_range = [ 150, 425 ] # example channel range to keep channel_range = [ 150, 425 ] # example channel range to keep
 pre_calibrate_targets( uvfits_file_name, channel_range = channel_range ) pre_calibrate_targets( uvfits_file_name, channel_range = channel_range )
 +</code>
 +
 +-----
 +
 +==== Old hardware-correlator observations ====
 +
 +If you downloaded data from cycle 17 or earlier, it is likely correlated using the GMRT Hardware Backend (GHB; a.k.a. the hardware correlator). For frequencies of 325 MHz and higher, the 32 MHz bandwidth is typically split over an upper-side band (USB) and lower-side band (LSB), both captured in separate LTA files (typical extensions are .lta and .ltb). In SPAM, both sideband LTA files need to be pre-processed separately:
 +<code python>
 +convert_lta_to_uvfits( lta_file_name )
 +convert_lta_to_uvfits( ltb_file_name )
 +</code>
 +The next step is run using the //keep_channel_one// option to enforce frequency continuity between the sidebands:
 +<code python>
 +pre_calibrate_targets( uvfits_file_name_lta, flags_file_name = lta_file_name + "##*.FLAGS*", keep_channel_one = True )
 +pre_calibrate_targets( uvfits_file_name_ltb, flags_file_name = ltb_file_name + "##*.FLAGS*", keep_channel_one = True )
 +</code>
 +The resulting UVFITS files for USB and LSB per target can be combined:
 +<code python>
 +uvfits_file_name_usb = "./fits/<target>_GMRT<frequency>_<date>_<stokes>_USB.UVFITS"
 +uvfits_file_name_lsb = "./fits/<target>_GMRT<frequency>_<date>_<stokes>_LSB.UVFITS"
 +uvfits_file_name = "./fits/<target>_GMRT<frequency>_<date>_<stokes>.UVFITS"
 +combine_usb_lsb( uvfits_file_name_usb, uvfits_file_name_lsb, uvfits_file_name )
 +</code>
 +The output UVFITS file can be processed further in the main pipeline.
 +
 +Regarding the main pipeline, there are two options that may be relevant to get to better results. The first option related to the situation explained above, where two sidebands (USB and LSB) are joined together to cover 32 MHz of bandwidth. In that case, it may help to turn on an image-based flagging option that treats the joined USB and LSB separately. Reason for this is that the USB and LSB have separate signal chains, and thus there can be system problems that relate only to one of the two sidebands.
 +<code python>
 +process_target( target_uvfits_file_name, flag_image_usb_lsb = True )
 +</code>
 +
 +The second option is to turn on baseline-based calibration, BUT ONLY AFTER VERY CAREFUL CONSIDERATION. In an ideal world, baseline-based calibration would never be necessary. However, I have witnessed situations using hardware-correlator data in which residual gain errors on baselines did not seem to be solely antenna-based. Baseline-based calibration is something to try as a last resort, and can be switched on in the pipeline as shown below. Please note that this is implemented under strict limitations: baseline-calibration is applied only in the final stages of the processing, and is determined on a per-observation base (meaning only one correction per baseline per observe session).
 +<code python>
 +process_target( target_uvfits_file_name, do_blcal = True )
 </code> </code>
  
Line 145: Line 185:
  
 ----- -----
 +
 +==== EXPERIMENTAL: Processing uGMRT wideband data ====
 +
 +SPAM has some options to process uGMRT wideband data. SPAM does not support the processing of large fractional bandwidths (df/f>~0.2) in one run, but instead the bandwidth can be split up into smaller chunks (subbands) that can be processed independently. If done carefully, the calibrated output visibilities of SPAM pipeline runs on multiple subbands can be jointly imaged with a wideband imager (WSClean) as a final step. This approach has produced good results when applied on bands 3 (250-500 MHz) and band 4 (550-850 MHz) data. Processing band 2 data (120-250 MHz) has given mixed results. Since there is not yet a good way to apply the wideband primary beam corrections, this approach only works for observations where the target's angular size is small with respect to the primary beam size (size <~ 0.1 * FWHM).
 +
 +The first step is to convert LTA to UVFITS format:
 +<code python>
 +lta_file_name = "./<project name>_<observe date>.lta"
 +convert_lta_to_uvfits( lta_file_name )
 +</code>
 +Next, we split the UVFITS file into smaller frequency chunks (subbands):
 +<code python>
 +uvfits_file_name = "./fits/<project name>_<observe date>.UVFITS"
 +split_wideband_uvdata( uvfits_file_name )
 +</code>
 +The width of the frequency chunks is automatically set to a sensible value. The resulting 4 or 6 UVFITS files are also located in the fits subdirectory and named "./fits/<project name>_<observe date>.BAND<xx>.UVFITS", where <xx> is a 2-digit simple counter starting at 1 for the lowest frequency chunk. 
 +
 +From here, each frequency chunk is processed independently in a similar fashion as a narrow-band GMRT observations, but using slightly different function calls. The first step is the pre-calibration:
 +<code>
 +uvfits_file_name = "./fits/<project name>_<observe date>.lta.BAND<xx>.UVFITS"
 +reference_frequency = 450.e6
 +pre_calibrate_wideband_targets( uvfits_file_name, flags_file_name = lta_file_name + "##*.FLAGS*", reference_frequency = reference_frequency )
 +</code>
 +Setting a fixed reference frequency ensures that the frequency averaging of all frequency chunks is the same, which is important when jointly imaging the SPAM output visibilities later. Sensible values seem to be:
 +<code>
 +reference_frequency = 450.e6  # for uGMRT band 3 observations
 +reference_frequency = 650.e6  # for uGMRT band 4 observations
 +</code>
 +The pre-calibrated visibilities per target are located in the fits subdirectory and are named per subband.
 +
 +Next comes the SPAM main pipeline run. This is best done in separate project directories per subband. If possible, use a good, single reference sky model for all runs. For example, this reference model can be obtained from first running SPAM on the narrow-band GMRT (GSB) data that was recorded alongside the uGMRT wideband data, and extracting a sky model from the final SP2B image using PyBDSF.
 +<code>
 +target_uvfits_file_name = "./fits/<source_name>_UGMRT<band>-<subband>_<observation_reference_date>_<polarization(s)>_<sideband>.
 +UVFITS"
 +catalog_name = "<narrowband_project_dir>/fits/<source_name>.SP2B.PBCOR.pybdsm.gaul"
 +catalog = read_pybdsm_ascii_catalog( catalog_name )
 +source_list = create_source_list_from_catalog( catalog )               
 +resolution = 10. # representative resolution of model image in arcsec
 +process_wideband_target( target_uvfits_file_name, model_source_list = source_list, model_resolution = resolution )
 +</code>
 +
 +If all went well, each SPAM pipeline run on a subband yielded a final image and a calibrated visibility data set (.SP2B.CAL.RR.UVFITS). For use in WSClean, the calibrated visibilities all need to be collected in one directory and converted into measurement sets using CASA. Then WSClean can be used to do a final wideband imaging run. Here is an example:
 +<code>
 +wsclean -weight briggs 0 -pol RR -size 5000 5000 -scale 1.5asec -niter 15000 -auto-threshold 0.5 -auto-mask 3 -gain 0.25 -mgain 0.8 -weighting-rank-filter 3 -join-channels -channels-out 6 -j 8 -mem 80 -name SOURCE_UGMRT3 SOURCE_UGMRT3-01.MS SOURCE_UGMRT3-02.MS SOURCE_UGMRT3-03.MS SOURCE_UGMRT3-04.MS SOURCE_UGMRT3-05.MS SOURCE_UGMRT3-06.MS
 +</code>
 +Here, SOURCE_UGMRT3-0x.MS are the input measurement sets as produced by CASA. Make sure that "-scale" is set to the proper pixel scale, and that "-channels-out" matches the number of input measurement sets. WSClean produces a lot of output files, but probably you are interested in the final wideband image "*-MFS-image.fits".
 +
 +
 +
 +-----
 +
 +Feedback: [[intema@strw.leidenuniv.nl|Click here]]
  
huibintemaspampipeline.1498553730.txt.gz · Last modified: 2017/06/27 10:55 by huibintema

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki