User Tools

Site Tools


huibintemaspamfaq

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
huibintemaspamfaq [2017/12/13 01:40]
huibintema [Imaging with WSClean]
huibintemaspamfaq [2019/09/10 06:36]
huibintema [SPAM crashes in screen session]
Line 2: Line 2:
 ===== SPAM frequently asked questions ===== ===== SPAM frequently asked questions =====
  
-This ever-expanding section is based on user feedback. For questions on installing SPAM, see [[huibintemaspaminstall|this page]]. If you encounter a problem not listed below, please feel free to [[intema@strw.leidenuniv.nl|contact me]].+This ever-expanding section is based on user feedback. For questions on installing SPAM, see [[huibintemaspaminstall|this page]]. If you encounter a problem not listed below, please feel free to [[huib.intema@curtin.edu.au|contact me]]. 
 + 
 +----- 
 + 
 +==== Obit is missing ==== 
 + 
 +Q: Why is Obit reported to be missing during the ParselTongue install? 
 + 
 +A: The ParselTongue configuration command may give you the following error message: 
 +<code bash> 
 +checking for Obit... missing 
 +configure: error: Obit is not available 
 +</​code>​ 
 +This is likely to be caused by a missing python package or system library, not because Obit is missing (the script looks for Obit.so in the ${SPAM_PATH}/​Obit/​python/​ directory. To find what is causing the problem, please do the following:​ 
 +In bash: 
 +<code bash> 
 +. <​spam_install_path>/​setup.sh 
 +export PYTHONPATH=${SPAM_PATH}/​Obit/​python:​${PYTHONPATH} 
 +export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:​${SPAM_PATH}/​lib 
 +</​code>​ 
 +Or in (t)csh: 
 +<code bash> 
 +source <​spam_install_path>/​setup.csh 
 +setenv PYTHONPATH ${SPAM_PATH}/​Obit/​python:​${PYTHONPATH} 
 +setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:​${SPAM_PATH}/​lib 
 +</​code>​ 
 +Then 
 +<code bash> 
 +${PYTHON} 
 +</​code>​ 
 +And in python: 
 +<code python>​ 
 +import OSystem, OErr 
 +</​code>​ 
 +Something will fail here, probably a missing python package or a missing library. Some of the missing libraries can be found [[ftp://​ftp.strw.leidenuniv.nl/​pub/​intema/​spam/​lib/​|here]]. Another possible command that may reveal missing libraries is through the following shell command: 
 +<code bash> 
 +ldd ${SPAM_PATH}/​Obit/​python/​Obit.so 
 +</​code>​ 
 +This will report all libraries that are needed by Obit. If one is missing, the line will contain the text "=> not found"​. 
 + 
 +----- 
 + 
 +==== libncurses.so.6 is missing ==== 
 + 
 +Q: Why is libncurses.so.6 reported to be missing during the AIPS install? 
 + 
 +A: The AIPS install may fail because the libncurses.so.6 library cannot be found. This library is available in the lib subdirectory of the spam install directory. The library can be copied to one of the standard locations (e.g., /usr/lib) or the LD_LIBRARY_PATH environment variable can be modified to include the spam/lib directory before running AIPS install again. The latter can be done in bash: 
 +<code bash> 
 +export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:​${SPAM_PATH}/​lib 
 +</​code>​ 
 +Or in (t)csh: 
 +<code bash> 
 +setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:​${SPAM_PATH}/​lib 
 +</​code>​
  
 ----- -----
Line 17: Line 70:
  
 ----- -----
 +
 ==== Too many open files ==== ==== Too many open files ====
  
Line 40: Line 94:
  
 UPDATE: The latest ParselTongue distributed with SPAM has an improvement in place to reduce the occurence of this error. UPDATE: The latest ParselTongue distributed with SPAM has an improvement in place to reduce the occurence of this error.
 +
 +-----
 +
 +==== Out of pty devices ====
 +
 +Q: SPAM crashes with a message about "pty devices"​. What do I do?
 +
 +A: Under certain circumstances,​ the SPAM pipeline allocates pty devices without freeing them after use. This was a flaw that has been fixed in ParselTongue. Installing ParselTongue version 2.3d or higher, distributed with SPAM, will resolve this issue.
  
 ----- -----
Line 64: Line 126:
     wprojplanes = <some number, e.g. 128>, stokes = '​RR',​ weighting = '​briggs',​ usescratch = True ... )     wprojplanes = <some number, e.g. 128>, stokes = '​RR',​ weighting = '​briggs',​ usescratch = True ... )
 </​code>​ </​code>​
 +
 +For more guidance on the options of CASA clean, go to the [[https://​casa.nrao.edu/​casadocs/​latest|online CASA documentation]].
  
 ----- -----
Line 77: Line 141:
 </​code>​ </​code>​
  
------+For more guidance on the options of WSClean, go to the [[https://​sourceforge.net/​p/​wsclean/​wiki/​Home/​|WSClean wiki]].
  
 +-----
  
 ==== Problems running SPAM on new Intel CPUs, possibly after Linux update ==== ==== Problems running SPAM on new Intel CPUs, possibly after Linux update ====
Line 84: Line 149:
 Q: Why doesn'​t the SPAM pipeline work (anymore), with AIPS generating NaNs in certain tasks? Q: Why doesn'​t the SPAM pipeline work (anymore), with AIPS generating NaNs in certain tasks?
  
-A: Updates of the Linux operating system in 2017 has triggered floating point problems when running AIPS tasks on a new line of Intel Xeon E5-xxx CPUs. This is likely caused by an outdated Intel compiler used to build the AIPS binary install. A GNU compiled version of AIPS 31DEC13 fixes the problem and is available [[ftp://​ftp.strw.leidenuniv.nl/​pub/​intema/​spam/​AIPS_31DEC13A.tgz|here]]. It may run a bit slower than the Intel compiled version.+A: Updates of the Linux operating system in 2017 has triggered floating point problems when running AIPS tasks on a new line of Intel Xeon E5-xxx CPUs. This is likely caused by an outdated Intel compiler used to build the AIPS binary install. A GNU compiled version of AIPS 31DEC13 fixes the problem and is available [[ftp://​ftp.strw.leidenuniv.nl/​pub/​intema/​spam/​AIPS_31DEC13_Gnu.tgz|here]]. It may run a bit slower than the Intel compiled version. 
 + 
 +Q: Why does the SPAM pipeline generate signal 11 errors in certain AIPS tasks? 
 + 
 +A: What is causing this is not completely clear, but probably has to do with the newer Intel hardware and Linux installs. A work-around seems to be to use the GNU-compiled version of AIPS 31DEC13 available [[ftp://​ftp.strw.leidenuniv.nl/​pub/​intema/​spam/​AIPS_31DEC13_Gnu.tgz|here]]. It may run a bit slower than the Intel compiled version. 
 + 
 +(Kindly tested by Gerrit Schellenberger and Arnab Chakraborty) 
  
 ----- -----
Line 101: Line 173:
 ==== SPAM crashes in screen session ==== ==== SPAM crashes in screen session ====
  
-Q: Why does my SPAM pipeline run crash when the screen session in which I run SPAM is interrupted?​+Q: Why does my SPAM pipeline run crash when the screen session in which I run SPAM is interrupted, or when I reconnect?
  
 A: As the SPAM pipeline can take several hours to days to run, it is useful to execute it in a screen session when working remotely. However, some users found that the SPAM pipeline crashed during a disconnect from the screen session with the error "Fatal IO error 11 (Resource temporarily unavailable) on X server localhost:​1.0.",​ or something similar. This is likely due to the fact that the screen is started in an environment with the X server available that becomes unavailable after the disconnection,​ causing the crash of SPAM. A solution might be to use a "​fake"​ X server, such as xvfb, in the screen session. Another solution is to run SPAM remotely via a VNC session. Note that checking the progress of the pipeline can also be done by checking the spam*.log in the datfil subdirectory. A: As the SPAM pipeline can take several hours to days to run, it is useful to execute it in a screen session when working remotely. However, some users found that the SPAM pipeline crashed during a disconnect from the screen session with the error "Fatal IO error 11 (Resource temporarily unavailable) on X server localhost:​1.0.",​ or something similar. This is likely due to the fact that the screen is started in an environment with the X server available that becomes unavailable after the disconnection,​ causing the crash of SPAM. A solution might be to use a "​fake"​ X server, such as xvfb, in the screen session. Another solution is to run SPAM remotely via a VNC session. Note that checking the progress of the pipeline can also be done by checking the spam*.log in the datfil subdirectory.
  
-(Kindly contributed by Andrea Botteon)+Another problem with using screen was found when reconnecting,​ leading to an "​error: ​(4, '​Interrupted system call'​)"​. A fix has been added to ParselTongue which captures this exception without crashing the SPAM pipeline. 
 + 
 + 
 +(Kind contributions from Andrea Botteon ​and Błażej Nikiel-Wroczyński)
  
 ----- -----
 +
 +==== Manual flagging ====
 +
 +Q: Can I manually flag bad data in SPAM?
 +
 +A: Manual flags can be inserted in both the //​pre_calibrate_targets()//​ and the //​process_target()//​ steps. Both functions allow for a list of python dictionaries to be passed as the argument //​manual_flags//​. The dictionaries consist of keys and values that correspond to the adverbs and values of the AIPS task //UVFLG//. Here is an example:
 +<code python>
 +manual_flags = [ { '​antennas'​ : [ 5, 12 ] },
 +                 { '​antennas'​ : [ 1 ], '​stokes'​ : '​RR'​ },
 +                 { '​antennas'​ : [ 23 ], '​timerang'​ : [ 1,23,45,0, 99,0,0,0 ] } ]
 +pre_calibrate_targets( uvfits_file_name,​ manual_flags = manual_flags )
 +</​code>​
 +In this example, the antennas with AIPS indices 5 and 12 (corresponding to CASA indices 4 and 11) will be flagged completely (all times, both polarizations),​ for antenna 1 only stokes RR will be flagged (all times), and antenna 23 will be flagged from timestamp 1,23,45,0 [=day,​hour,​min,​sec] onwards (both polarizations). ​
 +
 +(Kindly suggested by Chris Riseley)
 +
 +-----
 +
 +==== Problems running SPAM on new Intel CPUs, possibly after Linux update ====
 +
 +Q: Why doesn'​t the SPAM pipeline work (anymore), with AIPS generating NaNs in certain tasks?
 +
 +A: Updates of the Linux operating system in 2017 has triggered floating point problems when running AIPS tasks on a new line of Intel Xeon E5-xxx CPUs. This is likely caused by an outdated Intel compiler used to build the AIPS binary install. A GNU compiled version of AIPS 31DEC13 fixes the problem and is available [[ftp://​ftp.strw.leidenuniv.nl/​pub/​intema/​spam/​AIPS_31DEC13_Gnu.tgz|here]]. It may run a bit slower than the Intel compiled version.
 +
 +-----
 +
 +Feedback: [[huib.intema@curtin.edu.au|Click here]]
  
huibintemaspamfaq.txt · Last modified: 2020/10/05 17:48 by huibintema