PyFstat merge requestshttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests2017-12-22T09:36:36Zhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/9Adds the basic header with simple information2017-12-22T09:36:36ZGregory AshtonAdds the basic header with simple informationFixes #10 (at least for the gridded searches)
This adds a basic header with the data, username, and hostname information. Example output (from `examples/transient_examples/short_transient_search_gridded.py`)
```
# date:2017-12-20 11:54...Fixes #10 (at least for the gridded searches)
This adds a basic header with the data, username, and hostname information. Example output (from `examples/transient_examples/short_transient_search_gridded.py`)
```
# date:2017-12-20 11:54:51.736768;user:greg;hostname:greg-len-ubuntu
# _ _ F0 F1 F2 Alpha Delta
1.000000000000000000e+09 1.000172800000000000e+09 2.999998991056105524e+01 -1.000000000000000036e-10 0.000000000000000000e+00 5.000000000000000000e-01 1.000000000000000000e+00 4.206904220581054688e+01 1.000097200000000000e+09 1.980000000000000000e+04
1.000000000000000000e+09 1.000172800000000000e+09 2.999999011234983470e+01 -1.000000000000000036e-10 0.000000000000000000e+00 5.000000000000000000e-01 1.000000000000000000e+00 4.204999923706054688e+01 1.000097200000000000e+09 1.980000000000000000e+04
```
@dkeitel Am I missing anything from the header?David Keiteldavid.keitel@ligo.orgDavid Keiteldavid.keitel@ligo.orghttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/8add transient F(t0,tau) file output2017-12-19T08:05:31ZDavid Keiteldavid.keitel@ligo.orgadd transient F(t0,tau) file outputThis adds two bits of transient output:
1. two extra columns for the maximum-likelihood t0,tau values to the standard gridSearch output file, iff a `transientWindowType` was set
1. one extra file per Doppler candidate with the full t0,...This adds two bits of transient output:
1. two extra columns for the maximum-likelihood t0,tau values to the standard gridSearch output file, iff a `transientWindowType` was set
1. one extra file per Doppler candidate with the full t0,tau grid iff a new explicit option `outputTransientFstatMap` is set to True.
The main low-level change is that get_fullycoherent_twoF now returns 3 results, two being `None` by default except for the transient case, and so I added `[0]` to its calls from other usecases. Tests and examples seem to be unaffected, so I hope I got all of them, but a double-check can't hurt. Or maybe you now a more elegant solution @GregAshton ?Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/7transients: user-configurable windows, and some minor things2017-12-13T13:04:20ZDavid Keiteldavid.keitel@ligo.orgtransients: user-configurable windows, and some minor thingsHi @GregAshton this is finally the first part of the merge request I've been promising since summer.
The main first commit replaces the boolean 'transient' flag of the main classes by transientWindow, t0band, tauband. The main use case ...Hi @GregAshton this is finally the first part of the merge request I've been promising since summer.
The main first commit replaces the boolean 'transient' flag of the main classes by transientWindow, t0band, tauband. The main use case is a coherent grid search; I hope I've managed to put sane defaults to all other use cases, but not 100% sure I haven't overlooked any in the semicoherent/glitch world, so it's definitely worth a careful look from you.
Next commit suppresses transient parameters from the injection .cff file if window=none, which I think I needed to fix some data-reuse corner cases, though I can't remember anymore quite which cases those were. Still seems reasonable to do, though...?
Third commit just updates an example.
I split this off from my devel branch; the next part will require a small filepointer handling fix in lalpulsar, and for the final one (where this is all leading to) I need to clean up my CUDA code. I'll make separate merge requests when ready.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/6Renames sftfilepath -> sftfilepattern2017-07-18T14:24:57ZGregory AshtonRenames sftfilepath -> sftfilepatternThis renames the input sftfilepath to sftfilepattern and adds
documentation on how that should be used, i.e. a colon separated list of
wildstring or exact matches. In globbing for all matches, the colon
split is added in. sftfilepath is ...This renames the input sftfilepath to sftfilepattern and adds
documentation on how that should be used, i.e. a colon separated list of
wildstring or exact matches. In globbing for all matches, the colon
split is added in. sftfilepath is still used by `Writer` since
an exact path is known.
@dkeitel I've submitted this as a merge request to get your input, both because it effects a previous bug you fixed, and it may require you to rename a variable in all scripts (`sftfilepath` -> `sftfilepattern`). My reasoning is that the name better reflects its use: it will be more typical to provide a pattern than a path. Are you happy with this change?
Cheers,
GregGregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/5Fix sftfilenames and sftfilepath for multiple detectors2017-07-17T16:51:07ZGregory AshtonFix sftfilenames and sftfilepath for multiple detectorsIf there are multiple detectors, then multiple .sft files are generated.
This updates the attribute 'sftfilename' -> 'sftfilenames', a list of
those files while also updating 'sftfilpath' to a colon separated list
of the relevant sft files.If there are multiple detectors, then multiple .sft files are generated.
This updates the attribute 'sftfilename' -> 'sftfilenames', a list of
those files while also updating 'sftfilpath' to a colon separated list
of the relevant sft files.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/4add SFT-vs-output timestamp check to gridsearch, and fix bug in _get_list_of_...2017-07-17T08:11:13ZDavid Keiteldavid.keitel@ligo.orgadd SFT-vs-output timestamp check to gridsearch, and fix bug in _get_list_of_matching_sftsAnother infrastructure issue: GridSearch doesn't notice when SFTs have been updated. This just ports the check from MCMC, but along the way I noticed it had a bug for single-string path patterns.
```
-ported check from MCMC search
-bug: ...Another infrastructure issue: GridSearch doesn't notice when SFTs have been updated. This just ports the check from MCMC, but along the way I noticed it had a bug for single-string path patterns.
```
-ported check from MCMC search
-bug: for single-str SFT path, glob list comprehension
was trying every single character
```
(I hope multiple merge requests don't cause headaches through gitlab...? Let me know if you prefer patches instead.)Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/3new helper function run_commandline()2017-07-17T07:57:04ZDavid Keiteldavid.keitel@ligo.orgnew helper function run_commandline()Hi Greg,
another convenience issue I noticed is that the call to lalapps_Makefakedata_v5 is not properly error-checked: if I call pyfstat.Writer from a script (originally derived from your examples, I think), and lalapps fails, it will...Hi Greg,
another convenience issue I noticed is that the call to lalapps_Makefakedata_v5 is not properly error-checked: if I call pyfstat.Writer from a script (originally derived from your examples, I think), and lalapps fails, it will print the LAL error messages, but the script will continue running afterwards. E.g. this contrived example with duration=0:
``
14:55 INFO : No SFT file matching fake_tCW_Tobs_86400_tau_8640.0_h0_1e-24/H-0_H1_1800SFT_faketCW-1000086400-0.sft found
14:55 INFO : Executing: lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir="fake_tCW_Tobs_86400_tau_8640.0_h0_1e-24" --outLabel="faketCW" --IFOs="H1" --sqrtSX="1e-23" --startTime=1000086400 --duration=0 --fmin=29.9 --Band=0.2 --Tsft=1800 --injectionSources="fake_tCW_Tobs_86400_tau_8640.0_h0_1e-24/faketCW.cff"
XLAL Error - XLALMakeTimestamps (SFTutils.c:372): Check failed: Tspan > 0
XLAL Error - XLALMakeTimestamps (SFTutils.c:372): Input domain error
XLAL Error - XLALMakeMultiTimestamps (SFTutils.c:430): Check failed: (ret->data[X] = XLALMakeTimestamps ( tStart, Tspan, Tsft, Toverlap ) ) != ((void *)0)
XLAL Error - XLALMakeMultiTimestamps (SFTutils.c:430): Internal function call failed: Input domain error
XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:479): Check failed: ( cfg->multiTimestamps = XLALMakeMultiTimestamps ( uvar->startTime, uvar->duration, uvar->Tsft, uvar->SFToverlap, cfg->multiIFO.length )) != ((void *)0)
XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:479): Internal function call failed: Input domain error
XLAL Error - main (makefakedata_v5.c:180): Check failed: XLALInitMakefakedata ( &GV, &uvar ) == XLAL_SUCCESS
XLAL Error - main (makefakedata_v5.c:180): Internal function call failed: Input domain error
Running and timing search next...
``
And after the last line, it continues to start the search, and only then actually fails with a runtime error.
As I noticed that you have a few more commandline calls throughout, which are not quite consisteny anyway (variously with os.system() and subprocess, and not all try-except-ed), I tried a more general solution in this commit, introducing a run_commandline() helper function that wraps the whole try-except so that it is used consistently throughout.
(Most of the diff lines are from renaming all "cl"-like strings to something unique, just for better debuggability, as at one point I had gotten the check_cached_data_okay_to_use() confused. I hope it's still readable enough. And I haven't tested the PFS calls.)
Greetings,
David
commit:
```
-try-except wrapper for subprocess.check_output()
-use this for MFDv5 call (properly terminate at error!)
-and for other small utility calls
-use unique variable names for cl strings too
```Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/2workaround for matplotlib on X-less remote logins2017-07-06T08:22:55ZDavid Keiteldavid.keitel@ligo.orgworkaround for matplotlib on X-less remote loginsHi Greg,
logged into a remote (e.g. LDG cluster) without X-forwarding, importing pyfstat fails for me:
```
import pyfstat
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyfstat/__init__.py", line 3, in <...Hi Greg,
logged into a remote (e.g. LDG cluster) without X-forwarding, importing pyfstat fails for me:
```
import pyfstat
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyfstat/__init__.py", line 3, in <module>
from .core import BaseSearchClass, ComputeFstat, Writer, SemiCoherentSearch, SemiCoherentGlitchSearch
File "pyfstat/core.py", line 9, in <module>
import matplotlib.pyplot as plt
File "/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py", line 97, in <module>
_backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
File "/usr/lib64/python2.7/site-packages/matplotlib/backends/__init__.py", line 25, in pylab_setup
globals(),locals(),[backend_name])
File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_gtkagg.py", line 10, in <module>
from matplotlib.backends.backend_gtk import gtk, FigureManagerGTK, FigureCanvasGTK,\
File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_gtk.py", line 13, in <module>
import gtk; gdk = gtk.gdk
File "/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/__init__.py", line 64, in <module>
_init()
File "/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/__init__.py", line 52, in _init
_gtk.init_check()
RuntimeError: could not open display
```
I cobbled together the following workaround, let me know if you have a better solution:
```
-first check $DISPLAY environment variable
-then use non-interactive "AGG" backend if necessary
```Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/1peakutils missing in requirements2017-06-16T14:20:03ZDavid Keiteldavid.keitel@ligo.orgpeakutils missing in requirementsHi Greg, just updated your PyFstat and had to add peakutils to my local installation, which was not in your requirements yet. A nice trivial change to test my very first merge request with gitlab, hope it works as intended...Hi Greg, just updated your PyFstat and had to add peakutils to my local installation, which was not in your requirements yet. A nice trivial change to test my very first merge request with gitlab, hope it works as intended...Gregory AshtonGregory Ashton