PyFstat issueshttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues2020-02-05T10:53:27Zhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/19missing zero-initialisation of FstatResults2020-02-05T10:53:27ZDavid Keiteldavid.keitel@ligo.orgmissing zero-initialisation of FstatResultsIn !26 I noticed that if `self.whatToCompute = lalpulsar.FSTATQ_2F` isn't set, F-statistic output will be nasty uninitialised random numbers. It seems we're not initialising the FstatResults struct properly. Not sure what the proper way ...In !26 I noticed that if `self.whatToCompute = lalpulsar.FSTATQ_2F` isn't set, F-statistic output will be nasty uninitialised random numbers. It seems we're not initialising the FstatResults struct properly. Not sure what the proper way to do so through SWIG is...https://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/17install issues2020-01-14T10:52:51ZDavid Keiteldavid.keitel@ligo.orginstall issuesI'm unable to install from the current version 6057a8edc5efde55b7cfde774c2905acd249c7ad under either python 2 or 3. I'll make a branch for fixing it under py3 first, then see if py2 compatibility is still possible too.I'm unable to install from the current version 6057a8edc5efde55b7cfde774c2905acd249c7ad under either python 2 or 3. I'll make a branch for fixing it under py3 first, then see if py2 compatibility is still possible too.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/14Version mismatch between examples2020-02-05T10:50:58ZGregory AshtonVersion mismatch between examplesThe example `semi_coherent_search_using_MCMC.py ` is currently broken with the error message
```
../lib/python2.7/site-packages/pyfstat/mcmc_based_searches.py in run(self, proposal_scale_factor, create_plots, window, **kwargs)
526 ...The example `semi_coherent_search_using_MCMC.py ` is currently broken with the error message
```
../lib/python2.7/site-packages/pyfstat/mcmc_based_searches.py in run(self, proposal_scale_factor, create_plots, window, **kwargs)
526 logl=self.logl, logp=self.logp,
527 logpargs=(self.theta_prior, self.theta_keys, self.search),
--> 528 loglargs=(self.search,), betas=self.betas, a=proposal_scale_factor)
529
530
TypeError: __init__() got an unexpected keyword argument 'a'
```Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/13Potentially dangerous (on clusters) temp-file naming policy in Writer.predict...2020-02-05T10:48:39ZReinhard PrixPotentially dangerous (on clusters) temp-file naming policy in Writer.predict_Fstat()In debugging some confusing behavior/problems of pyFstat on a cluster, I noticed that the method predict_fstat() of the class Writer, in make_sfts.py:257 uses the 'label' argument to construct the filename for the temporary internal fil...In debugging some confusing behavior/problems of pyFstat on a cluster, I noticed that the method predict_fstat() of the class Writer, in make_sfts.py:257 uses the 'label' argument to construct the filename for the temporary internal file ('temporyFile') in the *user's home-dir* they launched the jobs from.
This is potentially confusing/dangerous, as it's not documented (I believe) and as there's also the argument 'outdir', which is more obvious to be required to be unique on cluster runs.
What can happen here is that someone uses the same label for all jobs of a cluster run, gives every job a unique 'outdir', and still jobs will fall over themselves when all trying to read-read-delete the same temporary file in predict_fstat().
As as fix I would simply suggest to place that temporary file inside the 'outdir' as well, then there should be no problems.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/12test failures on CIT2020-01-14T10:52:01ZDavid Keiteldavid.keitel@ligo.orgtest failures on CITI'm trying to get a PyFstat installation running on CIT, and getting test failures. (Though some basic examples to work already.)
1. something related to matplotlib backends again, MR incoming:
`[david.keitel@ldas-pcdev13 PyFstat]$ pyt...I'm trying to get a PyFstat installation running on CIT, and getting test failures. (Though some basic examples to work already.)
1. something related to matplotlib backends again, MR incoming:
`[david.keitel@ldas-pcdev13 PyFstat]$ python tests.py
Traceback (most recent call last):
File "tests.py", line 5, in <module>
import pyfstat
File "/home/david.keitel/git/PyFstat/pyfstat/__init__.py", line 3, in <module>
from .core import BaseSearchClass, ComputeFstat, SemiCoherentSearch, SemiCoherentGlitchSearch
File "/home/david.keitel/git/PyFstat/pyfstat/core.py", line 15, in <module>
import pyfstat.helper_functions as helper_functions
File "/home/david.keitel/git/PyFstat/pyfstat/helper_functions.py", line 20, in <module>
import matplotlib.pyplot as plt
File "/home/david.keitel/.local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 116, in <module>
_backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
File "/home/david.keitel/.local/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 60, in pylab_setup
[backend_name], 0)
File "/home/david.keitel/.local/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py", line 6, in <module>
from six.moves import tkinter as Tk
File "/home/david.keitel/.local/lib/python2.7/site-packages/six.py", line 203, in load_module
mod = mod._resolve()
File "/home/david.keitel/.local/lib/python2.7/site-packages/six.py", line 115, in _resolve
return _import_module(self.mod)
File "/home/david.keitel/.local/lib/python2.7/site-packages/six.py", line 82, in _import_module
__import__(name)
ImportError: No module named Tkinter
`
2. After fixing that one, some `Directory not empty` or `Device or resource busy` errors in test 12. Full log attached. Might be a shutil version issue?[test_ldas-pcdev13_20180111.log](/uploads/2e28da6129b662f65952dfcbcc60dbed/test_ldas-pcdev13_20180111.log)Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/6example failures2020-01-14T10:50:29ZDavid Keiteldavid.keitel@ligo.orgexample failuresHi @GregAshton, two issues I found with running examples on current master, the first in transient_search_using_MCMC.py and apparently related to the ptemcee move and hence I'm not sure how to debug it:
`12:12 INFO : Running final bu...Hi @GregAshton, two issues I found with running examples on current master, the first in transient_search_using_MCMC.py and apparently related to the ptemcee move and hence I'm not sure how to debug it:
`12:12 INFO : Running final burn and prod with 200 steps
0%| | 0/200 [00:00<?, ?it/s]Traceback (most recent call last):
File "examples/other_examples/transient_search_using_MCMC.py", line 47, in <module>
mcmc.run()
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 522, in run
sampler = self._run_sampler(sampler, p0, nburn=nburn, nprod=nprod)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 388, in _run_sampler
total=nburn+nprod):
File "/home/dkeitel/.local/lib/python2.7/site-packages/tqdm/_tqdm.py", line 862, in __iter__
for obj in iterable:
File "/home/dkeitel/.local/lib/python2.7/site-packages/ptemcee/sampler.py", line 355, in sample
raise ValueError('Attempting to start with samples outside posterior support.')
ValueError: Attempting to start with samples outside posterior support.
`
Probably more trivial to fix is fully_coherent_search_using_MCMC.py which runs mostly through but gives at the end:
`Traceback (most recent call last):
File "fully_coherent_search_using_MCMC.py", line 60, in <module>
mcmc.run(subtractions=[F0, F1])
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 524, in run
fig, axes = self._plot_walkers(sampler, nprod=nprod, **kwargs)
TypeError: _plot_walkers() got an unexpected keyword argument 'subtractions'`
Is it safe to just remove that argument, or is there a replacement?Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/4Confusion between F and 2F2020-01-14T10:50:23ZGregory AshtonConfusion between F and 2FAs pointed out by David:
> Meanwhile, I noticed something in the core that made me pause: run_computefstatistic_single_point() returns 2F.
> MCMCTransientSearch.logl() calls this "FS" and does "return FS + self.likelihoodcoef" where fr...As pointed out by David:
> Meanwhile, I noticed something in the core that made me pause: run_computefstatistic_single_point() returns 2F.
> MCMCTransientSearch.logl() calls this "FS" and does "return FS + self.likelihoodcoef" where from L~exp(F) I think it should >be F not 2F...?
>(Same for the base MCMC class, it just seems to have another wrapper function inbetween.)
>I guess the sampler only ever needs relative likelihoods, anyway, so it doesn't make a difference...?
This needs to be cleaned upGregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/3tests.py failure: no attribute 'exponential_time'2020-01-14T10:52:21ZDavid Keiteldavid.keitel@ligo.orgtests.py failure: no attribute 'exponential_time'Hi Greg, returning to PyFstat after a while. With 5837254ca73df2e5de432e7e409ddbbb3edf10da the examples still seem to work, but tests.py throws the following:
```
12:30 INFO : The config file TestData/Test.cff has been modified since...Hi Greg, returning to PyFstat after a while. With 5837254ca73df2e5de432e7e409ddbbb3edf10da the examples still seem to work, but tests.py throws the following:
```
12:30 INFO : The config file TestData/Test.cff has been modified since the sft file TestData/H-4800_H1_1800SFT_Test-700000000-8640000.sft was created
12:30 INFO : Now executing: lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir="TestData" --outLabel="Test" --IFOs="H1" --sqrtSX="1" --startTime=700000000 --duration=8640000 --fmin=28 --Band=4 --Tsft=1800 --injectionSources="TestData/Test.cff"
.
======================================================================
ERROR: test_fully_coherent (__main__.TestMCMCSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests.py", line 227, in test_fully_coherent
search.run(create_plots=False)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 433, in run
sampler = self._run_sampler(sampler, p0, nburn=nburn, nprod=nprod)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 345, in _run_sampler
self._run_sampler_with_conv_test(sampler, p0, nprod, nburn)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 326, in _run_sampler_with_conv_test
**self.convergence_kwargs):
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 316, in _test_convergence
return self._get_convergence_test(i, sampler, **kwargs)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 282, in test_autocorr_convergence
acors[temp, :] = emcee.autocorr.exponential_time(x)
AttributeError: 'module' object has no attribute 'exponential_time'
----------------------------------------------------------------------
Ran 11 tests in 142.569s
FAILED (errors=1)
```Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/2why int(fmin) in run_makefakedata?2020-01-14T10:50:08ZDavid Keiteldavid.keitel@ligo.orgwhy int(fmin) in run_makefakedata?Hi again Greg, last one from me for now, and since I don't know if it's a simple bug or an intended hack, this is just an issue and not a merge request yet: Writer.run_makefakedata() has the line
`cl.append('--fmin={}'.format(int(self.f...Hi again Greg, last one from me for now, and since I don't know if it's a simple bug or an intended hack, this is just an issue and not a merge request yet: Writer.run_makefakedata() has the line
`cl.append('--fmin={}'.format(int(self.fmin)))`
which, if I'm trying to work with a relatively narrow band around the injection Freq, will make MFDv5 fail with a signal/datarange mismatch error, because the --Band is not increased and hence the whole SFT range is just shifted downwards, away from the injection.
For me just removing that int() and making sure myself that the band is wide enough seems to work fine in all use cases so far, though I guess you might have introduced it as an extra security padding? But in that case I'd suggest to increase the Band as well, and ideally to make it optional.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/1PicklingError on saving sampler2020-01-14T10:49:47ZGregory AshtonPicklingError on saving samplerMessage from David Keitel:
Hi Greg,
is it generally advised to install all dependency packages through pip, instead of relying on some system-installed versions? Everything except corner is in my Debian repositories, so I tried only pi...Message from David Keitel:
Hi Greg,
is it generally advised to install all dependency packages through pip, instead of relying on some system-installed versions? Everything except corner is in my Debian repositories, so I tried only pip-ing that one, but then to be able to mcmc.run() also had to install bashplotlib which was not in your list.
After that, the transient example runs mostly through, though I get some emcee RuntimeWarnings (see below) and after the apparently still-successful MCMC run comes a fatal PicklingError (see further below).
If you think using all-pip packages is a likely cure, I'll simply try that.
Greetings,
David
```
11:59 INFO : Running final burn and prod with 200 steps
0%| | 0/200 [00:00<?, ?it/s]/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:335: RuntimeWarning: invalid value encountered in subtract
- lnprob[:, jupdate::2]
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:340: RuntimeWarning: invalid value encountered in less
accepts = logrs < logpaccept
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:384: RuntimeWarning: invalid value encountered in subtract
paccept = dbeta * (logl[i, iperm] - logl[i - 1, i1perm])
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:389: RuntimeWarning: invalid value encountered in greater
asel = (paccept > raccept)
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:402: RuntimeWarning: invalid value encountered in subtract
- dbeta * logl[i - 1, i1perm[asel]]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [01:38<00:00, 2.03it/s]
12:00 INFO : Mean acceptance fraction: [ 0.16785 0.3322 0.0562 ]
12:00 INFO : Tswap acceptance fraction: [ 0.10255 0.10935 0.11615]
12:00 INFO : Plotting temperature 0 chains
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyfstat/mcmc_based_searches.py", line 419, in run
self.save_data(sampler, samples, lnprobs, lnlikes)
File "pyfstat/mcmc_based_searches.py", line 971, in save_data
pickle.dump(d, File)
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 236, in dump
pik.dump(obj)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 1004, in save_instancemethod0
obj.im_class), obj=obj)
File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 1231, in save_type
StockPickler.save_global(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <type 'lalpulsar.PulsarDopplerParams'>: it's not found as importlib.PulsarDopplerParams
```
(Note that
import lalpulsar
lalpulsar.PulsarDopplerParams
is getting found, so probably(?) the error is not from the SWIG-LAL installation side.)Gregory AshtonGregory Ashton