PyFstat issueshttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues2020-01-14T10:49:47Zhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/1PicklingError on saving sampler2020-01-14T10:49:47ZGregory AshtonPicklingError on saving samplerMessage from David Keitel:
Hi Greg,
is it generally advised to install all dependency packages through pip, instead of relying on some system-installed versions? Everything except corner is in my Debian repositories, so I tried only pi...Message from David Keitel:
Hi Greg,
is it generally advised to install all dependency packages through pip, instead of relying on some system-installed versions? Everything except corner is in my Debian repositories, so I tried only pip-ing that one, but then to be able to mcmc.run() also had to install bashplotlib which was not in your list.
After that, the transient example runs mostly through, though I get some emcee RuntimeWarnings (see below) and after the apparently still-successful MCMC run comes a fatal PicklingError (see further below).
If you think using all-pip packages is a likely cure, I'll simply try that.
Greetings,
David
```
11:59 INFO : Running final burn and prod with 200 steps
0%| | 0/200 [00:00<?, ?it/s]/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:335: RuntimeWarning: invalid value encountered in subtract
- lnprob[:, jupdate::2]
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:340: RuntimeWarning: invalid value encountered in less
accepts = logrs < logpaccept
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:384: RuntimeWarning: invalid value encountered in subtract
paccept = dbeta * (logl[i, iperm] - logl[i - 1, i1perm])
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:389: RuntimeWarning: invalid value encountered in greater
asel = (paccept > raccept)
/usr/lib/python2.7/dist-packages/emcee/ptsampler.py:402: RuntimeWarning: invalid value encountered in subtract
- dbeta * logl[i - 1, i1perm[asel]]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [01:38<00:00, 2.03it/s]
12:00 INFO : Mean acceptance fraction: [ 0.16785 0.3322 0.0562 ]
12:00 INFO : Tswap acceptance fraction: [ 0.10255 0.10935 0.11615]
12:00 INFO : Plotting temperature 0 chains
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyfstat/mcmc_based_searches.py", line 419, in run
self.save_data(sampler, samples, lnprobs, lnlikes)
File "pyfstat/mcmc_based_searches.py", line 971, in save_data
pickle.dump(d, File)
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 236, in dump
pik.dump(obj)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 1004, in save_instancemethod0
obj.im_class), obj=obj)
File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 835, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/dist-packages/dill/dill.py", line 1231, in save_type
StockPickler.save_global(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <type 'lalpulsar.PulsarDopplerParams'>: it's not found as importlib.PulsarDopplerParams
```
(Note that
import lalpulsar
lalpulsar.PulsarDopplerParams
is getting found, so probably(?) the error is not from the SWIG-LAL installation side.)Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/3tests.py failure: no attribute 'exponential_time'2020-01-14T10:52:21ZDavid Keiteldavid.keitel@ligo.orgtests.py failure: no attribute 'exponential_time'Hi Greg, returning to PyFstat after a while. With 5837254ca73df2e5de432e7e409ddbbb3edf10da the examples still seem to work, but tests.py throws the following:
```
12:30 INFO : The config file TestData/Test.cff has been modified since...Hi Greg, returning to PyFstat after a while. With 5837254ca73df2e5de432e7e409ddbbb3edf10da the examples still seem to work, but tests.py throws the following:
```
12:30 INFO : The config file TestData/Test.cff has been modified since the sft file TestData/H-4800_H1_1800SFT_Test-700000000-8640000.sft was created
12:30 INFO : Now executing: lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir="TestData" --outLabel="Test" --IFOs="H1" --sqrtSX="1" --startTime=700000000 --duration=8640000 --fmin=28 --Band=4 --Tsft=1800 --injectionSources="TestData/Test.cff"
.
======================================================================
ERROR: test_fully_coherent (__main__.TestMCMCSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests.py", line 227, in test_fully_coherent
search.run(create_plots=False)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 433, in run
sampler = self._run_sampler(sampler, p0, nburn=nburn, nprod=nprod)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 345, in _run_sampler
self._run_sampler_with_conv_test(sampler, p0, nprod, nburn)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 326, in _run_sampler_with_conv_test
**self.convergence_kwargs):
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 316, in _test_convergence
return self._get_convergence_test(i, sampler, **kwargs)
File "/home/dkeitel/git/PyFstat/pyfstat/mcmc_based_searches.py", line 282, in test_autocorr_convergence
acors[temp, :] = emcee.autocorr.exponential_time(x)
AttributeError: 'module' object has no attribute 'exponential_time'
----------------------------------------------------------------------
Ran 11 tests in 142.569s
FAILED (errors=1)
```Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/7add switches to override check_old_data_is_okay_to_use?2020-01-14T10:50:41ZDavid Keiteldavid.keitel@ligo.orgadd switches to override check_old_data_is_okay_to_use?Both the grid and MCMC search classes have check_old_data_is_okay_to_use methods, which are a good idea in principle, but might need to get quite a bit more involved to check all possible differences between search calls in all use cases...Both the grid and MCMC search classes have check_old_data_is_okay_to_use methods, which are a good idea in principle, but might need to get quite a bit more involved to check all possible differences between search calls in all use cases. (E.g. the simple grid output format currently cannot be used to distinguish between a search with and without t0,tau bands, which I'll open another issue about.)
So a nice convenience feature, at least for use during development, would be a 'reusedata' argument, default True, but when set to False just skips that check. If you like it, I can make a patch.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/issues/10reminder: add header to output files2020-02-05T10:47:15ZDavid Keiteldavid.keitel@ligo.orgreminder: add header to output filesLow-priority reminder issue for text file headers, as briefly discussed in #8:
> David: generally add a header with all search parameters (also beneficial for data archiving and external postprocessing
> Greg: Nevertheless, adding a ...Low-priority reminder issue for text file headers, as briefly discussed in #8:
> David: generally add a header with all search parameters (also beneficial for data archiving and external postprocessing
> Greg: Nevertheless, adding a header as is done by CFSv2 is something on my to-do list (also for the MCMC output)David Keiteldavid.keitel@ligo.orgDavid Keiteldavid.keitel@ligo.org