PyFstat merge requestshttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests2020-01-27T13:08:36Zhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/29Improve the version file information2020-01-27T13:08:36ZGregory AshtonImprove the version file informationThis adds the version information (and the git tag), e.g.
```
>>> import pyfstat
>>> pyfstat.__version__
'1.3: (CLEAN) 4fdfc02 2020-01-22 08:40:17 +1100'
```This adds the version information (and the git tag), e.g.
```
>>> import pyfstat
>>> pyfstat.__version__
'1.3: (CLEAN) 4fdfc02 2020-01-22 08:40:17 +1100'
```https://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/28set minimum python version to 3.52020-01-19T22:59:53ZDavid Keiteldavid.keitel@ligo.orgset minimum python version to 3.5@GregAshton as discussed last year with @ReinhardPrix in #20, I'd like to
1. set the minimum version to 3.5 right now (which is what I still have on Debian Stretch)
1. make a Zenodo release (will need you to do this)
1. bump again to ...@GregAshton as discussed last year with @ReinhardPrix in #20, I'd like to
1. set the minimum version to 3.5 right now (which is what I still have on Debian Stretch)
1. make a Zenodo release (will need you to do this)
1. bump again to 3.6 (minimum required by flake8).Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/27CI: add --diff to black2019-11-08T09:38:42ZDavid Keiteldavid.keitel@ligo.orgCI: add --diff to black@ReinhardPrix : This should make a failed job report back what are the actual formatting issues. This would be useful as it seems that black is getting stricter in higher python versions, e.g. the changes [here](https://gitlab.aei.uni-ha...@ReinhardPrix : This should make a failed job report back what are the actual formatting issues. This would be useful as it seems that black is getting stricter in higher python versions, e.g. the changes [here](https://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/commit/5fccdd3e2a75fcbfb94dfd8aa04dc32804610e86) passed black 19.10b0 for me on CIT (python 3.6.8) but fail what the CI is running.
Do you have any concerns that the larger job logs might be a problem?Reinhard PrixReinhard Prixhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/26ComputeFstat: extra flag computeAtoms independent of transientWindowType2019-11-06T11:01:45ZDavid Keiteldavid.keitel@ligo.orgComputeFstat: extra flag computeAtoms independent of transientWindowType@GregAshton what it says on the label; tested that this allows to get full atoms file output even if no transient window is set.@GregAshton what it says on the label; tested that this allows to get full atoms file output even if no transient window is set.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/25ephemerides handling fixes2019-11-06T11:04:00ZDavid Keiteldavid.keitel@ligo.orgephemerides handling fixes@GregAshton 2 fixes to ephemerides handling:
1. Fix a problem I introduced in !18 - for some reason I cannot remember (maybe just "high numbers look good") I picked DE421 as the standard to look for, while in lalapps it's currently a...@GregAshton 2 fixes to ephemerides handling:
1. Fix a problem I introduced in !18 - for some reason I cannot remember (maybe just "high numbers look good") I picked DE421 as the standard to look for, while in lalapps it's currently actually DE405 for both CFSv2 and MFDv5.
1. Teach Writer.run_makefakedata to actually pass through the chosen ephem files to the MFDv5 command line. The search class already properly handles them through InitBarycenter, while for making SFTs it seems we never actually did this properly so far.
I'll also need to add instructions for manual ephemerides download to the README, until a better way to provide those to pip users can be figured out; but I'll do that after !23 to avoid conflicts.
(fixed-up version of !24)
cc @ReinhardPrix in case you have opinions on the default version?Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/23Fix py3 setup2019-11-06T10:14:22ZDavid Keiteldavid.keitel@ligo.orgFix py3 setupSee #17 for more details, this allows me to install again under both Python 3.5.3 on my Debian laptop and Python 3.6.8 on CIT.
@GregAshton I'm not really fluent in packaging and all changes are ad-hoc hacks, so please have a careful loo...See #17 for more details, this allows me to install again under both Python 3.5.3 on my Debian laptop and Python 3.6.8 on CIT.
@GregAshton I'm not really fluent in packaging and all changes are ad-hoc hacks, so please have a careful look whether these are all reasonable.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/22Gitlab ci runner setup2019-08-06T15:06:44ZReinhard PrixGitlab ci runner setupok, I've attached a little runner (an old laptop of mine ;)), and written a gitlab-ci script that runs the tests and also checks for 'black' code-style, all seems to be passing right now (after another small fix):
https://gitlab.aei.uni-...ok, I've attached a little runner (an old laptop of mine ;)), and written a gitlab-ci script that runs the tests and also checks for 'black' code-style, all seems to be passing right now (after another small fix):
https://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/pipelines
/cc @dkeitelGregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/21Make tests more robust2019-08-06T08:21:47ZReinhard PrixMake tests more robust@GregAshton I noticed a few minor portability issues when running the tests on different platforms (cluster nodes, MacOSX), these patches fix all those I've come across.
/cc @dkeitel@GregAshton I noticed a few minor portability issues when running the tests on different platforms (cluster nodes, MacOSX), these patches fix all those I've come across.
/cc @dkeitelGregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/20Use black pep8 codestyle2019-07-01T13:10:14ZReinhard PrixUse black pep8 codestyle@GregAshton another change I'd like to do on my own fork: use 'black' code formatter, which I'm a fan of to fully get rid of formatting-style ambiguities resulting in uniform-looking code.
[this should be merged only after the previous ...@GregAshton another change I'd like to do on my own fork: use 'black' code formatter, which I'm a fan of to fully get rid of formatting-style ambiguities resulting in uniform-looking code.
[this should be merged only after the previous one !19 as it sits on top of it]
see https://black.readthedocs.io/en/stable/Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/19Port to python3 and new lalsuite2019-07-01T13:09:57ZReinhard PrixPort to python3 and new lalsuiteHi [Greg](@GregAshton), I've recently been playing around with pyFstat, and wanted to update/modernize various aspects a bit.
If you're happy to integrate such changes into your upstream repo, I'll send you a few merge-requests, otherwi...Hi [Greg](@GregAshton), I've recently been playing around with pyFstat, and wanted to update/modernize various aspects a bit.
If you're happy to integrate such changes into your upstream repo, I'll send you a few merge-requests, otherwise I'm also ok to keep this stuff on my fork.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/18ephemerides finding improvement2019-11-05T11:52:43ZDavid Keiteldavid.keitel@ligo.orgephemerides finding improvementHi @GregAshton , I was trying to completely isolate a PyFstat installation inside a venv without relying on anything in my home directory, and so I came up with this proposed solution to get ephemerides paths from the $LALPULSAR_DATADIR ...Hi @GregAshton , I was trying to completely isolate a PyFstat installation inside a venv without relying on anything in my home directory, and so I came up with this proposed solution to get ephemerides paths from the $LALPULSAR_DATADIR env variable. In principle this could be made smarter about which version of the files it chooses, but for now I've just told it to try the new 00-40-DE421 and old 00-19-DE421 standards, then fall back to asking for manual user input. Any other use case can still be covered by the conf file approach, which is still checked first and hence can override the env variable. I also added an error catching for the case where the file exists, but is missing one or both of the actual path definitions, and updated the README.
Do you think this covers all relevant cases?Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/17add pycuda as optional dependency2018-08-07T00:04:19ZDavid Keiteldavid.keitel@ligo.orgadd pycuda as optional dependencyEventually remembered to document this...Eventually remembered to document this...Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/16internal timing of transient F-stat map function2018-05-07T22:01:35ZDavid Keiteldavid.keitel@ligo.orginternal timing of transient F-stat map functionFor `TransientGridSearch` class, saves an internal attribute that adds up timing of all calls to `call_compute_transient_fstat_map()`. Also printed out (on info level) at the end, I might change that to debug at a later point, but info i...For `TransientGridSearch` class, saves an internal attribute that adds up timing of all calls to `call_compute_transient_fstat_map()`. Also printed out (on info level) at the end, I might change that to debug at a later point, but info is good for me at the moment.
@GregAshton Pipeline failed as usual while you're offline, so please give it a 2nd try.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/15transient search: user option tauMin2018-03-20T05:21:07ZDavid Keiteldavid.keitel@ligo.orgtransient search: user option tauMin@GregAshton : Just another relatively straight-forward user input extension I forgot in the earlier big tCW MR.
The equivalent t0Min is not needed as minStartTime can play its role in each case I've thought about.@GregAshton : Just another relatively straight-forward user input extension I forgot in the earlier big tCW MR.
The equivalent t0Min is not needed as minStartTime can play its role in each case I've thought about.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/14writer function for atoms (TransientGridSearch)2018-02-05T11:28:51ZDavid Keiteldavid.keitel@ligo.orgwriter function for atoms (TransientGridSearch)Here's a simpler merge request that can go in before !13, just adding a write_atoms_to_file method for debugging purposes to the standard ComputeFstat class, wrapping lalpulsar.write_MultiFstatAtoms_to_fp().
It's another feature that I ...Here's a simpler merge request that can go in before !13, just adding a write_atoms_to_file method for debugging purposes to the standard ComputeFstat class, wrapping lalpulsar.write_MultiFstatAtoms_to_fp().
It's another feature that I only really need for TransientGridSearch at the moment, but in principle other search classes could request the `whatToCompute = lalpulsar.FSTATQ_ATOMS_PER_DET` flag too, so I think it makes sense to have this at the ComputeFstat level.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/13pyCUDA version of transient F-stat2018-02-09T02:11:14ZDavid Keiteldavid.keitel@ligo.orgpyCUDA version of transient F-statThis adds a pyCUDA implementation of lalpulsar's ComputeTransientFstatMap function, to be used for now only from the TransientGridSearch() class but in principle portable to other search classes too.
There are two separate kernel fil...This adds a pyCUDA implementation of lalpulsar's ComputeTransientFstatMap function, to be used for now only from the TransientGridSearch() class but in principle portable to other search classes too.
There are two separate kernel files (.cu) installed as 'package_data', and a source file tcw_fstat_map_funcs.py that includes both the direct wrappers to those GPU kernel and some setup acrobatics.
@GregAshton Could you have a preliminary look and first let me know if you think this is all in scope for PyFstat in the first place, or if you'd prefer the kernels and wrappers to live in an external package?
Then here's a (non-exhaustive) list of hacks or possibly controversial implementation details:
1. Since pycuda won't be installed by default on many systems, I'm doing optional imports in init_transient_fstat_map_features(), for which there might be a more elegant and pythonic solution.
1. I've had to add an explicit `__del__` destructor to the ComputeFstat class. pyCuda has a nice autoinit feature which would do the cleanup too, but for multi-GPU hosts (e.g. CIT head nodes) I need to manually work with what it calls "contexts" and then clean that up in the end, too.
1. The lal and pycuda versions currently use slightly different FstatMap objects, which I could unify in future if the if-elses looks too ugly.
1. The last fixup commit adds explicit defaults for the new ComputeFstat attributes also to the derived SemiCoherentSearch and SemiCoherentGlitchSearch classes; I was hoping they could somehow inherit the defaults from their base class, but I guess that would require explicitly calling its `__init__` from the child...?
1. The commit history is a bit messy, e.g. the "choose from multiple GPUs with CUDA_DEVICE environment variable" is superseded by later commits, but if it's not too much of a bother I'd like to keep this history, because if any issues arise it might become necessary to return to that simpler implementation.
I'm also still chasing some larger-than-expected differences between CPU and GPU versions for exponential windows, though those could just be due to Reinhard's "FastExp" functions. Which is why this is `WIP:` and self-assigned for now. Feedback already much appreciated, though!Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/12displayless matplotlib import workaround also in helper_functions.py2018-01-31T05:07:16ZDavid Keiteldavid.keitel@ligo.orgdisplayless matplotlib import workaround also in helper_functions.pyAddresses the first test failure on CIT mentioned in #12. The same workaround is already present in core.py, not sure why on other systems I hadn't seen the import problem triggered from within helper_functions.py like it now does.
Note...Addresses the first test failure on CIT mentioned in #12. The same workaround is already present in core.py, not sure why on other systems I hadn't seen the import problem triggered from within helper_functions.py like it now does.
Note other files still have the plain import, but in all scenarios I tried, fixing these two seems to be sufficient. I'd assume enforcing agg once per session is sufficient, and only in some special circumstances are the helper_functions called before the import from core, whereas grid_based_searches and mcmc_based_searches always have had the import from core before they call matplotlib themselves.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/11transients: fix window=none case and allow custom dt0, dtau2018-02-08T04:15:48ZDavid Keiteldavid.keitel@ligo.orgtransients: fix window=none case and allow custom dt0, dtauThis fixes the `windowRange` struct for type=none, in accordance with XLALComputeTransientFstatMap. And since that meant I needed to touch the if-else construct for default t0, tau spacings anyway, I chose this opportunity to make them u...This fixes the `windowRange` struct for type=none, in accordance with XLALComputeTransientFstatMap. And since that meant I needed to touch the if-else construct for default t0, tau spacings anyway, I chose this opportunity to make them user-configurable too.
It seems somewhat messy and contrary to the classes idea to keep adding extra transient parameters to the core ComputeFstat class. But we talked in the past about a "hybrid" MCMC search that only samples in Doppler params, with the transient params marginalised over a grid, and I think to keep that option open, it's probably best to continue in this way.Gregory AshtonGregory Ashtonhttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/10Create TransientGridSearch to provide transient grid searches2018-01-09T15:32:12ZGregory AshtonCreate TransientGridSearch to provide transient grid searchesPreviously, this functionality was part of GridSearch, this splits this
into a separate subclass to ease future development.
Closes #11
@dkeitel - let me know if there are any problems with this. I've hardcoded in the `nsegs=1` as I d...Previously, this functionality was part of GridSearch, this splits this
into a separate subclass to ease future development.
Closes #11
@dkeitel - let me know if there are any problems with this. I've hardcoded in the `nsegs=1` as I don't think there is any reason or plan to do semi-coherent transient searches right?David Keiteldavid.keitel@ligo.orgDavid Keiteldavid.keitel@ligo.orghttps://gitlab.aei.uni-hannover.de/GregAshton/PyFstat/-/merge_requests/9Adds the basic header with simple information2017-12-22T09:36:36ZGregory AshtonAdds the basic header with simple informationFixes #10 (at least for the gridded searches)
This adds a basic header with the data, username, and hostname information. Example output (from `examples/transient_examples/short_transient_search_gridded.py`)
```
# date:2017-12-20 11:54...Fixes #10 (at least for the gridded searches)
This adds a basic header with the data, username, and hostname information. Example output (from `examples/transient_examples/short_transient_search_gridded.py`)
```
# date:2017-12-20 11:54:51.736768;user:greg;hostname:greg-len-ubuntu
# _ _ F0 F1 F2 Alpha Delta
1.000000000000000000e+09 1.000172800000000000e+09 2.999998991056105524e+01 -1.000000000000000036e-10 0.000000000000000000e+00 5.000000000000000000e-01 1.000000000000000000e+00 4.206904220581054688e+01 1.000097200000000000e+09 1.980000000000000000e+04
1.000000000000000000e+09 1.000172800000000000e+09 2.999999011234983470e+01 -1.000000000000000036e-10 0.000000000000000000e+00 5.000000000000000000e-01 1.000000000000000000e+00 4.204999923706054688e+01 1.000097200000000000e+09 1.980000000000000000e+04
```
@dkeitel Am I missing anything from the header?David Keiteldavid.keitel@ligo.orgDavid Keiteldavid.keitel@ligo.org