gridsearch output format and (t0,tau)band use case
For the implementation of explicit t0,tau search bands in !7 (merged) I'm a bit in conflict with PyFstat's output file format from gridded searches, which so far is a simple headerless file with columns
startTime endTime F0 F1 F2 Alpha Delta 2F
.
First a question: the original transient search style you had in mind was probably to pass arrays of minStartTime, maxStartTime and then the search is an (up to) 7D grid of ComputeFStat calls over all those columns? That's how I parse GridSearch.get_input_data_array too.
Whereas the !7 (merged) version is closer to CFSv2, max-5D search and transient postprocessing over a t0,tau grid at each point. I can see value in both approaches and we should be able to support both in the same interface and output format.
In the !7 (merged) implementation, it now still writes out the same overall minStartTime and maxStartTime into the first 2 columns for every Doppler candidate, and in the last column max(2F) maximised over the t0,tau grid. Problem with that is information loss, as a symptom of which running a new GridSearch with added transientWindowType, t0Band, tauBand arguments in a directory with an existing output file from identical Doppler parameters will not trip check_old_data_is_okay_to_use and hence not actually rerun the search.
Simple solutions:
- in the (t0Band,tauBand) usecase, replace the first 2 columns by the ML estimators (t0,t0+tau at which F=maxF)
- in the (t0Band,tauBand) usecase, add extra columns
- generally add a header with all search parameters (also beneficial for data archiving and external postprocessing)
More complicated solution:
- wrap lalpulsar file-writing functions
What do you think @GregAshton ?