Jim Roe posted to the AAVSO-Photometry discussion group about his problem with poor bandwidth and data usage quotas from his new dark sky site. I have heard this from other members and think it will be a growing issue. As CCD images get larger and bandwidth becomes restricted, it will be harder for some people to use VPhot.
He proposed two options. One is to turn VPhot into a standalone application. The other is to allow it to load star lists. I replied that the first option is likely not feasible but the second makes some sense.
Can VPhot be coded to accept a list of sources in some standard format? It can plot a field with the sources (similar to VSP?) and then the user can do their normal photometry with it. The first question is how much additional effort would it take Geir? The second is what should that format be?
Aaron
Hi Aaron,
Interesting ideas.
Uploading have always been a concern, and we have played with different ideas, including a standalone application. While possible, it will be quite a bit of work as you suggest, and also probably an increased amount of support concerning installation. So I rather not ...
I have not thought of uploading and plotting star lists as such. That is a really good idea. Only problem I see is plate solving. That is done by PinPoint, not VPHOT, and to my knowledge PinPoint does not do that with anything else than FITS files. But maybe I'm wrong?
There would be quite a bit of work involved with this method too, but if the idea is good enough and offers a fun challenge, then the amount of work is secondary :-)
Geir
PS: It is possible to create a client application that uses PinPoint to plate solve images, create star lists and upload them to VPHOT. I already have the code for most of that. But it means that users have to buy PinPoint and install it on their local machine. Perhaps an AAVSO deal could be made?
Sorry, I didn't see this thread until now (can't we set up a separate mail list that pushes messages out - it's the modern thing to do.)
A couple of notes on implementing your code to create star lists.
1) Pinpoint light is included in MaximDL. I usually run it (in batch mode) on my images before uploading them to VPHOT anyway. How about a special accomodation for us who can do it this way?
2) Astrometry.net offers a FREE solver that will work on images OR data lists (including x-y lists of objects). It will run in Cygwin on reasonably powerful computers (ie, any from the last 5-10 years). X-Y lists of objects can be generated by Source Extractor, which will also run on Cygwin (I believe - if not, then in a virtual machine). In other words, it should be possible to set up a free "appliance" that will process FITS images into ordered lists of objects by RA and Dec. I hereby volunteer to create such an appliance that can be freely downloaded by any AAVSO member if you will provide the VPHOT interface to accept the files and do the photometry and reporting. Deal?
This is an interesting concept, and of course is exactly how APASS is run - process the images locally, create star lists, transfer those star lists instead of images to HQ, and then continue the analysis here. This reduces the data upload by 10-100x, depending on the star density of the field. I've developed a standardized format that has served me personally for decades, and now serves APASS and all of the pipeline data from AAVSOnet.
In theory, once you have star lists, VPHOT should be able to process them. After all, it does go through submitted images and extract all stars from each one, creating a star list that is used for subsequent steps. You could insert your locally-generated star list at some intermediate step and let VPHOT continue afterwards. In practice, there would be a lot of programming effort from Geir to make this happen - the list of images would then have to serve both images and star lists; the image display would then have to look more like VSP for the star list option, but retain the greyscale display for the image option. I've written software before, and this update would be non-trivial, and not worth Geir's time unless enough people used it. Personally, I'd rather see his next efforts concentrated on doing an even better job of handling the transformation process.
The other large effort concerns the star lists themselves. Generating a star list in CCD coordinate space, with instrumental magnitudes, is relatively straight-forward. The AAVSO could specify a Standard Format for such lists, as each needs to have a number of parameters in the header to be able to further process. Adding coordinate transformations (WCS) is an additional step that can be difficult; Jim mentions astrometry.net as one option.
The way I've always handled this is to break the process into steps, and have a suite of programs that handle each step, generating an output file that is used as input for the next step. For example, Sextractor could generate the CCD coordinate file, which is then input to the astrometry.net routines. The problem is that you can do this for your own files, if you are a programmer or at least a competent systems-level administrator so that you can get things like Cygwin running under Windows 7. It is far more difficult to extend that so that anyone running Windows XP, or using a different vendor's CCD camera, can use the same software. Sure as rain, someone will want to process DSLR images next - are you ready? Supporting operating systems and hardware is a never-ending task!
What I would suggest instead is to leverage the enormous effort already made by the image-processing vendors such as MaximDL, AIP4WIN, etc., and suggest that they produce a standardized output format for multi-object photometry files. That is the hard part of star-list creation. Once you have that standardized CCD-coordinate, instrumental magnitude, file, something on the Cloud can add the WCS as a pre-processing step, much like what Geir is doing with Pinpoint now. So here would be my proposal: let me work with the vendors and see if we can get them to write another Standardized Format output. That is the first and most important step. I'll be at SAS in May; I know several of the vendors will also be there; maybe we can sit down at that time and discuss the project. If the vendors aren't on board, I think the project is defeated immediately.
Note that all AAVSOnet images are already run through a photometric pipeline, generating WCS-corrected star lists. For AAVSOnet at least, uploading star lists instead of images would be straight-forward.
Arne
Sounds like a good plan. You already have the output format defined, no need to re-create that, and if you could convince vendors to produce photometry files to be submitted to VPHOT, I'll do my part to have VPHOT use those files.
This is of course a project that is a bit into the future. But I'd love to hear the vendors response to your suggestion.
Geir
Arne wrote
"The way I've always handled this is to break the process into steps, and have a suite of programs that handle each step, generating an output file that is used as input for the next step. For example, Sextractor could generate the CCD coordinate file, which is then input to the astrometry.net routines. The problem is that you can do this for your own files, if you are a programmer or at least a competent systems-level administrator so that you can get things like Cygwin running under Windows 7. It is far more difficult to extend that so that anyone running Windows XP, or using a different vendor's CCD camera, can use the same software. Sure as rain, someone will want to process DSLR images next - are you ready? Supporting operating systems and hardware is a never-ending task!"
I see at least some of the difficulties, but can't never accomplished anything ;), right? I know there is a lot of specialized work involved but maybe one or more volunteers can undertake a project (and, yes, I would be willing to do what I can). We would need guidance and help. As you say, break the development process into steps. Maybe start with a file description for what is needed to interface with VPHOT (or other existing software). Let us prepare a preliminary outline for review as to sufficiency for the purpose, step by step.
As to system level knowledge, I would think an amateur who can put together a telescope/ccd system capable of generating target images in large quantities would have some basic skills and/or be willing to pick up what is needed. In particular, I'm thinking that such a person could set up Virtual Box (free virtual machine monitor) in a Windows (XP or 7) computer and boot up a specially prepared Linux virtual machine with all the necessary programs and scripts to accomplish the job. These VMs can share a folder in the host (Windows) machine such that any image file(s) can be imported into the VM and the output star list saved back to the same folder for uploading to VPHOT. Getting calibrated image files into a particular Windows folder can be accomplished with relatively simple VB scripts. I develped one that reads a simple CSV target list, commands an ASCOM telescope to the target, commands an ASCOM camera to take the appropriate images, annotates the FITS header with info needed by VPHOT, saves the calibrated images to a folder of my chosing and does it all night long. I plan to offer a presentation at the fall meeting and will make the script freely available (GPL license?).
It is the "specially prepared Linux virtual machine" that will need programming skills which, I would propose, would be the output from a volunteer team. Source Extractor and the astrometry.net software run in Linux. Once the VM is set up, working and verified a master copy can be made available for downloading by whomever wants to use it. Updates, bug fixes, etc take place in one operating system and released as necessary. If we can only handle FITS files at the beginning, it's a good start. It's been a while since I played with Source Extractor but it seems to me that is where we will have to handle DSLR images as once the sources have been extracted, the files will all look the same from there on out.
Comments? (You can't hurt my feelings!)