VPhot Load - When it rains it pours!

Affiliation
American Association of Variable Star Observers (AAVSO)
Fri, 11/06/2015 - 22:26

VPhot users:

We have reached a level of VPhot use that has never occurred before. smiley  sad  I guess the recent forum posts and the survey have made more observers try/use the program/tool. The number of images in the queue have exceeded 2,000 per day. This is more than twice what we have seen in the past.

Frankly this has caused a condition that we cannot resolve in a few minutes. Currently many large images (16 MB) are present. These are apparently not pre-solved and may take as many as 90 seconds to pass through the queue. This means that some recent images in the queue will take more than 1 day to process. This means that when tomorrow's new images are uploaded the queue will grow. Ouch!

So what to do? The VPhot team has been actively discussing options.  Both software and hardware alternatives are possible. Most of them cannot be executed in a day or so. In fact, some will take much longer. In the interim, expect delays in processing. Try to plate-solve your images before uploading. This changes processing time from 90 sec to less than 10 secs! Yes, that much faster! Think about selecting subframes. If your image has a FOV much greater than 30 arcminutes, your target and comps will only fill the middle of the image and the remainder of the field is wasted! Look at the size of the queue before uploading your images. Hold off uploading if you can wait a while. Waiting will happen anyway!

I suspect that this issue will cause some of you to give up. This is obviously NOT what I would like to see happen. frown The VPhot team WILL work on resolving this problem. Please try some of the recommendations above, but please do not give up. 

Unhappy at this moment, Ken

 

 

 

Affiliation
American Association of Variable Star Observers (AAVSO)
VPhot Load

Hi Ken,

I have been participating in some of the VPhot loads the past week.  I have loaded time series images for five different fields and it adds up.  Mine have been a part of the 16MB images.  My field is 20' on a side and I'm using all of it for searches.  The past couple of weeks, I have been plate-solving all of my images because I have also had problems with many images not solving or solving incorrectly on a random basis.  It has been kind of a mystery.  Anyway, I checked the times for my images to process during the upload and it looks to average about 21 seconds per frame.  I have wondered why it is so long with them already being plate-solved.  Just an FYI from my recent experience.

Cheers,

Mike

Affiliation
American Association of Variable Star Observers (AAVSO)
VPhot Delays

Such a great tool that a lot of people like using it! smiley

I like the ideas of sub-frame, compressing images, and pre-solving before upload.  My unbinned images are almost 33MB, and 43 arc min square, so 2x2 binning, shooting a sub-frame, and compressing often gets that down to about 3.5MB each.  I need to work on plate solving before upload, though.

Thank you and the whole team for all your excellent work!

Clear skies,

Brad Vietje

Newbury, VT

Affiliation
American Association of Variable Star Observers (AAVSO)
"That's a problem I wish we had"

During my career, I've heard people say, "that's a problem I wish we had", usually in response to some engineer who pointed out the problems that would exist if the project as designed was really successful. I'll bet some engineering-type said something similar when the design for VPHOT included a completely cloud based operation. Well, now the problem has come home to roost, but I'm sure a work-around will be found.

I uploaded 43 images to VPHOT yesterday. All were plate solved, and each image is only 6MB. At 17x11 arcmin, cropping isn't really an option for me. I'm not sure exactly when they showed up in my work space, but it was between 18 and 24 hours later.  The uploads themselves really didn't seem to take long, but I wasn't watching them closely. So, Ken's points are well taken.

The only additional option I can think of to reduce the load on the VPHOT server is to use something other than VPHOT to do the analysis. I've "played" with using MaxIm and AIP4WIN, but wasn't impressed. Maybe I need to try to be more serious about learning the ins-and-outs of these programs. OR I could really bite the bullet and gear up for IRAF.

I'm sure the future will be brighter, one way or another.

Whit

Local version of VPhot

Hi All,

I took a CHOICE VPhot course earlier this year and think it is a fantastic tool. I've used it to analyse quite a few BSM South and BSM Berry images and find VPhot to be very convenient. However I won't be using it for my own work because it is just impractical to upload all the images from an all night time series (typically 500 DSLR frames x 4 color channels = 2000 images, each ~5MB).

But more and more observers are starting to use VPhot so the problems with shear volume of uploads will only get worse. In my opinion the best solution would be a version of VPhot that could be downloaded to the observer's computer and run locally, along the lines of VStar. Images then would not need uploading at all but would have to be plate solved locally. Obviously some data needs to be tranmitted to and from VSX and other AAVSO databases, but very much less than all those images.

I have no idea how this would be impiemented, I just put the thought forward for consideration. Cheers,

Mark

Affiliation
American Association of Variable Star Observers (AAVSO)
VPHOT loading

During discussions before I left, we were aware of the possible overloading of the system.  Several possibilities were discussed:

- moving the plate-solving to a separate computer; basically a pre-processor.  That frees up the main computer for analysis.  One thought there was a linux computer running the astrometry.net library, which is more robust than PinPoint, and then perhaps a cross-mounted disk.

- using an increasingly faster Windows instance on the cloud.

- including an option to upload only a star list, rather than an image.  You do the star finding/extraction on your local computer.  AIP4WIN, for example, can create a photometry list for all stars in an image.  Text files are easy to compress and transfer.  However, this involves a major change to VPHOT, as it would have to provide an artifical image based on the star list.

- including an option to upload a partial image.  I used to do this at USNO-Flagstaff, where I set up small cutouts around the target and comparison stars, and only stored and processed that small fraction of the entire image.  Great for high-cadence time series.  You still have real pixels, but only those pixels of interest.  This again is a major rewrite of the VPHOT software, and probably of your acquisition software as well.

- creating a local version of VPHOT.  While you might think this is the logical step, it means a total rewrite of VPHOT, something Geir was not willing to do with his limited volunteer time.  It also means a dramatic up-tick in the support effort, as there will be people wanting this on various platforms.  It would also be more difficult to maintain as a members-only benefit.

So the first two solutions were deemed our best choices for now, with knowledge that image transfer will be the slow link and the most frustrating part for those not using AAVSOnet resources.  Option #2 is what we initially chose.  Option#1 is pretty straightforward, but we didn't have the staff resources available to do it at that time.

Bottom line:  the team has been aware of the issue, and hoped that a faster processor and asking people to locally plate-solve as many images as possible would delay the day when a more complex alternative would be required.  For example, when you are doing a time series, you can usually window the image down to the region of interest before transfer so that the entire image doesn't have to be shipped.  It is rare when you need an entire 4kx4k image for measuring a single target.

Arne