Skip to main content

Automated image prossesing

akjalam2's picture
akjalam2
Offline
Joined: 2012-01-05

 

As the number of Robotic Telescopes increase amongst amatures i would like to see The AAVSO having more information on how to setup 

an pipeline for image processing / reduction / photometry ....

There is not much information on how for ex the AAVSO robotic network works behinde the scene.

Ive seen some (very little info) in the forum on this subject about Iraf doing some of the work but else we are let out in the dark.

I have looked into just IRAF and Sextractor using my Mac and after a great deal of agony over installationproblems i finally got them running.

IRAF is a fantastic tool .. and when you learn to type again :) (command line ) it is real fun.

When you do Robotic Telescope mini surveys using for ex CCD Commander a night you get a heavy load of data and i think that its just a matter of time before we will se more observers go into the Robotic field so it would be nice if AAVSO would take the step to be the "leaders" of info even in this exiting

uppcomming field.

 

So let us in on the scripting secrets:)

 

//Pierre

Observatory Automation
STHO's picture
STHO
Offline
Joined: 2010-09-01

Pierre,

I can't help much with the current pipeline processing of the data taken with APASS but I can assist in the automation of the observatory from a hardware and environmental telemetry standpoint. I think it is a very good suggestion that we start discussing the data pipeline process and that we get onboard with the current processing system that is being used in the AAVSONet systems.

If there is anything that I can do to assist in the hardware automation or integration please let me know.

Regards,

Thomas C. Smith

Dark Ridge Observatory

Automated processing
roe's picture
roe
Online
Joined: 2010-07-25

Well, we have VPHOT to do photometry.  I wouldn't call it a pipeline yet but it has features that can speed up processing a lot.  Geir has implemented a procedure whereby with the correct FITS header items (especially one identifying which sequence to use) images are processed upon receipt and sent directly to the Analysis Log.  From there the results can be examined and the AAVSO report generated with a few clicks.  It is a feature I requested some time back and I have been "testing" it ever since (I wouldn't tackle the number of targets I do without it).  You'll have to ask Geir if he is ready to release the information for general use.

There are several manual steps that are necessary, some of them one-time setup type things.

  • The proper items of the FITS header have to be set up (I use a script to do this for each image)
  • The images have to be calibrated (BDF) (and that can be automated to an extent)
  • The telescope parameters have to be set up in VPHOT with an identifying entry in the FITS header
  • The correct sequences have to be set up in VPHOT (every time you add a new target a new sequence has to be set up for it)
  • The results VPHOT comes up with have to be reviewed carefully as spurious results can happen
  •  The results to be reported to AAVSO have to be selected
  • The VPHOT output is a text file in the AAVSO format which must be manually uploaded to WebObs

The worst part for me is the images must be uploaded, there is no pre-processing (yet).  For those with a big data pipe that may not be a concern.  For those with slower connections or data quotas, maybe more of a problem.

We have discussed this topic before and It seems to be just another item on already full plates.  The description above is my adaptation until something else comes along.

Jim Roe

Sextractor based pipelines
Bikeman's picture
Bikeman
Offline
Joined: 2010-07-30

akjalam2 wrote:

I have looked into just IRAF and Sextractor using my Mac and after a great deal of agony over installationproblems i finally got them running.

IRAF is a fantastic tool .. and when you learn to type again :) (command line ) it is real fun.

I'm experimenting with SExtractor as well, I found it easiest to just take the SciSoft collection from ESO, and install it inside its recommended habitat, Fedora Linux. All of it runs happily inside a VirtualBox VM on my Mac, my Windows notebook and a Linux host. Scisoft has Sextractor, Scamp (for plate solving), IRAF (which I'm not using) and tons of other useful stuff.  

I'm doing DSLR photometry, producing wide fields which are tedious to reduce, so an automatic pipeline (w/o image calibration, tho: the inputs are calibrated images in FITS) is what I have in mind as well. It works reasonably well so far but needs more work to make it robust and easy to use.

CS

HBE

AAVSOnet pipeline
HQA's picture
HQA
Offline
Joined: 2010-05-10

The current processing pipeline, used by both APASS and AAVSOnet is a mixture of IRAF, plus Fortran programs based on DAOPHOT aperture photometry routines and SLALIB astrometric libraries, cobbled together with scripts.  I've developed this over a couple of decades, which is why it is non-homogeneous.  However, we need to update this to a system that others can support; I won't be around forever!

Because APASS is run by the same pipeline, and I need better deblending in the APASS images, I am switching over its processing to either Sextractor or PSFextractor this summer.  At that time, we will be removing the IRAF portion by recoding the image processing in Python, and adding in Astrometry.net WCS routines instead of the method currently used.  Then we get to reprocess about a thousand nights of APASS imaging. :-)

For AAVSOnet, my wish is for everything to be platform independent, probably Python based.  That way, the Windows machines can process data during the daylight hours, or transfer the images over to the linux machine for parallel processing while imaging is underway.  I think such a pipeline could then be fruitfully used by others for their own robotic setups.  If some programmers want to give Matt and I a hand at this, we can generate a design document and assign programming tasks.  My understanding of Windows environment is pretty limited, so help there would be appreciated!  However, I'd then request that this thread resume in the software forum.

I want to stay clear of the image acquisition/observatory automation loop, as I feel there are vendors who are actively supporting that aspect.

Arne

Amateur automated photometry loop
akjalam2's picture
akjalam2
Offline
Joined: 2012-01-05

"I want to stay clear of the image acquisition/observatory automation loop, as I feel there are vendors who are actively supporting that aspect."

I cant agree on that.

I see it as AAVSO should look into progress and wishes amongst amateurs for future observation tequnices.

There will always be vendors of this and that but what drives the amateur is to be able to do what others sell on his own. At least if you are interested in tequnice.

Automated photometry pipleines will be there for amateurs ..its just a matter of time.

Ive looked into IRAF and more on PYRAF (iraf in a python enviorment.

Installing Sextractor on MAC is not as easy as you wish (just read the forum of Astromatic.net :( )

Iraf/Pyraf is alot easier when you understand that you have to setup an Iraf user account.

And with the help of Astrometry.net code you can easliy do the WCS part too.

I will try to set up my own pipline just for the fun of it and share my scripts and progress here.

//Pierre

AAVSO 49 Bay State Rd. Cambridge, MA 02138 aavso@aavso.org 617-354-0484