Hello All,
I appreciate all of the feedback with respect to LCGv1 on the other forum. However, I feel it is more appropriate for LCG v1 to have it's own discussion thread. First let's give everyone a better understanding of logistics. The AAVSO has to update its infrastructure. I want to make sure everyone understands that up front. Most of the softwared on our Web server is so old that it no longer supported. This keeps us stuck, unable to make changes for fear of breaking the entire site. You may have noticed that certain things have stopped working or work erratically and this will only become worse over time. Keeping things exactly the same simply isn't an option.
That being said we aren't trying to make a unilateral decision on the matter. We want to provide our users with what they find to be the most useful. However, we only have so many resources at our disposal. LCGv2 and Vstar were graciously written and maintained by volunteers who we are very happy to have. LCGv1 was written so long ago that not only is that person no longer on staff, but we aren't even sure who wrote it in the first place. As you can see this is problematic. We might be able to port the tool with resources and money, but more importantly it requires sustained maintenance and updating. For example, soon APASS DR10 will be available. It has a whole new set of filters which LCGv1 doesn't support. While we have people to make these changes for other tools, LCGv1 doesn't have this and won't for the forseeable future. This is just one example of many I could give, such as support for exoplanets or spectra which will also soon be available. Addtionally, it is a replication of resources as many of LCGv1s functions overlap with other tools, even if they are not entirely the same. Given our small staff we have to make sure we are not overtaxing our abilities here or other areas will suffer.
Okay, now that everyone has proper context, I want to come up with a compromise that best addresses everyones wants and needs. LCGv1 was already effectively deprecated before I even joined the staff, so I have had very little occasion to use it. It is also clear that how I would use the tool is not how many of you are. For this reason, I need to know how you are using the tool, what you do with it and why the other tools don't meet your needs. The two things I have noticed so far is that speed is a concern, also the ability to make publishable plots. I am looking into this. However Can you please expound upon these issues and features from your own experience so that I have a better understanding of what functionality everyone wants? We want to serve you as a community as best we can and we need your input to do it.
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
Just for clarification, LCGv1 was written using gnuplot by Aaron Price in the early 2000's. It has two redeeming features: it is very fast, and it produces png files that can then be easily included in publications and posters. As Bert mentions, it has not been maintained and so does not support new AID features like additional bandpasses. While a volunteer programmer might be found to update and maintain LCGv1, my personal feeling is that it would be better to spend some time improving LCGv2 so that it includes those missing features from v1 that power users desire.
Arne
Bert,
just try to plot ASASSN-18ey as I mentioned in my post to the other thread. LCG2 did not plot anything and VSTAR after more than 45 min did it. However one can not select the own measurements in VSTAR (at least it is not obviuos). Speed is definitley an issue and LCG2 is not up to it even if it would plot the data.
I am preparing a poster for a conference and it6 took me 5 min to generate the figures for three stars with LCG1 and put them into the poster. I could highlight my observations (with blue squares). It would be much more cumbersome with LCG2 to start with. I have added a screen shot of that part of the poster. Also a forum should be part of the website like the one we are using.
Josch
i like lcg1 bc it is fast, real fast. i use lcg1 for planning my observing sessions and for checking that an observed value i posted makes sense. for example using lcg1 i have caught my own logging and analysis errors and edited my observation to correct it. time for me is of the essence, doesnt matter to me what happens to the old lcg1 but i would recommend that somehow someone can step up and create a very similar program that is compatible with the newer infrastructure upgrades or if that is not possible to make the changes so that lcg2 is almost instantaneous. the thing to remember is that we are all volunteering our time and my fear is that if the tools aavso provides us are not fast many folks will find something else to do with their time and/or the number of observations will decline.
One of the annoyances is just that the handoff from other parts of the website to LCG2 require retyping of the star name and typing in a logical number of days in the recent past to plot. If you are researching a star it would be better to be able to land on the light curve without those annoyances, for instance from the star map. It seems to me that this handoff could be handled much more eligantly.
You cannot satisfy everyone in the way tools are setup. Hoeever you can setup configuration settings in the member profiles so that tools work the way the member wants. For instance preferred filter and day settings in LCG2.
To what extent is LCG2 dependent on the rest of the site? Does it have to harmonise with the CMS used (Drupal I believe)? If so, does it use a plug-in, or is it coded from scratch? I have to say I don't find it particularly slow, and I don't have a blindingly fast connection either. I do think that the layout and styling could do with the attentions of someone with a nodding acquaintance with aesthetics however.
Hey Mike,
To answer your question, LCGv2 has no connection to the rest of the site. It is a stand alone javascript application which simply is linked to various places in the site. This independence means that it is not dependent at all on the CMS, but also means that it is difficult to pass information contained within the CMS to it.
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
As I mentioned in the other thread, I was using LCGv1 for observation planning. I would love to have a "last X days" feature to be able to plot the current behavior of a particular star. I called the LCGv1 from a link in my Excel spreadsheet and used it to determine if a star was in range of a particular instrument and which chart would be appropriate. I usually plot the last three years for a Mira and three to five times the average period for a UG for instance. You get the idea.
Tim
Hi Tim,
I just asked the author of the LCGv2 if this could be added to the API options. Will let you know what happens.
Many thanks,
Sara
Thank you Sara. I appreciate that.
Tim
Hi Sara,
I've managed to make a workaround in my spreadsheet using the TODAY() function and calculating the current JD using that. I've been able to pass that as the tojd parameter. The fromjd parameter is then a simple matter of subtracting the desired number of days.
Thank you again for your help.
Tim
Excellent news Tim!
I'm glad that you were able to come up with this workaround because it didn't look like it was going to be all that straightforward on our end.
Many thanks,
Sara
FWIW, my main LCG V1 usage: after every processing session, I look at my every submitted data point in context of the global lightcurve data. In the winter this reaches 100 targets and 250 data points a night. LCG V2 is beyond hopeless for this.
What my observing program requires--what AAVSO has always provided--is a very fast app, in which I can simply type the names of target stars sequentially--nothing else--and get clear, simple lightcurve plots. The value of V1's speed and clarity-at-a-glance is not about convenience, but about staying in the flow of one's work. LCG V2 breaks that flow at every turn, disqualifying it completely. It must be designed for other users, and that's fine, but LCG V1 (or as power users refer to it, "LCG") satisfies all the criteria right now, always has. Perhaps a subset of V2, radically streamlined beyond all recognition, could be made to work--great. But if V1 is simply trashed with nothing useful to replace it, my only options are to write my own local LCG V1 that polls webobs to make plots I can use, or find something else to do in astronomy. I don't know how to put it more plainly than that.
I completely agree with Eric Dose.
Another point is the very useful (for me) field
WHAT SHOULD THE SIZE OF THE PLOTTED POINTS BE?
Enter a pixel width
I often use 0.3 for edition purpose, the default size of 1 is too big.
NB: Because of this LCG announce, and for the first time, I was looking for a replacement solution for my variable observations, just in case, just to be ready if LCGv1 desapear.
I fully understand AAVSO's reasons for the decision and I believe it's time to move forward.
I also understand the needs of the users that are used to LCG (v1), and I believe that those of us who find it nice, useful, indispensable, or any other, can look for alternatives.
In fact, one does not need to do too much. LCG v1 is based on gnuplot (gnuplot.info), which is free and open source software. As one can download the data, there is in principle no hinder to make one's own plots as one likes, that is, very similar to LCG v1 if one wishes to.
This is in fact what I have done. I have written a short program (in Perl) that plots lightcurves using gnuplot as plotting routine. It generates plots that look much like LCG v1. If someone finds this interesting you can download it from this repository:
https://figshare.com/articles/LCplot_-_a_program_to_plot_light_curves_of_variable_stars
/6964760
The plots that it generates look like the attached file. All parameters can be tweaked using gnuplot's own syntax.
Please pay attention that this program needs gnuplot and a few perl modules. Also it is only tested in Linux.
LCplot is free and open source software distributed under LGPL v.3.
Hope you enjoy!
Could that replace LCGv1 on the updated website?
Would be great and independent of operating system.
Josch
Thanks for your interest, Josch. I believe it could, but not without a bit of coding. Afterall, both LCGv1 and LCplot act as interfaces for plotting AAVSO data using gnuplot.(EDIT: There is however the issue of the integration with the technology behind the new site. That's also a decision of the developer team. Perhaps, this little program can better live its life as a standalone thing).
The only difficulty that needs to be overcome to port this to windows is to modify a step that is currently needed to eliminate a new line that is introduced when generating the first chunk of gnuplot config file (in the case no observer highlight is required). To do this I use the shell (bash) tool "truncate", which is an essential part of *NIX but I believe is absent in Windows. Another workaround is to find a pure Perl solution, or rewrite the program in another language.
I do not claim that this is the most beautiful or efficient code. There are surely coders out there that can do better. But LCplot is free and open source software, so anyone is free to investigate the code and improve it!
and work in platforms other than Linux if dependencies are met (Perl, some Perl modules, and gnuplot).
Code can be found here:
https://figshare.com/articles/LCplot_-_a_program_to_plot_light_curves_o…
Hope it works for you.
Hernán [DHEB]
As the second, I guess, of what will probably become a chain-posting of folks' LCG V1 replacements, I'll note here that I built a rudimentary python Lightcurve plotter this afternoon, as a demo only. Much, much easier than I had ever anticipated.
I'll attach here the first straw-man plot of this woefully incomplete code (total programming time so far ~ 2.5 hours; unfair advantage as I could adapt code from my current processing system). I plan to even have the code read my uploaded submission file and so I can plot all that night's stars by hitting Enter from one to the next.
If anyone's interested, I'll throw the (probably rapidly evolving) code onto my github site. the usual cautions: No warranties for the time being. It will require importing pandas and matplotlib but so far nothing else important. In time (maybe this weekend) I'll tweak it, especially observer highlighting, plotting uncertainties or not, change point size, select ...anything else that people suggest that makes sense.
I'm calling it 'pylcg' for now--better names earnestly solicited!
Eric,
Thank you for your kind offer. I would be very interested in a Python version of LCG.
Frank Schorr (SFRA)
Thanks Eric.
I for one would like to see the code.
Thanks for the interest. I now plan to push the python-based local-LCG V1 clone (working name pylcg) to github.com/edose by late Sunday, earlier partial pushes possible. It will probably have the multi-plot feature, which extracts star IDs directly from a WebObs upload file, very useful for rapidly reviewing recently uploaded data. Still looks like the only dependences will be matplotlib, pandas, and python 3.
Hi Eric,
very interesting, please also provide an executable of the code. I am not a programmer and do not have Python installed. My son prepared a Python program for me to sort my multitude of star data into the specific directories, which beforehand I did manually which was a pain in the neck. Now it works like a charm. Looking forward to see the executable.
Josch
An executable--yes, for general use that's probably a good idea though if I've never done it. But if I can learn tkinter this weekend, surely I (or someone else out there--hint, hint) could do this too. Probably after beta testing by our python gurus.
And I assume the executable would be Windows-only?
Forum monitors: Do we need to move this sub-thread to a separate forum topic, still under Software Dev?
Hi Eric,
since I am not a programmer, I have asked my son to have a look at the code and generate a Win executable. Sorry to say but he was not very exited abut your code. Apprently you hardcoded two examples. Also if you read e.g. a large number of data (like e.g. for ASASSN-18ey) your code will be very slow as well.
We will have a look at a solution as soon as I am back home and if we succedd to read out the data from the AAVSO database (I hope at least this should be useful in the code) we will post a possible program here.
Josch
The code is not finished, and the github repository is quite clear about this. Naturally there is some temporary hardcoding, but this comes out as the pieces start working together. That's how it's done.
Even so, my apologies if this wasted your time. I have taken the repo private for now, and I will post a note here if and when the software is more generally useful to others.
While I'm here, let me attach the latest (unfinished) screenshot.
Hi Eric,
sorry for me earlier post. This looks much better now. However you should try to load a big dataset e.g. ASASSN-18ey) to also see how this behaves.
Anyway maybe we can see how to improve the speed of the program in case of needs.
Josch
We're all good, Josch. Actually, the code itself is superfast, it's downloading the data from AAVSO (VSX API) that takes ~ 1 second. But for most datasets that's already as fast as or faster than the current web LCG V1. Of course ideas about speeding that download are always welcome.
Good idea about super-big datasets--that is now on my Smoke Test list.
Eric, I'm sorry that Josch's unfortunate remarks made you take the code private. Of course, it is most instructive and interesting to follow the code as it develops.
I hope you will reconsider.
Jim Jones
Actually, Josch's comments were useful. Going public is a compromise between usefulness to coders/hackers vs usefulness to users-only. I was too early. The repo will go public later, when it's well enough developed to elicit suggestions or even pull requests. And as git works, the dev history you seek will still be there--likely to my embarrassment but there it is.
Today I'm stuck on a bona fide tkinter bug (garbage collection is disconnecting checkbuttons during "long" wait for data download); nothing more develops until I can hack around that.
Mostly--thank you for the encouragement.
Hi Eric,
could you please share any progress you made in the recent weeks?
Thanks,
Josch
Gee, all this great activity to replace LCG1 functionallity. I suppose there are options available.
Having volunteered for various dubious tasks over the years, I can attest that a volunteer will generally loose energy or interest after a time. I can only thank them for the marvelous job they have done so far.
If HQ is dogged in their persuit of LCG2, why not build all of that functionality into LCG2 so we don't have to avoid using LCG2?
Also, LCG2 will soon become unmanageable code just like LCG1 is now. Is there an LCG_3 in the works that returns all of the funtionality of LCG1 and adds the good stuff from LCG2 and VSTAR plus spectra?
The data download interface could certainly be optimized for large datasets and choice of file format. It's a pain to have to down-load and parse or reformat or cut-and-paste to re-upload or plot data. Maybe the download format could have an option to match the WEBOBS upload format.
Pehaps an AAVSO gnuplot or pyplot program with AAVSO data hooks can be created. It could be downloaded, then fed from the lage-dataset-optimized download facillity.
Maybe VSTAR with a little better start/stop date interface. The variety of specialized plots in VSTAR are fantastic. Also the ability to source other large databases. I actually use them. Maybe all that funtionality can be combined with good things from the LGC's to make an LCG_3. Then add the new spectra data.
Seems like we need a single, really fast, functional, all purpose, data up-load, down-load, plotting for publications program that can be farmed out to a pro-shop (if money exists).
Does anyone have a dollar/time estimate for a pro-shop to start from scratch and meet our specifications? Can AAVSO recoup some expenses by repurposig such new code for other large data-base users?
Big data is big business. Maybe what we need is already out there.
That could be compared with relying on the good graces of volunteers to patch WEBOBS, LCG1, LCG2. Like me, and the code, those volunteers are aging.
Ray
I have 90% completed my locally-run LCG V1 replacement. I think I have also figured out how to make a lightweight Windows 7/8/10 executable from it to ensure that it runs locally without needing python or anything else.
As of now this plotter has:
Plotting time is ~ 1 second or less including download. Redraws including zoom in/out and resetting filter/band choices are essentially instantaneous. Later we can add: clicking sequentially through the stars in an upload file for review (probable), and showing table of observers for plotted data (if there's interest).
I expect to complete the beta version by this Thursday. Python 3 code will be on github when stable, and I will be happy to send the executable file to perhaps 2-4 interested beta-testers. (No problem if none, I'm still quite glad I did this for my own program's use.)
EDIT: added current screenshots, still subject to tweaks.
Eric,
I would be interested in beta testing although I am out of town next week.
Also to show the code to my son as he has investigated to load big data quickly some time ago and could maybe hekp in speeding up for large datasets.
EDIT: One feature would be also interested to select only a certain user to plot the data of.
Josch
Here's my ten cents on this issue. Firstly, I end up using LCG1 vs LCG2 almost all the time. The simple reason, as already stated by multiple users, is it is fast, easy to use, presents a nice straightforward "clean" plot suitable for publication, etc. LCG2 just doesn't work so well, it still seems to have some "weird quirks", and fails for really large datasets. It's not as user-friendly, nor intuitive as LCG1, and somewhat cumbersome to use.
I feel the best way forward, is to rewrite LCG1 in C++, incorporate a few of the useful new features of LCG2, eg. new bandpasses, ability to select observations, etc. It should be a server side app, not a local one, because most users prefer it running right on the website, than having to download executables on all their devices.
Why C++? Well, despite it being an "ancient" language by today's standards, being around for about 40 years now, it still ranks as one of the top general languages used for almost every type of application! It continues to improve, and remain "modern", with the current C++17 release, and it will remain backwards compatible to older C/C++, thanks to GNU. It has a mix of very fast low-level features, as well as the modern object-oriented style constructs. A C program written in 1979 still compiles and runs on today's systems, can that be guaranteed with Python, Javascript, Ruby, other new languages?
Mike
Considering that Python, Javascript, and Ruby didn't exist in 1979, the answer must logically be no.
I know that. Clearly I meant would these modern languages be around 40 years from now, would apps written in them today, still compile and function going way forward? As we have seen so far, the stuff we have based our core functionality on, hardly lasts a few years, before its deprecated!
Nah. PHP which drives LCG V1 has been around since 1994, still runs fine. Choice of language isn't even the main problem with LCG V1, anyway. The required Drupal updates causing this disruption would be just as required even if everything had been written in C++.
Perhaps a better language-preference criterion would be: "10-20 years from now, how many available volunteer programmers will be conversant in C++ vs say python or even (gag) JavaScript?" Well, just look in the schools. The user groups. Github. Web site code everywhere. Books on Amazon. Recent astronomy code by SCSI (astropy), and others. There's even an annual conference "Python in Astronomy"--is there one in C++? So we can guess what and who will be widely available in 2030.
I'm no overly big fan of python and its weird culthood, but python does get the job done quickly, is extremely easy to test automatically and to modify, it will have plenty of supporters 10-20 years from now, and it's hardly "new" (1991) or "deprecated". Some of my first programs were in IBM 360 assembly, but even I can see that scripting "glue" languages drive the future especially of GUIs and web front ends, and that compiled languages like C++ will be relegated to some speed-critical packages imported by...the scripts in charge. Indeed, looking at python and R, that's already happened.
Not really, Can you be sure Python or any of the dozens of new languages will still be supported in 30-40,50 years? Of course IBM assembly is gone, being a specific vendor, but the whole idea behind C/C++ and Java, are to make them universal standards which will continue forever. Using different languages to write apps makes maintenance a big headache. So, I recommend standardize to C++ or Java for everything. The web stuff may need periodic updates, but that would be the minimal amount of work to keep things operational. Keep in mind, the speed issue has been raised by numerous users, so a language which incorporates the low level ability points toward C++ over the others as the obvious choice!
BTW, This is the successful principle Southwest airlines uses, they fly just one type of airplane the 737, so maintenance and training costs are least headaches :)
Mike
Hi, all.
I did indeed write the first versions of the LCG. Gary Walker e-mailed me with some questions about it a couple of weeks ago. While I cannot maintain it, I'm happy to answer any questions that would help others maintain it or move to a newer version.
I agree fully with Dr. Pablo. Even before I left in 2012 we were planning the next version of the LCG and identified java as its likely platform. There are issues that require a new and better version, which Bert described well.
With that said, since the new LCG and the older one are completely independent of each other I don't see why the old one cannot be kept around until the new one is brought up to speed. The LCG is a unique beast with a unique, idiosyncractic interface. There is no way you can get it completely right directly out of the box. It took about 14 years of near constant customization for it to reach the point it was in '12. To this date, I feel awe for all of our users who constantly tested it and sent in detailed reports, while being patient with me to fix it (largely via trial and error). The result was a fine program that did just about everything the user wanted at the time.
The downside is that, as Bert mentioned, the code was a mess. I'm a self taught coder and it was truly the definition of spaghetti code. I commented like crazy, but otherwise knew nothing about best practices in the field. While it is easy to fix it when it breaks, it would be an utter pain to expand in any real sense.
Re: Speed. I recall the initial issues with LCG also concerned speed. Although the main issue was not so much LCG processor time as much as the back end database processing time. We fixed it through the use of a dozen or so keyed indexes. So it is possible the solution to the LCG2 speed problem is to create the appropriate indexes on the database to match the most used queries for the LCG2. That will greatly expand the database resources needed to host it. But with cloud computing it shouldn't be a problem. And when the old LCG is finally nuked from orbit, you can delete all those old keys and reclaim much of those resources.
A bit of technical history of the LCG: The first version was written in 1998 over a weekend as a task to teach myself Perl. Over time it was slowly expanded as users (and staff - Arne in particular was great about making suggestions and predicting what people would want in the future) made requests. In the early 2000's I refactored it to make the code slightly better. Then did so again when we moved to MySQL (from ascii files!) around 2006 or so. When I started graduate school, we eventually hired a webmaster and he was tasked with formally taking over the code. I don't recall if he recoded it from scratch (he was a Python man) or maintained the Perl code.
And I can't leave out Chris Watson, who did a brilliant job on the UI. I would recommend connecting with him regarding any new version of the LCG.
There is nothing conceptually hard about the LCG at all. Any competent coder could do the job. What's hard is the large amount of customization you have to do to make it useable to a couple of thousand people using a vast array of computer systems. For example, I would spend weeks adding single pixels here and there to the plot outputs so that observers could print it out with one click and on one page. It took forever, but I understood why that feature is important. As an observer (at the time), I knew the process of plotting curves and charts to assemble an observing notebook a few hours before a run.
IMHO, the LCG is too critical of a tool for the organization to leave up to volunteers alone. An open source approach could work, but only if a staff member is in charge and coordinating it. It's just too important and too weird. It takes a ton of resources, thought and time to get it right. A few minutes here and there is not going to do it justice. You need to eat and breathe it.
Be well,
Aaron