Skip to main content

pylcg (LCG V1 replacement run locally)

40 posts / 0 new
Last post
Eric Dose
Eric Dose's picture
pylcg (LCG V1 replacement run locally)

And...pylcg v0.1 BETA limps across the finish line and is available today for the intrepid.

The present form is a Windows executable, tested on WIn 7 and 10. It is available as a zip file that can be unzipped to a directory and run from there as app.exe. You do not need python installed nor anything else, and it does not seem to need Windows installation, registry changes, or permissions. The app download is a bit slow, but then the data download and plotting are pretty fast indeed.

To begin, consider starting in about this order:

  1. Please look at the README file at www.github.com/edose/pylcg -- yes, hackers, the repo and source code are public again. This README is as concise an intro as I could write; it will save you time.
  2. Download the zip file from www.awfulgood.com/pylcg/pylcg_app.zip (using a FTP program like Filezilla, or even just a recent browser--I've verified downloads in Firefox and Chrome). And sorry about the size (205.5 MB) of this thing--pyinstaller 3.3.1 is supposed to render the program into a single file, but I can't make that option work. (If someone knows why this happens with pyinstaller, do let me know.)
  3. Just run app.exe found in the main (unzipped) folder. The first time you run it, you may get a couple of delaying messages: (a) your anti-virus may complain that pylcg or app.exe doesn't have a digital signature, and (b) if your firewall is very fussy it may ask for permission to access the internet.
  4. When you've compiled your inevitable list of bugs and change requests, send me a message through AAVSO (Eric Dose, observer DERA) or any other way you care to. If you'll be as specific as you can (esp. Star ID & JD range for data or plotting problems), I'll clear up the problem as fast as I can.

I really want the high-throughput AAVSO contributors to have a streamlined light curve plotter for continuity of workflow. Let's hope this is a start.

Thanks,

Eric Dose, obs code DERA

 

Tonisee
Functionality requests

First thank you for taking that path!
I have a comment and a suggestion about possible functionality.

Comment about highlighting an observer - I personally use that quite often when planning my data usage or observation strategy or just when trying to get overview of existing observations. How I do that? I plot e.g. visual and V data. That last one often contains non-transformed observations mixed with transformed ones. Also, often there are observers who have contributed long(er) time-series collected using exactly the same setup and data processing approach (selection of comparison stars mostly, I guess). So I highlight observations of those observers, typically one by one, who have contributed with many datapoints - to see how nightly observations scatter and/or how longer time period looks like. After such visual inspection I usually select observer(s) whose data I will take as the reference. But also I use highlighting to display my own observations against others.

Problem with especially LCGv2 is that the symbol is very large (actually just different colour of symbols would do well, when it would be shown on/as foreground layer!) and that doesn't make visual inspection easier. Still, that's better than no such functionality :-)

My small request for functionality is about finding stars. I would suggest or ask what you think about making possible to use any Simbad resolvable name? For that, e.g. Astroquery (it's part of Astropy module, https://astroquery.readthedocs.io/en/latest/) could be really usable. I'm not sure if it can be used directly for AAVSO VSX or photometry databases, but definitely it is very useful to get coordinates based on almost any name a star might have.

Best wishes,
Tõnis

Eric Dose
Eric Dose's picture
Hello Tõnis

Hello Tõnis

Thanks for the notes.

Yes, highlighting observers will probably happen sooner rather than later. We will start with highlighting only one observer.

I also like the idea of resolving Simbad names, but I will have to learn how this would work. I also hesitate to import astropy entirely (it's big) for only one function, but perhaps I can persuade pyinstaller to import only the part needed.

Cheers

SGEO
SGEO's picture
Eric,

Eric,

I have code from the Eggen project that would check Simbad for aliases. If it would be of use I can send it.

Cheers,
George

 

hambsch
hambsch's picture
Thanks a lot

Thanks a lot Eric,

for the continuing development of the program. Together with my son I will have a look and give feedback.

Regards,

Josch

clittlefield
Great work!

Hi Eric,

Great work! I have a few minor suggestions.

  1. When selecting the date range, it would be nice to be able to enter calendar dates so that the user doesn't have to convert from calendar date into JD.
  2. I recommend that you consider listing a truncated JD along the x-axis (i.e., subtract an integer constant from all JDs). I think that this would make the x-axis labels more readable, particular when the user zooms in. The x-axis label could be updated to indicate the offset (e.g., JD - 2458119).
  3. When you hover the cursor over the plot, matplotlib displays the current x and y values of the cursor in the bottom of the window, but it is possible to customize this information by specifying a custom coordinate format. As an example, the default output might report the cursor position as x=2.45811e+06, y=10.000, but with a small amount of customization, this could be changed to something like: Closest observation to cursor: JD = 2458119.5001, V = 10.010 +/- 0.010, Obs = DERA, Date = 2018-01-01.

These are very minor points, though, and it's possible that other users might not care about them. Your code combines the speed of LCGv1 with the interactivity of LCGv2, and I find the user interface intuitive.

Best,

Colin

(Original post edited for clarity)

hambsch
hambsch's picture
Great program

Dear Eric,

I used the EXE version a bit today as I am travelling until Sunday night. On my laptop I cannot see the whole screen and a slider bar in case of need would be advisable.

The speed is great, it plotted all 18ey data in about  1 minute. I also would like to be able to enter calendar dates and also either plot calendar dates as x-axis or JD like in LCG1.

The feature to zoom is very handy. It would be great if all data are loaded and one could chose a single (or even multiple) observer, not only highlighting them but maybe select them and only show the data of the selected user(s). Sometimes the data are scattered a lot due to problems some observers have in getting decent photometric data. It would be great to exclude them.

Furthermore I would need to test it more in detail but presently my time is limited to do so.

Anyway it is a real improvement in view of LCG2 and it would be good to be taken over by AAVSO.

My son told me that the speed for the present amount of data is dependent on the AAVSO API to download the data and not by your code. Only if the amount of data goes into millions then one could use other means to speed things up.

So a great tool and I am convinced it will be useful for many observers.

Regards,

Josch

Eric Dose
Eric Dose's picture
Issue list is started

Yes, the speed is remarkable. For example, when you click the Grid checkbox, it's actually redrawing the entire plot from scratch.

Separate subject: I've opened a tracking list of enhancements to add and issues to correct, on the "Issues" page of pylcg's Github repo, at:

     https://github.com/edose/pylcg/issues

I'm still assembling that, but let me know if I've forgotten something.

Eric Dose
Eric Dose's picture
pylcg 0.2 beta is available

I'm pleased to make available a new, improved pylcg--version 0.2 beta--is now available for download. It has lots of goodies for high-volume users, including:

  • import of star IDs from upload files,
  • Next and Previous plot buttons,
  • entry of Start and End dates in calendar format (US or International) as well as Julian Dates,
  • check marks next to dates entered to ensure correct entry,
  • On/Off toggling of Less-than observations, and
  • cache clearing in case something goes woefully wrong (I haven't seen that yet).

Sorry, but plotting Calendar Dates on the x-axis will have to wait until the next (November) version. For days I chased a bug in Tkinter's zoom toolbar in which switching x-units makes it lose track of all x-axis info. It's hopeless, and I must code my own zooming and x-axis management, both. Sigh.

Download pylcg as a zip file, from www.awfulgood.com/pylcg/pylcg_app.zip (using a FTP program like Filezilla, or even just a recent browser--I've verified downloads in Firefox and Chrome). I would always run an antivirus scan on any download. Then you just unzip the main folder anywhere and run app.exe. and then the first time you actually run app.exe, (1) your anti-virus may complain about signatures, and (2) your firewall may ask for permission to access the internet. That's pretty normal. If you've run a scan, then just click OK and enjoy.

The README file at www.github.com/edose/pylcg will probably be helpful. This README is as concise an intro as I could write; it will save you time. The issues page (both pending and closed) is at https://github.com/edose/pylcg/issues . The repo and source code are public.

Comments, bug reports, and suggestions for enhancement are solicited! You may private message me via AAVSO, or just post to this thread.

Pylcg 0.3 beta, with x-axis Calendar Dates and a few other enhancements, is scheduled for early November, hopefully before the AAVSO meeting in Flagstaff--see you there!

Thanks,

Eric Dose, AAVSO observer DERA

hambsch
hambsch's picture
Thanks a lot

hi Eric,

I look forward seeing you in Flagstaff at the AAVSO meeting.

The progam works great and indeed I use the election of user (myself) very often when plotting data as LCG1 does not allow to plot only one user. It is also sometimes useful to show about the precission of the observations compared to other users. Hence this feature (select user) would be definitely useful. One other possibility to be of paramount importance is to be able to use the image in other programs like Powerpoint for presentations. I do not see the possibility to copy the image except making a screen shot.

Josch

 

Eric Dose
Eric Dose's picture
All doable...

Hello Josch, I think the toolbar's "floppy disk" icon will save the plot as a PNG file. Let me know if that does not work for you. In any case, I will probably have to replace this in 0.3 Beta if I can't get Tkinter's toolbar to work reliably.

Could you please let me know what you mean by "election of user"--do you want (1) the option to plot only one user's data, or (2) the option to highlight observations from one user? "Highlight Observer" is already scheduled for 0.3 Beta; option (1) plot only one user's data is quite possible as well if that would be useful. I have in mind a very rapid toggle between one user vs all users.

Cheers & see you in Flagstaff,
Eric

hambsch
hambsch's picture
Highlight and only one user would be great

Hi Eric,

highlighting a user amongst many datasets and only plotting a single user, if those possibilities would be there that would be great. For presentations I sometimes want to show only my own data and since I regularly suibmit those to the AAVSO database, the easiest to find is in there. LCG1 did not had this option neither has LCG2.

I am not yet at a stage at home to plot my own data from several months or even years easily with my own tools, although I make progess as well.

It is also nice a tool to see whether there is a need to clean up submitted data. My son wrrote me a Python script to do so based on a given uncertainty above which the data are discarded in the AAVSO file I submit. Very handy as trhere are always some flaws which makes some points pop out.

Great software anyway. I am going to try the save possibility. Have to prepare the presentation for the AAVSO meeting.

Josch.

 

Eric Dose
Eric Dose's picture
Highlight Observer next

OK then, thanks: Highlight Observer moves toward the top of the issue list. I hadn't considered One Observer plots, but that should be easy enough.

This will make for a lot of option buttons on the screen, and a screen layout reorg may eventually be required--tabs or something. I don't look forward to that, but mostly I don't want to make the control area any larger or it won't fit on laptop screens.

I have tamed the default toolbar--the documentation forgot to mention that a toolbar refresh is required at multiple points, but now it works. So in the next release the zoom and pan function will work fine, and the save-image function will remain stable as well. Updated code and issues list (both open and closed) are on the github repo so you can see real time what's going on. Still aiming for early November release of 0.3 Beta.

Tonisee
Control window

Hi Eric,

what do you think about the solution of having separate control window, e.g. such one that can be freely positioned?

That could keep graph window more or less clean and maybe even simplifies development.

Best wishes,

Tõnis 

Eric Dose
Eric Dose's picture
Not so simple

Rearranging all the controls will not simplify development...for all its execution speed and reliability, Tkinter does not make major refactoring easy. But if the number of controls (buttons, entry boxes, checkboxes) continues to grow, yes, some of the lesser-used controls may have to move to a separate window and/or into the menu bar.

Rearranging all the controls would suspend other development and testing for a month or two. And frankly it's tedious beyond belief whereas right now we have some momentum. I'll open a github issue to track the idea, but this is for later.

 

Tonisee
Controls etc addons

Hi Eric,

I have to agree with your comments on separate controls window in Tkinter.

I also wanted to share a nice finding. Because of other projects at work, I was looking for fast updating graphing (real-time data from network analyzers and vibration/shock testing setups) possibility in Python and found http://www.pyqtgraph.org/ Seems that pyqtgraph doesn't need any particular specific add-ons from Python, except PyQt that is very often already installed (when installing e.g. matplotlib). Also, PyQt allows to create almost any kind of GUI.

On pyqtgraph page, they point to one other project that illustrates very well what I was thinking about when asking about controls window - e.g this particular screenshot: http://acq4.org/screenshots/photostim.png (there is more eye-candy screenshots).

Probably it did not help much but hopefully was interesting :-)
Best wishes,
Tõnis

Tonisee
Proposal for additions to documentation

When trying to keep installation minimum under Windows, I installed python3 with miniconda and then added from Anaconda prompt:  conda install numpy matplotlib
Probably that installed also everything that is needed for pylcg

For minimum footprint, I downloaded latest git release as .zip-file and extracted to a directory c:\users\tonis\pylcg-master. To get it running I had to add it's directory to PYTHONPATH:

(base) C:\Users\tonis\pylcg-master>set PYTHONPATH=%PYTHONPATH%;C:\Users\tonis\pylcg-master

and after that in the directory C:\Users\tonis\pylcg-master\pylcg : python -m app

Probably it works similarly under e.g. Linux when one is not using any development environments etc. Eric, what do you think, would similar example be worth to mention in Readme?

Best wishes,
Tõnis

Eric Dose
Eric Dose's picture
I can't support Linux too

Tõnis: It's not quite clear to me just what you're asking for. It's wonderful if you've been able to run pylcg's python code in Linux, but I will confess here that I am wholly unqualified to advise others (in a readme file, for example) on how to run python code in a wide gamut of environments, and especially not in Linux. I would be happy to consider adding to the repo a text file with your Linux installation advice, if you'd care to write it up.

As all the initial user requests were for a Windows executable, that is what I have provided. So far Windows executables seem to provide uniform installation and high performance. If someone wants to provide pylcg (in a more mature version) as a web service, that would be great but again I'm not qualified to do so.

Pylcg development is strictly subordinate to my busy observing program; I am keeping strict limits on the pylcg effort. My own plan continues to be: (1) pushing python code intermediate revisions (every few days) to Github, and (2) providing links here to Window executables for more major releases (~monthly) until the current issues are addressed. Then about January, my development will probably stop. If someone is moved to fork the repo to continue development of pylcg and its readme from there, they are always welcome to do so.

clkotnik
clkotnik's picture
pylcg access to webobs

Eric,

Very nicely done!  I was looking at the code - I know, I know.  Sorry.

The funky web response parsing with various separators surprised me.  Is this all driven by the free-form comments?  No way to get the web service to escape the separator?

thanks,

Cliff

Eric Dose
Eric Dose's picture
"funky" is probably too generous of you

Hey, Cliff, thanks for the kind words.

The funky web parsing is designed to be fast and failsafe, probably overkill but it must be said I haven't gotten it to fail yet. In fact I have some e-mails from AAVSO that I have yet to fully parse, and those suggestions will likely lead to streamlining. But such code changes should not affect pylcg users at all, might help future maintainers. I'm adding this to the issue list (github) for 0.3 beta (Nov 8); it may fall back to 0.4 beta end of year.

See you in Flagstaff!

Eric Dose
Eric Dose's picture
pylcg 0.3 beta is now available

The third and last beta (advanced test) version of pylcg is now available. Zoom, pan, observer highlighting, one-observer plots, cursor position in date and magnitude, calendar dates on horizontal axis, and observer list popup table are now added.

Download the Windows executable from http://awfulgood.com/pylcg/pylcg_0.3_beta.zip .

Source Code (python 3) and Readme file at https://github.com/edose/pylcg . You want to read the readme file first (whence its name).

Screenshot at https://github.com/edose/pylcg/blob/master/screenshot_pylcg_03beta.png . (It might be clearer in https://raw.githubusercontent.com/edose/pylcg/master/screenshot_pylcg_03... .)

Release 1.0 with fewer rough edges is expected mid-January 2018.

Enjoy!!

hambsch
hambsch's picture
Problems on my laptop

Hi Eric,

great improvements, however on my laptop I cannot see the full screen. I cannot see and check the error bar and less than boxes nor can I see the observers. You loose a lot of space by having blank areas between the boxes to the right where one can enter information (star, plot range, checkboxes etc.). Unfortunately if the screen does not fit into the resolution of the e.g. laptop screen then there is no slider to be able to access also the options hidden below the Quit pylcg field.

I can show you the issue during the AAVSO meeting the next days.

Josch

That is of course a drawback and maybe you could add this to your to-do list.

Eric Dose
Eric Dose's picture
laptop screen res

Josch, what screen resolution is your laptop?

See you in Flagstaff...

hambsch
hambsch's picture
resolution

Eric,

1920x1080

See you soon, Josch

Eric Dose
Eric Dose's picture
OK, I don't understand,

OK, I don't understand, because I just measured this version's window at a total size of 1231 x 896, and it does display well within my 1920x1080 test monitor--but you can show me in Flagstaff.

Eric Dose
Eric Dose's picture
Point taken

After testing pylcg on some laptops etc and reconsidering, yes in a week or two I probably do need to release a 0.31 beta to shrink minimum vertical size a bit, down to 800 if I can fit everything.

Josch, it would be very helpful for me to see pylcg on own your laptop. I think that's you across the conference room, so perhaps we can find each other today and do that.

Eric Dose
Eric Dose's picture
pylcg 0.31 beta (smaller window) now available

Of course as soon as I write that 0.3 is the last pylcg beta, we discover that Windows 10 on laptops tends to expand the native size of certain GUI apps so that they no longer fit on the screen. Pylcg got caught in this. Gee, thanks, Microsoft. Josch and George both showed it to me in Flagstaff, so I had to do something. That something is...0.31 beta, now available as Windows executable at from http://awfulgood.com/pylcg/pylcg_0.31_beta.zip . No other changes from 0.3 beta. Remember that you can resize the plot window from the initial size at launch. If 0.3 works for you, no need to go to 0.31. They will converge at release 1.0 mid-January.

Source Code (python 3) and Readme file (please read it before installing) continue to be available at https://github.com/edose/pylcg .

Let me know how it fits now. If it requires a 0.32 and/or 0.33 to get the sizing right, I'll do it.

Happy Thanksgiving to my American friends, Happy Thanksgiving in arrears to my Canadian friends, and Joyeux Jeudi-comme-les-autres etc to my Euroamis.

hambsch
hambsch's picture
Just downloaded

Hi Eric,

thank you for the quick fix. It was a pleasure meeting you in Flagstaff and talking to you. I downloaded the new version to my desktop just to see what all I missed on my laptop :-)

I need to do this on my laptop as wel to test your fix, however I see a possible imnprovement for V1.0.

In the list of observers it would be great to also add the filters they observe in like it is done on the new LCGV2 where I find this a good addon.

Josch

hambsch
hambsch's picture
Seem not to work

Also downloaded on my laptop and I see no difference to the pevious version.

Actually you did not change the version number nor the date, did you? Would be better to see whether the nbew version is loaded. On my desktop of course I have larger resolution screens and see the full window. On the laptop not.

Josch

Eric Dose
Eric Dose's picture
My fault. It works now.

My fault. I somehow corrupted the link so that it showed 0.31 but directed you to 0.3.

It works now. I have corrected the link in my previous post, and I'll repeat it here to be sure: http://awfulgood.com/pylcg/pylcg_0.31_beta.zip . Sorry for mistakenly pointing you to the old version.

I will add to the Issues a summary of filters per observer in the List Observers table. thanks.

hambsch
hambsch's picture
Still does not work

Eric,

sorry to say but it still does not show the full screen.

If I open it and make it full screen on the laptop I see until the less than and error bars checkbox.

If I have it smaller than the full screen I can boxes preferences and VSX but the icons in function of the plot area (zoom etc) are gone.

Only on my desktop I see all (still with V0.30). I would need to download the new 0.31 also for the desktop.

Josch

Eric Dose
Eric Dose's picture
This looks like a Win 10 setting

Josch: v 0.31's window is much smaller than v.0.30's window. So if large 0.30 appears fine on your desktop, but small 0.31 (772 pixels vertical) still doesn't fit on your laptop (1080 pixels vertical), then your laptop's OS must be enlarging them, in this case over-enlarging them. George's new Windows 10 laptop with stupendously large pixel count shows the same pylcg problem, which is quite impossible at its native resolution. But I have never seen this problem on my Windows 10 Pro desktop, Windows 7 desktop, or Windows 7 Pro laptops, for any pylcg version.

This app enlargement seems limited to Windows 10 laptops. I don't have one to test on, but I can recommend you try to stop or limit app window resizing (for pylcg only) by:

  • In the shortcut's Properties > Compatibility Tab, try unchecking  “Disable display scaling on high DPI settings”. This setting has caused big problems in certain Windows 10 variants. This will apply only to pylcg, and you can always reverse it.
  • Try running pylcg in a Compatibility Mode for Win 7 (same Compatibility Tab, thus limited to pylcg). This may also defend against problems arising from future Win 10 updates.
  • If that fails, you will want to examine other settings in the same Tab under the "Change high DPI settings" button, which looks suspicious too.

At my end, I will continue to investigate such Win 10 settings. But within pylcg I know of nothing more I can do: the more I shrink pylcg, the more these Windows 10 laptops will simply over-compensate to continue causing the problem that you and George see, while markedly worsening the GUI for all native-resolution users.

Please let me know if one of the above changes works, and I will definitely insert that specific advice in pylcg's installation advice (README file).

 

hambsch
hambsch's picture
Will try on Monday

Eric, thanks for the recommendations. Will try on Monday as during the weekend I am too busy and out of the house.

Josch

hambsch
hambsch's picture
Got it to work

Eric,

I got it to work by checking both boxes in the high DPI settings and putting the drop down menues to application and when I open the program.

Josch

 

Eric Dose
Eric Dose's picture
pylcg 1.00 (1st main release) is available

Hello, all,

The new and first main release 1.00 of pylcg is available for download at:

      http://www.awfulgood.com/pylcg/pylcg_1_00.zip.

To install, download the zip file and unzip it anywhere. Has been tested on Windows 7 and 10.

I would recommend you first read the updated README file at:

     https://github.com/edose/pylcg/blob/master/README.md

For the intrepid, the source code is available at https://github.com/edose/pylcg as well.

This will be the last release for several months. Minor bug-fix releases are of course possible.)

Enjoy,

Eric

hambsch
hambsch's picture
New version and new request

Hi Eric,

thank you very much for the new version. I will test it the coming days.

I would also like to put forward a new request. Would it be possible to have the option to save the data which are selected in the window? I mean if I select a certain user and filter than only those data are being downloaded and sved or if no user but a given filter is seleted all data in that filter are being downloaded and sved.

That would help in selecting certain users and/or filters for plots and further analysis of the avalílable data.

Thank you for considering this.

Best regards,

Josch

hambsch
hambsch's picture
Window issues on my laptop

Hi Eric,

sorry for the long delay to the V1 testing on my laptop. I still cannot see all of the window even if I use the SMALLER option in the preferences. That seems not to work whatever I try in resizing the window.

On te desktop all works fine due to the larger screens.

Josch

 

Eric Dose
Eric Dose's picture
Josch, this is unfortunate.

Josch, this is unfortunate. But I understood that your laptop actually had much larger screen size in terms of pixel count. At first sight, I suspect that your laptop's high-pixel-count settings have been reversed. If others have a solution, please post it here. Thanks.

I can't reproduce this problem on my desktops or laptops, but I'll check again. As of this moment, I can only suggest that you verify that Windows 10 Update has not reset your graphics settings (as it is known to have silently reset/corrupted certain USB Suspend and other settings).

hambsch
hambsch's picture
Right side info does not scale

Eric,

I see that if I change the size of the window only the part where the LC is shown is scaling and the text below the program name with all the input boxes and radiobuttons does not scale.

Ok I have found the issue. In the settings you can choose a size of the text in APP'S and that was set to the recommended version of 150%. If I switch to 125% then I see the full info from the pylcg. SO I need to stay with the text size to 125%.

Thanks for your concern.

Josch

 

Eric Dose
Eric Dose's picture
And thank you for writing

And thank you for writing back. That's very useful info, and I will update the README file to include it.

Log in to post comments
AAVSO 49 Bay State Rd. Cambridge, MA 02138 aavso@aavso.org 617-354-0484