Skip to main content

Extended Text Format Plugin

19 posts / 0 new
Last post
mpyat2
mpyat2's picture
Extended Text Format Plugin

Hi all,

Using VStar to analyze light curves for newly found variables I often needed load data from simple text files and join them with surveys’ data. One of the features I missing was lack of band (filter) specification in VStar simple text format. Well, it is possible to create text files in AAVSO Download format, which supports filters however that format requires too many extra columns to be created.
Also, there is no way to tell VStar what kind of Julian Day is in a file of a simple format, JD or HJD (for short-period variables it is important).
I decided to create my own plugin which supports (a) filter specification and (b) HJD. The plugin should be as compatible as possible with existing Simple Text Format.
I did it, probably the plugin might be interesting for other members of the community (some other people were interested in such a feature as filer specification in the input file: https://www.aavso.org/vstar-self-generated-file)

The plugin can be downloaded here (the folder contains compiled plugin along with sources): https://drive.google.com/open?id=1qgq5GWfcF9SHQ87LsoA__beu9GTDwa6r 
Download “pmak.vstar.plugin.ExtendedTextFileFormatObservationSource.jar” and put it into vstar_plugins folder, then restart VStar. Now you should see an additional item in the File menu “New Star from Extended Text Format File...” (see attached picture).

The plugin can load comma-separated VStar Simple Text Format (see VStar Manual) without any modifications. Unlike VStar, it does not validate (intentionally) ObsCode value so you may use non-standard observer codes, like ASAS-SN, for example. It could be useful for filtering.
The plugin does not recognize value separator automatically (like VStar does), so you should specify a separator to load tab-separated VStar Simple Text Format using a directive #DELIM at the beginning of the file:
#DELIM=tab

(#DELIM directive has the same meaning as for AAVSO Upload Format)

To specify filter (band) you can use global directive #FILTER. For example, to tell the plugin that all observations in a file were made using Johnson V filter, put the following line at the top of a file:
#FILTER=V

You can also specify a filter for each observation separately in the 6th column of a data file (doing so you will make the file incompartible with VStar Simple Text Format Reader):
2457007.701,13.143,0.014,PMAK,,V
2457007.702,13.116,0.014,PMAK,,V
2457008.7,13.098,0.015,PMAK,,V
2457009.695,13.104,0.015,PMAK,,V

The plugin supports #NAME directive, as VStar reader does:
#NAME=Some Star
2457007.701,13.143
2457007.702,13.116
2457008.7,13.098

To specify that time values are HJD, use #DATE directive:
#Date=HJD

(Directive names are case-insensitive, #Date is the same as #DATE) 

Full list of directives:
#NAME=(object name). May appear in any line.
#DATE=(date column type: JD or HJD, default: JD). May appear before observations only.
#DELIM=(delimiter, default: comma). May appear in any line (i.e. to change delimiter)
#FILTER=(default value, if not specified in FILTER column). May appear in any line (i.e. to change the filter)
#OBSCODE=(default Observer Code value, if not specified in ObsCode column). May appear in any line (i.e. to change ObsCode)
#MAGSHIFT=(value to be added to magnitudes, default = 0.0). May appear in any line (i.e. to change magnitude shift)
#DATEADD=(value to be added to date values, default = 0.0). May appear in any line (i.e. to change the value to be added to dates)
#ERROR=(N to ignore all validation errors but time and mag; Y: do not ignore errors; any other values are ignored). May appear in any line.
#FIELDS=(list of fields, see explanation beneath)
#any comment

Using the plugin you can read files of different formats. Existence and position of data fields can be set with #FIELDS directive. The following #FIELDS directive corresponds to a default set of fields:
#FIELDS=Time,Mag,MagErr,ObsCode,Flag,Filter

To read AAVSO download format, put the following line at the top of a file:
#Fields=time,mag,magerr,,filter,obscode,,,,,,,,flag

Using this directive I was able to read some example data files from ‘data’ VStar folder, which cannot be read by VStar itself! (Those files include ch_cyg_data_2454600_to_2455000.csv, eps_aur_2445000.csv, ss_cyg_nov_5_2007_to_2009.csv, etc.)

Another example: the following directives allow us to read ASAS-SN Sky Patrol files:
#Date=hjd
#Fields=time,,,,,mag,magerr,,,filter
#ObsCode=ASAS-SN
#HJD,UT Date,Camera,FWHM,Limit,mag,mag_err,flux(mJy),flux_err,Filter
2457007.70083,2014-12-16.2012822,bb,1.67,16.107,13.063,0.014,22.844,0.299,V
2457007.70229,2014-12-16.2027437,bb,1.78,16.108,13.036,0.014,23.416,0.301,V
2457008.69975,2014-12-17.2002721,bb,2.49,16.031,13.018,0.015,23.812,0.311,V
2457009.69530,2014-12-18.1958917,bb,1.76,15.976,13.024,0.015,23.683,0.328,V
2457009.69670,2014-12-18.1972825,bb,1.62,16.058,13.046,0.015,23.187,0.306,V
<...>

You can find other examples in “samples” folder. “TestForErrors” subfolder contains sample files which generate different errors (for test purposes).

If you are interested in this plugin and find any bugs or errors, please let me know about them.

 

Best regards,
Max

P.S. Please note that currently, the plugin may not properly recognize quoted fields with delimiters inside quotes that may appear in a data file (i.e. comments in AAVSO Download Format File). "Comment" field appears between "ObsCode" and "Flag" field. To overcome this, use #FIELDS directive without "flag" field when reading AAVSO Download Format:

#Fields=time,mag,magerr,,filter,obscode

oopfan
oopfan's picture
Brilliant!

I'll give it a try, Max, thanks very much.

Brian

oopfan
oopfan's picture
Trouble loading plug-in

Hi Max,

Quick question: Are you using Windows, Linux, or Mac? I am on Windows 7. Java Version 8 Update 211 (build 1.8.0_211-b12).

I created a local folder for plug-ins and pointed VStar to it (see attachment) using the Preferences dialog. In the attachment I am showing where VStar stores the settings in the Windows registry.

Unfortunately when I restart VStar your plug-in does not show in the File menu. I also tried rebooting. No luck. Also nothing in the VStar log file.

Any ideas?

Thanks,

Brian

 

mpyat2
mpyat2's picture
VStar plugin

Hi Brian,

I'm using  Windows 10 and Windows 7 (on different machines).

VStar looks for its plugins in C:\Users\(user-name)\vstar_plugins folder. If you have downloaded any plugin (see https://www.aavso.org/vstar-plugin-library) this folder should exist (with downloaded plugins in it).

If you do not have "vstar_plugins" folder in your home folder, create it and put the plugin in it. VStar should load new plugins after restart.

Best regards,

Max

P.S. I tried to change Plugin Location Base URL using Preferences (see attached pictures). Nothing changed, VStar still looks for plugins in C:\Users\(user-name)\vstar_plugins. It seems those settings are related to the download location (from where to download), not to the destination.

oopfan
oopfan's picture
It works!

Hi Max,

The good news is that your plug-in works although I barely scratched the surface of all the options you offer. So far I just tested the "#NAME" directive and adding the "V" filter to my observations.

With regards to the plug-in file location, the magic was like you said creating the folder "vstar_plugins" in my User directory though it appears as if VStar looks there regardless of the setting in Preferences. In fact I returned the setting to its default location at "https://www.aavso.org/sites/default/files/vstar-plugins/vstar-plugins-2...." but it still picks up your plug-in in my local folder!

Thanks again,

Brian

 

mpyat2
mpyat2's picture
Thank you, for info, Brian!

Thank you for info, Brian!

If you have any suggestions/comments please let me know!

Best regards,

Max

 

oopfan
oopfan's picture
Check Star

Hi Max,

Is there a way to describe a Check Star (name, magnitude, error, filter) and if so does it adversely affect Phase Plot?

Brian

 

mpyat2
mpyat2's picture
Check Star

Hi Brian,

You could make a separate file for check star (if checkstar data is in the same file as your target, simply make a copy and define required fields with #FIELDS directive). Then you can load both files in VStar to see both curves. 

Se attached example: file1_target.txt for target, file2_check.txt for check star (actually the only difference between the files is content of #FIELDS directive (and also #NAME). The result is in a picture attached: green points are for the target, red (filtered by NAME) are for checkstar.

Max

P.S. There are no problems with the phase plot, however, for period analysis (DFT, AoV) you should exclude checkstar data first.

David Benn
David Benn's picture
Handling quoted fields

Hi Max

This is really good. Thanks.

Some time ago I decided that when complex fields (e.g. quoted fields containing quotes) are present, it's best delegate CSV parsing in VStar to a library. Handling all aspects of CSV parsing properly can take some effort.

You may find the ObservationFieldSplitter class I wrote useful for this purpose and should be able to replace your call to line.split(...) with a call to ObservationFieldSplitter.getFields().

I agree that what you've done is heading in the direction of what I was thinking here:

  https://www.aavso.org/comment/53790#comment-53790

Re: what you were wondering about with VStar plugins directory, yes, the plugin Prefs setting only affects where plugin files can be loaded via the Plugin Manager. The simplest thing is to take your jar and add it to vstar_plugins, as you mention to Brian.

If I have further feedback on the plugin or code I'll let you know.

David

mpyat2
mpyat2's picture
CsvReader

Thank you, David, I will try CSV reader.

Max

mpyat2
mpyat2's picture
Extended Text Format Plugin

Hi all,

I've made some changes in the Extended Text Format Plugin https://drive.google.com/drive/folders/1qgq5GWfcF9SHQ87LsoA__beu9GTDwa6r

1) Now it handles quoted fields (instead of using CSV reader I wrote my own short method (based on my old code come from Delphi years), I'm not sure this was the best way yet it allows me better control of what the method does).

2) A new directive #ESCAPINGQUOTES= has been added. It can be #ESCAPINGQUOTES=N (default value), which implies that internal double quotes in quoted fields should be represented as "" (double double-quotes) or #ESCAPINGQUOTES=Y, in such case double quotes should be represented as \". Using this directive (along with #FIELDS directive) I can read "SN2011FE.csv" example file without errors, while the standard reader cannot read several lines. The "SN2011FE.csv" example file with additional directives is here https://drive.google.com/open?id=1us1CzwOOLHezsRc9EddR-kh6sPGiWQ-f Please note that the directives do not interfere with the standard plugin so the file can be read by the standard plugin too.

3) #FIELDS directive now recognizes 'name' and 'comments' fields. See "SN2011FE.csv" example above.

4) The plugin is compatible with both stable 2.20 and unstable VStar 2.21 Rev. 1630, I've removed DateInfo.setJulianDay() call.

Well, probably it's time to write some documentation. Now a short description of directives is in the source code available by the link at the beginning of the message.

David, what do you think about including this plugin into the standard bundle (this will need renaming package), if you find it useful? Besides its main functionality (filter specification in simple files etc.) It could be used to read new data sources if plugins for them not yet created (using appropriate directives).

Sincerely,

Max

David Benn
David Benn's picture
Extended Text Format Plugin

Hi Max

I think including it will be great! I was planning to suggest it. I can imagine using it.

When the new release is made, there will be a few new plugins.

I like to get independent testing of my plugins so I can point a tester at this if you like.

Thanks for thinking about writing doco too. That's useful and important.

David

mpyat2
mpyat2's picture
Testing

Hi David,

Yes, independent testing would be great.

Max

daveh
daveh's picture
Extended Text Format Plugin Testing

Max,

David Benn asked me to test your plugin. I tested all of the files you had in your sample files download. Everything works great!! Nice job. Is there anything else that you need to have tested?

Dave

HDHA

mpyat2
mpyat2's picture
Testing

Hi Dave,

Thank you for your job! Now I have no additional things to be tested.

Best regards,

Max

David Benn
David Benn's picture
In repo

Hi Max

The code is in SourceForge now.

Let us know when you have a user document you'd like feedback on or want help with.

David

mpyat2
mpyat2's picture
In repo

Thank you very much, David!

I will prepare a doc, probably slightly later. Sorry for the delay,.

Best regards,

Max

David Benn
David Benn's picture
Doc

No worries Max. There's still plenty to do before the next VStar release.

David

David Benn
David Benn's picture
Doc and VStar release

Hi Max

I have made a candidate release available for testing which includes the plugin jar.

Do you want any assistance with writing a doc or feedback?

Any changes you want to make to the plugin? 

I should also check that you want the plugin to be available in this release.

One other thing that occurred to me was that the name "Extended Text Format" may confuse some people when they see that, thinking about AAVSO Upload Format which has Visual and Extended modes. Thoughts?

Thanks. 

David

Log in to post comments
AAVSO 49 Bay State Rd. Cambridge, MA 02138 aavso@aavso.org 617-354-0484