JP2Gen

From Helioviewer Wiki

Jump to: navigation, search

Contents

Summary

The JP2Gen software converts solar FITS files to JP2 files for use with the Helioviewer project.



Getting started

The source code can be downloaded from here - you will need the version control software Bazaar installed in order to download the latest stable trunk. You will also need IDL and Solarsoft with at least the following instrument packages : 

gen hessi spex xray goes eit cds mdi lasco trace eis chianti stereo secchi euvi

Note that some of these packages require environment variables to be set (usually to find databases). Please consult the details for each instrument to correctly install the software.



Other Requirements

You will also need access to the relevant FITS files. The current list of supported datasets can be found here. We explicitly support code to convert these FITS files to JP2 files that can be used with the Helioviewer Project.



Using JP2Gen

1. Create a directory where you are going to store the JP2GEN programs, i.e., /home/jsmith/hv.

2. Create a directory where you are going to store the output from JP2GEN i.e., /home/jsmith/PUT_JP2_FILES_HERE.

3. Update your IDL path to include the location of the JP2GEN programs, i.e., in your .idl_startup file include the line !PATH = !PATH + ':' + EXPAND_PATH('+/home/jsmith/hv',/all_dirs).

4. Get JP2Gen from Launchpad and put it in the directory specified by the environment variable HV_JP2GEN.

5. Copy the file /home/jsmith/hv/jp2gen/local/EXAMPLE_hv_writtenby.pro into /home/jsmith/hv/jp2gen/local/hv_writtenby.pro . This file is where you will store your local default settings. Edit the file as indicated by the header information in the file.

6. Start up SSWIDL.

AIA

IDL> hv_aia_list2jp2,list,/copy2outgoing

Converts AIA FITS files into JP2 files. The variable list is a string array describing the full path (directory plus filename) of the FITS file to be converted.

LASCO C2

IDL> hv_lasco_prep2jp2,'yyyy/mm/dd','YYYY/MM/DD',/c2,/copy2outgoing

Writes LASCO C2 JP2 files for data between the dates yyyy/mm/dd and YYYY/MM/DD . Requires an active X windows session to be running. (Note that a subset of the data is converted to JP2, namely files that have filter = Orange, polar = Clear and have the full 1024 x 1024 FOV). If yyyy/mm/dd = YYYY/MM/DD then all the relevant data for that single day is converted to JP2. Copy all the JP2 files created to the outgoing directory.

LASCO C3

IDL> hv_lasco_prep2jp2,'yyyy/mm/dd','YYYY/MM/DD',/c3,/copy2outgoing

Writes LASCO C3 JP2 files for data between the dates yyyy/mm/dd and YYYY/MM/DD . Requires an active X windows session to be running. (Note that a subset of the data is converted to JP2, namely files that have filter = Clear, polar = Clear and have the full 1024 x 1024 FOV). If yyyy/mm/dd = YYYY/MM/DD then all the relevant data for that single day is converted to JP2. Copy all the JP2 files created to the outgoing directory.

EIT

IDL> hv_eit_prep2jp2,'yyyy/mm/dd','YYYY/MM/DD',/copy2outgoing

Writes EIT JP2 files for data between the dates yyyy/mm/dd and YYYY/MM/DD . (Note that only the full FOV subset of the data is converted to JP2). If yyyy/mm/dd = YYYY/MM/DD then all the relevant data for that single day is converted to JP2. Copy all the JP2 files created to the outgoing directory.

MDI

IDL> hv_mdi_prep2jp2,<subdir>,/int,/mag,/copy2outgoing

Writes MDI JP2 files for all the MDI data located in <subdir>. One can optionally write the continuum (white-light) data using the /int switch, or the longitudinal data, using the /mag switch, or both. Copy all the JP2 files created to the outgoing directory.



Converting quicklook data to JP2

JP2Gen also includes code to turn quicklook data files into JP2. This code is intended to run indefinitely. The commands for these are as follows


LASCO C2

IDL> hv_lasco_prep2jp2_auto,/c2,/copy2outgoing

Converts today's LASCO C2 quicklook FITS files to JP2 . Requires an active X windows session to be running. (Note that a subset of the data is converted to JP2, namely files that have filter = Orange, polar = Clear and have the full 1024 x 1024 FOV). Copy all the JP2 files created to the outgoing directory. Requires that the device setup file is edited to point to where the quicklook FITS files are kept.

LASCO C3

IDL> hv_lasco_prep2jp2_auto,/c3,/copy2outgoing

Converts today's LASCO C3 quicklook FITS files to JP2. Requires an active X windows session to be running. (Note that a subset of the data is converted to JP2, namely files that have filter = Clear, polar = Clear and have the full 1024 x 1024 FOV). Copy all the JP2 files created to the outgoing directory. Requires that the device setup file is edited to point to where the quicklook FITS files are kept.

EIT

IDL> hv_eit_prep2jp2_auto,/copy2outgoing

Converts today's EIT FITS files to JP2. (Note that only the full FOV subset of the data is converted to JP2). Copy all the JP2 files created to the outgoing directory.

MDI

IDL> hv_mdi_prep2jp2_ql,/copy2outgoing

Converts today's MDI quicklook FITS files to JP2. One can optionally write the continuum (white-light) data using the /int switch, or the longitudinal data, using the /mag switch, or both. Copy all the JP2 files created to the outgoing directory. Requires that the device setup file is edited to point to where the quicklook FITS files are kept.



Transferring data to a remote location

JP2Gen can push the JP2 files you have written to another computer. You need to edit the file local/hv_writtenby.pro to change the from and to computers, users and file directories as appropriate. There are two commands that may be used to do the transfer. The first one looks in the outgoing directory once only and transfer whatever is there

IDL> hv_jp2_transfer

The second command repeats indefinitely transferring whatever is in the outgoing directory 15 minutes after the completion of the previous transfer.

IDL> hv_jp2_transfer_schedule,15

The amount of time (in minutes) between consecutive searches of the outgoing directory can be changed.



Monitoring JP2Gen

JP2Gen can write out a simple webpage where the current status of all your quicklook processes can be monitored. You need to edit the file local/hv_writtenby.pro to point the structure tag "webpage" to your web space. The created webpage is called jp2gen_monitor.html , You can activate the monitoring system by using the JP2Gen command:

IDL> hv_webpage,<minutes>

where <minutes> is the cadence (in minutes) for the re-creation of the webpage, i.e., this command will update the status of JP2Gen once every 15 minutes.




Helioviewer device setup files

Each instrument has a device setup file associated with it. This is actually a function which defines all the necessary parameters (JP2 compression factors for example) for that particular device. You can write your own if you want to use something other than the default choices, and you can call it through:

IDL> hv_eit_prep2jp2,'yyyy/mm/dd','YYYY/MM/DD',details = "my_setup_file.pro"

JP2Gen will look for this file and it will expect to find an IDL function called "my_setup_file" that defines a single structure through which one can define all the parameters you need. If you need to add any extra functionality to write out the JPEG2000 files you want, please let us know. Full details on the structure that is defined in a Helioviewer device setup file can be found here.

Each device branch in JP2Gen has a file

hvs_<device name>_default.pro

which contains the relevant default information.



What JP2Gen does

For each supported dataset, JP2Gen uses standard SSW routines to do all the standard pre-processing and calibration to produce a useful science quality image. That image is then written as a JPEG2000 file. The routines are written to process batches of files, delimited by time. The structure of the JP2Gen programs and their output is as follows:

locationcontents
$HV_JP2GEN/ subdirectory containing JP2GEN
$HV_JP2GEN/gen general purpose routines called by all the device-specific routines.
$HV_JP2GEN/<nickname> the routines specific to a given device known as nickname.
$HV_JP2GEN_WRITE/write the directory where the output files, including JP2s, are stored.
$HV_JP2GEN_WRITE/write/outgoing/<nickname>/<measurement>/<directory structure> the outgoing directory. Files and directory structure in here are put here to be copied to remote locations.
$HV_JP2GEN_WRITE/write/web web notices directory. This directory stores text files that are picked up by other routines to create a web page which monitors the state of JP2Gen processes.
$HV_JP2GEN_WRITE/write/vX.YY version X.YY of the JP2 files and related output.
$HV_JP2GEN_WRITE/write/vX.YY/jp2 All the JP2 files written by JP2Gen.
$HV_JP2GEN_WRITE/write/vX.YY/jp2/<nickname> JP2 files for the device known as nickname.
$HV_JP2GEN_WRITE/write/vX.YY/jp2/<nickname>/<measurement>/<directory structure> the JPEG2000 files stored in the correct directory structure.
$HV_JP2GEN_WRITE/write/vX.YY/log All the log files written by JP2Gen.
$HV_JP2GEN_WRITE/write/vX.YY/log/<nickname> log files concerning the conversion of FITS to JP2 for the device known as nickname.
$HV_JP2GEN_WRITE/write/vX.YY/log/<nickname>/<measurement>/<directory structure> Log files for each JP2 file written.
$HV_JP2GEN_WRITE/write/vX.YY/db All the database files written by JP2Gen.
$HV_JP2GEN_WRITE/write/vX.YY/db/<nickname> database files concerning the conversion of FITS to JP2 for the device known as nickname.
$HV_JP2GEN_WRITE/write/vX.YY/db/<nickname>/<measurement>/<directory structure> A database file for the files written with <nickname> and <measurement>.


The nickname is the common everyday name used in conversation for a particular device.



JP2 file design: decisions and reasons

The Helioviewer project has been exclusively focused on using an archive of JPEG2000 formatted images to deliver solar physics image data over the web since September 2008. Since then, the design of the JPEG2000 files for use with the Helioviewer has undergone many revisions. The design of these files, and the reason for the choices we made are described here. This page will be updated as reviews of the functionality of the Helioviewer Project make different demands on the design of our JPEG2000 files.



Supported datasets

The Helioviewer Project supplies IDL code that converts FITS data from a range of instruments into JP2 files. The current list of instruments, and the time-range of data for which JP2 files are available, is given here.



Adding new datasets to the Helioviewer Project

You want to have your data written into JP2 files for use with the Helioviewer Project - how do you do that? Full information can be found here.



FITS2HV

The clients need to know key properties of the original FITS files. When we convert a FITS file to a JP2 file, we copy the FITS keywords as XML keywords and store them in an XML box at the top of the JP2 file. This meta-data is accessed by various components of the Helioviewer Project. A key describing the currently used FITS header keywords is maintained on the following page: FITS to HV conversion key.



FITS to JP2 Benchmarking

We have performed some benchmarking concerning how long it takes to read a FITS file, prep it using IDL SSW routines, and write it to disk. Current benchmarks can be found here.



JP2 File Versions

  • v0.8

Version v0.8 of the JP2s released 2010-03-09. Simple database output written every day for every nickname/measurement combination. This aids with the transfer of files from one place to another, and also means we can keep track of which input file was converted into which JP2 file.

  • v0.7

Version v0.7 of the JP2s released 2010-02-24. Output directory structure now <nickname>/<measurement>/YYYY/MM/DD. Each JP2 file gets an individual log file. Device setup files provided for each device - use can use default file provided by the Helioviewer Project or write their own. JP2 files themselves contain two new HV_ tags describing the setup file, but are otherwise unchanged.

  • v0.6

Version v0.6 of the JP2s released 2009-11-19. Fixes errors in EIT file creation. Sub-sampled images 512 x 512 are no longer resampled up to 1024 x 1024, Subsampled EIT FITS files remain at 512 x 512 pixels and have a correspondingly larger number of arcseconds per pixel.

  • v0.5

Version v0.5 of the JP2s released 2009-11-16. This set removes the LASCO transparency mask. Also, introduced version numbers NOT based on dates. New version numbers and Bazaar revision number (of JP2GEN) now introduced via keywords into the JP2 XML headers.

  • v20090917

Version v20090917 (released 2009-09-17) is the first version that removes the image rescaling/recentering done before writing to JP2. These trasnformations are now done via the client.

  • v20090511

Version v20090511 (released 2009-05-11)written to demonstrate client capabilities at the Solar Physics Division Meeting in Boulder, CO, June 14-18, 2009, (http://spd2009.boulder.swri.edu/).



See also

Retrieved from "http://www.helioviewer.org/wiki/index.php?title=JP2Gen"