JPEG2000 file design for the Helioviewer Project
From Helioviewer Wiki
The Helioviewer Project has concentrated on using JPEG2000 exclusively as the basis of our image archive since September 2008. Since then, the files have undergone many revisions as the project itself has evolved. In each of the topics below, changes to the design of each aspect of the file are given, along with the reasons for those changes.
Contents |
Summary of current file design
- 3 September 2009
Each file consists of an XML header and the JPEG2000 encoding of the image. The XML header is structured thus:
<?xml version="1.0" encoding="UTF-8" ?> <meta> <fits> ...original FITS keywords as XML elements... ...extra keywords inserted by the Helioviewer Project... </fits> </meta>
The JPEG2000 encoding is described below.
General design philosophy
We want to keep the JPEG2000 file as close to the original FITS file as possible, both in terms of the data and metadata. Hence we really want to minimize the amount of processing we do to the FITS file in order to turn it into a file useful to the Helioviewer Project.
Recent code reviews (July-August 2009) revealed that much of the existing image manipulation to turn FITS data to HV-JPEG2000 data (for example re-centering, re-scaling to a hierarchy of pixel sizes) was unnecessary since the Kakadu library is used by HV/JHV to extract the required portions of the JPEG2000 file at the required spatial resolution. If one is slightly smarter about extracting the necessary subfields from a JPEG2000 file, one does not need to re-center and re-scale the image.
Hence much of the image manipulation done before September 2009 has been dropped. The only exception is rotation. FITS data is rotated so solar north is always in the same direction in all HV-JPEG2000 files. It was felt that NOT derotating would mean that HV/JHV would have to de-rotate every image (or rotate the extraction region) on the fly. This appeared to be computationally expensive, both on the server for HV, and in the client for JHV, and so very little would be gained by a strict adherence to our general design philosophy. An XML tag, <HV_ROTATION> has been added to indicate the amount by which the HV-JPEG2000 data has been rotated in order to move solar north to the required position (see also the metadata section below).
One aspect of keeping the JPEG2000 file as close to the original FITS files means that the JPEG2000 files do not contain different types of measurements when compared to the original FITS files. For example, if the original FITS file was an EIT 304 file, then the JPEG2000 file will contain an image of that EIT 304 file, and nothing else.
Filename format
- 3 September 2009
The filename format has gone through several iterations. The present format is
date__time__observation.jp2
We separate the date, time and observation components by a double underscore. The date, time and observation are constructed as follows
- date = yyyy_mm_dd (year_month_day)
- time = hh_MM_ss_SSS (hours_minutes_seconds_milliseconds)
- observation = observatory_instrument_detector_measurement (the observatory, instrument, detector and measurement strings can contain any character except an underscore ('_') and a dot ('.') ).
Some example filenames are
2003_10_12__08_59_46_277__SOHO_EIT_EIT_171.jp2
2008_02_09__12_34_56_789__STEREO-A_SECCHI_EUVI_304.jp2
1998_05_22__12_56_34_934__SOHO_MDI_MDI_continuum.jp2
1999_12_02__00_09_52_010__SOHO_LASCO_C2_white-light.jp2
Metadata information
- 3 September 2009
In line with the general philosophy of changing as little as possible, all the FITS keywords are copied into the HV-JPEG2000 file as XML tags i.e., <THIS_TAG>value</THIS_TAG>. Variable types are preserved. Some extra XML tags, written by the JP2Gen software are included in the header also.
JPEG2000 compression
The Helioviewer Project is intended to let users visually inspect solar data from heterogeneous data sources over the web. Therefore the size of the files we create is a major consideration, and must be balanced against fidelity to the original data. Early experiments showed that the default choices we have made (as defined in the file JI_HV_OBSERVER_DETAILS.pro in the JP2GEN code) seemed to give an acceptable visual impression for EIT, LASCO-C2 and MDI data. We have settled on 0.5 bits per pixel compression in order to reduce the total amount of data transferred in order to give a reasonable view of the aforementioned data. We found that LASCO-C3 had to have considerably more bits per pixel in order to give a good visual impression at a significant extensions from disk centre.
