Written by Julie Davis (SUVI summer intern 2011)
(From Experience with AIA*)
*some instructions may not apply to instruments other than AIA
**Lockheed interior data access only. For exterior consult the
SDO User's Guide
Important Basic Programs:
ssw_jsoc_time2data
ssw_jsoc_time2data_sample
mreadfits
read_sdo
utplot
outplot
About Fits:
Fits (Flexible Image Transport System) files are a NASA endorsed file type used for
transport, analysis, and archival storage of scientific data sets. The fits file contains
two major pieces, header keywords and image data. The header contains an array of structures.
These structures are metadata—useful data about the image file. Examples (drawn from AIA) include:
Data about time, origin, exposure, count, and quality of the data:
These will give strings containing dates/times, integer and float values of exposure times,
counts of data in the image in pixel units, number of saturated pixels, etc.
These will give statistical calculations using the full image data, such as the average
pixel value for the entire image, deviations, skew, low and high percentiles, etc.
These will give information about instrument temperature and ccd/image processing data, etc.
Other keywords include spacecraft orientation and physical data about the sun.
A complete list of keywords and their descriptions can be found on the AIA team site.
(Password required):
Document AIA02840 AIA/SDO FITS Keyword Documents
Working with Header Data:
Prior to using these procedures, ensure that the ssw tree is installed/accessible on your machine.
Also, if IDL has not been aliased to directly access the tree, you must enter sswidl to begin your IDL session.
Header data provides a broad basis from which to begin analysis of fits data.
Using IDL (with the solarsoft tree installed), the user can access powerful sample tools,
the ssw_jsoc_time2data and ssw_jsoc_time2data_sample programs. Details of these programs can be
found using the IDL which and UNIX more commands.
IDL> which, 'ssw_jsoc_time2data'
/ssw/site/idl/jsoc/idl/ssw_jsoc_time2data.pro
IDL> $ more /ssw/site/idl/jsoc/idl/ssw_jsoc_time2data.pro
The calling examples listed in the ‘more’ output assume a fair degree of knowledge with the various
parameters and keys. The basic and most useful formula tends to be:
Contained in this call are the desired start and end times, given as two strings (with time being an
optional argument); a variable name (in this case index) into which the data will be written,
and a list of desired fits header data values. These are the keywords described in section one.
Keywords should be able to be entered as a comma delimited string, though in some environments a
vector notation may be required instead.
This procedure will return an array of structures, based on which key words were chosen.
The ssw_jsoc_time2data differs from the ssw_jsoc_time2data_sample in that it does not currently
take a cadence argument directly. At the time of writing, a new slotted AIA lev1 series access is
in development, which takes a cadence argument. However, as of now, the more intuitive procedure is
ssw_jsoc_time2data_sample. Calling formats for time2data_sample are very similar:
Like before, the call contains strings holding start and end times (hour/min/sec
specification optional), a variable into which data will be written, and a set of keywords to pull
from the header. However, time2data_sample allows here the specification of a cadence, sample width,
and data series. Cadence can be input in days, hours, minutes, or seconds. Sample width refers to the
amount of time examined for each cadence segment. If the cadence is ‘1d’ (one day) and the sample
width is 120 (seconds), ssw looks at the first two minutes of every day. Setting the /uniq_wave
switch will cause it to return only one image for each specified wavelength. Ds stands for data series,
and allows the user to specify which level of processed data to search through. Level 0 is raw,
level 1 is processed, level 1p5 has more finalized calibration, etc. Information about the various
switches is explained in the program documentation, accessed as seen in the beginning of this section.
A note on cadences: Due to the massive file archive sizes, it may be necessary to sample data at
certain data cadences or time intervals. A lower cadence quickly multiplies the sample size, and
thus a smaller date range should be selected to avoid long waits. For example, a year of fits header
DATAMEAN data taken at a one day sampling cadence will take approximately 7 to 10 minutes.
The overheads are not terrible, though it is recommended that smaller date ranges be used with
cadences smaller than one day. This will be detailed in section 3.
The utility of ssw_jsoc_time2data_sample alone ends at sampling the header data for simple print
and help statements. To obtain plots or work with the images themselves, ssw_jsoc_time2data_sample
must be used in conjunction with procedures like read_sdo and utplot.
Plotting Header Data
Header data may have values useful in the form of plots. For this purpose, utplot and outplot are
convenient. These two procedures function much like IDL’s plot and oplot. All of the common keywords
of the IDL plot functions such as title work with utplot and outplot. These functions graph header
data versus time, with the x-axis labled with UT (Universal Time) dates and times. Using
ssw_jsoc_time2data_sample to gather the required file range then sorting for a single wavelength,
users may produce plots such as the following, a simple DATAMEAN plot that reveals solar flares.
The calls for utplot and outplot are simple. For example:
IDL> utplot, data.t_obs, data.datamean
IDL> outplot, data.t_obs, data.datamean
Here, data is the variable into which ssw_jsoc_time2data_sample wrote the header data.
It is important to remember that when you call time2data_sample, all the needed keys are specified.
In the above plotting example, time2data_sample keys required included T_OBS and DATAMEAN.
Time2data_sample only populates the structure fields specified in the key if a key is called.
If no key is specified, it will populate all of the fields (every possible keyword).
Reading and Working with Image Data
Originally, mreadfits was written to read in FITS image data for manipulation. With uncompressed
files, this still works. However, SDO data comes in a compressed form. Read_sdo was written as a
wrapper code for mreadfits to first decompress the images then read them in. The call for read_sdo
contains a few useful optional components that are included in the documentation but not well explained.
This results in an array of structures containing the header data and an image array. With AIA, a
help statement on image will result in [4096,4096], the size of an AIA image in pixels.
Multiple files may also be read in:
IDL> ff = file_search('*0193.fits')
IDL> read_sdo, ff, headers, images
Using file search and wildcards, it is possible to obtain large quantities of images. However,
take note that read_sdo can quickly become a system killer. It is best not to read in more than
perhaps fifty images or so at one time, especially at a file size like AIA, where one image is 32MB.
The full read_sdo call contains useful optional parameters to select only portions of the images.
This feature allows the user to look at only a region of interest, or create movies of the images
with reasonable runtime.
The full read_sdo call (also found in the documentation):
In this example, the file stored in the variable infile is read in, the header data goes into index,
image data goes into data, and an area from pixel 1 to 100 in both the x and y direction is selected.
The two ones refer to the initial x and y coordinate, and the two 100 values are how many pixels wide.
Thus this segment would go from 1 to 101 in both the x and y direction.
The preceding procedures constitute a basic overview of how to use SSW. The appendix contains other
useful example routines that rely on the basics of this guide.
APPENDIX
(With thanks to Paul Boerner (boerner@lmsal.com) for providing these examples)
Plot a lightcurve using JSOC query
IDL> good171 = where(currentheaders.wavelnth eq 171 and currentheaders.quality eq 2l^30l)
; Note that the quality is set to 2^30 for near real time data
IDL> set_line_color
IDL> utplot, currentheaders[good171].t_obs, currentheaders[good171].datamean/currentheaders[good171].exptime,
ytitle = 'Mean pixel value [DN/sec]'
Get a chunk of data off the local cache
IDL> cd, '/archive/sdo/AIA/lev1/2011/05/01'
IDL> ff = file_search('*/AIA*_[012][0-9]00*_0193.fits') ;
Get all the 193 images from the first minute of every hour
Then,
go to http://jsoc2.stanford.edu/ajax/lookdata
click on RecordSet Select
in the box, enter:aia.lev1[25056141-26833011][?AIFWEN=74?]
Click GetRecordCount
Check desired keywords, and click Fetch KeywordValues for RecordSet