The long term trending site
has month, year, and mission long plots for various types of health keeping data for both AIA and HMI. The plots and the trending site
should be updated by about the 15th of every month. This document has detailed instructions on how to perform this update.
The Local Database
Every mnemonic that is used for trending is stored as a structure in its own IDL save file in the DATABASE_DIR. For example, all of the data
since the mission_start for the voltage of the AIA ATA1 ISS MY diode is in a file called: DATABASE_DIR/aia/E_AIA_IS1_DIODE_MY.sav
One can load and view the contents of this file as follows:
** Structure <26066f8>, 4 tags, length=246273872, data length=215489642, refs=1:
T STRUCT -> ANYTIM2INTS Array[15392115]
V DOUBLE Array[15392115]
ST STRING '18-Mar-2010'
EN STRING '1-Sep-2012'
IDL>
All of the mnemonics saved in the database have the same structure as shown by the help command above. Specifically, each has the times T with corresponding values V.
Also shown are the dates that the structure has data for, between start (ST) and end (EN) date. With the structure loaded one can easily plot the time series as
follows:
IDL> el_plot_mnem, 'Mission Long ATA1 ISS MY Diode Voltage', time = E_AIA_IS1_DIODE_MY.t, values = E_AIA_IS1_DIODE_MY.v, psym=3
Whenever it's convenient in the first or second week of each month, one should update all of the mnemonics structures to append the previous months data
using the following procedure.
In the description below, files indicated in red will need to be modified or used directly.
common_vals.pro - This file contains all of the global and default variables used by the trending and database routines. The variables aia/hmi_mnems contain
the full list of all of the mnemonics in the database.
extract_1mnem.pro - This procedure is used to pull down one mnemonics data for a specified time range.
extract_all.pro - This procedure is used to pull down ALL of the mnemonics data from mission_start to end_time. DO NOT USE FOR NORMAL PROCEDURE.
get_data.pro - This function loads from the local database one or several mnems data and returns it as a structure.
make_trending_plots.pro - This procedure is used to plot month, year, and mission long plots.
trend_aia.pro - This procedure sets-up and calls make_trending_plots to make housekeeping plots for AIA.
trend_hmi.pro - This procedure sets-up and calls make_trending_plots to make housekeeping plots for HMI.
trend_lev0_aia.pro - This procedure sets-up and calls make_trending_plots to make lev0 plots for AIA.
trend_lev0_hmi.pro - This procedure sets-up and calls make_trending_plots to make lev0 plots for HMI.
trend_fits_aia.pro - This procedure sets-up and calls make_trending_plots to make lev1 plots for AIA.
update_1mnem.pro - This procedure updates one mnemonic to the date specified.
update_all.pro - This procedure updates all the mnemonics for which_inst to the date specified. If you wish to add a mnemonic to the database,
simply append it to the respective list (aia/hmi_mnems) and use update_all to download all of the data for that mnemonic since mission_start.
Procedure for HK database
To begin updating the local HK database, start a new IDL session or reset the current session using the IDL command .reset_session. In the following example
I will assume that today's date is 10-Sep-2012 and that we are first going to update AIA mnemonics. The first step in the database update procedure is to
modify the globals in the file CVSROOT/IDL/tmack/common_vals.pro:
Update end_time to the first of the current month, ( '1-Sep-2012' )
Update month_start to the first of the previous month, ( '1-Aug-2012' )
Update year_start to the first of the current month and the previous year, ( '1-Sep-2011' )
Make which_inst equal to 'aia' (lowercase is important)
Ensure that data_path, events_file, and image_path all point to the proper directories. For example, the variable data_path should point to
DATABASE_DIR/which_inst/ and image_path should be TRENDING_DIR/trending_plots/which_inst/
Now that the global variables are configured, in the new IDL session run:
IDL> update_all, '1-Sep-2012'
Once this completes, change which_inst to equal 'hmi' and do the following:
IDL> .reset_session
IDL> update_all, '1-Sep-2012'
Now both AIA and HMI databases should have data current to 1-Sep-2012. You can check any given mnemonic by simply using the restore and help commands as above.
Procedure for Lev0 database
You can update the lev0 database at the same time as the HK database is being updated. To begin updating the lev0 database, do the following:
Run script archive_files.csh with the input of the month and year extension you want to
add to the plots. Since the plots that are currently in the trending_plots directory are from last month
(July since our example is for creating August plots), use 2012_07 as the argument.
> cd TRENDING_DIR/trending_plots/aia/
> ../archive_files.csh 2012_07
> cd TRENDING_DIR/trending_plots/hmi/
> ../archive_files.csh 2012_07
If you mess up and need to rename plots, there is a python script: TRENDING_DIR/trending_plots/fixwoops.py
that has the basic structure of searching for and renaming files. Use with care (comment out subprocess.call to see what actions will be taken first).
Update the events.txt file located in DATABASE_DIR. This file is used to make the event key
in the month-long and year-long plots.
Plots Using Data from the HK Database
trend_aia.pro and trend_hmi.pro - The two main programs used to make plots. They grab the data from
the local database and do calculations if necessary, set the plotting parameters, then pass everything to make_trending_plots.pro.
Then make_trending_plots grabs the data if it has not already been done, then calls el_plot_mnem to make
month-long, year-long, and mission-long plots.
You can simply run trend_aia and trend_hmi in an IDL session to generate the plots. These plots will automatically be saved to image_path.
Make sure that you have the correct which_inst set before you run these. Make sure you .reset_session after changing which_inst.
Plots Using Data from the Lev0 Database
First you should compile the get_lev0_data module with: .comp get_lev0_data
trend_lev0_aia.pro and trend_lev0_hmi.pro - These procedures make plots from the local lev0 database data.
You can simply run trend_lev0_aia and trend_lev0_hmi in an IDL session to generate the plots. These plots will automatically be saved to image_path.
Make sure that you have the correct which_inst set before you run these. Make sure you .reset_session after changing which_inst.
Note these take some time to pull all of the data; so I suggest running one at a time, overnight, for two nights.
Plots Using Data from Lev1
To make the couple of plots from the lev1 database for aia, open the trend_fits_aia procedure and look at the top. There are three blocks
that have to do with the different time scales of plots: month, year, and mission. Ensure that one of these blocks is uncommented and the other two are commented.
In an IDL session, compile and run the trend_fits_aia procedure for each month, year, and mission, commenting out the other blocks respectively.
At the time of this writing, there were no lev1 plots for hmi.
AIA Front Filters and Focal Plane Plots
Jake makes these and will email the spreadsheets when requested.
You must save each plot from the spreadsheet separately. To ensure you have the proper name for each plot, use the names
from the links on the long term trending site. E.g. save the 94 Focal Plane Filters plot to TRENDING_DIR/trending_plots/aia/94_focal_plane.png
AIA Focus Plots
After a full focus sweep you should be able to generate the focus plots for AIA. The following procedure to generate
the focus plots will assume that the most recent focus sweep was done on 25-Sep-2012 @ 20:28:00.
Hint: You can find when the last full focus sweep was run by using the History Search on the H&S and
searching for 3107. Be sure to note the time that script 3107 was run.
> cd TRENDING_DIR/aia/focus_trend
> mkdir 20120925
> idl
IDL> start='25-Sep-2012 20:28'
IDL> zbuff=1 ;0 if you would like to see the plots as they are generated.
IDL> .run aia_focus_astig_gfit
IDL> .run aia_focus
IDL> .run aia_get_focus_files
IDL> .run aia_focus_sequence
IDL> .run aia_focus_plot_mosaic
IDL> .run aia_focus_do_mosaics_new
IDL> .comp focus/focus_trending
IDL> focus_trending
After the last command is run you should see the location of the 5 pngs that were created by the focus_trending procedure.
These should be automatically put into your TRENDING_DIR/trending_plots/aia/ as mission_long_aia_XXXXfocus.png where XXXX = the wavelength number for a given plot.
Note: There are path specific constants in aia_focus_do_mosaics_new and focus_trending that will have to be changed if you move
these procedures.
Now that the new plots have been made it's time to copy them to the long_term_trending directory. Use the following commands to copy all the new plots to the long_term_trending directory:
The most recent plots are now in the long_term_trending directory structure. One can check how the new site looks now by opening one of html files in the long_term_trending directory and verifying that
the plots are correct. The final step is to release the long_term_trending directory by zipping (e.g. tar -czf long_term_trending.tar.gz TRENDING_DIR/long_term_trending) and following the Releasing the website
instructions below.