Errors and Problems You Might Encounter

On this Page:

• EGSE
• Instrument
• Test Bed

EGSE Errors

"Log file open" error in the event viewer

Could mean...
• If you abort a running STOL (such as aia_ops or hmi_ops), user log files are not automatically closed. This may include more than one log file. The event viewer history log file (yyyymmdd.hst) or files with other names or “handles”.
To Fix it....
• perf sdo_log_close in the EGSE command window to close the event viewer log file:
perf sdo_log_close
• Or enter the command close user “handle” in the EGSE command window to close log files with other names, if known
close log ”LOG”
• Or select the "Handles" tab on the EGSE screen to see a list of open log files. Select the file (or files) in question and close it.

"Invalid Line ID" error in the event viewer

Could mean...
The HMI wavetune set ID is not set correctly. This is displayed on the HMI sequencer screen (HMI_SEQ_WL_SET_INDEX) , and refers to the line in the wavelength_set table (FLIGHT_V02.s as of 7/20/11) that the sequencer should use to get the tuning index. This parameter gets set in hmi_reconfig.src and hmi_testbed_config.src, but is not set in script 9000. It can show the wrong value if the testbed (or instrument) was shutdown and then not brought up with one of these STOLs.
To Fix it....
• Send the command "/HMI_SQ_CONFIG PARAMETER=WAVELENGTH_SET_INDEX VALUE=14"
• Here, 14 is the current value asof 7/20/11. If you don't know what the value should be, check it with IDL:
IDL> sag_plot_mnem, win=1, mnem='HMI_SEQ_WL_SET_INDEX'

"Child name in use" error in the event viewer

Could mean...

You tried to load something that contained a file that was already onboard. Do an sdo_dir to verify.
To Fix it...

Run the STOL sdo_clear_error and select "FS" (for File System). Then either erase the conflicting load that is already on board and try again, or rename the new load to something unique.

"Max number of procedures (6) already loaded" error in the event viewer

Probably means...

The EGSE can only handle a depth of 5 STOLS (Calling a STOL that calls another STOL that calls another STOL that calls another STOL...) and you exceeded that limit.
To Fix it...

Separate out some STOLs to reduce the depth.

"Socket died / hmi_ssim_tc is not ready" error in the event viewer

Could mean...

The one time I saw this it was because I had an extra space in my command. The line in the STOL said: set $ls_cmd = " /hmi_table...etc." It was the extra space after the open quotation mark that killed the socket.
To Fix it...

Remove the extra space. If that doesn't work, talk to Roger Chevalier.

STOL thinks you are commanding HMI, but you are commanding AIA (on SDOIOC-SPR)

Could mean...

• See email below

From: Lehman, Emma M (N-INTRINSYX TECHNOLOGIES CORPORATION)
Sent: Wednesday, July 27, 2011 9:01 AM
To: Paul Boerner; wolfson@lmsal.com
Cc: 'Zoe Frank'; Allard, Brett A
Subject: STOL issue on SDOIOC-SPR

For the record:
We (Paul, Jake, Emma) ran into a problem on SDOIOC-SPR while trying to use the sdo_oh_control STOL to command AIA yesterday. The problem was caused by the way the STOL determined which instrument we were commanding: it checked whether HMI or AIA telemetry was coming through, rather than what the global $GS_SDO (which is either AIA or HMI) was set to. This caused a problem because SDOIOC-SPR receives telemetry from both instruments, so whichever instrument it checked first would return a positive result (current telemetry). In our case, this meant the STOL thought it was commanding HMI rather than AIA. I think we did not run into this problem before because we usually stick to commanding on HMI to test out the SPR machine, and the STOLs that use this method check HMI first, see that the telemetry is current, and if so do not go on to check AIA, so the problem only occurs if you want to command AIA.

I found and fixed a handful of other STOLs with the same problem:
• Sdo_dir
• Sdo_dir_dump_test
• Sdo_dir_menu
• Sdo_oh_cmd_for _scripts
• Sdo_script_config

In addition to the ones Paul and I fixed:
• Sdo_oh_control
• Sdo_oh_control_menu
• Sdo_oh_construct tlm

It is still possible that there are other STOLs that do the same thing, but that I missed in my search because they use different syntax or check different telemetry packets. So, in case anyone runs into this again, the fix is below.

Thanks,
Emma

FIX:
The issue can be fixed by replacing the following code (usually at the beginning of the STOL):

			
#Determine which instrument: AIA or HMI
			
IF ( isStale(0x05) == 0 ) THEN
			
	set $ls_prefix = “HMI_”
			
ELSE
			
	IF ( isStale(0x25) == 0 ) THEN
			
		set $ls_prefix = “AIA_”
			
	END IF
			
END IF
		

With


Set $ls_prefix = CONCAT($GS_SDO, “_”)


Note that there are a couple variations of this check, but it should be obvious what the differences are and if you need to leave out the underscore, etc.

Instrument Errors

AIA/HMI "Camera Anomaly" ANS Message

Meaning and fix…
• Messages containing the term “Camera Anomaly” such as “Bit Flip Camera Anomaly for HMI Camera 2” for example, should be acted on promptly. See AIA and HMI Camera Interface Anomaly Recovery procedure for instructions on recovering from these errors.
• The messages indicate a particle hit or SEU (single event upset) in the AIA or HMI camera electronics affecting the image data. See Camera, DCHRI, and Watchdog Reset Anomalies for an up to date listing.
• Note that a camera link failure message will also be generated by the ANS during the camera recovery procedure.

AIA/HMI "TABLE_PARITY_ERR" ANS Message

Meaning and fix…
• AIA and HMI Table Parity Errors are reported as multiple limit trip conditions in the EGSE History log but a single notification will be sent to the operations team. Messages containing the term “TABLE_PARITY_ERR”, such as “EGSE message: LIMIT_TRIP :tm_mgr :02:09:36: HMI_HRB_TABLE_PARITY_ERR:” for example, should be acted on promptly. See Table Error Recovery Procedure for instructions on recovering from these errors.
• The messages indicate a particle hit or SEU (single event upset) in the AIA or HMI on-board table memory section of the DCHRI and may or may not affect the science data. Affected data may or may not be recoverable in data post-processing. See Camera, DCHRI, and Watchdog Reset Anomalies for an up to date listing.

AIA "Readout timer has expired, but shutter has not closed" ANS Message

Could mean...
• These messages are sent when the camera takes a 40 msec exposure. This only happens when there is a large flare and the AEC drives the exposure time down to 40 msec, nominal exposure times are much longer. The notification code for these is “IMC220”. See Readout Errors for an up to date list.
More Info
• As of 9/22/11, Gary Linford updated the ANS code for both the EGSE Alerts and the NASA ANS alerts to stop sending the "IMC220" messages (Readout Timer Errors) to the phones. The NASA ANS messages will now go to the "info alert" list.

AIA "Frame 4 from framelist 3343 had error BUSY_TS1" ANS Message

Could mean...
• Framelist error messages are sent when the currently running AIA framelist is interrupted. BUSY_TS indicates a busy telemetry stream. We see them when the sequencer is stopped for maneuvers, particularly GT/PZT calibration maneuvers. The numbers (4 and 3343 in this example) reported on the line are related to the framelist interrupt position. The notification code for these is “ISQ203”. See Framelist Errors for an up to date list.

AIA "Camera Link Not Ready" ANS Message

Could mean...
• These errors go to the ANS alert page but do not go through to the phones because we asked Gary L. to filter them out in March 2011.
• On the testbed, this error usually means you have to reset the HSB atmel. Brett Allard knows how to do this.
• We usually see this message on HMI when the camera settings are changed for a special framelist (like Regulus, for example). In this case, the camera readout delays are changed, and when you switch back to the old framelist and settings sometimes you sometimes get this error for the first couple frames. It is not a big deal.
• On AIA, we see this occasionally when high solar activity makes the images hard to compress efficiently, so NPACKETS and NSPIKES is driven up (see lookdata screenshots below) and it takes longer to downlink each image. This throws off the timing and sometimes causes us to get "Camera Link Not Ready" messages. The notification code for these informational messages is “IMC216”. See Camera Link Errors for an up to date list.

HMI "Timer Violation" ANS Message

Could mean…
These messages are sent when the currently running HMI framelist is interrupted. We see them most often during eclipse season and lunar transits. The numbers reported on the line are related to the framelist interrupt point. The notification code for these is “HSQ204”. See Timer Violations for an up to date list.

HMI "Phase Max Error" ANS Message

Could mean…
• We see these messages when we are trying to upload information (like new versions of scripts or tables) at the same time the HMI image stabilization system is trying to download information from regular aliveness tests it runs.
• They are not harmful but we try to avoid them by breaking large uploads into smaller units when possible. The notification code for these is “HIS312”. See Phase Max Errors for an up to date list.

"Aborting script with error" ANS Message

Could mean...
A scheduled script was scheduled twice at the same time and so aborted one of the runs. If this causes runaway messaging on the ANS, see ANS for information on remote lock-file removal.

AIA/HMI "SS Unable to find wildcard filename match for *7004.*.SCR" error in event log

Could mean...
The 700* scripts are called automatically by TSM at the start of S/C calibratons. If the script is not loaded to the instrument then you get that message. It is nothing to worry about, but can be avoided by uploading an empty "dummy script".

Clearing AIA/HMI file system or other onboard memory errors.

Occasionally you may encounter an onboard memory error which presents itself as an "ERROR_STATE". For instance:

ERROR INVALID_STATE: THE FILE /RAM/H_ECLIPSE.1.61.GZ CANNOT BE ERASED FROM RAM WHILE IN THE ERROR_STATE

This error occurred because H_ECLIPSE was already onboard and the operator attempted to upload a new version before erasing the old version. To clear errors of this type, first be sure you understand the cause and resolution. Then one can clear the ERROR_STATE by using the sdo_clear_error stol.

Data Stream Errors

"EARLY FILE WARNING" ANS Messages

Means…
• We occasionally receive messages including the words “FOP1 EARLY FILE WARNING FOPEarlyFileWarningTransmit:”. These are due to delays in verifying backup tapes at Stanford. Hao usually fixes of these quickly.
• Note from Hao, “ It's nothing to get concerned over. It means the telemery backup tape on our end hasn't been verified in 15 days. … FYI, the type of message that should raise concern are the "FNL" Final File Warnings. It means a telemetry file hasn't been verified in 25 days and something might be wrong.”

"Socket died to: bmw/audi on port" ANS Message

Could mean...
Machines in the IOC were disconnected from telemetry. Most likely due to workstation reboots.

Test Bed Errors

Pickscreen Doesn't Work

Might be because...
• The test bed runs an old version of the GSE, which requires the port to be specified by pickscreen in order to open a screen. The new version of the GSE does not need the port to be specified by pickscreen, and since pickscreen is under CVS control, if you do a CVS update in the egsesw/EGSE/BIN area, the new version of pickscreen will come through, which does not specify the port, so you won't be able to open any screens.
• To fix this (or at least get around it), edit pickscreen (in /egsesw/EGSE/BIN/pickscreen) by changing the command it sends (should be line 56) from:
  echo "Starting screen: /egsesw/egse/tasks/screen ${GSE_SCREEN_DIR}/$screen_name"
  to
  echo "Starting screen: /egsesw/egse/tasks/screen 1004 ${GSE_SCREEN_DIR}/$screen_name"
  1004 is the port that the EGSE should use to open the screen.

You get spacecraft telemetry, but no instrument telemetry

Might be because...
• IRIS was using the testbed and forgot to switch the cables back.
• The cables are located behind the hardware table in the corner. They are labeled. If you don't know how to switch them back, ask Gary Heyman or David Sabolish to do it for you.