======================================================================= Hewlett-Packard OpenVMS ECO Cover Letter ======================================================================= ECO NUMBER: VMS722_MEM_CHAN-V0200 PRODUCT: OpenVMS Alpha OPERATING SYSTEM V7.2-2 UPDATE PRODUCT: OpenVMS Alpha OPERATING SYSTEM V7.2-2 1 KIT NAME: VMS722_MEM_CHAN-V0200 2 KIT DESCRIPTION: 2.1 Installation Rating: INSTALL_2: To be installed by all customers using the following feature(s): - Memory Channel This installation rating, based upon current CLD information, is provided to serve as a guide to which customers should apply this remedial kit. (Reference attached Disclaimer of Warranty and Limitation of Liability Statement) 2.2 Reboot Requirement: Reboot Required. HP strongly recommends that a reboot is performed immediately after kit installation to avoid system instability. If you have other nodes in your OpenVMS cluster, they must also be rebooted in order to make use of the new image(s). If it is not possible or convenient to reboot the entire cluster at this time, a rolling re-boot may be performed. 2.3 Version(s) of OpenVMS to which this kit may be applied: OpenVMS Alpha V7.2-2 2.4 New functionality or new hardware support provided: No. 3 KITS SUPERSEDED BY THIS KIT: - VMS722_MEM_CHAN-V0100 4 KIT DEPENDENCIES: Page 2 4.1 The following remedial kit(s), or later, must be installed BEFORE installation of this, or any required kit: - VMS722_PCSI-V0100 - VMS722_UPDATE-V0100 4.2 In order to receive all the corrections listed in this kit, the following remedial kits, or later, should also be installed: - None. 5 FILES PATCHED OR REPLACED: o [SYS$LDR]SYS$MCDRIVER.EXE (new image) Image Identification Information image name: "SYS$MCDRIVER" image file identification: "X-59" image file build identification: "X71Z-0050170001" link date/time: 23-MAY-2002 11:14:56.38 linker identification: "A11-39" o [SYS$LDR]SYS$PMDRIVER.EXE (new image) Image Identification Information image name: "SYS$PMDRIVER" image file identification: "X-31" image file build identification: "X71Z-0050170012" link date/time: 21-MAY-2003 09:49:40.31 linker identification: "A11-39" 6 PROBLEMS ADDRESSED IN THIS KIT 6.1 New problems addressed in the VMS722_MEM_CHAN-V0200 kit Page 3 6.1.1 PMDRIVER FORK_THREAD TQE DOUBLE-INSERT FIX 6.1.1.1 Problem Description: After installation of the VMS722_MEM_CHAN-V0100 ECO kit, systems may hang when using the Memory Channel SCS-port. The system will hang and not crash, requiring manual intervention and a system-HALT (Console ^P) to recover. This hang only occurs if there is high SCS-data-transfer activity (MSCP/TMSCP disk/tape serving) with high IPL-8 fork latency on the Memory Channel target node. A forced operator crash-dump and analysis will reveal the OpenVMS EXEC looping within the following routines: + SYSTEM_PRIMITIVE*.EXE: EXE$SWTIMER_FORK Primary SMP CPU stuck scanning EXE$GL_TQFL TQE-queue; check PCs on CPU-0 stack. + SYS$PMDRIVER.EXE: PM$COMQ_RETRY V7.2-2: TQE$L_FPC: SYS$PMDRIVER+13CC0 SDA> FORMAT/TYPE=TQE @EXE$GL_TQFL SDA> FORMAT/TYPE=TQE @. SDA> REPEAT .......... The OpenVMS EXE$GL_TQFL TQE-timer-queue will be corrupted, typically with the first TQE linked back to itself: + SDA> VAL QUE EXE$GL_TQFL Occasionally, there will be an ACCVIO within TIMESCHDL_xxx (SYSTEM_PRIMITIVES) while servicing TQE-queue. Images Affected: - [SYS$LDR]SYS$PMDRIVER.EXE 6.1.1.2 CLDs, and QARs reporting this problem: 6.1.1.3 CLD(s) CFS.97083,CFS.97225,CFS.98090,CFS.98928 6.1.1.4 QAR(s) 75-83-734,75-83-993 Page 4 6.1.1.5 Problem Analysis: A PMDRIVER.C retry (and TQE-stall) mechanism reschedules servicing of stalled SCS-port commands on PMx0: PDT COMQ (cmd. queue). The TQE used for the TQE-stalled retry, in routine FORK_THREAD, was not synchronized, allowing multiple insertions into VMS system TQE-queue by successively stalled PM commands. 6.1.1.6 Work-arounds: None 6.2 Problems addressed in the VMS722_MEM_CHAN-V0100 kit 6.2.1 Memory Channel virtual-hub (VHUB) can fail to come "ONLINE" 6.2.1.1 Problem Description: 1. A Memory Channel virtual-hub (VHUB) will fail to come "ONLINE" and form SCS-virtual-circuitlink-up if the Memory Channel VHUB VH0/Master node is not booted first, prior to booting the VHUB VH1/Slave MC-node 2. If a VH0/Master Memory Channel node crashes and/or reboots while the VH1/Slave Memory Channel node remains running, the Memory Channel link will fail and both VHUB Memory Channel nodes MCA0 (and MCB0 if applicable) will remain "OFFLINE" This MCx0 "OFFLINE" problem may also occur during MCA0/MCB0 adapter/link error-handling/recovery. The following symptoms are manifestations of this MC VHUB BOOT "OFFLINE" problem: OPA0: console errors: -------------------- %MCA0 CPU00: 19-SEP-2000 04:17:50 Slave but adapter_ok off, retrying. %MCA0 CPU00: 19-SEP-2000 04:17:50 MC re-init 5 second timer. %MCA0 CPU00: 19-SEP-2000 04:17:55 Slave but adapter_ok off, retrying. %MCA0 CPU00: 19-SEP-2000 04:17:55 MC re-init 5 second timer. . . ....... after 20 retries............... . . %MCA0 CPU00: 19-SEP-2000 04:18:00 Slave but adapter_ok off, retrying. %MCA0 CPU00: 19-SEP-2000 04:18:00 MC re-init 10 minute timer. Page 5 %MCA0 CPU00: 19-SEP-2000 04:28:00 Slave but adapter_ok off, retrying. %MCA0 CPU00: 19-SEP-2000 04:28:00 MC re-init 10 minute timer. . . . ON REMOTE NODE ATTEMPTING SW INIT......... MCA0 CPU00: 19-SEP-2000 04:27:50 node state retries exceeded" DCL SHOW DEVICE command output: ------------------------------- $ DCL SHOW DVICE MCA0: & PMA0: (& MCB0:/PMB0:) = OFFLINE: $ SHOW DEVICE MC Device Device Error Name Status Count MCA0: Offline 2 MCB0: Offline 16 $ SHOW DEVICE PM Device Device Error Name Status Count PMA0: Offline 0 PMB0: Offline 0 Images Affected: - [SYS$LDR]SYS$MCDRIVER.EXE 6.2.1.2 CLDs, and QARs reporting this problem: 6.2.1.3 CLD(s) CFS.76589,CFS.82476,CFS.82453,CFS.88426 6.2.1.4 QAR(s) None. 6.2.1.5 Problem Analysis: The initial MC-1.0 Memory-channel hardware adapter design and the original SYS$MCDRIVER Memory Channel device driver (from V7.1-2 through V7.3) assumed that the Memory Channel adapter "HUB_OK" and "ADAPTER_OK" status bits function symmetrically on both sides of a Memory Channel VHUB configuration. The later MC-1.5 and MC-2 adapter hardware design used the VH1/Slave node to assert ADAPTER_OK on both Memory Channel VHUB adapters. They also used the VH0/Master Memory Channel node to assert HUB_OK on both Memory Channel VHUB nodes. Consequently, Page 6 the initial MCDRIVER design incorrectly tested the Memory Channel adapter status, and mis-sequenced the Memory Channel adapter "ONLINE" control-commands. 6.2.1.6 Work-arounds: None. 6.2.2 MC_INCONSTATE (SYS$MCDRIVER) bugcheck 6.2.2.1 Problem Description: An MC_INCONSTATE (SYS$MCDRIVER) bugcheck may occur during local/remote Memory Channel node reboot or Memory Channel adapter/Memory Channel link- error-recovery. This bugcheck can occur regardless of the Memory Channel hub configuration: VHUB or real-HUB. The MC_INCONSTATE bugcheck will typically occur when a "nested error (MCDRIVER-internal or MC-adapter HW-error)" is encountered while recovering from a memory channel link error or local/remote memory channel node crash/reboot. The "MC_INCONSTATE" bugcheck is obvious, and is nearly always caused by this "nested error-handling" bug. A typical MCx0: error-log event sequence, and SDA> crash summary are shown below: MCx0: ERROR-LOG SUMMARY: Unsuccessful events: --------------------------------------------- MCB0 - Hardware error, reinitializing. MCB0 - Node 0: State: Uninitialized Node 1: State: Uninitialized MCB0 - Memory channel link online failure 2 MCB0 - We shouldn't be here. CRASH - MC_INCONSTATE Crashdump Summary Information: ------------------------------ Bugcheck Type: MC_INCONSTATE, Fatal error detected by Memory Channel Failing PC: FFFFFFFF.E2983A44 SYS$MCDRIVER+0BA44 Failing PS: 30000000.00000804 Module: SYS$MCDRIVER (Link Date/Time: 29-DEC-1999 04:09:37.99) Offset: 0000BA44 Images Affected: - [SYS$LDR]SYS$MCDRIVER.EXE Page 7 6.2.2.2 CLDs, and QARs reporting this problem: 6.2.2.3 CLD(s) CFS.83678,CFS.84469,CFS.72132,CFS.88426, 6.2.2.4 QAR(s) None. 6.2.2.5 Problem Analysis: The MC_INCONSTATE crash occurs due to internal SYS$MCDRIVER "nested error handling" synchronization problems between competing Memory Channel error handling and Memory Channel community node termination fork-threads. Memory Channel community node termination and MC-community-crashing status was incorrectly set and tested within numerous MCDRIVER error-handling routines. 6.2.2.6 Work-arounds: None. 6.2.3 Memory Channel Receive channel (RX_MESS_CHAN) message processing may hang 6.2.3.1 Problem Description: Memory Channel Receive channel (RX_MESS_CHAN) message processing may hang after processing 512 RX_MESS_CHAN messages during a single fork-thread ([MEM_CHAN]MC$HANDLE_MESS_CHAN_INT routine). This could occur with heavy Memory Channel SCS-traffic and high IPL-8 fork-thread scheduling latency. A Memory Channel RX_MESS_CHAN message-handling hang will lead to CNXMGR/LOCK_MGR stalls (and potential cluster hangs) as well as SCS "virtual-circuit timeouts". OPA0: CONSOLE PM/MC ERROR MESSAGES: ----------------------------------- %PMA0 CPU00: ... MC$_CHAN_QUE_EMPTY channel = 541C8 ppd = 83DD4CC0 %PMA0 CPU00: ... stall state CLEAR channel = 541C8 ppd = 83DD4CC0 %MCA0 CPU00: ... Timeslice exceeded while in workque for node RM763A %MCA0 CPU00: ... Timeslice exceeded while in workque for node RM763A %MCA0 CPU00: ... Timeslice exceeded while in workque for node RM763A %PMA0, Virtual Circuit Timeout - REMOTE PORT xxxx Page 8 SCS VC-TIMEOUT ERRLOG ENTRY: ---------------------------- . . . Error Type/SubType x4009 Signaled via Packet, Virtual Circuit Timeout. The "... Timeslice exceeded" error may continue to occur after this fix is applied. However, MC RX_MESS_CHAN processing will no longer hang after this event. Images Affected: - [SYS$LDR]SYS$MCDRIVER.EXE 6.2.3.2 CLDs, and QARs reporting this problem: 6.2.3.3 CLD(s) CFS.83846, CFS.74421 6.2.3.4 QAR(s) 75-3-3950 6.2.3.5 Problem Analysis: The SYS$MCDRIVER/MCCHANNELS.C mc$handle_mess_chan_int routine failed to refork after processing 512 RX_MESS_CHAN messages. It, instead, relied on a subsequent "MESS_CHAN_INTerrupt" to resume RX_MESS_CHAN processing. Under certain symmetrical SCS-credit-stall conditions, the specific remote Memory Channel node may not send any subsequent Memory Channel messages due to Memory Channel message-free-queue (MFQ) exhaustion: i.e., all messages are stalled in RX_MESS_CHAN processing due to re-fork failure. 6.2.3.6 Work-arounds: None. 6.2.4 MCDRIVER enters an infinite Hardware/Software initialization error-retry loop Page 9 6.2.4.1 Problem Description: Following a boot-time Memory Channel C unit-init/self-test "LOOPBACK WRITE TEST" failure, which indicates a Memory Channel adapter PCI-DMA error, the MCDRIVER will enter an infinite HW/SW initialization error-retry loop. The following OPA0:/console errors will be issued at 5 second intervals, changing to 10 minute intervals after 20 retries: %MCA0 CPU00: ... MC loopback write interrupt test failed. %MCA0 CPU00: ... Couldn't get mgmt lock. %MCA0 CPU00: ... ERR - ucb offline and adapter not crashing . %MCA0 CPU00: ... Couldn't get mgmt lock. %MCA0 CPU00: ... ERR - ucb offline and adapter not crashing . %MCA0 CPU00: ... Couldn't get mgmt lock. %MCA0 CPU00: ... ERR - ucb offline and adapter not crashing . Note: The first error message occurs on the first pass only. Images Affected: - [SYS$LDR]SYS$MCDRIVER.EXE 6.2.4.2 CLDs, and QARs reporting this problem: 6.2.4.3 CLD(s) CFS.91841, CFS.82837 6.2.4.4 QAR(s) 75-52-211 6.2.4.5 Problem Analysis: MCDRIVER unit-init self-tests failed to place MCA0: (or MCB0:) UCB permanently "OFFLINE" after PCI-DMA "loopback write test" failure. The PCI-DMA "loopback write" self-test is only performed on the first pass through unit-init, and skipped for all subsequent Memory Channel reinitialization/start-up attempts. Consequently, subsequent Memory Channel reinitializations attempted to perform Memory Channel software-link-up and software-init, but failed in an infinite loop. The MCDRIVER should have assumed that inability to complete a basic local PCI-DMA "loopback write" guarantees that no communication is possible with other Memory Channel cluster nodes on any subsequent Memory Channel reinit/restart attempt. Therefore, no Page 10 subsequent/successive Memory Channel software-init can succeed. 6.2.4.6 Work-arounds: None. 6.2.5 System crashes with a CPUSPINWAIT, CPU spinwait timer expired bugcheck. 6.2.5.1 Problem Description: CPUSPINWAIT bugchecks may occur on any GSxxx Alphaserver platform (GS140,GS80/160/320) with a Memory Channel-adapter. The bugchecks occur due to an eror in the SYS$MCDRIVER "MC$ALLOCATE_MESSAGE" routine performing Memory Channel message free-queue-header "loopback WRITE", and an incorrect timer implementation. The CPUSPINWAIT bugcheck will always involve an SMP$TIMEOUT acquiring the SCS-spinlock while another SMP-CPU is holding the SCS-spinlock within the SYS$MCDRIVER / [MEM_CHAN]MCCHANNELS.C MC$ALLOCATE_MESSAGE routine. Crashdump Summary Information: ------------------------------ Bugcheck Type: CPUSPINWAIT, CPU spinwait timer expired Failing PC: FFFFFFFF.8007A384 SMP$TIMEOUT_C+00064 Failing PS: 28000000.00000804 Module: SYSTEM_SYNCHRONIZATION_MIN Offset: 00000384 NOTE: The "MC loopback write interrupt test failed" error is typically due to a leftover/stale Memory Channel adapter PCI-logic error-state that will only clear with a CONSOLE >>> INIT operation (to perform PCI-bus RESET). Users who frequently reboot without using the CONSOLE >>> BOOT_RESET = ON switch (Environment Variable) or without performing a CONSOLE >>> INIT command are susceptible to this "MC loopback write test" error. Images Affected: - [SYS$LDR]SYS$MCDRIVER.EXE 6.2.5.2 CLDs, and QARs reporting this problem: Page 11 6.2.5.3 CLD(s) CFS.91841, CFS.82837 6.2.5.4 QAR(s) 75-52-211 6.2.5.5 Problem Analysis: "Loopback write" is used to synchronize remote Memory Channel node free-queue headerpointers. This MC$ALLOCATE_MESSAGE SCS-spinlock hang CPUSPINWAIT bugcheck occurs due to two design problems within [MEM_CHAN]MCCHANNELS.C while performing "Loopback write" on Memory Channel MESS_CHAN free-queue-header: 1. Insufficient EVAX_MB (Memory Barriers) were used to ensure timely posting of CPU->PCI->MC-adapter "LOOPBACK WRITE" and local cache flushing ("stale cache" problem) to avoid Alpha speculative-execution "pre-fetching". EVAX_MB are required to serialize CPU-vs-MC/PCI memory-writes; and for conformance with ALPHA SRM (AXP spec.) Para. 5.6.4.3. 2. The "LOOPBACK WRITE" timer was incorrectly implemented with EVAX_RPCC ("Read Process Cycle-Counter"), a 32-bit CPU-clock-based timer susceptible to short-term overflow when using an 8-second timeout value (timeout based on MC_SERVICES_P5 value). 6.2.5.6 Work-arounds: None. 6.2.6 System can crash with a INVPTEFMT, Invalid page table entry format 6.2.6.1 Problem Description: Any SCS-data-transfer of "0-length", using the Memory-Channel/MC SCS-port will result in an "INVPTEFMT, Invalid page table entry format" bugcheck The bugcheck is within IOC_STD$PTETOPFN, as a result of a call to IOC_STD$FILSPT from PMDRIVER.C/SETUP_COPY. Crashdump Summary Information: ------------------------------ Bugcheck Type: INVPTEFMT, Invalid page table entry format Current Process: NULL Page 12 Current Image: Failing PC: FFFFFFFF.800B88FC IOC_STD$PTETOPFN_C+0008C Failing PS: 38000000.00000804 Module: IO_ROUTINES (Link Date/Time: 13-DEC-2000 00:39:37.49) Offset: 000048FC Images Affected: - [SYS$LDR]SYS$PMDRIVER.EXE 6.2.6.2 CLDs, and QARs reporting this problem: 6.2.6.3 CLD(s) CFS.83593,CFS.83772 6.2.6.4 QAR(s) 75-66-1330 6.2.6.5 Problem Analysis: The PMDRIVER.C (and historically cloned PBDRIVER.C) SETUP_COPY routine mishandles "0-length" SCS-buffer-transfer packets. This results in the SETUP_COPY routine "walking" off of the end of a user- page-table, while successively calling IOC_SPT$FILSPT for each page, resulting in an INVPTEFMT bugcheck. 6.2.6.6 Work-arounds: None. 6.2.7 SCS "SEND MESSAGE" and SCS data transfer commands can stall or hang 6.2.7.1 Problem Description: SCS "SEND MESSAGE" (typically LOCK_MGR and MSCP disk commands) and SCS data transfer commands, issued over a PM/MC SCS virtual circuit (VC), can stall or hang following exhaustion of Memory channel "channel-free-queue" entries. The duration of this stall or hang is entirely dependent on SCS-sysap traffic and flow-control (SCS "credit") patterns and will persist until one of the following occurs: Page 13 o SCS VC timeout error closes the VC o SCS-sysap sends a message that breaks the stalemate o SCS VC timeout mechanism sends a message that breaks the stalemate o PMx0: SCS-port timeout occurs, crashing the MC port This SYS$PMDRIVER MC-SCS-command processing hang/stall can occur under the following two conditions: - HANG: Under heavy and primarily unidirectional loads; - STALL: Under more bi-directional loads, stalls will create low performance over the Memory Channel VC, drastically reducing Memory Channel performance under load. Because this hang/stall will block internode SCS-sysap cluster communications, symptoms can be obscure and numerous, or may manifest as: o Performance degradation over Memory Channel based SCS VCs o A SCS VC-timeout o A LOCK_MGR stall/hang or performance loss o MSCP served disk command timeouts or disk I/O slowdowns o Customer LOCK_MGR-dependent application stalls, hangs, or slowdowns Images Affected: - [SYS$LDR]SYS$PMDRIVER.EXE 6.2.7.2 CLDs, and QARs reporting this problem: 6.2.7.3 CLD(s) None. Page 14 6.2.7.4 QAR(s) None. 6.2.7.5 Problem Analysis: SYS$PMDRIVER's command processing routine, "FORK_THREAD:", stops processing commands when it cannot allocate a "channel-free-queue" entry from MCDRIVER. This hang occurs because the "FORK_THREAD:" routine neglects to refork after a "MC channel-free-queue" exhaustion stall. Command processing will only resume when a new command is inserted on PMDRIVER's command queue. If all the SYSAPS have exhausted their send credits and are awaiting responses from the other side, this hang can persist indefinitely, or until higher-SCS- layer timeout occurs. 6.2.7.6 Work-arounds: None. 7 INSTALLATION INSTRUCTIONS: 7.1 Installation Command Install this kit with the POLYCENTER Software installation utility by logging into the SYSTEM account, and typing the following at the DCL prompt: PRODUCT INSTALL VMS722_MEM_CHAN /SOURCE=[location of Kit] The kit location may be a tape drive, CD, or a disk directory that contains the kit. Additional help on installing PCSI kits can be found by typing HELP PRODUCT INSTALL at the system prompt 7.2 Scripting of Answers to Installation Questions 7.2.1 Scripting of Answers to Installation Questions During installation, this kit will ask and require user response to several questions. If you wish to automate the installation of this kit and avoid having to provide responses to these questions, you must create a DCL command procedure that includes the following definitions and commands: - $ DEFINE/SYS NO_ASK$BACKUP TRUE Page 15 - $ DEFINE/SYS NO_ASK$REBOOT TRUE - Add the following qualifiers to the PRODUCT INSTALL command and add that command to the DCL procedure. /PROD=DEC/BASE=AXPVMS/VER=V2.0 - De-assign the logicals assigned For example, a sample command file to install the VMS722_MEM_CHAN kit would be: $ $ DEFINE/SYS NO_ASK$BACKUP TRUE $ DEFINE/SYS NO_ASK$REBOOT TRUE $! $ PROD INSTALL VMS722_MEM_CHAN/PROD=DEC/BASE=AXPVMS/VER=V2.0 $! $ DEASSIGN/SYS NO_ASK$BACKUP $ DEASSIGN/SYS NO_ASK$REBOOT $! $ exit 8 COPYRIGHT AND DISCLAIMER: (C) Copyright 2003 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP and/or its subsidiaries required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. Neither HP nor any of its subsidiaries shall be liable for technical or editorial errors or omissions contained herein. The information in this document is provided "as is" without warranty of any kind and is subject to change without notice. The warranties for HP products are set forth in the express limited warranty statements accompanying such products. Nothing herein should be construed as constituting an additional warranty. DISCLAIMER OF WARRANTY AND LIMITATION OF LIABILITY THIS PATCH IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND. ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE HEREBY EXCLUDED TO THE EXTENT PERMITTED BY APPLICABLE LAW. IN NO EVENT WILL COMPAQ BE LIABLE FOR ANY LOST REVENUE OR PROFIT, OR FOR SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, WITH RESPECT TO ANY PATCH MADE AVAILABLE HERE OR TO THE USE OF SUCH PATCH.