NSE system crash. SEBI puts onus on NSE, NCL MDs for Feb’ 21 system crash, 4-hour trading halt

A probe by market regulator SEBI into the four-hour-long trading disruption witnessed by the National Stock Exchange (NSE) on February 24, 2021, has revealed that exchange’s critical trading infrastructure suffered a massive breakdown due to faulty design and lower than required capacity to handle the peak load. 

In a show-cause notice on the subject sent a few months ago, SEBI has charged the exchange’s MD and CEO Vikram Limaye , chief of operations and technology Shiv Kumar Bhasin and National Clearing Ltd (NCL) MD and CEO Vikram Kothari of a casual, lackadaisical and less-than-honest approach.

NSE is a trading platform but NCL does risk management and settlement of trades. NSE had blamed digging of roads around its data centre and issues with telecom links for the systems failure. But SEBI has found that NSE and NCL lacked the essential safeguards and their systems failed on various counts.  

On February 24, NSE suffered the longest ever duration of trading halt due to a tech glitch at around 10.06 am. The Bank Nifty index price feeds to brokers were delayed and stopped completely in 30 minutes. Later, the Nifty index also stopped. NSE suspended derivatives trading at 11.40 am and minutes later, halted trading in the cash equity and currency segment. Trading recommenced at 3.30 pm and market timing was extended up to 5 pm for the first time.

Flawed technology

NSE told SEBI that its service provider Hitachi had made unknown changes to its storage area network (SAN) and the exchange was unaware of its faulty ‘failover logic,’ which caused the tech glitch. NSE said, “failover logic was contrary to the exchange’s design and requirements. If it had been properly designed by Hitachi, the systems would have continued to function despite the telecom link failure.” 

Failover logic is an automated process of switching to a standby computer server, system, hardware component or network upon the failure or abnormal termination of the previously active system or network..

SEBI has held NSE and NCL top bosses responsible for the glitch on the reasoning that the management failed in adequate testing and supervision before implementing Hitachi’s failover logic, a system critical to market functioning.

NSE had telecom links from Tata Communications and Airtel to replicate data between its primary data centre (PDC) at BKC and the near disaster recovery (DR) site. The telecom links were unstable since 10.06 am, which impacted the storage replication between the two sites.

Since the Hitachi SAN also failed, everything including index, risk management, clearing and settlement, surveillance stopped functioning. SEBI rules say that critical trading technology should not be outsourced and even when done for the need of a specialist, its responsibility and control vests with the exchange management.

Lacked peak load capacity

Apart from the failover logic, the lower capacity of telecom leased lines also prevented NSE from automatic switchover of operations to the backup site on February 24.

SEBI prescribes that exchanges have leased line capacity to the tune of 1.5 times their projected peak load. NSE was supposed to calculate the projected peak load for 60 days ahead based on the per second peak load of the past 180 days. All the systems involving trading, clearing, settlement were required to be considered in this process. Further, the technical components like network, hardware, software etc too were required to adequately meet such capacity requirements.

SEBI has specified that once the load exceeds 75 percent of the installed capacity, immediate measures should follow. Also, the actual 1.5 times peak load capacity should be in excess of the backup peak load for emergency situations. But NSE and NCL had a total capacity of only 16gpbs, which consisted of four telecom lines of 4gbps each. All these four lines were utilised, leaving nothing for the backup. 

“It means NSE, NCL did not have any backup capacity. To manage the load between PDC and DR site, the capacity requirement of NSE and NCL should have been much higher than 16gbps. The bandwidth available on February 24 was not sufficient to manage the load. NSE and NCL did not account for the normal lead time delivery of additional link capacity. Therefore, the load exceeded the capacity of the links,” the SEBI notice said.

SEBI said NSE, NCL would have been left with only 8gbps capacity if any one of the two telecom operators faced a disruption when the total peak load on the telecom leased lines was 9.49 gbps and 10.12 gpbs in January and February 2021, respectively.  

Delayed action 

Systems failure was noticed at 10.06 am but NSE continued trading till 11.40 am sans risk management, which also violated rules.

“NSE, NCL were completely aware that allowing trades without any risk management is a situation of disaster, born from their own decision to halt trading at 11.25 am. There is no valid argument for trading without any risk management between 10.06 am and 11.40 am,” SEBI said.

SEBI refuted NSE’s arguments that it allowed trades sans risk management to avoid collateral damage. SEBI said NSE and NCL management never gave serious thought to the option of moving at the DR site. Also, non replication of data and inconsistency showed there was lack of preparedness of the systems. Moreover, the management was not forthcoming in their replies to SEBI. The regulator was in the dark about the system’s failure at NSE and was informed about halting the market trading only at 11.30 am.  

“Responsibility is always coterminous with power. MD and CEO at all points of time has to be responsible for all the decisions and acts of omission and commission by the exchange or any of its employees as well for the technology aspects. Same is applicable to the clearing corporation. Such lackadaisical and less than honest approach cannot be accepted,” SEBI said, charging NSE, NCL and its top management with violation of several rules. 

A probe is further on regarding another major technical glitch on March 6 where prices for several stocks stopped updating on the screen in minutes after the opening bell. Then, as the stock prices moved, price updates for Nifty and other indices stopped reflecting on the screen.

NSE’s response

“NSE/NCL and its management received a show cause notice in August 2021 relating to the technology incident of February 24, 2021, an incident which is now more than a year old. Extensive reviews on this matter have been conducted by various committees, and necessary actions have been taken. NSE/NCL have submitted a detailed response to SEBI on all aspects of this matter several months ago.

“Significant human and financial resources continue to be committed by NSE/NCL to ensure robustness of technology. NSE/NCL and management acted in good faith and the decisions and actions taken were bonafide decisions based on the facts and circumstances prevalent at the time of the incident and in the best interests of markets and investors. NSE/NCL and management are committed to the highest standards of governance, integrity and compliance with laws and regulations. We are unable to provide additional detailed comments on this matter,” a spokesperson for NSE said.

COMMENT NOW