Using reinforcement learning for stock trading

It was when he was working for Silicon Valley-headquartered start-up Accredible that Deepender Singla, Co-founder, Agvest Technologies Pvt Ltd, learnt about reinforcement learning (RL), a branch of artificial intelligence that was being used by technology giants for online games. They were teaching computers to play games such as chess, poker and Go. The computers were programmed to play with the top players and defeat them.

Deepender wondered if he could use RL to do intra-day trading in stocks. “I took a week off from my work when I combined reinforcement learning with Nifty trading and after a week or two, I open-sourced this project on the Internet,” says Deepender, 30, an electrical engineering graduate from Thapar Institute of Engineering & Technology, Patiala, Punjab, and a Liberal Arts degree holder on a Young India fellowship from Ashoka Univesity. It was when he open-sourced the project he came in touch with his Co-founder Rahul Goyal, an M.Tech in computer science engineering from IIT-Delhi and an MSc in Financial Economics from Oxford. Both of them decided to collaborate and form a venture that will use RL for intra-day trading strategies for shares.

Quantitative research

According to Deepender, they moved to his home town Panipat in January 2018 and started Agvest Technologies, which is known by the brand name Niveshi. Rahul was involved in doing quantitative research at that time. The two decided to create a system that automated quantitative research. “Can we build the system using reinforcement learning based on my open-sourced project, which will take inputs from the market and create trading strategies,” says Deepender, on their thinking.

Niveshi’s engineers built the core infrastructure platform that allows strategy research, capital allocation and market execution to be automated.

Its data scientists then experimented on top of this infrastructure with the latest in RL to iterate viable strategies. Current quant strategies, according to him, are expensive to generate and full of human bias. Niveshi aims to eliminate this by having machines create a new class of strategies that use mathematical computations and crunch numbers to identify trading opportunities. They decide to start with the banking sector because it is heavily traded. They got the data of all the listed banking stocks on their system, including prices at regular intervals and got the system to use RL and come up with trading strategies for leading public sector banks and private sector banks, one after the other. They went live in September 2018 with their quant trading strategy. Within a month, says Deepender, they lost half the capital they had deployed. That was when the IL&FS crisis happened and they realised that even though their system was able to come up with the right trading strategy, it was not robust enough to handle outlier events like the IL&FS crisis.

Deepender says they realised they needed money to work on the product. Niveshi raised ₹4.70 crore from 3One4 Capital and a few angel investors in January 2019. The first part was coming up with an automated trading strategy. The second part they worked on was to put the strategy through a series of tests, literally hundreds of them – what if the market starts a little earlier, what happens when data of one bank is inter-changed with that of another bank, what if the trade is delayed by a few seconds or minutes – to see if the system was able to survive all those tests. “I have to make sure that those strategies which my system creates, even if they don’t make money, should not bleed money,” says Deepender.

They went live with the new system and found that the system was now robust enough to handle all possible mathematical computations. He answers with a lot of pride that even on days when the Sensex crashed heavily, their system made money for their clients. Niveshi, according to Deepender, operates with one of the largest trading desks in India, but doesn’t divulge the name. Niveshi has an exclusive agreement with that customer for some more time, after which it is free to sell or licence the product to others.

Business model

Deepender says Niveshi will continue with its quant trading for intra-day strategy. It gets a percentage of the profit as income. Niveshi will come up with similar systems for other sectors, followed by other asset classes such as bonds, foreign exchange and commodities.

The company is selling intra-day trading strategies to financial institutions, but aims to set up its own hedge fund later, says Deepender. The biggest challenge is getting the right talent; they need bright computer engineers to work on the product and these engineers have to be paid high salaries, he adds.