At least 50 per cent of GenAI projects will overrun their budgeted costs due to poor architectural choices and lack of operational know-how, by the end of 2028 with the aggregated costs of model inference (opex) reaching at least 70 per cent of total model lifetime costs.
In conversation with businessline Deepak Seth, Director Analyst at Gartner explained that companies often struggle to visualise all the cost elements associated with a new technology. In case of AI, the variables which influence the cost are being established slowly. Gartner observed how some AI interactions are already going over budget with each AI interaction leading to 4-5 iterations of a specific task. The information shared by user in their prompts is processed for every response, leading to an incremental cost with every iteration.
“This would be one thing where the demand for this can exceed the way companies have figured out what the cost would be,” said Seth.
Impact on returns varies by use-case
IOn impact on investment, Seth said the same would depend on the use-case for the AI. Gartner categorised AI-usage into three categories: defend - where you take an AI assistant and roll it out to employees, 2 extend - where you layer in some capability of your own and uppend - where the company builds its own LLM or fine tunes existing ones. Defend is the least cost-intensive, followed by extend and then uppend. Seth said defend will provide return on employee, extend on investments and uppend ensures return on the future.
Gartner estimated that the total cost of ownership for defend-use case will vary between $115,000-800,000 giving an estimated cost of $1,700-12,300 per user per year across coding, business and marketing purposes. Extend-use case cost of ownership can vary from $760,000 to $4.4 million for $6,000-27,300 per agent per year. Uppend use-case costs will vary from $3.3 to 16.6 million with a use-specific value.
“The key is to move beyond defend-use case into the extend category. You have to think of what AI can do rather than focusing on what AI is. Companies are slowly trying to figure out this transition,” said Seth.
Need to drive down inference cost
Model inference which is similar to the operating expense in the case of a model must go down considering 70 per cent of AI model cost will be the same, added Seth. He suggested companies can catch responses and respond to questions from the same rather than repeatedly sending the question to the large language model.
“Think of it like the cost of putting the plumbing in for your faucets, tap, water and flush, etc. We are saying at some point, 70 per cent of your cost will be the cost of the water, not the cost of putting the plumbing in place,” said Seth.
SLMs to give better returns
Buildings SLMs rather LLMs will be a costlier investment initially but provide better returns considering recurring costs, maintenance costs will be lower, Birgi Tamersoy, Sr Director Analyst, Gartner said.
“With LLMs, you can start very quickly just by prompt engineering, and get good results. So the entry point is very low. SLMs will require more investment for customisations and accuracy. But in the long term, your operating costs will be dropping,” said Tamersoy.