By Vipul Gaur, Advocate
Imagine asking an artificial intelligence assistant to prepare a research report. Within seconds, it creates a structured document with charts, references and a detailed explanation of the subject. The writing is polished, the presentation looks professional and the information appears ready to use. A closer review, however, reveals that some figures do not match available records and some references cannot be found. The report looks complete, but parts of it are built on information that never existed.
That is the challenge emerging as artificial intelligence becomes more capable and more widely used. AI systems have become remarkably good at producing responses that sound convincing, but a confident answer is not always a reliable one. The central question around AI is therefore beginning to change. It is no longer only about what these systems can create, but about how much confidence people should place in their output.
A recent Supreme Court judgment showed how this concern can move from theory into real-world consequences. While hearing an appeal arising from insolvency proceedings, the Supreme Court set aside orders passed by the National Company Law Tribunal after finding that the decisions had relied on non-existent legal precedents generated through artificial intelligence. The court did not question the usefulness of AI as a tool, but highlighted the need for careful verification and human judgment when technology is used in areas where accuracy has serious consequences.
The incident reflects a wider shift taking place across sectors. Artificial intelligence is moving from being a technology people explore out of curiosity, to one they increasingly depend upon in professional and everyday settings.
Also Read – From Enlightenment to Experience: Bihar’s Master Plan to Make Bodh Gaya a Global Buddhist Destination
When AI moved from curiosity to dependence
The early appeal of generative AI came from its ability to perform tasks that once appeared beyond the reach of ordinary software. It could draft content, create images, explain complex topics and assist with programming. For many users, it felt like a powerful assistant that could improve productivity and reduce the time spent on routine work.
That relationship is now becoming more complex. Businesses are integrating AI into their operations, developers are using AI tools during software development, and professionals are relying on these systems for research, analysis and summarising large volumes of information. Organisations are also experimenting with AI systems that can manage more complex workflows instead of simply responding to individual questions.
As AI becomes more involved in real-world processes, the consequences of mistakes become more significant. An incorrect answer in a casual conversation may be inconvenient, but an inaccurate legal reference, financial analysis or medical suggestion can have much wider implications.
The difficulty is that AI errors do not always look like errors. A person who is unsure may express doubt or ask for more information. An AI system can produce incorrect information in a clear and confident manner, making it difficult for users to immediately recognise that something is wrong. This creates a unique challenge because the technology can appear dependable at the very moment it requires closer examination.
The next stage of AI development is likely to make this question even more important. Earlier systems mainly responded to prompts. Newer AI tools are being designed to take on more independent tasks, including organising information, working across applications and supporting multi-step workflows. The shift is significant because the risk changes when AI moves from providing answers to taking actions.
A wrong response from a chatbot can be corrected by a user. A wrong action taken by an AI system operating within a business process, however, raises more difficult questions about responsibility and oversight. As these systems become more capable, organisations will need to think carefully about where automation should end and human review should begin.
Why human judgment remains central to AI’s future
The solution is not to treat artificial intelligence as a technology that cannot be trusted. AI has already shown considerable value by helping people process information, identify patterns and complete tasks more efficiently. The challenge is building the right relationship between machine capability and human responsibility.
A lawyer may use AI to speed up legal research, but the lawyer remains responsible for checking whether the information is accurate. A doctor may use AI-assisted analysis, but medical decisions still require professional experience and understanding of context. A company may rely on AI-generated insights, but accountability for business decisions cannot be transferred to a machine.
Keeping a human in the loop is therefore not a limitation on AI adoption. It is the process that allows organisations to use powerful systems while ensuring that important decisions remain connected to human judgment.
Trust in AI will also depend on issues beyond accuracy. As these systems become more common, people will want greater clarity about how their information is collected, processed and protected. AI systems depend heavily on data, making privacy an important part of the wider conversation around responsible adoption.
India’s Digital Personal Data Protection framework is part of this broader discussion. While it is not an artificial intelligence law, it addresses a fundamental issue behind many AI applications the responsible handling of personal information. As organisations continue to develop and deploy AI-driven systems, confidence in the technology will depend not only on the quality of its responses but also on the way it manages the data that supports those responses.
Artificial intelligence has already demonstrated that it can assist humans in ways that were difficult to imagine only a few years ago. The next phase will depend on how effectively people and institutions learn to work along with these systems. The goal is not to replace human judgment with AI, but to ensure that technological progress continues with accountability, verification and responsibility built into the process.
AI may continue becoming more intelligent, but its long-term success will depend on something more difficult to build confidence that grows alongside its capabilities. Building trust in artificial intelligence will require a shift in how the technology is adopted. Organisations may need to move beyond simply measuring what AI systems can produce and focus equally on how those systems are reviewed, monitored and used. Clear verification processes, transparency around limitations and meaningful human involvement in important decisions can help ensure that AI remains a tool that strengthens human capability rather than a system that replaces human responsibility.
(The author holds an LL.M. in Business Laws from NLSIU, Bengaluru, and is currently pursuing a Ph.D. in Cyber Laws at RGNUL, Patiala)
(Disclaimer – The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the views of Indian Masterminds. For feedback or queries, please write to [email protected].)
