Generative AI (“Gen AI”) products are known as Large Language Models (“LLMs”), of which several well-known tools are commercially available (e.g., ChatGPT, an OpenAI tool; Claude, an Anthropic tool; Gemini, a Google tool; and CoPilot, a Microsoft tool).  There is a growing body of research surrounding how LLMs operate, including in writing software code.¹  However, there are a number of significant legal and practical challenges, including those related to protection of Intellectual Property, stemming from the use of Gen AI that complicate how a service provider should go about navigating these waters.

Software development is a major focus of AI because of the need for faster delivery, innovative features, and cutting-edge functionality; however, the digital transformation explosion has cast aside practical elements of the process, including security.  This post identifies legal issues to address and articulates a number of practical guidelines to move ahead with the use of Generative AI in writing code.

1. Transparency (for both employees and users)

Perhaps the most fundamental question is how to ensure that both employees and customers understand the decision to use Gen AI tools.  Internally, a service provider should be transparent about its intentions with employees who may be impacted, especially existing coders. A successful AI strategy requires leaders to treat employees as partners in this innovation and involve them in the planning process from the start. Starting with an open approach will build trust and is a practical necessity because employees possess the deep institutional knowledge about the tasks being automated – this is crucial for training the AI system accurately.

For external use, where the generated software is public-facing, the Gen AI use must be evaluated from the regulatory perspective that will apply to its development and use.²  At its most fundamental level, regulators are concerned about consumers understanding that the software has been generated using AI tools, whether the regulations require disclosure of the source or merely inform users that they are interfacing with a Gen AI chatbot. Giving notice to customers about the use of AI-build and/or AI-driven software is required.  If customers will be using an AI chatbot to communicate about services questions/operations, the service provider needs to disclose that information.

2. Minimizing/Eliminating Hallucinations

A major problem with Gen AI tools, especially when code is being generated, is the errors that accompany its use – so-called “hallucinations.”³

This phenomenon occurs when an LLM produces outputs that are not grounded in its training data or are incorrectly decoded, resulting in statements that may appear coherent but are factually wrong.⁴

In essence, the Gen AI is “imagining” patterns, facts, or relationships that do not exist — similar to how humans might perceive shapes. For example, “code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency.”  While recent LLM-based approaches have shown promising results and revolutionized code generation tasks, LLMs often generate hallucinations, especially in code generation scenarios requiring the handling of complex contextual dependencies in practical development process.⁵ Although some studies have analyzed hallucinations in LLM-powered code generation, other studies are limited to standalone function generation.  [2409.20550] LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation.

Several factors contribute to coding hallucinations:

• Training data limitations: AI models are trained on incomplete or biased datasets, leading to errors in generated content;
• Overconfidence in predictions: AI systems may predict an answer or generate content even when they’re unsure, often fabricating details instead of admitting uncertainty; and
• Model complexity: larger models, while powerful, are also more prone to hallucinations due to their sheer complexity and volume of data they process.

Mitigation of AI Hallucinations

Given the risks associated with AI hallucinations, a service provider looking to use Gen AI for coding needs to take proactive steps to mitigate the creation of hallucinations:

a.Human Oversight

No matter how advanced Gen AI becomes, human oversight remains crucial. A user should ensure that Gen AI outputs, particularly in critical areas (e.g., coding) are reviewed by human experts/SMEs before being acted upon.

b.Continuous Model Improvement

Regular updates and improvements to Gen AI models can help minimize hallucinations. By refining training datasets and incorporating new techniques such as reinforcement learning, companies can reduce the frequency of hallucinations.

c.Transparency and Accountability

It is essential for each user to be transparent about the potential for errors. As described above, this includes setting clear expectations for customers/employees and implementing accountability measures in case Gen AI-generated content leads to issues.

3. Protecting IP – the Distinction Between Open/Enterprise Versions

Sound uses of Gen AI require a clear understanding of the distinction between tools that are “open” to public training (including Internet) and output, and those that have the ability to limit training data only to that provided by the user/enterprise that deploys it (“siloed” to limit using only the user’s own data, assuming the enterprise contracts with a developer for use of that Gen AI tool).  As suggested, this distinction normally is discernable from the terms under which the user/deployer obtains access to the tool from the developer/supplier.  A word of caution, however, deserves to be articulated – sometimes developers do NOT actually follow the terms set forth in the agreement, so it behooves a user to monitor how the Gen AI tool is performing to identify if the supplier is adhering to the legal terms of its enterprise system and actually protecting its proprietary information.⁶

This distinction is relevant so that every service provider can manage several legal issues.  First, it matters in terms of how the user’s proprietary information is protected – if coders enter proprietary information into an open Gen AI tool, that information is almost certainly going to end up in the future training data for that tool.  In other words, proprietary coding information will lose its protections and become part of the training data for other users of the tool.   And that in turn will result in incorporation of proprietary information in “new” coding outputs generated by the open Gen AI tool.  Of course, that result also could lead to the potential for the new code to infringe previously copyrighted code.

The reverse problem also exists, where use of an open Gen AI tool could easily result in outputs that incorporate the intellectual property of others – leading to potential infringement claims against the user.  Given these persistent infringement issues, it is critical that each user evaluate up front whether it is appropriate to consider using an open Gen AI tool.

But even if a business decides to use an enterprise version of the Gen AI tool to attempt to ensure that its proprietary information is protected and does not end up in the future training database, that decision does not end the legal issues surrounding the use of Gen AI. Under both EU/US patent laws and copyright laws, code generated purely by the Gen AI tool cannot be patented or copyrighted – only “natural persons” are eligible to have their inventions/content protected, so a machine generated product does not qualify.  The US Copyright Office — the US Patent and Trademark Office has shared similar thoughts — has provided some possible relief to this result by ruling that “sufficient human oversight” may overcome the presumption that AI has generated the content, but neither agency has provided any meaningful guidance as to what constitutes “sufficient” documentation that humans have contributed enough to the content to warrant it being protected.

Conclusion

Hallucinations in Gen AI coding are more than just occasional glitches; they represent a significant challenge for businesses adopting AI technologies.⁷ By understanding common examples, a user can better prepare itself to handle these issues. With human oversight, improved training models, and a focus on transparency, businesses can continue to leverage Gen AI while minimizing the risks associated with hallucinations.  The foundational solution is for the user to establish a robust AI governance program to ensure best practices are followed.