The Last Conversation

The last person Pierre ever confided in was not a person.

For six weeks in the early spring of 2023, Pierre, a young Belgian father, spoke almost every day to an AI chatbot named Eliza. He was a health researcher frightened by the climate and died by suicide. After his death, his wife, Claire, preserved the conversation logs. According to her reading of those exchanges, the software listened, agreed and never once offered crisis resources or suggested he seek professional help. Instead, it appeared to validate his feelings and suggested that his death might help save the planet. Then he was gone. The logs remained. The questions remained too.

Eliza was, and still is, available on the Chai app, a platform developed and operated by Chai Research, a US-based company. Three years on, as of publication, no civil claim arising from his death has been publicly reported. ¹

Think of Pierre’s story as an edge-case test. Engineers design such tests to reveal where a system may fail. In AI development, anticipating extreme situations that deviate from trained data is far more complex. Pierre’s story reveals similar fault lines in the law. It shows that when a distributed software system operates across jurisdictions and harms a person, the rules meant to protect that person are not merely thin. For the most vulnerable users in certain locations, they are almost non-existent. Was Pierre reckless? Was he foolish? Or was he simply doing what millions now do: turning, late at night and alone, to something that simulates understanding?

From Tool to Confidant

We used to picture artificial intelligence as a hammer: a neutral tool. It did whatever the hand that held it wanted. The hand was in charge; the hammer obeyed and had no opinions. That picture is now outdated.

No longer just a tool, an AI chatbot increasingly functions as a companion, confessor, and, for some, a therapist. People share thoughts with it they would not divulge to anyone else, drawn in by a private confidant that always answers at any hour and, in some cases, is free of charge. Quietly and without a licence, it has begun doing work once reserved for trained professionals: comfort in a crisis, advice about life.²

Pierre was not a rare use case. Every day, millions reach for conversational AI over the most ordinary things: a strained relationship, a sleepless night, a job they are afraid to leave, a grief that will not lift, a worry they have not yet taken to a doctor. Most come to no harm. That is exactly the point. What happened to Pierre did not begin with something unusual. It began with something almost everyone now does.

There is a reason people turn to a conversational AI. Across much of the world, including in resource-rich countries, real psychiatric help is scarce. In poorer countries, the great majority of people in serious mental distress receive no proper care at all.³ An AI companion does not judge. It does not tire. It does not send a bill. For millions, it is the only confidant within reach. Into that emptiness walks a friendly voice that never sleeps.

The voice is not built to heal you. It is built to keep you. The business behind most modern technology rewards one thing above all others — attention. A companion that keeps you talking is a monetiseable success. That it may be loosening your grip on the world is, to the company, a side effect, not a fault to be fixed.⁴

The Sentence Nobody Wrote

Here is the strangest part. When Eliza told Pierre that death could be beautiful, no engineer had written that line. No one approved it. And afterwards, no one could fully explain why it appeared.

This is the famous ‘black box’. An AI teaches itself, from billions of conversations, how to operate. Its inner workings are opaque, not only to the people who use it, but often to the people who built it.⁵ You cannot simply open it up and point to the precise instance when the model began to drift toward an unexpected pattern.

For the law, this is a deep problem. Courts are built to ask who was at fault, and to trace a clean line from a careless act to an injury. Blame needs an author. Here, there is not a distinct one. When the act is intertwined inside a system its own makers cannot dissect, that line is unfathomable. Establishing a clear causal link and attributing fault is equally difficult in Brussels, London, and San Francisco alike.⁶ Some argue the answer is not to demand a confession from the AI companion, but to make the people who deploy it explain themselves instead.⁷

Everywhere and Nowhere

Old law has an old habit. It likes to know where things are.

When a faulty car is built in one city, sold in another, and crashes in a third the law can plant its feet on solid ground. Artificial intelligence pulls that ground away. A single system might be trained by a company on one continent, stored on servers on a second, tuned by a team on a third, and then opened, late at night, by a grieving man at his kitchen table. So where did the harm happen? Everywhere. Nowhere. Or, and this is the danger, wherever suits the company best.⁸

Legal tradition offers three places to stand: where the company lives, where the harm occurred, and where the pain was felt. The middle option breaks down in cases where AI is used across jurisdictions because it obscures place of injury and source of negligence. With a defective drug, the making and the swallowing are separate events. With a chatbot, the words are produced and received in the very same instant. Cause and effect fold into one. The true locus delicti is not where the engine interacted with the user, but somewhere along the supply chain where the model was built. Some would push the wrong back upstream, to the design choices made at headquarters, a tidy theory that happens to drag the case far from the injured user and makes proving negligence harder.⁹

The firmest ground is the third: where the harm manifested. Here, the European Union is relatively clear. A European who is hurt can, in principle, sue close to home, but only if the foreign company truly reached into that market, with local advertising, local prices, and a local launch. Merely being accessible within a market is not enough.¹⁰ As for which country’s law then governs the dispute, the EU’s rule points, broadly, to the place where the damage was felt: Belgian law for Pierre. Frame the harm instead as a faulty product, and a different chain of rules clicks into place.¹¹

A user in Istanbul has less recourse. Türkiye has no special AI law, and its ordinary rules echo Europe’s. However, a foreign company with no office, no staff, no assets in the country, and no treaty forcing its hand has little reason to appear before a Turkish court at all. Civil litigation, though, is not Türkiye’s only lever: under the Internet Law (No. 5651) the authorities can compel a large foreign platform to appoint an in-country legal representative, backed by advertising bans, bandwidth throttling, and administrative fines of up to three percent of global turnover. Yet this is an instrument of content regulation, not compensation: it can compel a platform to establish a local presence, but it neither creates a civil claim nor makes an injured user whole, and it reaches only ‘social network providers’ above the high user thresholds that a niche companion app may never cross.¹²

Three Legal Systems, Three Dead Ends

Let’s examine how three different legal systems have attempted to hold those responsible for harmful AI frameworks accountable.

The European Union has been the boldest. With the EU AI Act, it has written the most comprehensive AI law in the world, and tried to do so before the worst harms could occur. The law draws a bright line around systems that prey on the vulnerable, the young, the ill, and the desperate, and forbids those that exploit such weaknesses from doing real damage. It also says an AI system must, as a rule, clearly announce itself as such. The fines for breaking these rules are substantial. And yet companion chatbots were never placed on the law’s list of the most dangerous systems, so how firmly the act regulates them is still being debated. The ambition is real. The gaps are real too.¹³

For Claire, it all comes down to a simpler question: must she prove the unprovable, or must the company explain itself?

More telling is what Europe chose not to do. It had drafted a second law, the AI Liability Directive, which was quieter but more powerful. That law would have eased the victims’ daunting task: instead of forcing them to prove exactly how the ‘black box’ did its damage, it would have presumed the link and left the company to disprove it. For someone like Claire, it might have proved a less challenging route to filing a claim. In the summer of 2025, it was withdrawn. What remains is a narrower rule – the revised EU Product Liability Directive – which expressly defines software and AI systems as ‘products’ that can be deemed defective, just like a faulty physical good. This is a real step forward, yet a much smaller one.¹⁴ The boldest regulator in the world has, for now, walked away from a law built precisely to address AI-driven negligence.

America took the opposite path: it wrote almost nothing. There is no national AI-liability law. The entire debate plays out in the shadow of Section 230 of the Communications Decency Act. Written a quarter of a century ago, long before conversational AI was made available to the general public, it is a law designed to shield online platforms from liability for third-party content – leaving a massive legal gray area over who is to blame when the AI itself generates harmful output.¹⁵ This is why social networks cannot be sued for content posted by third-party users. Whether that old shield also covers what a conversational AI itself invents is now the billion-dollar question. In the most closely watched case – Garcia v. Character Technologies, Inc. – brought by the mother of a fourteen-year-old American boy who died after months of talking to a chatbot, the plaintiff creatively bypassed that shield by suing the company for defective product design.¹⁶ Moreover, the presiding judge made a sharp observation that threatens the industry’s entire defense: words strung together by an AI chatbot, with no human author behind them, may deserve no free-speech protection at all.¹⁷ If that idea holds, AI tech’s ability to rely on the old shield will be weakened. Section 230 was designed to protect a host who platforms other people’s speech. A system that writes the potentially harmful words is not hosting anyone.¹⁸

There is a deeper point here. When an AI system performs a role that resembles therapy while wearing the label of a ‘wellbeing app’, it slips past the rules that bind licensed therapists. While the function may seem similar, the legal and ethical duty of care is entirely different. Some argue this is by design: expertise sold cheaply, dressed as a product, to sidestep a century of professional regulation.¹⁹

Türkiye occupies a unique regulatory vacuum between the EU and the US. It has no dedicated AI statute, no limited liability shield for social platforms, and no chatbot case law. What it has is the Turkish Code of Obligations (TBK) – rooted in an old civil law tradition – and a handful of scholars trying to craft new remedies for novel digital harms from old precedents. Call it legal alchemy. The traditional path of fault-based tort asks the victim to establish the defendant’s negligence and runs straight back into the opacity of the ‘black box’.²⁰ A more viable path applies strict hazard liability under Article 71 of the TBK, which holds operators liable for inherently dangerous activities regardless of fault, like running an industrial chemical plant. The argument is bolder than it sounds: an AI companion built to simulate deep bonds with people in crisis is inherently dangerous.²¹ Alternatively, Türkiye’s Consumer Protection Law No. 6502 treats the harmful chat as a defective service.²² Further, the KVKK (Law on the Protection of Personal Data No. 6698) extends to any company, foreign or not, that handles a Turkish citizen’s data.²³ It mirrors the EU’s GDPR, but lacks its extraterritorial reach.

Who Is to Blame?

Even once you know which law applies, a final obstacle remains: pointing at the liable party.

A chatbot is not always made by one company in one factory; it is built in layers. One firm may build the raw engine, another may shape it for its purpose, and a third may put it in front of you. Each layer refines the large-language model with new data and features, and no single entity always owns it completely. When harm occurs, it emerges from these interlocking layers at once, and responsibility slips into the gaps between. Ask each contributor along the supply chain, and each will point, reasonably, at the others.²⁴

What Claire Is Up Against

Imagine Claire decides to fight. She will meet three walls.

The first is power. Her strongest weapon is not the law of injury at all – it is the law of data. Europe’s data rules reach across the ocean – any company serving Europeans must obey, wherever it sits, under penalty of fines measured against its worldwide income. A data regulator can investigate without a courtroom and without the company’s consent. That is real leverage.²⁵ Suing the company, by contrast, is far harder, unless it truly marketed itself directly within her country.

The second wall is enforcement. Say she wins. A judgment is only worth the assets on which it can actually collect. A European court order is not automatically enforceable in America; an American court must agree to honour it. The SPEECH Act was written precisely to refuse foreign judgments that offend American free speech principles. On paper she has won; in practice, she only holds a piece of paper. A well-funded company would raise every objection. The fight would be slow, costly, and uncertain.²⁶

The third wall is incentive. Perhaps the clearest lesson about extraterritorial AI liability enforcement comes not from a courtroom but from a regulator. In March 2023, Italy’s data protection authority, Garante, suspended ChatGPT (owned and operated by OpenAI) for Italian users, citing the lack of a lawful basis for processing data and inadequate age verification. OpenAI complied within weeks, not because the Garante demonstrated legal superiority, but because it controls access to the Italian market.²⁷ Extraterritorial enforcement can be effective when the regulator controls something the company needs: market access, market share, valuation, or reputation. Smaller firms, like Chai Research, without comparable resources or a European footprint, lack equivalent leverage, offering little reason to engage. AI developers, in effect, choose their regulator by choosing where to incorporate, host, and deploy. The user is rooted in one place; the developer chooses where they want to be.

Algorithms Travel, yet Accountability is Elusive

Consider a woman in Istanbul opening a wellness app on her phone. She has done nothing wrong. She simply reached for help, and help reached back from several countries at once. Where she sits, the safety promises lack strong regulatory teeth. If the app harms her, her odds of a remedy are close to zero. This is not because the harm is imaginary, but because the cross-border machinery to establish and enforce liability does not exist.

A decade ago, co-authors Mayer-Schönberger and Cukier warned that turning everyday human interactions into quantifiable digital data would breed harms our laws were never built to catch. AI’s reach into the fragile mind may be the sharpest proof of that yet.²⁸

None of this is inevitable. It persists because the law still assumes harm has an address.

Algorithms are borderless; accountability is not. That gap is not a glitch awaiting a clever fix. It is an asymmetry that must be addressed via legislative mandates, regulatory action, and effective cross-border enforcement. Until it is closed, the cost will keep mounting, quietly, among the people least able to bear it: those who cannot readily access professional help, who lack the resources to pursue complex cross-border claims, and whose injuries risk falling into the gaps between legal systems.

Permission required for reprinting, reproducing, or other uses.

Disclaimer: This analytical insight is provided for general informational purposes only and does not constitute legal advice.

Zeynep TUNCER is an attorney and mediator based in Istanbul with experience in international law, cross-border transactions, and mergers & acquisitions. She is fluent in Turkish, French, and English, operating seamlessly across multiple jurisdictions and business cultures.

References