For AI developers, the privacy risks of generative AI may extend well beyond the intended uses of their products. A recent finding by the Office of the Privacy Commissioner of Canada (“OPC”) against X Corp. and xAI underscores that organizations may remain accountable when their AI systems enable the processing of personal information for harmful or foreseeable misuse.

In its June 2026 findings, the OPC concluded that X Corp. and xAI contravened the Personal Information Protection and Electronic Documents Act (“PIPEDA”) by enabling the generation of sexualized deepfakes of identifiable individuals through the Grok AI platform, an AI chatbot integrated into the X social media platform.

While the investigation focused on sexually explicit deepfakes, the decision has broader implications for organizations developing, deploying, or integrating generative AI technologies that process personal information. Significantly, it signals that privacy considerations may need to be addressed not only in how AI systems are intended to operate, but also in how they could foreseeably be misused.

The investigation

The OPC launched a commissioner-initiated investigation of X Corp. and xAI in January 2026 following widespread reports that Grok had generated large volumes of non-consensual sexualized deepfakes, including content depicting minors.

The investigation examined two central questions:

  1. Whether X Corp. and xAI obtained valid consent for the collection, use, and disclosure of personal information used to create sexualized deepfakes; and
  2. Whether a reasonable person would consider the collection, use, and disclosure of personal information for an image-generation service capable of producing such deepfakes to be appropriate in the circumstances.

The OPC answered both questions in the negative.

Deepfakes as personal information

While the OPC's conclusion that deepfakes depicting identifiable individuals constitute personal information is consistent with existing privacy law principles, the decision provides important insight into how accountability obligations apply in the context of generative AI. The OPC rejected the respondents' argument that users were solely responsible for generating the content, finding instead that organizations that develop, deploy and commercialize AI systems remain accountable for how those systems process personal information. The decision therefore reinforces the importance of implementing appropriate safeguards and governance measures to address foreseeable misuse of AI technologies.

Meaningful consent remains central

The OPC determined that organizations would require express consent from the depicted individual before collecting, using, or disclosing personal information to generate sexualized deepfakes. According to the OPC, such deepfakes involve highly sensitive personal information—particularly where children are depicted—and fall well outside the reasonable expectations of individuals whose images may be used. The significant risks of reputational, psychological, financial, and other harms reinforce the need for express consent in these circumstances.

Accountability cannot be delegated to users

A key takeaway from the decision is the OPC's rejection of the argument that responsibility rests solely with users. The respondents argued that users initiated the prompts and therefore bore primary responsibility for generating the objectionable content. The OPC disagreed, finding that organizations that design, deploy and commercialize AI systems remain accountable for how those systems collect, use and disclose personal information.

According to the Commissioner, the fact that a user initiates a prompt does not relieve an organization of its obligations under PIPEDA. The decision reinforces that AI developers and platform operators cannot avoid privacy accountability by characterizing themselves as neutral intermediaries where their technologies actively enable the processing of personal information and foreseeable privacy harms.

Privacy by design expectations are increasing

Beyond the consent issues, the OPC found that the respondents' practices failed PIPEDA's "appropriate purposes" test. Regardless of whether consent has been obtained, under section 5(3), organizations may collect, use or disclose personal information only for purposes that a reasonable person would consider appropriate in the circumstances.

The Commissioner accepted that AI image-generation tools can serve legitimate business purposes. However, the OPC found that less privacy-invasive alternatives were available.

Specifically, the decision criticizes the respondents for deploying image-generation and editing capabilities without adequately anticipating foreseeable misuse scenarios.

The OPC concluded that stronger safeguards implemented before deployment could likely have reduced or prevented the generation of harmful deepfakes while preserving the utility of the technology.

This aspect of the decision reinforces a growing regulatory expectation that organizations incorporate privacy, safety, and misuse prevention measures during product design rather than after harm occurs.

Privacy impact assessments under scrutiny

The decision also highlights the importance of meaningful privacy impact assessments ("PIAs") in the development and deployment of AI systems. The OPC criticized both organizations' assessments, finding that key risks associated with the technology were either overlooked or underestimated.

In particular, the Commissioner noted that the respondents failed to adequately anticipate how users might misuse the technology to generate sexualized deepfakes at scale and the heightened risks associated with publicly sharing AI-generated content through the @Grok functionality on the X platform.

While PIPEDA does not expressly require PIAs, the decision suggests that regulators will expect organizations to identify foreseeable privacy risks, assess potential harms to both users and non-users, and revisit those assessments as products evolve. Organizations relying on generic or static AI risk assessments may find those assessments insufficient when subjected to regulatory scrutiny.

What this means for AI developers

The decision offers several practical lessons for organizations developing or deploying AI tools:

  • Organizations remain accountable for AI-enabled processing.

    The OPC rejected the argument that responsibility rests solely with users who provide prompts. Organizations that develop, deploy and commercialize AI systems remain responsible for how those systems collect, use and disclose personal information.

  • Foreseeable misuse must be addressed proactively.

    The OPC criticized the respondents for failing to adequately anticipate and mitigate the risk that their technology could be used to generate harmful sexualized deepfakes. Organizations should assess not only intended uses of AI systems, but also reasonably foreseeable misuse.

  • Risk assessments and safeguards should evolve with the technology.

    Privacy impact assessments and governance processes should be updated as AI systems acquire new functionalities or are deployed in new contexts. Organizations should also be prepared to demonstrate that their safeguards are effective in practice, rather than merely existing on paper.

  • Traditional privacy principles continue to apply to AI.

    The decision demonstrates that established concepts such as consent, accountability and appropriate purposes remain central to the assessment of AI systems under Canadian privacy law.

Looking ahead

Although the decision focuses on sexualized deepfakes, its implications extend beyond that context. The findings illustrate how the OPC is applying established privacy principles, including consent, accountability and appropriate purposes to emerging AI technologies.

The decision also comes at a time when Canadian lawmakers are increasingly addressing the harms associated with non-consensual intimate imagery, including AI-generated content. Across Canada, several provinces have enacted legislation providing civil remedies for victims, reflecting a broader trend toward stronger protections against technology-enabled image abuse.

For organizations deploying AI systems, the decision underscores the importance of identifying foreseeable privacy risks, implementing effective safeguards and regularly reassessing those measures as technologies evolve. Organizations developing or using generative AI tools should consider whether their governance frameworks, risk assessments and technical controls are sufficient to address the privacy risks associated with their particular use cases.

Stay informed

For tailored guidance on how the report’s findings might impact your organization’s privacy practices, please contact a member of our Cyber Security and Data Protection and Artificial Intelligence groups.