Understanding the Impact of Data Privacy Laws

This new white paper by Stanford Institute for Human-Centered Artificial Intelligence (HAI) titled "Rethinking Privacy in the AI Era" addresses the intersection of data privacy and AI development, highlighting the challenges and proposing solutions for mitigating privacy risks. It outlines the current data protection landscape, including the Fair Information Practice Principles, GDPR, and U.S. state privacy laws, and discusses the distinction and regulatory implications between predictive and generative AI. The paper argues that AI's reliance on extensive data collection presents unique privacy risks at both individual and societal levels, noting that existing laws are inadequate for the emerging challenges posed by AI systems, because they don't fully tackle the shortcomings of the Fair Information Practice Principles (FIPs) framework or concentrate adequately on the comprehensive data governance measures necessary for regulating data used in AI development. According to the paper, FIPs are outdated and not well-suited for modern data and AI complexities, because: - They do not address the power imbalance between data collectors and individuals. - FIPs fail to enforce data minimization and purpose limitation effectively. - The framework places too much responsibility on individuals for privacy management. - Allows for data collection by default, putting the onus on individuals to opt out. - Focuses on procedural rather than substantive protections. - Struggles with the concepts of consent and legitimate interest, complicating privacy management. It emphasizes the need for new regulatory approaches that go beyond current privacy legislation to effectively manage the risks associated with AI-driven data acquisition and processing. The paper suggests three key strategies to mitigate the privacy harms of AI: 1.) Denormalize Data Collection by Default: Shift from opt-out to opt-in data collection models to facilitate true data minimization. This approach emphasizes "privacy by default" and the need for technical standards and infrastructure that enable meaningful consent mechanisms. 2.) Focus on the AI Data Supply Chain: Enhance privacy and data protection by ensuring dataset transparency and accountability throughout the entire lifecycle of data. This includes a call for regulatory frameworks that address data privacy comprehensively across the data supply chain. 3.) Flip the Script on Personal Data Management: Encourage the development of new governance mechanisms and technical infrastructures, such as data intermediaries and data permissioning systems, to automate and support the exercise of individual data rights and preferences. This strategy aims to empower individuals by facilitating easier management and control of their personal data in the context of AI. by Dr. Jennifer King Caroline Meinhardt Link: https://lnkd.in/dniktn3V

The rapid advancement of AI technologies, particularly LLMs, has highlighted important questions about the application of privacy laws like the GDPR. As someone who has been grappling with this issue for years, I am *thrilled* to see the Hamburg DPC's discussion paper approach privacy risks and AI with a deep understanding of the technology. A few absolutely refreshing takeaways: ➡ LLMs process tokens and vectorial relationships between tokens (embeddings), fundamentally differing from conventional data storage and retrieval. The Hamburg DPC finds that LLMs don't "process" or "store" personal data within the meaning of the GDPR. ➡ Unlike traditional identifiers, tokens and their embeddings in LLMs lack the necessary direct, targeted association to individuals that characterizes personal data in CJEU jurisprudence. ➡ Memorization attacks that extract training data from an LLM don't necessarily conclude that personal data is stored in the LLM. These attacks may be practically disproportionate and potentially legally prohibited, making personal identification not "possible" under the legislation. ➡ Even if personal data was unlawfully processed in developing the LLM, it doesn't render the use of the resulting LLM illegal (providing downstream deployers some comfort when leveraging third-party models). This is a nuanced and technology-informed perspective on the complex intersection of AI and privacy. As we continue to navigate this rapidly evolving landscape, I hope we see more regulators and courts approach regulation and legal compliance with a deep understanding of how the technology actually works. #AI #Privacy #GDPR #LLM

⚠️Privacy Risks in AI Management: Lessons from Italy’s DeepSeek Ban⚠️ Italy’s recent ban on #DeepSeek over privacy concerns underscores the need for organizations to integrate stronger data protection measures into their AI Management System (#AIMS), AI Impact Assessment (#AIIA), and AI Risk Assessment (#AIRA). Ensuring compliance with #ISO42001, #ISO42005 (DIS), #ISO23894, and #ISO27701 (DIS) guidelines is now more material than ever. 1. Strengthening AI Management Systems (AIMS) with Privacy Controls 🔑Key Considerations: 🔸ISO 42001 Clause 6.1.2 (AI Risk Assessment): Organizations must integrate privacy risk evaluations into their AI management framework. 🔸ISO 42001 Clause 6.1.4 (AI System Impact Assessment): Requires assessing AI system risks, including personal data exposure and third-party data handling. 🔸ISO 27701 Clause 5.2 (Privacy Policy): Calls for explicit privacy commitments in AI policies to ensure alignment with global data protection laws. 🪛Implementation Example: Establish an AI Data Protection Policy that incorporates ISO27701 guidelines and explicitly defines how AI models handle user data. 2. Enhancing AI Impact Assessments (AIIA) to Address Privacy Risks 🔑Key Considerations: 🔸ISO 42005 Clause 4.7 (Sensitive Use & Impact Thresholds): Mandates defining thresholds for AI systems handling personal data. 🔸ISO 42005 Clause 5.8 (Potential AI System Harms & Benefits): Identifies risks of data misuse, profiling, and unauthorized access. 🔸ISO 27701 Clause A.1.2.6 (Privacy Impact Assessment): Requires documenting how AI systems process personally identifiable information (#PII). 🪛 Implementation Example: Conduct a Privacy Impact Assessment (#PIA) during AI system design to evaluate data collection, retention policies, and user consent mechanisms. 3. Integrating AI Risk Assessments (AIRA) to Mitigate Regulatory Exposure 🔑Key Considerations: 🔸ISO 23894 Clause 6.4.2 (Risk Identification): Calls for AI models to identify and mitigate privacy risks tied to automated decision-making. 🔸ISO 23894 Clause 6.4.4 (Risk Evaluation): Evaluates the consequences of noncompliance with regulations like #GDPR. 🔸ISO 27701 Clause A.1.3.7 (Access, Correction, & Erasure): Ensures AI systems respect user rights to modify or delete their data. 🪛 Implementation Example: Establish compliance audits that review AI data handling practices against evolving regulatory standards. ➡️ Final Thoughts: Governance Can’t Wait The DeepSeek ban is a clear warning that privacy safeguards in AIMS, AIIA, and AIRA aren’t optional. They’re essential for regulatory compliance, stakeholder trust, and business resilience. 🔑 Key actions: ◻️Adopt AI privacy and governance frameworks (ISO42001 & 27701). ◻️Conduct AI impact assessments to preempt regulatory concerns (ISO 42005). ◻️Align risk assessments with global privacy laws (ISO23894 & 27701).   Privacy-first AI shouldn't be seen just as a cost of doing business, it’s actually your new competitive advantage.

Insightful Sunday read regarding AI governance and risk. This framework brings some much-needed structure to AI governance in national security, especially in sensitive areas like privacy, rights, and high-stakes decision-making. The sections on restricted uses of AI make it clear that AI should not replace human judgment, particularly in scenarios impacting civil liberties or public trust. This is particularly relevant for national security contexts where public trust is essential, yet easily eroded by perceived overreach or misuse. The emphasis on impact assessments and human oversight is both pragmatic and proactive. AI is powerful, but without proper guardrails, it’s easy for its application to stray into gray areas, particularly in national security. The framework’s call for thorough risk assessments, documented benefits, and mitigated risks is forward-thinking, aiming to balance AI’s utility with caution. Another strong point is the training requirement. AI can be a black box for many users, so the framework rightly mandates that users understand both the tools’ potential and limitations. This also aligns well with the rising concerns around “automation bias,” where users might overtrust AI simply because it’s “smart.” The creation of an oversight structure through CAIOs and Governance Boards shows a commitment to transparency and accountability. It might even serve as a model for non-security government agencies as they adopt AI, reinforcing responsible and ethical AI usage across the board. Key Points: AI Use Restrictions: Strict limits on certain AI applications, particularly those that could infringe on civil rights, civil liberties, or privacy. Specific prohibitions include tracking individuals based on protected rights, inferring sensitive personal attributes (e.g., religion, gender identity) from biometrics, and making high-stakes decisions like immigration status solely based on AI. High-Impact AI and Risk Management: AI that influences major decisions, particularly in national security and defense, must undergo rigorous testing, oversight, and impact assessment. Cataloguing and Monitoring: A yearly inventory of high-impact AI applications, including data on their purpose, benefits, and risks, is required. This step is about creating a transparent and accountable record of AI use, aimed at keeping all deployed systems in check and manageable. Training and Accountability: Agencies are tasked with ensuring personnel are trained to understand the AI tools they use, especially those in roles with significant decision-making power. Training focuses on preventing overreliance on AI, addressing biases, and understanding AI’s limitations. Oversight Structure: A Chief AI Officer (CAIO) is essential within each agency to oversee AI governance and promote responsible AI use. An AI Governance Board is also mandated to oversee all high-impact AI activities within each agency, keeping them aligned with the framework’s principles.

Before diving headfirst into AI, companies need to define what data privacy means to them in order to use GenAI safely. After decades of harvesting and storing data, many tech companies have created vast troves of the stuff - and not all of it is safe to use when training new GenAI models. Most companies can easily recognize obvious examples of Personally Identifying Information (PII) like Social Security numbers (SSNs) - but what about home addresses, phone numbers, or even information like how many kids a customer has? These details can be just as critical to ensure newly built GenAI products don’t compromise their users' privacy - or safety - but once this information has entered an LLM, it can be really difficult to excise it. To safely build the next generation of AI, companies need to consider some key issues: ⚠️Defining Sensitive Data: Companies need to decide what they consider sensitive beyond the obvious. Personally identifiable information (PII) covers more than just SSNs and contact information - it can include any data that paints a detailed picture of an individual and needs to be redacted to protect customers. 🔒Using Tools to Ensure Privacy: Ensuring privacy in AI requires a range of tools that can help tech companies process, redact, and safeguard sensitive information. Without these tools in place, they risk exposing critical data in their AI models. 🏗️ Building a Framework for Privacy: Redacting sensitive data isn’t just a one-time process; it needs to be a cornerstone of any company’s data management strategy as they continue to scale AI efforts. Since PII is so difficult to remove from an LLM once added, GenAI companies need to devote resources to making sure it doesn’t enter their databases in the first place. Ultimately, AI is only as safe as the data you feed into it. Companies need a clear, actionable plan to protect their customers - and the time to implement it is now.

Yesterday, Colorado’s Consumer Protections for #ArtificialIntelligence (SB24-205) was sent to the Governor for signature. If enacted, the law will be effective on Feb. 1, 2026, and Colorado would become the first U.S. state to pass broad restrictions on private companies using #AI. The bill requires both developer and deployer of a high-risk #AI system to use reasonable care to avoid algorithmic discrimination. A High-Risk AI System is defined as “any AI system that when deployed, makes, or is a substantial factor in making, a consequential decision.” Some computer software is exempted, such as AI-enabled video games, #cybersecurity software, and #chatbots that have a user policy prohibiting discrimination. There is a rebuttable presumption that a developer and a deployer used reasonable care if they each comply with certain requirements related to the high-risk system, including Developer: - Disclose and provide documentation to deployers regarding the high-risk system’s intended use, known or foreseeable #risks, a summary of data used to train it, possible biases, risk mitigation measures, and other information necessary for the deployer to complete an #impactassessment. - Make a publicly available statement summarizing the types of high-risk systems developed and available to a deployer. - Disclose, within 90 days, to the attorney general and known deployers when algorithmic discrimination is discovered, either through self-testing or deployer notice. Deployer: - Implement a #riskmanagement policy that governs high-risk AI use and specifies processes and personnel used to identify and mitigate algorithmic discrimination. - Complete an impact assessment to mitigate potential abuses before customers use their products. - Notify a consumer of specified items if the high-risk #AIsystem makes a consequential decision concerning a consumer. - If the deployer is a controller under the Colorado Privacy Act (#CPA), it must inform the consumer of the right to #optout of profiling in furtherance of solely #automateddecisions. - Provide a consumer with an opportunity to correct incorrect personal data that the system processed in making a consequential decision. - Provide a consumer with an opportunity to appeal, via human review if technically feasible, an adverse consequential decision concerning the consumer arising from the deployment of the system. - Ensure that users can detect any generated synthetic content and disclose to consumers that they are engaging with an AI system. The law contains a #safeharbor providing an affirmative defense (under CO law in a CO court) to a developer or deployer that: 1) discovers and cures a violation through internal testing or red-teaming, and 2) otherwise complies with the National Institute of Standards and Technology (NIST) AI Risk Management Framework or another nationally or internationally recognized risk management #framework.

🧠 “Data systems are designed to remember data, not to forget data.” – Debbie Reynolds, The Data Diva 🚨 I just published a new essay in the Data Privacy Advantage newsletter called: 🧬An AI Data Privacy Cautionary Tale: Court-Ordered Data Retention Meets Privacy🧬 🧠 This essay explores the recent court order from the United States District Court for the Southern District of New York in the New York Times v. OpenAI case. The court ordered OpenAI to preserve all user interactions, including chat logs, prompts, API traffic, and generated outputs, with no deletion allowed, not even at the user's request. 💥 That means: 💥“Delete” no longer means delete 💥API business users are not exempt 💥Personal, confidential, or proprietary data entered into ChatGPT could now be locked in indefinitely 💥Even if you never knew your data would be involved in litigation, it may now be preserved beyond your control 🏛️ This order overrides global privacy laws, such as the GDPR and CCPA, highlighting how litigation can erode deletion rights and intensify the risks associated with using generative AI tools. 🔍 In the essay, I cover: ✅ What the court order says and why it matters ✅ Why enterprise API users are directly affected ✅ How AI models retain data behind the scenes ✅ The conflict between privacy laws and legal hold obligations ✅ What businesses should do now to avoid exposure 💡 My recommendations include: • Train employees on what not to submit to AI • Curate all data inputs with legal oversight • Review vendor contracts for retention language • Establish internal policies for AI usage and audits • Require transparency from AI providers 🏢 If your organization is using generative AI, even in limited ways, now is the time to assess your data discipline. AI inputs are no longer just temporary interactions; they are potentially discoverable records. And now, courts are treating them that way. 📖 Read the full essay to understand why AI data privacy cannot be an afterthought. #Privacy #Cybersecurity #datadiva#DataPrivacy #AI #LegalRisk #LitigationHold #PrivacyByDesign #TheDataDiva #OpenAI #ChatGPT #Governance #Compliance #NYTvOpenAI #GenerativeAI #DataGovernance #PrivacyMatters

President Biden’s recent Executive Order on AI leaves one key issue open that remains top of mind for most organizations today – data privacy. The order calls Congress to pass “bipartisan data privacy legislation” to protect Americans’ data. As we embrace the power of AI, we must also recognize the morphing challenges of data privacy in the context of data sovereignty. The rules are constantly changing, and organizations need flexibility to maintain compliance just in their home countries but also in every country in which they operate. Governments worldwide, from the European Union with its GDPR to India's Personal Data Protection Bill, are setting stringent regulations to protect their citizens' data. The essence? Data about a nation's citizens or businesses should only reside on systems within their legal and regulatory purview. We all know AI is a game-changer but also a voracious consumer of data and a complicating factor for data sovereignty. Especially with Generative AI, which consumes data indiscriminately, often stored and processed at the AI companies' discretion. This collision between AI's insatiable appetite for data, the temptation for organizations to use it, and global data sovereignty regulations present a unique challenge for businesses. With the right approach, businesses can harness the power of AI while respecting data sovereignty. Here are a few ideas on how: Mindset: Make data sovereignty a company-wide priority. It's not just an IT or legal concern; it's a business imperative. Every team member should understand the risks associated with non-compliance. Inventory: Know your data. With large enterprises storing data in over 800 applications on average, it's crucial to maintain an inventory of your company's data and be aware of the vendors interacting with it. Governance: Stay updated with regional data laws and ensure compliance. Data sovereignty requires governance to be local also. Vendor Compliance: Your external vendors should be in lockstep with your data policies. Leverage Data Unification Solutions: Use flexible, scalable tools to ensure data sovereignty compliance. Data unification and management tools powered by AI can detect data leakages, trace data lineage, and ensure data remains within stipulated borders. I’ve witnessed how this can be accomplished in many industries, including healthcare. Despite stringent privacy and sovereignty policies, many healthcare management systems demonstrate that robust data management, compliant with regulations, is achievable. The key is designing systems with data management policies from the outset. To all global organizations: Embrace the future, but let's do it responsibly. Data privacy and sovereignty are not a hurdle; it's a responsibility we must uphold for the trust of our customers and the integrity of our businesses. Planning for inevitable changes now will pay dividends in the future. #data

During seed round due diligence, we found a red flag: the startup didn’t have rights to the dataset used to train its LLM and hadn’t set up a privacy policy for data collection or use. AI startups need to establish certain legal and operational frameworks to ensure they have and maintain the rights to the data they collect and use, especially for training their AI models. Here are the key elements for compliance: 1. Privacy Policy: A comprehensive privacy policy that clearly outlines data collection, usage, retention, and sharing practices. 2. Terms of Service/User Agreement: Agreements that users accept which should include clauses about data ownership, licensing, and how the data will be used. 3. Data Collection Consents: Explicit consents from users for the collection and use of their data, often obtained through clear opt-in mechanisms. 4. Data Processing Agreements (DPAs): If using third-party services or processors, DPAs are necessary to define the responsibilities and scope of data usage. 5. Intellectual Property Rights: Ensure that the startup has clear intellectual property rights over the collected data, through licenses, user agreements, or other legal means. 6. Compliance with Regulations: Adherence to relevant data protection regulations such as GDPR, CCPA, or HIPAA, which may dictate specific requirements for data rights and user privacy. 7. Data Anonymization and Security: Implementing data anonymization where necessary and ensuring robust security measures to protect data integrity and confidentiality. 8. Record Keeping: Maintain detailed records of data consents, privacy notices, and data usage to demonstrate compliance with laws and regulations. 9. Data Audits: Regular audits to ensure that data collection and usage align with stated policies and legal obligations. 10. Employee Training and Policies: Training for employees on data protection best practices and establishing internal policies for handling data. By having these elements in place, AI startups can help ensure they have the legal rights to use the data for training their AI models and can mitigate risks associated with data privacy and ownership. #startupfounder #aistartup #dataownership

Connecticut amended its comprehensive privacy law. Here are the changes that may require new practices.   The State of Connecticut - Office of the Governor signed the amendment to the #DataPrivacy law last week, and it takes effect July 1, 2026. Changes include: 🔸Adding sensitive data types: mental or physical health disability and treatment info; financial, credit, or debit account or card numbers with required codes or passwords; government identification numbers; and info about a minor (<18). 🔸Expanding individual rights to permit access to inferences derived from personal data, and to confirm whether personal data is processed for profiling to make decisions that produce legal or similarly significant effects. If it is, there are new rights regarding the profiling, including to question results, understand reasoning, review personal data used, and correct data used and have the decision reevaluated. 🔸Detailed new documented assessment requirements for profiling for purposes of making decisions that produce legal or similarly significant effects. 🔸Requiring privacy policies to indicate whether personal data is used for training large language models (for #ArtificialIntelligence). 🔸Requiring companies to provide, upon consumer request, a list of the third parties personal data is sold to. 🔸Requiring consent sell sensitive data. 🔸Prohibiting personal data of minors from being sold or used for targeted #OnlineAdvertising. Consent is no longer a basis for such sales or usage.   The law will also apply more broadly to companies that: process any personal data of more than 35,000 consumers; process any sensitive personal data (other than for payment transactions); or sell personal data.   As next steps before the law takes effect next July: ✔️ Confirm all the data types indicated are treated as sensitive in in your company's policies and procedures, including those that trigger data protection assessments. With California still considering regulations, this will be the first state to require a documented data protection assessment when government identification numbers are processed. ✔️ If your company makes in-scope profiling decisions, update assessment processes (such as those currently used for Colorado) to include required elements, and revise individual rights processes to address the new requirements. ✔️ Update individual rights processes to allow consumers to receive lists of all third parties that personal data is sold to (leverage Oregon processes as appropriate). ✔️ Address the new #privacy policy content requirements in your next privacy policy review and update. ✔️ Confirm sensitive data types will not be sold without consent, such as in targeted advertising efforts. ✔️ If your company deals with personal data of minors, plan to omit such data from targeted advertising efforts, or from practices that constitute "sales".