Master Your AI Act Role and Avoid Risk
You’ve integrated a powerful AI tool to screen candidates for your HR department. It promises to save hundreds of hours and identify the best talent. But when a candidate claims they were unfairly rejected due to algorithmic bias, who is held accountable? Is it the company that developed the AI, or is it you, the company that used it?
This isn’t a hypothetical scenario; it’s the central compliance question that thousands of businesses are facing. The intertwined regulations of the GDPR and the AI Act create a complex web of responsibilities. Simply signing a contract with an AI vendor doesn’t absolve you of your obligations. To truly master your AI Act role and avoid risk, you must grasp these nuances.
In our Ultimate Guide to GDPR and AI Act Risks, we laid out the roadmap. Now, we zoom in on the most crucial crossroads: the division of labor between the AI provider and the AI user (or “deployer”). This guide will dissect each role, clarify their specific duties, and provide a practical framework to manage this shared responsibility effectively.
Identify Your Role: The First Step to Compliance
Before you can manage your obligations, you must correctly identify your position on the regulatory map. The AI Act and the GDPR use different terminologies, but they often describe overlapping functions. Getting this right is the foundational step if you want to master your AI Act role and avoid risk.
Under the AI Act, the key players are:
Provider: The entity that develops an AI system with the intention of placing it on the market or putting it into service under its own name or trademark. Think of the companies behind tools like ChatGPT, Midjourney, or specialized B2B AI software.
Deployer (User): Any natural or legal person, public authority, agency, or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity. If you use a third-party AI tool for your business operations, you are a deployer.
Under the GDPR, the roles are defined by data processing activities:
Controller: The entity that determines the “purposes and means” of the processing of personal data. In essence, they decide why and how personal data is processed.
Processor: The entity that processes personal data on behalf of the controller, following their instructions.
The Crucial Overlap:
Here’s where it connects. When you, as a company, use an AI tool to process customer or employee data, you are typically the Deployer (under the AI Act) and the Data Controller (under the GDPR). You are making the decision to use the AI for a specific business purpose (e.g., to analyze CVs).
The AI vendor is the Provider (under the AI Act) and usually the Data Processor (under the GDPR), as they are providing the technology that processes data based on your instructions. However, be aware: in some complex scenarios, a provider could also be a joint controller or even a separate controller, making contractual clarity paramount.
Provider Obligations: Building a Safe and Compliant AI
To avoid risk under the new regulation, the AI Act places the primary burden of ensuring an AI system is safe, transparent, and robust on its provider, especially for high-risk systems. Their obligations are extensive and continuous.
1. Technical Documentation & Risk Management:
Before a high-risk AI system can even touch the market, the provider must create and maintain extensive technical documentation. This is the system’s “passport,” detailing everything from its intended purpose and architecture to the datasets used for training, validation, and testing. It must include a robust risk management system that identifies, analyzes, and mitigates potential risks throughout the AI’s lifecycle.
2. Conformity Assessment & CE Marking:
For high-risk systems, the provider must complete a conformity assessment and issue an EU Declaration of Conformity before CE marking. The deployer must verify the CE marking and declaration before putting the system into service. Reassess after any substantial modification.
3. Post-Market Monitoring (PMM):
Providers must operate a proactive PMM system and report serious incidents. Deployers must retain logs, monitor performance in their context of use, and promptly inform the provider of anomalies or incidents.
Deployer/User Obligations: Using AI Responsibly
As a deployer, you are on the front lines. To master your AI Act role and avoid risk, your main responsibility is to use the AI system in a way that respects both the law and the rights of individuals. The AI Act and GDPR assign you several key duties.
1. Use in Accordance with Instructions:
You must use the high-risk AI system according to the provider’s technical documentation and instructions for use. Using a tool for a purpose it wasn’t designed or tested for can void warranties, break contracts, and expose you to significant liability.
2. Effective Human Oversight:
This is a cornerstone of the AI Act. For high-risk systems, deployers must ensure appropriate human oversight is in place. This is not a passive role. The designated individuals must have the necessary competence, training, and authority to understand the system’s outputs, question them, and ultimately decide whether to use, disregard, or override them. The goal is to prevent or minimize risks that the AI system might produce.
3. Input Data & Record-Keeping:
You are responsible for the quality and relevance of the data you feed into the AI system. The principle of “garbage in, garbage out” has significant legal implications here. Furthermore, for high-risk systems, you are required to maintain automatically generated logs of the system’s operation to ensure a level of traceability. These logs are crucial for audits and incident investigations.
4. Transparency and Information Duties:
As controller, provide GDPR notices (Arts. 13/14) and meaningful information about automated decision-making where applicable. The AI Act also requires informing people when they interact with AI systems such as chatbots and labeling AI-generated content that could be mistaken for real, notably deepfakes.
The Cross-Responsibility Matrix: A Practical Guide
The relationship between provider and deployer is a partnership built on shared responsibility. Defining these duties clearly is essential to master your AI Act role and avoid risk. This matrix clarifies who typically does what, but remember: these duties must be formally defined in your contracts.
The Cross-Responsibility Matrix: A Practical Guide
The relationship between provider and deployer is a partnership built on shared responsibility. Defining these duties clearly is essential to master your AI Act role and avoid risk. Below is a bullet-point version of the matrix, optimized for responsive layouts:
- Lawfulness of Training Data
Provider: Ensures the original training dataset was compiled legally (respecting copyright, privacy, etc.).
Deployer: N/A (for the core model).
Evidence: Provider’s technical documentation, data summaries, contractual warranties. - Technical Documentation
Provider: Creates, maintains, and provides access to the system's technical documentation.
Deployer: Reviews documentation to ensure proper use and understand limitations.
Evidence: The technical documentation itself, access rights specified in the contract. - Conformity Assessment (CE)
Provider: Conducts the assessment and affixes the CE marking to high-risk AI.
Deployer: Verifies the AI system has a valid CE marking before putting it into service.
Evidence: Declaration of Conformity, CE marking, contract clauses. - Human Oversight
Provider: Designs the system to enable human oversight (e.g., with clear explainability features).
Deployer: Implements the oversight processes, trains staff, and makes final decisions.
Evidence: Internal policies, training records, decision logs. - Data Protection Impact Assessment (DPIA)
Provider: Provides necessary information about the system to help the deployer conduct their DPIA.
Deployer: Conducts and documents the DPIA for the specific use case, as they are the Data Controller.
Evidence: Completed DPIA report, provider's security/technical information. - Incident Reporting
Provider: Reports serious incidents to market surveillance authorities. Informs deployers.
Deployer: Reports operational anomalies or suspected incidents to the provider.
Evidence: Post-Market Monitoring plan, incident logs, communication protocols. - Transparency to End-Users
Provider: Provides clear information on the system’s capabilities and limitations.
Deployer: Informs individuals they are interacting with AI and provides GDPR-required notices.
Evidence: Provider's instructions for use, deployer's privacy policy, on-screen notices.
Risk Indicators and Escalation: Your Early Warning System
Proactive governance is the best strategy to avoid risk. You need an internal early warning system to flag potential issues with your AI tools before they become major compliance breaches.
Trigger an immediate legal/technical review if you observe:
Technical Red Flags:
Performance Drift: The AI’s accuracy or performance consistently degrades over time.
Anomalous Outputs: The system generates bizarre, biased, or completely nonsensical results that deviate from its expected behavior.
High Bias Metrics: Regular testing reveals the system is systematically favoring or disadvantaging a particular demographic group.
Legal & Contractual Red Flags:
Provider Reluctance: The provider is evasive or unwilling to share necessary parts of the technical documentation or conformity assessment evidence.
Unilateral Contract Changes: The provider alters terms of service, particularly those related to data processing, liability, or sub-processors, without proper notification or your consent.
Geographic Data Transfer Issues: You discover data is being processed in a jurisdiction not covered by your TIA or the contract, without a valid legal basis.
Operational Red Flags:
Spike in Complaints: A noticeable increase in customer or employee complaints specifically mentioning “the algorithm,” “the system,” or “unfair automated decisions.”
Complex DSARs: You receive Data Subject Access Requests (DSARs) demanding an explanation of a specific automated decision.
Security Incident: The AI system is involved in any kind of data breach or security incident, either as the target or the cause.
These are not just operational hiccups; they are critical signals that your compliance framework is under stress. Each one should trigger a pre-defined escalation path, involving your Data Protection Officer (DPO), legal counsel, and technical teams to assess the risk and define a mitigation plan.
The era of “plug and play” AI is over. The AI Act and GDPR demand a sophisticated, collaborative approach to compliance. Responsibility is not a hot potato to be passed along; it’s a chain that links the provider and the user. The only way to truly master your AI Act role and avoid risk is by forging strong links through clear contracts, diligent oversight, and proactive monitoring. This transforms regulatory complexity into a foundation of trust and a true competitive advantage.
In your organization, how do you currently verify the compliance of your AI providers?
Master Your AI Act Role and Avoid Risk
At GDPR AI Consulting we support lawyers, companies, and data protection consultants in achieving GDPR compliance in a practical, secure, and always up-to-date way. Our AI assistant, trained with the latest European regulations, is available 24/7 to answer complex queries, draft policies and clauses, analyze internal documents, identify compliance risks, and translate legal texts into multiple languages in seconds.
Designed to complement and streamline the work of legal and compliance teams, it brings confidence, accuracy, and efficiency to every step of the process.
👉 See how we can help: View GPT plans
#GDPRAiConsulting #AIAct #AIGovernance #AICompliance #TechLaw #RiskManagement #DataPrivacy #RiskAvoidance #ProviderVsUser #Deployer
