Wrong decisions are the biggest AI security risk. A guest post by Assaf Keren, Chief Security Officer at Qualtrics.
Companies are deploying artificial intelligence for operational decisions—from customer communication to talent management. But the more autonomously AI systems act, the more serious the consequences of faulty or manipulated input data become. Security expert Assaf Keren, CSO at Qualtrics, analyzes why traditional IT security concepts are reaching their limits and what organizations need to do now.
Context Loss as a System Failure
Artificial intelligence is no longer an experimental tool. In organizations worldwide, AI systems handle tasks every day with immediate consequences for people: answering customer inquiries, assessing employee attrition risk, or routing patient concerns to medical professionals. As long as these systems operate on properly structured data, they deliver precise, scalable results. But therein lies the core problem of modern AI security.
If the context is flawed—manipulated, biased, or simply incomplete—an AI produces wrong outputs with the same apparent confidence as correct ones. And it does so at a speed that systematically overwhelms human oversight. The consequences are borne by customers, employees, and ultimately the organizations themselves. This is precisely the scenario that Assaf Keren, Chief Security Officer at Qualtrics, describes as the most underestimated security risk of the AI era.
Unsupervised Data: A Structural Risk
Concrete scenarios illustrate the scale of the problem: an AI agent shares sensitive information with the wrong person because access controls are based on outdated parameters. A chatbot generates an incorrect recommendation because the underlying feedback data has been systematically corrupted. An analytics system misattributes employee feedback to the wrong person—with direct consequences for career decisions.
The risk is amplified by usage patterns: according to Qualtrics, only 20 percent of employees use company-approved AI solutions. The rest rely on external, unmonitored tools—and base decisions on these inputs in ways that cannot be tracked or secured. At the same time, consumer surveys show that more than half of respondents cite misuse of personal data as their greatest concern when companies deploy AI.
Why Classical Security Concepts Fail
Encryption, access controls, compliance checks—the established toolkit of IT security was built for a world where data sat static in systems and humans made the resulting decisions. That world no longer exists.
Today, AI systems act autonomously. They make decisions in real time, based on dynamic data inputs, without human review of each individual case. This has fundamental implications for security professionals: the focus shifts from protecting stored data to ensuring the integrity of data that AI systems rely on at the moment of processing.
This is especially critical in experience management—wherever organizations collect feedback from customers, patients, or employees and analyze it with AI support. Open feedback channels are accessible by design and therefore inherently vulnerable to manipulation. Standard validation procedures rarely detect coordinated data corruption reliably, because they lack a reference point for what “normal” behavior looks like in a specific context.
Four Strategic Questions for Security Leaders
Keren proposes measuring organizational maturity with four concrete questions:
- Which business decisions does the platform influence? Looking at technical integrations alone is insufficient. What matters is end-to-end visibility from data input to triggered action—including all automated steps in between.
- How is the authenticity of incoming data ensured? Open input channels in particular require a clear picture of what normal behavior looks like within the feedback program—because only against this backdrop can anomalies be detected.
- Has the potential business damage been quantified? Security risks must be evaluated in business terms, not just technical ones. Decisions made on the basis of manipulated data carry concrete financial, legal, and reputational consequences.
- How quickly can anomalies be identified and addressed? The more autonomously AI operates, the shorter the window between faulty input and faulty decision. Monitoring systems must be able to flag anomalies in real time and enable targeted intervention before problems cascade across the organization.
Security as the Foundation for Trust
Keren puts it sharply: in the AI era, security is no longer a technical side constraint but the fundamental prerequisite for trustworthy decisions. It allows organizations to combine speed with reliability—and to act decisively without amplifying errors.
Whether AI ultimately builds trust or erodes it is determined not by processing power or model quality, but by the quality of the data flowing into the systems—and the processes that ensure that quality. For security professionals, this means redefining their role: from guardians of data to architects of trustworthy AI decision processes.
Assaf Keren has served as Chief Security Officer at Qualtrics since 2024, overseeing global product security and regulatory compliance. He previously served as CISO at PayPal.
Dr. Jakob Jung is Editor-in-Chief of Security Storage and Channel Germany. He has been working in IT journalism for more than 20 years. His career includes Computer Reseller News, Heise Resale, Informationweek, Techtarget (storage and data center) and ChannelBiz. He also freelances for numerous IT publications, including Computerwoche, Channelpartner, IT-Business, Storage-Insider and ZDnet. His main topics are channel, storage, security, data center, ERP and CRM.
Contact via Mail: jakob.jung@security-storage-und-channel-germany.de