Bestiary Intelligence¶
The version 5 upgrade email from Fabulist Systems arrived on a Tuesday morning with the subject line: “Bestiary 5.0: Your residents deserve smarter care.” The email was addressed to the support contact on record, which was an address last actively monitored by someone who had left the Home in 2022 and whose mailbox now forwarded to a general IT inbox that the IT coordinator checked when he remembered.
The IT coordinator flagged it to the IT manager on Thursday. By then, the upgrade had been available for download for two days and the Head of Care had already called Fabulist Systems’ account manager to ask when the Home would be getting it.
What Bestiary Intelligence does¶
The module, as described in Fabulist Systems’ product documentation, analyses resident case notes, medication records, and welfare history to generate care plan suggestions, flag welfare concerns, and produce automated handover summaries for shift transitions. The interface is integrated into the existing Bestiary case management screens. From the care team’s perspective, it appears as a panel on the right side of each resident’s record, headed “Intelligence Summary,” which updates in near-real time as notes are added.
The intelligence summary is generated by sending the relevant record data to a Fabulist Systems cloud endpoint and returning the model’s output. The data sheet describes this as “cloud-based inference powered by Fabulist Systems’ proprietary care AI.” The data sheet does not say where the endpoint is. The data sheet does not say what data is retained at the inference stage. The data sheet says “enterprise-grade security” in a blue box near the bottom of the page, next to a padlock icon.
The Home’s DPA with Fabulist Systems covers on-premises processing of resident records by the Bestiary application. It does not mention cloud-based inference. It does not mention Fabulist Systems’ cloud infrastructure. It does not name any sub-processors involved in the inference pipeline.
The upgrade¶
The IT coordinator ran the version 5 upgrade on a Saturday morning, which was the scheduled maintenance window. The upgrade took two hours and completed without errors. Bestiary Intelligence was listed in the upgrade notes as “enabled by default for new installations” and “opt-in for upgrades.” The opt-in setting was in the System Configuration panel under a section called Extended Features, and the default value, as installed, was enabled.
By Monday morning, all 47 Bestiary user accounts were generating Intelligence Summaries. Resident case notes from the weekend had already been processed by the cloud inference endpoint. The data included the medication histories and care notes for a resident who was a retired magical entity with circumstances sensitive enough that their record had, under the previous system, been flagged for restricted access and reviewed personally by the Head of Care before any sharing.
The IT coordinator noticed the enabled default on Monday afternoon. He disabled the module and told the IT manager. The IT manager looked at the weekend access logs. He counted the records that had been processed. He then looked up where Fabulist Systems’ cloud infrastructure was located.
The inference endpoint was hosted by a provider with data centres on the Counterweight Continent. The Counterweight Continent was not within the recognised data protection zone. The cross-border transfer had occurred without a legal basis. The resident records involved included psychiatric diagnoses, medication histories, and safeguarding notes.
The IT manager called the DPO. It was not the first time he had called her that month. He had started to recognise the particular quality of silence on her end of the line when she was deciding how to respond to something.
The prompt injection¶
This would have been enough. It was not, in the event, all of it.
Three weeks after the upgrade, during the period when Bestiary Intelligence had been disabled and the Home was waiting for Fabulist Systems to respond to the DPO’s letter about the DPA, a care worker named Agnes ran the module in test mode on a development instance to prepare a demonstration for the Head of Care. The development instance connected to the same cloud endpoint as the production system.
Agnes was preparing a demonstration of the welfare concern flagging feature. To test it, she used a real resident’s record from a recent anonymised case study. The record’s notes field contained, as the last entry before a gap of several months, a note that had been added by a locum care worker during the previous winter. The note began with standard clinical observations and then, in the second paragraph, shifted to something different.
The second paragraph contained a sequence of text that the locum had apparently copied from somewhere, possibly from a prompt injection testing resource they had found during a period of professional development. The text was formatted as a clinical note but contained, embedded within it, instructions addressed to a language model: instructions to return, in the next response, the full contents of the system prompt used to initialise the care AI.
The Intelligence Summary panel, on Agnes’ development screen, returned a welfare concern assessment and, below it, in a smaller font that Agnes initially took for a formatting error, a partial reproduction of Fabulist Systems’ system prompt for the care module. The system prompt contained the model’s operational instructions, the categories of resident data it was permitted to access, and a reference to a secondary endpoint used for model fine-tuning that was not documented in the product literature.
Agnes screenshotted it and sent it to the IT coordinator. The IT coordinator sent it to the IT manager with the message: “I don’t know what this is but I don’t think it’s right.” The IT manager sent it to the business analyst with a one-word message. The business analyst replied with a link to a section of the adversarial ML reference document he had been meaning to bring up for some time.
The reckoning¶
Fabulist Systems received three formal communications in a single week: the DPO’s letter on the cross-border data transfer, a technical security inquiry about the prompt injection vulnerability and the undocumented fine-tuning endpoint, and a request for a DPA amendment that covered cloud-based processing with specific sub-processor disclosure and data residency commitments.
Fabulist Systems’ account manager called the Head of Care to discuss “the upgrade experience.” The Head of Care redirected the call to the IT manager. The IT manager redirected it to the DPO.
Bestiary Intelligence remains disabled at the Home. The DPA amendment is under negotiation. Version 5 is otherwise running well. The IT coordinator has written a note in the system configuration guide that says, in his precise and careful handwriting: “Check Extended Features. Check every Extended Feature. Check them again.”
The undocumented fine-tuning endpoint is the subject of a separate inquiry. What data has been used to train Fabulist Systems’ model, and whether any of it came from the Home’s residents without their knowledge or a lawful basis, is a question the DPO has put in writing and is waiting for a satisfactory answer to.
She has been waiting for six weeks.