Kuan0

Canon MF Toolbox - solution if it's not working!

2025-03-16T16:02:00.000+00:00

Here's the fix for Canon multi-function devices' scanning software, MF Toolbox, not working since late 2024/2025. For me it opened, but nothing happened whatever button I clicked, PDF, scan, save....

I tried many online suggestions, none worked. The only effective one, finally, was one from 2012! For a Windows 11 (and probably 10) computer, add this to your path - C:\Windows\twain_32\MF4100 (or whatever is in the twain_32 subfolder for your device, although it's likely to be MF4100 - just look in that Windows twain_32 subfolder).

How to add something to your path in Windows?

Press a Win key on your keyboard
Type: env
You should see something like this:
Click on "Edit the system environment variables" (left or right one, it doesn't matter)
You should see something like this, entitled System Properties:
Click the Environment Variables button (bottom right)
You'll see something like this, entitled Environment Variables, with a list of items under Variable. One is named Path, highlighted below (I've blanked out the rest):
Click on Path. Then click the first Edit button just below its box (or, just doubleclick on Path)
You'll see something like this, entitled Edit environment variables, with a list under it (again I've blanked out some info):
Click on New, then type in or paste the path to the twain_32 MF4100 subfolder which, for me, was as above:
C:\Windows\twain_32\MF4100
You'll see something like this, with the added info now at the bottom of the list on the left:
Now, click OK and just keep clicking OK (I counted 3 times altogether clicking OK) till you come out of the Environmental Variables box
For me, that was it! MF Toolbox then started working properly again, I didn't even need to restart my computer.

I hope this helps others, it's been driving me mad for months!

AI literacy: EU AI Act

2025-02-09T14:42:00.004+00:00

The EU AI Act's "AI literacy" obligation applied from 2 Feb 25, alongside its prohibition on certain AI uses (commencement dates 1-pager). But what, if anything, should you do about it? Points to consider:

Who's caught? This obligation applies to "providers" and, especially, "deployers" (i.e. users) of AI systems

For non-compliance, you can't be fined yet (fining provisions don't kick in till 2 Aug 2025), or maybe even not at all (Art.4 isn't listed in Ch.12 on penalties, and it's unclear whether individual EU Member States can or will penalise breach of this obligation - we'll find out by 2 Aug 2026!)

But... train anyway? However, if you use any AI system caught by the EU AI Act, an AI upskilling/training and awareness program for staff is good practice and should help to boost your business's competitiveness as well as legal compliance, so you may want to roll it out anyway - if not yet, then ideally by 2 Aug 2026

Who? Train at least staff/contractors developing/adapting/integrating, operating/maintaining or using any high-risk AI systems (including third-party AI systems), and also train staff providing human oversight of AI; and ensure they have appropriate authority to perform their stasks properly. Train them similarly even if your AI systems aren't high-risk (the Art.4 AI literacy obligation applies to all AI systems)
What on? Train them (as appropriate to their role, technical knowledge, experience, education and training) on AI technicalities, use/safeguards, and interpretation of output, as well raising their awareness about AI's opportunities, risks and possible harms, taking into account the context the relevant AI system is to be used in, and the persons or groups of persons on whom your AI system is to be used

The "AI literacy" definition (below) mentions skills, knowledge and understanding, "taking into account" AI Act rights and obligations, to make an informed deployment of AI systems. This might imply that relevant staff should also be trained on what are your obligations under the AI Act as deployer/provider, at least to a basic level - even engineers who aren't in the Legal/Regulatory/Compliance teams

How?

Others' experiences. To see what other organisations are doing on AI literacy, you can review the Commission's "living repository" compilation of AI literacy practices of many organisations (15 currently) from different sectors and of different sizes (direct link). Added: on 2 Apr 25 the Commission conducted a survey to gather practices for the repository.
Also consider attending/viewing the AI Pact webinar on AI literacy on 20 Feb 25 (YouTube livestream). If you don't have enough internal resources/expertise to train your staff, external third-party resources are available, but do check that whoever you engage is sufficiently knowledgeable. There are now many out there who offer AI training (nice market for that since ChatGPT, and it can only get bigger!) - but how well qualified or expert are they? A lot of big well-known AI companies already provide online AI training (typically tailored to their own services but covering the basics too), those are often free, so it's worth checking them out.
AI jargon. My free YouTube video demystifying key AI jargon/terminology may also be of use😉, do incorporate it if you wish

What else? Consider contributing to any sectoral/industry initiatives on training/awareness of people ("affected persons") who may be affected by your use of AI systems, and/or of other actors in the AI value chain

Surely "affected person" won't be making any "deployment" of AI systems, so the "AI literacy" definition doesn't work very well in relation to them... it seems to be more awareness-raising on AI opportunities and risks/harms rather than training, there

What to monitor for?

Monitor relevant Member States' national laws for any local penalties that might be imposed for infringement of this obligation (seems unlikely, but you never know)
Watch out for any voluntary codes of practice on promoting AI literacy "facilitated" by the EU AI Office/relevant Member States under Art.95(2)(c), and take on board anything from them if you can
The AI Board is supposed to support the Commission in promoting AI literacy, public awareness and understanding of the benefits, risks, safeguards and rights and obligations in relation to the use of AI systems. If and when they put anything out, again see whether what they produce can usefully be incorporated into your own AI literacy program.

(Added) Update - more resources

Commission Q&A / FAQs on AI literacy - worth reviewing
Commission/OECD AI literacy framework for primary and secondary education (draft; feedback request, June 25)
Commission's general webpage on AI literacy, skills, talent
Cedefop policy brief on AI skills

Key background info for ease of reference

Art.4 AI literacy obligation: Providers and deployers of AI systems shall take measures to ensure, to their best extent, a sufficient level of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf, taking into account their technical knowledge, experience, education and training and the context the AI systems are to be used in, and considering the persons or groups of persons on whom the AI systems are to be used.

Rec.91: ... deployers should ensure that the persons assigned to implement the instructions for use [of high-risk AI systems] and human oversight as set out in this Regulation have the necessary competence, in particular an adequate level of AI literacy, training and authority to properly fulfil those tasks...

"AI literacy" definition: skills, knowledge and understanding that allow providers, deployers and affected persons, taking into account their respective rights and obligations in the context of this Regulation, to make an informed deployment of AI systems, as well as to gain awareness about the opportunities and risks of AI and possible harm it can cause

Rec.20: In order to obtain the greatest benefits from AI systems while protecting fundamental rights, health and safety and to enable democratic control, AI literacy should equip providers, deployers and affected persons with the necessary notions to make informed decisions regarding AI systems. Those notions may vary with regard to the relevant context and can include understanding the correct application of technical elements during the AI system’s development phase, the measures to be applied during its use, the suitable ways in which to interpret the AI system’s output, and, in the case of affected persons, the knowledge necessary to understand how decisions taken with the assistance of AI will have an impact on them. In the context of the application this Regulation, AI literacy should provide all relevant actors in the AI value chain with the insights required to ensure the appropriate compliance and its correct enforcement. Furthermore, the wide implementation of AI literacy measures and the introduction of appropriate follow-up actions could contribute to improving working conditions and ultimately sustain the consolidation, and innovation path of trustworthy AI in the Union. The European Artificial Intelligence Board (the ‘Board’) should support the Commission, to promote AI literacy tools, public awareness and understanding of the benefits, risks, safeguards, rights and obligations in relation to the use of AI systems. In cooperation with the relevant stakeholders, the Commission and the Member States should facilitate the drawing up of voluntary codes of conduct to advance AI literacy among persons dealing with the development, operation and use of AI.

Things cyber security, Q4 2024

2025-01-13T09:04:00.005+00:00

Selected things cyber security, mostly from Q4 2024, are listed below in reverse chronological order, some with descriptions. See also Things AI, Oct 2024, and Data protection & cyber security, Oct 2024

28 Dec 24

Software code, provenance: a thoughtful, telling post, Why it's hard to trust software, but you mostly have to anyway: "...the situation is fairly dire: if you're running software written by someone else—which basically everyone is—you have to trust a number of different actors. We do have some technologies which have the potential to reduce the amount you have to trust them, but we don't really have any plausible venue to reduce things down to the level where there aren't a number of single points of trust... Open source, audits, reproducible builds, and binary transparency are all good, but they don't eliminate the need to trust whoever is providing your software and you should be suspicious of anyone telling you otherwise"

21 Dec 24

International collaboration, critical infrastructure: the Critical Five (5 Eyes) countries reaffirmed their "vision of fostering collaboration across the private and government critical infrastructure communities in our five nations" (see the June 24 summary on how they plan to modernise their approach to critical national infrastructure security and resilience). CISA links

20 Dec 24

Consumer IoT: the UK Department for Science, Innovation & Technology (DSIT) will be undertaking an interim Post-Implementation Review of the Product Security and Telecommunications Infrastructure Act (PSTI), to be published by October 2026. To support this, DSIT has commissioned a consultancy to condcut preparations for evaluation and research projects on the product security elements of the PSTI Act

19 Dec 24

Training: the UK updated its webpage on cyber security training for business - this has links to many useful resources, such as free online staff training, a free online incident response exercise and personalised cyber action plan for SMEs/individuals, many from the UK National Cyber Security Centre

18 Dec 24

Supply chain, vendors, defence, national security: the UK Ministry of Defence wrote to Defence Industry CEOs/leads, asking them to review their organisations' performance against the Cyber Assessment Framework (CAF, developed by the UK National Cyber Security Centre (NCSC) for NIS Regulations assessments) particularly the areas of Govern, Identify, Protect, Detect, Respond and Recover; adopt Active Cyber Defence (ACD) with the NCSC and its tools including Early Warning (see 3 Dec); implement the March 24 Cyber Security Standard for Suppliers; and deliver Secure by Design

17 Dec 24

Cloud, SaaS, Microsoft 365: the US Cyber & Security Infrastructure Agency (CISA) issued a web-friendly verions of its directive Implementing Secure Practices for Cloud Services for US federal agencies, requiring deployment of its SCuBA tool - mentioned here because this open source tool, downloaded >30k times as at 13 Nov, automatically assesses Microsoft 365 (M365) configurations for security gaps (against CISA baselines): reportedly, misconfigurations were the initial access point for 30% of cloud environment attacks in the first half of 2024. "ScubaGear rapidly and thoroughly analyzes an organization’s M365 tenant configuration. It then delivers actionable security change insights and recommendations that allow the tenant administrator to close security gaps and attain a stronger defense within their M365 environment". So Microsoft 365 users could do worse than use this free tool!
Cybercrime, UN Convention: this Convention's privacy and security issues (law enforcement access to data) have been raised by many, including by the EDPB, referring to its Statement 5/2024 on the Recommendations of the High-Level Group on Access to Data for Effective Law Enforcement
Mobile comms: the US Cyber & Security Infrastructure Agency (CISA)issued Mobile Communications Best Practice Guidance after "identified cyber espionage activity by People’s Republic of China (PRC) government-affiliated threat actors targeting commercial telecommunications infrastructure, specifically addressing “highly targeted” individuals who are in senior government or senior political positions and likely to possess information of interest to these threat actors". While intended to assist highly-targeted individuals, its recommendations are obviously also relevant to everyone else who values their security and privacy (there were also iPhone and Android-specific recommendations, not reproduced here, see the link above):

Use only end-to-end encrypted communications (free messaging apps mentioned include Signal "or simlar apps")
Enable Fast Identity Online (FIDO) phishing-resistant authentication like hardware-based security keys
Migrate away from Short Message Service (SMS)-based MFA
Use a password manager
Set a telco PIN (for login etc)
Regularly update the operating system and other software (i.e. patch)
Opt for the latest hardware version from your cell phone manufacturer
Do not use a personal virtual private network (VPN): "Personal VPNs simply shift residual risks from your internet service provider (ISP) to the VPN provider"

16 Dec 24

Cybercrime, advanced persistent threats: a RUSI article points out that "...foreign government adversaries no longer have a monopoly on sophistication or persistence. Cybercriminals have just as much if not more of an impact on the Western world... Digital spying by foreign state adversaries is still important. However, in biasing themselves towards ‘APT versus cybercrime’, information security and cybersecurity practitioners create a false dichotomy that pushes resources, attention and support to areas that don’t always align with the greatest organisational or national risk and impacts"

13 Dec 24

CSAM, encryption: the EU Council agreed its general approach on the proposed CSAM Directive (this has notes on some amendments), based on which it can commence negotiations on the text with the European Parliament, but the Parliament hasn't agreed its own position internally yet, so it will be some months or longer before this Directive is adopted

Statements on the general approach by Austria, Austria and Slovenia, and Belgium, Finland, Ireland, Latvia, Luxembourg, Slovenia and Sweden
The age old debate continues about undermining encryption to allow checking of encrypted material for any CSAM, e.g. US litigation against Apple for dropping its planned CSAM scanning after privacy and surveillance concerns.
An excellent post about the planned "Chat Control" scanning under this Directive points out: "Chat Control is one example of mass screening for a low-prevalence problem — a dangerous mathematical structure. It requires breaking end-to-end encryption, the technological bedrock of digital privacy. Such a move would make mass surveillance cheap and easy again... false positives will overwhelm true positives in programs of this structure — mass screenings for low-prevalence problems under conditions of rarity, persistent uncertainty, and secondary screening harms. Under these conditions, even highly accurate such programs backfire by making huge haystacks (wrongly flagged cases, “false positives”) while missing some needles (wrongly cleared cases, “false negatives”)... when finite investigative resources are tied up processing CSAM possession tips from mass scanning, they cannot be used for other investigations... This is consistent with the possibility that children are endangered by such mass screening programs exhausting the investigative resources necessary to process tips that have a higher likelihood of being true positives and may otherwise be more relevant to current as opposed to past abuse. Curtailing targeted investigations that might stop ongoing abuse or bigger-fish distributors in favor of processing mass scanning tips that are overwhelmingly false positives does not serve the interests of vulnerable children or society... [but] it does serve the interests of those who would like a return to cheap, easy mass digital communications surveillance..."

Data, software, products: reminder from the US Federal Trade Commission (FTC) that to protect security it's important to have good data management (including enforcing mandated data retention schedules and mandating data deletion, so there's less unnecessary data that could be hacked), secure software development, and secure product design for humans (including least privilege, phishing-resistant MFA)
Financial services, incident reporting, vendors: the UK Prudential Regulation Authority (PRA) issued a consultation paper CP17/24 – Operational resilience: Operational incident and outsourcing and third-party reporting, on proposed "rules and expectations for firms to report operational incidents and their material third-party arrangements" (deadline 14 Mar 25), with reporting thresholds (quite subjective), and a phased approach to incident reporting: initial, intermediate, final, with certain minimum information
Product safety: the EU General Product Safety Regulation applies from this date. When assessing whether a product is a safe product, factors to consider include, "when required by the nature of the product, the appropriate cybersecurity features necessary to protect the product against external influences, including malicious third parties, where such an influence might have an impact on the safety of the product, including the possible loss of interconnection", particularly digitally connected products likely to have an impact on children (on top of sectoral laws on cybersecurity risks affecting consumers etc)

Detailed UK guidance, applicable to Northern Ireland summarises it as: "This Regulation requires that all consumer products placed on the NI and EU markets are safe and establishes specific obligations for businesses to ensure that safety. The Regulation applies to products placed on, or made available to, the market where there are no sector-specific provisions with the same objective"

12 Dec 24

Passkeys: these are touted as more secure than using passwords, and are increasingly being supported (see my book and the free PDF). Microsoft published its UX design insights to boost passkey adoption and security

10 Dec 24

EU Cyber Resilience Act (CRA): this Regulation entered into force (published in OJ 20 Nov 24, news item), requiring minimum cybersecurity requirements for any "product with digital elements" before they can be made available on the EU market, with certain cybersecurity vulnerability handling obligations on manufacturers, including on vulnerability disclosure

A "product with digital elements" is any software or hardware product and its "remote data processing solutions", including software or hardware components being placed on the market separately (where "remote data processing" is remote processing designed by the manufacturer, whose absence would prevent one of the product's functions from being performed - such as essential cloud processing). So, CRA catches not just IoT / smart devices but also software (and is not limited to consumer IoT, unlike the UK's Product Security and Telecommunications Infrastructure Act (PSTI))
CRA applies fully from 11 December 2027, but with some earlier applicable dates like 11 September 2026 for Art.14 on manufacturers' obligation to report any actively exploited vulnerability contained in such a product: with staggered deadlines of 24 hrs, 72 hrs etc.

9 Dec 24

Open source, malicious code, tools: open source code is increasingly incorporated into software, but open source packages can be malicious, or legitimate code can be accessed by attackers and "poisoned" to serve malicious purposes e.g. adding a backdoor for hackers. The Stack reported that a tool, DataDog, had been open sourced, termed as a [software] "supply chain firewall" that scans Python packages being installed, and blocks packages know to be malicious based on the tool provider's own observations or certain open source feeds

5 Dec 24

Consumer IoT: the UK published a survey it had commissioned before the Product Security and Telecommunications Infrastructure Act (PSTI) came into force, to map and analyse the market for consumer connectable products, and collect and analyse evidence on the compliance of manufacturers with the PSTI legal regime, as well as evidence on awareness and impacts of the legislation. It outlines well the PSTI compliance challenges (which many may be familiar with!). And see the related infographic. See also 25 Nov
Software patching, tools: one of the most critical security measures to take is patching, ensuring software is kept updated with new versions that address security vulnerabilities. Google released a new open source security patch validation automation tool Vanir, that helps Android developers "quickly and efficiently scan their custom platform code for missing security patches and identify applicable available patches". "While initially designed for Android, Vanir can be easily adapted to other ecosystems with relatively small modifications, making it a versatile tool for enhancing software security across the board"

4 Dec 24

EU digital identity wallets: technical standards, adopted by the Commission on 28 Nov, for cross-border eID wallets under the European Identity Framework that was updated in 2024, were published in the OJ under 5 implementing regulations with rules on eID Wallets' integrity and core functionalities, on eID Wallets solutions' protocols and interfaces and on person identification data and electronic attestations of attributes of eID Wallets, plus reference standards, specifications and procedures for a certification framework for eID Wallets, and obligations for notifications to the Commission concerning the eID Wallet ecosystem
Measures, metrics: how can organisations measure to what extent their cyber security measures are effective? The US National Institute of Standards and Technology (NIST) published updated guidance on how an organization can develop

"information security measures to identify the adequacy of in-place security policies, procedures, and controls. It explains the measures prioritization process and how to evaluate measures": Volume 1 — Identifying and Selecting Measures
"an information security measurement program with a flexible structure for approaching activities around the development and implementation of information security measures": Volume 2 — Developing an Information Security Measurement Program

3 Dec 24

Comms infrastructure: several US and other agencies published a joint guide Enhanced Visibility and Hardening Guidance for Communications Infrastructure, "that provides best practices to protect against a People’s Republic of China (PRC)-affiliated threat actor that has compromised networks of major global telecommunications providers. The recommended practices are for network engineers and defenders of communications infrastructure to strengthen visibility and harden network devices against this broad and significant cyber espionage campaign... Although tailored to communications infrastructure sector, this guidance may also apply to organizations with on-premises enterprise equipment"
EU cybersecurity, threats: EU security agency ENISA published its first NIS2 biennial report on the state of EU cybersecurity. This reported "substantial cyber threat level to the EU, highlighting discovered vulnerabilities exploited by threat actors targeting EU entities..." and made several policy recommendations on strengthening EU cyber skills/workforce and addressing supply chain security
UK cybersecurity, threats: the UK National Cyber Security Centre (NCSC) published its Annual Review 2024. Its head stressed in an accompanying speech the "clearly widening gap between the exposure and threat we face, and the defences that are in place to protect us... We need all organisations, public and private, to see cyber security as both an essential foundation for their operations and a driver for growth. To view cyber security not just as a ‘necessary evil’ or compliance function, but as a business investment, a catalyst for innovation and an integral part of achieving their purpose... Hostile activity in UK cyberspace has increased in frequency, sophistication and intensity... Actors are increasingly using our technology dependence against us, seeking to cause maximum disruption and destruction... And yet, despite all this, we believe the severity of the risk facing the UK is being widely underestimated... There is no room for complacency about the severity of state-led threats or the volume of the threat posed by cyber criminals. The defence and resilience of critical infrastructure, supply chains, the public sector and our wider economy must improve..."

The NCSC's incident management (IM) team issued 542 bespoke notifications to organisations of a cyber incident impacting them, providing advice and mitigation guidance (cf. 258 in 2023). Almost half related to pre-ransomware activity, enabling organisations to detect and remove precursor malware before ransomware was deployed.
Top sectors reporting ransomware activity into the NCSC were academia, manufacturing, IT, legal, charities and construction. "We received 317 reports of ransomware activity, either directly from impacted organisations, or from our partners (an increase on 297 last year). These were triaged into 20 NCSC-managed incidents, of which 13 were nationally significant. These included high-profile incidents impacting the British Library and NHS trusts"
The NCSC was made aware of 347 reports of activity that involved the exfiltration or extortion of data
The IM team issued ~12,000 alerts about vulnerable services through its Early Warning service (an automated NCSC threat notification service, free to UK organisations - do sign up!). Exploitation of zero-days CVE-2023-20198 (Cisco IOS XE) and CVE-2024-3400 (Palo Alto Networks PAN OS) also resulted in 6 nationally significant incidents which the IM team helped manage

2 Dec 24

Cybersecurity measures: the US Cyber & Security Infrastructure Agency (CISA) updated its Trusted Internet Connections (TIC) 3.0 Security Capabilities Catalog (SCC) version 3.2, based on the new National Institute of Standards and Technology (NIST) Cyber Security Framework (CSF) Version 2.0 mapping updates. TIC 3.0 SCC provides a list of deployable security controls, security capabilities, and best practices, intended to guide secure implementations and help US federal agencies satisfy program requirements within discrete networking environments, but is of more general use/interest
Encryption: this was Global Encryption Day: the Global Encryption Coalition reported on the support of policymakers and others for encryption , including for the protection of children
FS, DORA: the EU's DORA Regulation on digital operational resilience for the financial sector applies in the EU from 17 Jan 2025. Much secondary legislation on certain detailed requirements has been made under it (see list as at 4 Dec 24). On 2 Dec, an implementing regulation was published in the OJ on technical standards for standard templates for the register of information that in-scope financial entities must maintain in relation to their ICT services and ICT service providers, including providers' subcontractors in certain cases, such as details of their contracts

Financial service entities' contracts with their ICT service providers should also be updated to comply with DORA's requirements, and some providers are directly regulated under DORA - not discussed here

Managed security services, certifications; EU: the Council approved a directly-applicable Regulation amending the EU Cybersecurity Act (CSA) to enable future adoption of European certification schemes for managed security services (MSS, like incident handling, penetration testing, security audits, consulting advice on technical support), increasingly important for cybersecurity incidents' prevention, detection, response, and recovery. Awaiting the CSA's broader evaluation by the Commission, this targeted amendment aims to enable establishment of such European certification schemes to help increase MSSs' quality, comparability and trustworthiness, and avoid fragmentation as some Member States have initiated national certification schemes for such services

Not law yet, to awaiting OJ publication
The Council also approved a Cyber Solidarity Act Regulation (also awaiting OJ) to strengthen EU/Member State cooperation and resilience against cyber threats, e.g. creating a cyber security alert system pan-European infrastructure comprising national and cross-border cyber hubs responsible for detecting, sharing information and acting on cyber threats including cross-border incidents. It also creates a cybersecurity emergency mechanism (including a EU cybersecurity reserve: private sector incident response services "ready to intervene" on significant/large-scale incidents if requested by a Member State or EU body as well as associated third countries, and incident review mechanism

1 Dec 24

Cloud, access, MFA: previously, cloud service providers have tended to leave it to their customers to decide whether the customer wants to require MFA in order for its users to access its cloud service. A very positive trend is that providers are increasingly enforcing MFA, e.g. Snowflake will be blocking attempted sign-ins using single-factor authentication with passwords. It seems likely this move by Snowflake was influenced by >100 of its customers, who had not required MFA, being successfully attacked in 2024. While it would have behoved those customers to require MFA for access to their Snowflake services, these incidents did appear to lead to some negative comments about Snowflake

29 Nov 24

Online shopping, passwords: the US CISA issued tips with reminders on best security practices including updating software, using strong passwords ("at least 16 characters, random, and unique for each account"), using a password manager and enabling MFA where offered, plus phishing warnings etc.

(Interestingly, the US NIST's draft (for comment) of its fourth revision to the four-volume suite of Special Publication 800-63, Digital Identity Guidelines of August 2024 included SP 800-63B-4 on Authentication & Authenticator Management (another link), which recommended that passwords "shall" be at least 8 characters long and "should" be at least 15 characters.

Undersea cables: these are essential for Internet access/usage in a globally digitalised world. The UK, EU and 15 other partners endorsed shared principles in the New York joint statement on the security and resilience of subsea cables at the UN General Assembly, on working to ensure the security, reliability, interoperability, sustainability, and resiliency for the deployment, repair and maintenance of undersea cable infrastructure

28 Nov 24

Boards, directors: UK National Cyber Security Centre (NCSC)'s Cyber Security Toolkit for Boards: updated briefing pack released with insights on the ransomware attack against the British Library
EU NIS2 Directive: this Directive, updating and expanding the NIS Directive, should have been implemented by Member States by 17 Oct 24, but it wasn't (Europa list of those that have notified the Commission of their NIS2 transposition).

The Commission decided to open infringement procedures for not fully implementing NIS2 by sending formal notice to 23 Member States (Bulgaria, Czechia, Denmark, Germany, Estonia, Ireland, Greece, Spain, France, Cyprus, Latvia, Luxembourg, Hungary, Malta, Netherlands, Austria, Poland, Portugal, Romania, Slovenia, Slovakia, Finland and Sweden). The Commission has given them two months to respond, complete their transposition and notify their measures to the Commission. Ireland hasn't yet transposed NIS2 officially, but it wasn't in that list, for whatever reason

UK Cyber Security & Resilience Bill consultation: consultation closed, on UK DSIT's call for evidence on proposals to inform the Bill

26 Nov 24

Awareness raising, NIS: EU security agency ENISA updated its guide on how to promote cyber security awareness to C-level (part of its AR-in-a-box DIY awareness-raising toolbox, "a comprehensive solution for cybersecurity awareness activities designed to meet the needs of public bodies, operators of essential services, and both large and small private companies. It provides theoretical and practical knowledge on how to design and implement effective cybersecurity awareness") - still relevant to NIS2 of course

25 Nov 24

IoT, smart devices, vulnerability handling, PSTI: the IoT Security Foundation published The State of Vulnerability Disclosure Policy (VDP) Usage in Global Consumer IoT in 2024, including some coverage of the impact of the UK Product Security and Telecommunications Infrastructure Act (PSTI). "...the UK legislation has driven a bigger improvement [among UK retailers] than European and US retailers. Whilst the sample set maybe low, it is a consistent gauge moving faster in the right direction"

The survey indicated an increase in the proportion of manufacturers checked that had a vulnerability disclosure policy, from 23.99% in 2023 to 35.59% in 2024. Only ~21% of companies complied with PSTI's vulnerability disclosure requirements, though that's "increased significantly" from the previous year.
The picture's variable regarding proportion of retailers stocking products whose manufacturers support vuln disclosure. Over 50% of IoT products stocked by several UK retailers were from manufacturers that had vulnerability disclosure policies. John Lewis was the best, 93.33% of its products checked were from compliant manufacturers. The detail on specific manufacturers, their website statements of compliance and how some meet PSTI (or not) is worth a look
"There has clearly been some effect from the UK’s Product Security and Telecommunications Infrastructure Act (Part 1) requirements... but implementation seems fragmented and inconsistent. While some leading UK retailers are showing that around 90% of the IoT manufacturers they stock have vulnerability disclosure policies, there are some notable exceptions to this ‘dip test’ of the market and there are obvious differences in online marketplaces. The other regions showed less promising and variable data about the product manufacturers they stocked" (the report covers manufacturers and retailers in the EU, US and Asia too - not discussed here)
And there remains a "..gap in practice between the consumer and enterprise sectors. Whilst the consumer sector is firmly heading in the right direction, there is a stark contrast in market practice levels and continues to justify the need for consumer regulation" (I'd suggest enterprise IoT security could still improve)".
On individual product categories, "notable laggards being Health and Fitness, Lighting and, somewhat paradoxically, Security. Those manufacturer report cards read “must do better”"
See also 5 Dec

22 Nov 24

Fraud, data protection: the UK ICO emphasised that data protection is not an excuse when tackling scams and fraud, "warning that reluctance from organisations to share personal information to tackle scams and fraud can lead to serious emotional and financial harm. Data protection law does not prevent organisations from sharing personal information, if they do so in a responsible, fair and proportionate way". It published "new practical advice to provide clarity on data protection considerations and support organisations to share data responsibly to tackle scams and fraud", aimed at any organisation seeking to share personal information to identify, investigate and prevent fraud, especially banks, telecommunications providers and digital platforms"

The same also applies to organisations disclosing potential personal data, like IP addresses and domain names, as indicators of compromise (IOCs) in threat sharing initiatives/platforms regarding cyber threats/breaches, whether sectoral or otherwise, and it would have been helpful if the ICO had also made that point.

21 Nov 24

Critical infrastructure, red teaming: the US Cyber & Security Infrastructure Agency (CISA) published its insights from a red team assessment of a US critical infrastructure organisation including lessons learned (technical controls, staff training, leadership/board: "Leadership deprioritized the treatment of a vulnerability their own cybersecurity team identified, and in their risk-based decision-making, miscalculated the potential impact and likelihood of its exploitation") and technical details

18 Nov 24

Passwords: it's interesting that, following notification of personal data breaches, the Romanian data protection supervisory authority ordered a company to take measures including (machine translation) "password complexity and history policy on all customer accounts with a pre-established expiration interval". That is a decades old practice, which is no longer considered good. Technical experts including the UK NCSC and US NIST recommend longer rather than more complex passwords, indeed NIST's latest draft update recommends not enforcing any password complexity rules like one lowercase, one uppercase etc. Similarly with forced password changes every few months or year, which is now deprecated (e.g. New Scientist article) as it reduces security by resulting in people writing down passwords, using bad passwords they can remember, etc! So it seems that some GDPR authorities could still benefit from more technical assistance/education on cybersecurity...

15 Nov 24

UK NIS Regulations, notified incidents: the ICO is the regulator for digital service providers under NIS (cloud, online marketplaces, online search engines). Responding to a freedom of information request, the ICO stated that 37 incidents were reported to the ICO as NIS incidents, including 18 incidents that were not in fact NIS incidents and 2 incidents (reported in 2020 and 2021) that did not meet the mandatory threshold following its assessment. The figures suggest many incidents are reported as NIS incidents when they are not, but it's possible there were some actual NIS incidents that were not reported as the final total of 19 seems quite low...:
- 2020 - 2 (really 1, see above, but in fact the ICO did not consider it a NIS incident, so 0)
- 2021 - 3 (really 2, but the ICO did not consider 1 a NIS incident, so 1)
- 2022 - 4 (really 2, as the ICO did not consider 2 of those a NIS incident)
- 2023 - 19 (really 18, as one was incorrectly reported to the ICO as well as to the correct competent authority, but several were not considered NIS incidents, so 8)
- 2024 YTD - 9 (but 5 were not considered NIS incidents, so 4)

14 Nov 24

Product safety, IoT: in the first horizon scan report
by the UK Office for Product Safety & Standards (OPSS), privacy, data loss and wider cyber security issues like distributed denial of service (DDOS) attacks were considered as part of harms or benefits a technology may present in relation to non-physical aspects, and the scan's taxonomy of technologies included cybersecurity and data platforms: "combination of data, policies, processes, and technologies employed to secure information, protect organisations, and protect individuals' cyber assets, including specific biological research through omics, and financial activities through blockchain, like new data technology and PETs. Health data was at greater risk of being compromised by cyber threats. Trends across technologies included security issues, with new vulnerabilities created by increased automation and connected technology with IoT, and new ways of compromise; most IoT devices' relatively limited computing power limits cybersecurity complexity and effectiveness, their interconnectivity increases vulnerabilities (specific IoT rapid review guidance). Social commerce normalises online money transfer enabling cybersecurity scams. "Blockchain can potentially offer some solutions to these challenges". Online marketplaces also need consumer protection against scams.
- OPSS research on consumer attitudes/awareness indicated consumers are increasingly comfortable with manufacturers making changes remotely in the case of physical safety issues or cyber security vulnerabilities, but becoming less considerate of cyber security before initial purchase, particularly those with a low education level. Note that the OPSS is responsible for enforcing the UK's Product Security and Telecommunications Infrastructure Act (PSTI)

12 Nov 24

Financial services, vendors: UK FS regulators Bank of England (Bank), Prudential Regulation Authority (PRA) and the Financial Conduct Authority (FCA) issued PS16/24 FCA 24/16 – Operational resilience: Critical third parties to the UK financial sector, with final rules for FS use of critical third parties (CTPs) including operational risk and resilience requirements, and incident reporting and other notifications and enforcement
Security engineering, learning: all PDF chapters of the late, great Ross Anderson's seminal, very readable Security Engineering book (3rd edition 2020) are now available for free download via this link

7 Nov 24

NIS2, risk management: EU security agency ENISA issued a consultation on its detailed technical implementin gguidance (PDF no longer availabe on ENISA's website, but see Internet Archive) to support EU Member States and entities with implementation of the technical and methodological requirements of NIS2's required cybersecurity risk management measures. The final version is awaited. (On the implementing regulation for certain types of entities, see my October post.)

4 Nov 24

QR code phishing: this is phishing by tricking people into scanning malicious QR codes to take them to malicious websites or install/open malicious apps/files, and it's an increasing attack vector. Microsoft explained how it updated its Microsoft Defender for Office 365 to address this

31 Oct 24

Incident response, preparations, resilience: helpful lessons on the Jul 24 Crowdstrike outage from the UK Financial Conduct Authority (FCA) with its observations on how FS firms responded to the incident including infrastructure resilience, third party management, incident response and communications, with recommendations on what firms should be doing on these fronts
Cybersecurity measures: ending Cybersecurity Awareness Month, Microsoft published 7 cybersecruity trends and (same old, same old!) its tips for SMEs:

1in 3 SMBs have suffered a cyberattack (Microsoft tips: strong passwords, MFA, consider password manager, recognise/report phishing, keep software updated i.e. patching)
Attacks cost them >$250k on average and up to $7m (tip: risk assessment to understand gaps, determine measures to address)
81% of SMBs think AI increases need for additional security controls (tip: data security & data governance when adopting AI)
94% think cybersecurity is business-critical (tip: educate/train employees e.g. using Microsoft awareness resources)
<30% manage security in-house (tip: it's common to engage a Managed Service Provider (MSP) for security support)
80% mean to increase cybersecurity spending, prioritising "data protection" [NB broader than in the GDPR sense] (tip: prioritise data protection, firewall, anti-phishing, ransomware & device/endpoint protection, access control, identity management e.g. via DLP, EDR, IAM)
68% feel secure data access is a challenge for remote workers (tip: measures to protect data and Internet-connected devices, app store downloads; no credential sharing by email/text only phone in real time)

25 Oct 24

People: article Human-Centered Cybersecurity Revisited: From Enemies to Partners: "Focusing on “enabling approaches” that treat humans as partners adds another layer of protection to our cybersecurity defenses"

Recall the well known 1999 paper, "Users are Not the Enemy" (cleaner link). How organisations consider and treat their staff and customers can help improve cybersecurity - or the opposite!

24 Oct 24

Web security, standards: a Princeton news item discusses a new security standard their researchers worked on. "The change centers on how web browsers and operating systems verify a website’s identity when establishing a secure connection. They rely on third party organizations known as certification authorities, who issue digital certificates of authenticity based on a website owner’s ability to demonstrate legitimate control over the website domain, usually by embedding a random value that the certification authority has provided. ...bad actors could easily sidestep those hurdles to obtain a fraudulent certificate for a website they do not legitimately control... it could target any website on the internet. Users had no way to spot the fraud since the certificates were real, even if their underlying facts had been forged. With a fraudulent certificate, criminals could attack users and route traffic to fake sites without anyone knowing... the fake site would look every bit as legitimate as the real one... By adopting the Princeton standard, certification authorities have agreed to verify each website from multiple vantage points rather than only one... [multi-perspective validation]", which will improve Internet/web security

23 Oct 24

Cybersecurity measures, certifications, supply chain: the UK's Cyber Essentials certification scheme, to encourage organisations to implement key essential cybersecurity measures (cyber hygiene), reached its 10 year anniversary.

It's great to hear it has been effective in improving cybersecurity: "Recent insurance data shows us that organisations with Cyber Essentials are 92% less likely to make a claim on their insurance than those without it". The NCSC noted, "This statistic underscores the scheme’s effectiveness in mitigating cyber risks". "Additionally, where organisations require their third parties to get Cyber Essentials, we know they experience fewer third party cyber incidents".
The full impact evaluation noted that Cyber Essentials:

is providing cyber security protection to organisations of all sizes, including larger organisations that use other schemes, standards and accreditations
helps to improve organisations’ awareness and understanding of the cyber security risk environment
has stimulated wider actions, good practice and behaviours among organisations that use it
is being actively used as part of supply chain assurance to inform the supplier selection process, instil confidence and demonstrate basic cyber hygiene to the market

The NCSC also added, "Cyber Essentials has played a crucial role in raising awareness about cyber security. An evaluation conducted as part of the 10-year review revealed that 85% of certified organisations reported a better understanding of cyber risks. This increased awareness has empowered businesses to take proactive measures in safeguarding their digital assets", and said "The data is clear, implementing the five controls significantly lowers the risk of experiencing a cyber incident. For organisations lacking the necessary in-house expertise, support is readily available through companies offering the NCSC-recognised Cyber Advisor Service"
Also, to improve supply chain security, procurement efficiency, consistent minimum standards, UK financial entities Barclays, Lloyds Banking Group, Nationwide, NatWest, Santander UK and TSB have stated that they will promote and incorporate Cyber Essentials in their supply chain risk management and they encourage other businesses to incorporate Cyber Essentials into their supplier requirements. (This would also "Spread greater cyber insurance coverage across supply chains through the provision of free cyber insurance, and incident response services, included with Cyber Essentials certification to qualifying organisations")

Comment: Contractually requiring suppliers/vendors/service providers to be certified is a helpful move in the right direction. Cyber Essentials measures are the bare minimum that organisations should take, are not difficult to implement, and would go a long way towards preventing or reducing the impact of cyber incidents, so all organisations should be certifying, or at least implementing those measures even if they don't get certified! Unlike with ISO standards, Cyber Essentials measures are freely available, whether implemented through self-assessment or (Plus) third-party audit.
Note: Cyber Essentials funding is offered to small organisations in certain sectors like AI, quantum, semiconductors etc., with certain criteria

2 Oct 24

Ransomware: Counter Ransomware Initiative (CRI) guidance for organisations experiencing a ransomware attack and organisations supporting them
Scanning, testing: interesting study on how external cybersecurity scanning data can enhance underwriting accuracy for the (re)insurance industry. This compared companies’ security controls with actual insurance claims, identifying key predictive factors including the organisation's IP address count and patching cadence (the speed at which it updated software to address vulnerabilities), that help forecast claims. Single Point of Failure (SPoF) data also highlighted dependencies on third-party services like AWS (cloud) and VPNs

While aimed at (re)insurance, scanning/pen testing obviously is also helpful if not essential for insured organisations, and same issues obviously affect their susceptibility to successful attacks, so keep your number of IP addresses limited to reduce exposure, and (not a new recommendation) patch ASAP!

25 Sept but UK press release 17 Oct 24

Quantum computing, encryption, financial services - risks to security & FS: G7 Cyber Expert Group's statement on planning for quantum computing's opportunities and risks (including to public key cryptography) and steps that financial entities should take

10 Sept 24

Cybercrime, data sharing: the UK ICO and National Crime Agency (NCA) signed a memo of understanding on how they'll collaborate to improve the UK's cyber resilience, including "The NCA will never pass information shared with it in confidence by an organisation to us without having first sought the consent of that organisation" and "We will support the NCA’s visibility of UK cyber attacks by sharing information about cyber incidents with the NCA on an anonymised, systemic and aggregated basis, and on an organisation specific basis where appropriate, to assist the NCA in protecting the public from serious and organised crime"

Things AI, Q4 2024

2025-01-10T17:14:00.016+00:00

Selected things AI, mostly from Q4 2024, are listed below in reverse chronological order, some with descriptions. See also Things AI, Oct 2024, AI Liability Directive links, and Data protection & cyber security, Oct 2024. This blog illustrates how much is going on and the fast pace of AI-related developments, and I repeat that I've curated the below and by no means included all or even most of what's been happening. Italy's Garante seems the most active supervisory authority in enforcing against AI-related matters under GDPR. (Note that * is used in possible "rude words" to prevent this blog being auto-blocked, e.g. by AI!)

30 Dec 24

Tools, open source: chipmaker NVIDIA acquired run:ai, and plans to open source its software, that helps customers/users "to orchestrate their AI Infrastructure, increase efficiency and utilization, and boost the productivity of their AI teams", so that it can be used even with non-NVIDIA GPUs, whether on-prem or in-cloud

23 Dec 24

Future: Stanford University's Human-Centered Artificial Intelligence (HAI) center issued its Predictions for AI in 2025: Collaborative Agents, AI Skepticism, and New Risks

20 Dec 24

Data protection, enforcement, training, Italy: following its investigation into OpenAI's ChatGPT (news item), Italy's Garante fined OpenAI €15m (this amount was said to take account of its cooperative attitude, suggesting it could have been much higher!) and ordered OpenAI to conduct a "6-month institutional communication campaign on radio, television, newspapers and the Internet. The content, to be agreed with the Authority, should promote public understanding and awareness of the functioning of ChatGPT, in particular on the collection of user and non-user data for the training of generative artificial intelligence and the rights exercised by data subjects, including the rights to object, rectify and delete their data. Through this communication campaign, users and non-users of ChatGPT will have to be made aware of how to oppose generative artificial intelligence being trained with their personal data and thus be effectively enabled to exercise their rights under the GDPR... in view of the fact that the company established its European headquarters in Ireland in the course of the preliminary investigation, the Data Protection Authority, in compliance with the so-called one stop shop mechanism, forwarded the procedural documents to the Irish Data Protection Authority (DPC), which became lead supervisory authority under the GDPR so as to continue investigating any ongoing infringements that have not been exhausted before the opening of the European headquarters"
AGI, reasoning: OpenAI's new o3 model scored a breakthrough high score for its performance in one Arc Prize challenge, although still not reaching AGI...
Risks, testing: the US NIST published a paper, The Assessing Risks and Impacts of AI (ARIA) Program Evaluation Design Document. "ARIA (Assessing Risks and Impacts of AI) is a NIST evaluation-driven research program to develop measurement methods that can account for AI’s risks and impacts in the real world. The program establishes an experimentation environment to gather evidence about what happens when people use AI under controlled real-world conditions". Its testbed involves model testing, red teaming and field testing. "Dialogues collected in the ARIA environment will be curated and anonymized and are planned to be publicly released after each evaluation series. The publication of ARIA’s methods, metrics, practices and tools will facilitate adoption and scaling across industry and research settings"
Testing, researchers: OpenAI invited safety researchers to apply to receive API access to its forthcoming frontier models, including o3-mini, to advance frontier safety

19 Dec 24

Agents: Anthropic has shared its learnings on how to build effective AI agents
Agents, genAI: Microsoft researchers submitted a paper on the current resurgence of agents and argue that "While generative AI is appealing, this technology alone is insufficient to make new generations of agents more successful. To make the current wave of agents effective and sustainable, we envision an ecosystem that includes not only agents but also Sims, which represent user preferences and behaviors, as well as Assistants, which directly interact with the user and coordinate the execution of user tasks with the help of the agents"

Assistants seem to be software that controls agents on behelf of users. Sims, that simulate users, seem to require some kind of user profiling, so data protection again...

AI usage, health, tools: such a beneficial use case, an Indian group has developed AI tools to assist with tuberculosis diagnosis and treatment
EU AI Act: second draft of GPAI Code of Practice published, based on feedback on the first draft. This "remains a work in progress. They focused primarily on providing clarifications, adding essential details, and aligning to the principle of proportionality, such as the size of the general-purpose AI model provider." Verbal discussions on the second draft with Chairs and Vice-Chairs are planned, and workshops with general-purpose AI model providers and Member State representatives in the AI Board Steering Group are planned for the weeks of 20 and 27 January respectively. The third draft of the Code of Practice is expected to be out in the week of 17 February 2025

The Computer & Communications Industry Association (CCIA) criticised the draft as containing "measures already explicitly rejected by EU co-legislators during the AI Act negotiations. Ideas previously dismissed that have been resurrected include mandatory third-party assessment and differentiated treatment between smaller and larger GPAI developers. If left unchecked, the Code risks becoming an undemocratic vehicle that overturns the AI Act’s legislative process. This second iteration also contains measures going far beyond the Act’s agreed scope, such as far-reaching copyright measures"

18 Dec 24

Environment, tools: Microsoft announced "SPARROW: A Breakthrough AI Tool to Measure and Protect Earth’s Biodiversity in the Most Remote Places"
Intellectual property, laws: in the UK Parliament's debate on the Data (Use and Access) Bill, various amendments were proposed on AI and copyright, and more..., we'll have to wait to find out which ones if any get through (and see 17 Dec)
Models, testing: from the UK Artificial Intelligence Safety Institute (AISI) and US Artificial Intelligence Safety Institute, a joint pre-deployment evaluation of OpenAI's o1 model (see 5 Dec)
Neurosymbolic AI, LLM hallucinations: good article Generative AI can’t shake its reliability problem. Some say ‘neurosymbolic AI’ is the answer, on "neurosymbolic AI, which its advocates say blends the strengths of today’s LLMs with the explainability and reliability of this older, symbolic approach"

An example of a neurosymbolic system is Google's impressive AlphaGeometry AI system (certain code), that "surpasses the state-of-the-art approach for geometry problems, advancing AI reasoning in mathematics... solves complex geometry problems at a level approaching a human Olympiad gold-medalist - a breakthrough in AI performance...". "AlphaGeometry is a neuro-symbolic system made up of a neural language model and a symbolic deduction engine, which work together to find proofs for complex geometry theorems. Akin to the idea of “thinking, fast and slow”, one system provides fast, “intuitive” ideas, and the other, more deliberate, rational decision-making. Because language models excel at identifying general patterns and relationships in data, they can quickly predict potentially useful constructs, but often lack the ability to reason rigorously or explain their decisions. Symbolic deduction engines, on the other hand, are based on formal logic and use clear rules to arrive at conclusions. They are rational and explainable, but they can be “slow” and inflexible - especially when dealing with large, complex problems on their own. AlphaGeometry’s language model guides its symbolic deduction engine towards likely solutions to geometry problems..."
As the Turing Institute puts it, "“sub-symbolic” or “neuro-inspired” techniques only work well for certain classes of problem and are generally opaque to both analysis and understanding... “symbolic” AI techniques, based on rules, logic and reasoning, while not as efficient as “sub-symbolic” approaches, have much better behaviour in terms of transparency, explainability, verifiability and, indeed, trustworthiness... “neuro-symbolic” AI has been suggested, combining the efficiency of “sub-symbolic” AI with the transparency of “symbolic” AI. This combination can potentially provide a new wave of AI tools and systems that are both interpretable and elaboration tolerant and can integrate reasoning and learning in a very general way"
Also see 2023 paper Neurosymbolic AI -- Why, What, and How, May 2024 article Neuro-Symbolic AI Could Redefine Legal Practices

Future: is AI progress slowing down?

17 Dec 24

GenAI, contract, usage: Google updated its generative AI prohibited use policy to add "clear examples of conduct that is not acceptable. For example, we’re explicitly stating that using our tools to create or distribute non-consensual intimate imagery or to compromise security by facilitating phishing or malware is not allowed under the policy. Finally, we've added language that allows for exceptions for certain educational, artistic, journalistic or academic use cases that might otherwise violate our policies"
Data protection (and see next point): the EDPB adopted its Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models (see my blog on its stakeholder workshop, legitimate interests, and other important data protection issues)

This consistency opinion was requested by the Irish DPC, who welcomed it

Intellectual property, laws: the UK opened a public consultation on copyright and AI (news, deadline 25 Feb 25), seeking views on how to achieve its objectives for the AI and creative sectors of 1. Supporting right holders’ control of their content and ability to be remunerated for its use; 2. Supporting the development of world-leading AI models in the UK by ensuring wide and lawful access to high-quality data; 3. Promoting greater trust and transparency between the sectors (see also 18 Dec). It is considering:

Measures that would require increased transparency from AI developers. "This includes the content they use to train their models, how they acquire it, and any content generated by their models" (training data information including provenance, etc.), and
Introduction of an exception to copyright law for “text and data mining”, to improve access to content by AI developers but allow right holders to reserve their rights and thereby prevent their content being used for AI training - similar to the EU’s exception for text and data mining",under Art.4, Digital Single Market Copyright Directive (Directive (EU) 2019/790), but with rights reservations "using effective and accessible machine-readable formats" like robots.txt or metadata

Some thoughts/queries here on data protection:

The consultation document does mention data protection where personal data is used to train AI models or appears in AI-generated outputs (174), & that the ICO will issue further guidance on genAI (179)
It notes existing protection for personality: the passing off tort against misrepresentation in a commercial environment; some IP rights like copyright may help people control/prevent digital replicas, eg sound recording rights if training on a singer’s recorded voice, performance rights of film actors, singers.
The UK isn't seeking but would "welcome views" on intellectual property protection for personality rights (178).
Problem: IP rights, like the US right of publicity personality right, protect commercial proprietary interests, NOT human rights, and are often signed away to music/film companies. US rights of publicity/personality vary with the state, and generally protect only celebrities who can commercially exploit their well-known images, etc.
So, for ordinary UK/EU people, is GDPR adequate? Yes, using photo/video/audio to create a replica of someone's likeness is processing their personaldata, but could it be made clearer (e.g. by supervisory authorities) that deploying those replicas, e.g. in ads, scams etc, also constitutes "processing" of personal data, particularly if deepfakes are used in ways the person wouldn't agree to?
As regards GDPR remedies, shouldn't people have a right to prevent their deepfaked voice, image etc being used without their knowledge or consent, even if they're not famous? If legitimate interests is the claimed legal basis for using their likeness/voice, there's certainly a right to object, and compensation claims may be possible, but should there be additional positive rights there, e.g. to require hosting providers to take down deepfakes?
Real examples of deepfake misuse (there are many more): BBC presenter; social media user. Not to mention faces (almost invariably female) being used in deepfaked nudes, porn etc (Note: on 7 Jan, the UK government stated it would introduce a new offence for creating s**ually explicit deepfake images, plus other offences, as promised in the Labour Party manifesto p.35)

Policy, risks, US: the US Congress's bipartisan House Task Force on Artificial Intelligence published its report on AI (news item). This long report covered the full range of AI-related concerns: US government use of AI, federal preemption of state law, data privacy, national security, research, development & standards, civil rights & civil liberties, education & workforce, intellectual property, content authenticity, open and close systems, energy usage and data centers, small business, agriculture, healthcare and financial services
Transparency, AI usage: the UK government published information (using the UK's Algorithmic Transparency Recording Standard (ATRS); guidance, more on ATRS) on 14 further UK algorithmic tools used in public sector decision-making (news item). This may or may not have been in response to news reports about lack of transparency regarding UK government tools, see 28 Nov

The list is worth a skim, there have been many different uses, not limited to chatbots
The UK also published a mandatory scope and exemptions policy for the ATRS, setting out which organisations and algorithmic tools are in scope of the mandatory requirement to publish ATRS records (i.e. central government departments and Arm’s-length-bodies (ALBs): executive agencies and non-departmental public bodies which provide public or frontline services or routinely interact with the general public), and "the required steps to ensure that sensitive information is handled appropriately"

16 Dec 24

GenAI, images: Google released its experimental Whisk, allowing users to prompt using images instead of text to create new images. "Whisk lets you input images for the subject, one for the scene and another image for the style"

13 Dec 24

Agents: Google announced Google Agentspace for its cloud services, including "agents that bring together Gemini’s advanced reasoning, Google-quality search and enterprise data, regardless of where it’s hosted"
AI risks, taxonomy: the UK AI Standards Hub discussed developing a taxonomy of AI risks for organisations, including a proposed table mapping AI risk sources to AI hazards
Deepfakes, misinformation: ...Political Misinformation is not an AI Problem. "Technology Isn’t the Problem—or the Solution... There’s no quick technical fix, or targeted regulation, that can “solve” our information problems. We should reject the simplistic temptation to blame AI for political misinformation and confront the gravity of the hard problem". As always, it's humans, not tech - tech can't solve issues with how humans think and act! (see also 9 Dec)
Laws: Lord Clement-Jones's Public Authority Algorithmic and Automated Decision-Making Systems Bill had its second reading (debate); the date for its committee stage is TBA
SLMs, reasoning: Microsoft announced its Phi-4 "Small Language Model Specializing in Complex Reasoning", that "offers high quality results at a small size (14B parameters)" (and see 4 Dec). SLMs of course are more suitable for on-device processing than LLMs
SLMs: Epoch AI's article, Frontier language models have become much smaller. "...in 2023, the trend of frontier language models becoming bigger reversed... Should we expect frontier models to keep getting smaller? The short answer is probably not, though it’s harder to say if we should expect them to get much bigger than GPT-4 in the near term."

12 Dec 24

AI usage, LLMs, risks, privacy, PETs: Anthropic discussed its Claude insights and observations (Clio): "...an automated analysis tool that enables privacy-preserving analysis of real-world language model use. It gives us insights into the day-to-day uses of claude.ai... It’s also already helping us improve our safety measures... Clio takes a different approach, enabling bottom-up discovery of patterns by distilling conversations into abstracted, understandable topic clusters. It does so while preserving user privacy: data are automatically anonymized and aggregated, and only the higher-level clusters are visible to human analysts" (paper)
EU AI Act, jobs!: unsurprisingly, the EU AI Office is recruiting for legal or policy officers, deadline 15 Jan 25
GenAI, data protection: while publishing its response to its consultation series on generative AI,the UK ICO emphasised that genAI developers must provide proper privacy notices. "...it’s time to tell people how you’re using their information. This could involve providing accessible and specific information that enables people and publishers to understand what personal information has been collected. Without better transparency, it will be hard for people to exercise their information rights and hard for developers to use legitimate interests as their lawful basis".

However, unfortunately there's still little clarity from the ICO on how, if at all, the Art.14(4)(b) exemption could apply, where providing the information is "impossible or would involve a disproportionate effort", even though that was a key point raised by some consultation respondents

Testing: the AI Alliance's Trust and Safety Evaluations Initiative, still in draft, was updated to v0.3.1 (initial version was in Oct 24 so quite new), covering: terms glossary, user personae, taxonomy of evaluations & assessments, evaluators and benchmarks, leaderboards and evaluation platform reference stack

Data: the AI Alliance also has an Open Trusted Data Initiative (OTDI) with many datasets available; "our mission is to create a comprehensive, widely-sourced catalog of datasets with clear licenses for use, explicit provenance guarantees, and governed transformations, intended for AI model training, tuning, and application patterns like RAG (retrieval augmented generation) and agents"
Skills: the Alliance has produced a Guide to Essential Competencies for AI "to help bridge the AI divide... support a framework for education and training curricula and... help promote inclusive access to AI education. These competencies include the responsible and ethical use of AI, identifying data limitations, data analysis, machine learning, and AI logic, and range in levels from fluency to proficiency to expertise, and finally, mastery"

11 Dec 24

Agents: Google announced its Gemini 2.0, its "new AI model for the agentic era"
AI usage, deepfakes, comms: Wired article "OnlyFans Models Are Using AI Impersonators to Keep Up With Their DMs", "AI is replacing the humans who pretend to be OnlyFans stars in online amorous messages". If there's tech, people will use it - in all sorts of ways!
AI usage, health: Meta outlined use of its open source Llama model by researchers and developers to address various issues, "from gaps in clinical cancer trials to inefficiencies in agriculture"
OSs, mobile, genAI: Apple expanded its Apple Intelligence AI capabilities for iPhone, iPad, Mac

However, reportedly this AI has hallucinated inaccurate information when summarising breaking news alerts, and Apple stated it would update the AI feature

Sustainability, water, data centers: Techradar report on Microsoft's new "zero-water data center cooling design", including reuse of cooling water

10 Dec 24

Adoption, data quality: Ataccama Data Trust Report 2025 (registration wall), reported by Freevacy as finding, from a survey of 300 US, Canada, and UK data leaders, that "while 74% of responding organisations have adopted some AI-based solutions, only 33% have successfully integrated them across the company", suggesting that "54% of respondents, with 72% of data strategy decision-makers particularly concerned, feel the pressure of not implementing AI effectively, fearing a potential loss of competitive edge". 51% of executives prioritised data quality and accuracy improvements, 30% cited challenge of managing large data volumes
AI usage, education: the UK's Ofsted is researching how AI is used in education, barriers, challenges and potential benefits
Clean energy, data centres, sustainability: Google announced a strategic partnership with other organisations "to synchronize new clean power generation with data center growth in a novel way", developing "industrial parks with gigawatts of data center capacity in the U.S., co-located with new clean energy plants to power them"
Employment, recruitment, bias: IBM on AI in recruitment, how intersectionality (e.g. female AND non-white) can compound bias in AI-based systems, prioritising data diversity and ongoing monitoring
EU: the European High Performance Computing Joint Undertaking (EuroHPC) selected seven proposals to establish and operate the first AI Factories across Europe (Finland, Germany, Greece, Italy, Luxembourg, Spain and Sweden), most with "AI-optimised supercomputers": a major milestone for Europe in building a thriving ecosystem to train advanced AI models and develop AI solutions", that will "provide access to the massive computing power that start-ups, industry and researchers need to develop their AI models and systems. For example, European large language models or specialised vertical AI models focusing on specific sectors or domains" (news release)

9 Dec 24

GenAI, video: OpenAI moved its Sora video generation model out of research preview
Future, agents: interesting Quartz article on AI in 2025, tech companies touting AI agents cf returns on investment in AI, "Wall Street isn't convinced"
No panacea..., testing: Can AI Break the (Mathematical) Law? "...recent research claims to escape the inescapable — to exit the accuracy-error trade-off with artificial intelligence and machine learning (AI/ML). Are these breakthroughs, or illusions?... The promise of AI/ML is not to use data to escape the structure of the world, but to work more wisely within it. The challenge is knowing when and how to use it — recognizing the implications of universal mathematical laws, aligning with clearly defined policy goals and values, and testing as we go to assess what costs and benefits really accrue to whom". Again, how humans understand and use AI is critical! (See also 13 Dec)
Products, data centers, sustainability: physical products are increasingly AI-enabled. A letter sent by Lord Leong, Department for Business & Trade (DBT), on the UK Product Regulation & Metrology Bill, includes: "...The government and regulators will need to consider what specific requirements will be needed on products using Al, to safeguard their safety, as our understanding of their risks increases... the Bill ensures our product safety framework can take into account risks presented by the use of software and Al in physical products... product regulations made under the Bill will be able to regulate production processes as a whole if required. This will also include any use of Al as part of these processes and the regulations may also set labelling requirements for products containing Al. Regarding use of Al in the creative industries, including challenges to personal data and intellectual property, the Bill does not seek to regulate Al in and of itself. Nor does our product regulation framework cover the creative industries, beyond ensuring any physical product operates safely [refs to planned consultation on copyright & AI, see 17 & 18 Dec] ...Issues of Al and metrology were also raised, specifically on measurement of the power and water use of Al processes or data centres. The new powers set out in the Bill will allow metrology regulations to be updated to allow us to respond to technological advances, such as in Al... [and to ensure such measurements are accurate]"

6 Dec 24

Bias (age, disability, nationality, marital status): a tool to detect welfare fraud, used by the UK's Department for Work and Pensions (DWP), was reported as "showing bias according to people’s age, disability, marital status and nationality". Although humans make the final decisions on welfare payments, it was noted that no "fairness analysis has yet been undertaken in respect of potential bias centring on race, sex, sexual orientation and religion, or pregnancy, maternity and gender reassignment status"
Cybersecurity, LLMs, tools, cloud: Amazon researchers discuss a new model "that harnesses advanced AI capabilities to automate the creation of security controls, enabling faster, more efficient, and highly accurate generation of the rules [for AWS services' configuration and alerts processing] that help users safeguard their cloud infrastructures", in a post entitled "Model produces pseudocode for security controls in seconds"
GenAI, usage, law: the UK's Government Skills is piloting and seeking feedback on an AI-generated video used in its updated Civil Service expectations course as an optional module to help civil servants understand certain new legal responsibilities on equality: "the first time an AI-generated video has been used to enhance learning on one of the cross-government courses hosted on Civil Service Learning". Hmm, AI creating a video to explain/teach new laws... I'd like to know what the feedback was too! (recall that in a UK tax tribunal appeal, a party cited "supporting" rulings that were actually hallucinated by AI)
On-device ML, SLMs: Microsoft explained its "small but mighty" on-device Small Language Model, Phi Silica (and see 13 Dec)

5 Dec 24

Intellectual property, courts: Impact and Value of AI for IP and the Courts, speech by UK Deputy Head of Civil Justice
OpenAI, o1, testing, red teaming: system card released for OpenAI's o1 and o1-mini, describing safecoty work including external red teaming and frontier risk evaluations under OpenAI's Preparedness Framework (and see 18 Dec)
Training data, privacy, PETs: NIST discussion with certain winners of the UK-US PETs Prize Challenges, on "real-world data pipeline challenges associated with privacy-preserving federated learning (PPFL) and explore upcoming solutions. Unlike traditional centralized or federated learning, PPFL solutions prevent the organization training the model from looking at the training data. This means it’s impossible for that organization to assess the quality of the training data – or even know if it has the right format. This issue can lead to several important challenges in PPFL deployments"

4 Dec 24

Agents, gaming: Google introduced its Genie 2, "a foundation world model capable of generating an endless variety of action-controllable, playable 3D environments for training and evaluating embodied agents. Based on a single prompt image, it can [create a video game that can] be played by a human or AI agent using keyboard and mouse inputs". Fascinating and impressive
AI usage, science, weather: Probabilistic weather forecasting with machine learning article on Google's GenCast model (authors include many Google researchers; open GenCast code on GitHub including links for data and model weights)
Bias (gender), LLMs: Apple paper Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models. Language used in prompts matters. "Our findings highlight the importance of ensuring fairness in pre-trained LLMs, especially when they are later used to perform downstream tasks via prompt adaptation"
Employment, recruitment, data protection: the UK ICO published the 42 advisory notes that it issued following audits of organisations using AI in recruitment. Well worth a read for the points to consider/document if you're thinking of using AI for hiring
International cooperation: GPAI Belgrade Ministerial Declaration on AI issued (yes, it's somewhat confusing that "GPAI" here stands for "Global Partnership for Artificial Intelligence" and not general purpose AI as per the EU AI Act! There are currently 29 countries that are members of GPAI including the EU and many EU Member States, UK, US and also Australia, Canada, Singapore, New Zealand etc)
Health, medical devices, sandboxes: to "help test and improve the rules for AI-powered medical devices to ensure they reach patients quickly, safely and effectively" and improve diagnosis and patient care, the UK Medicines and Healthcare products Regulatory Agency (MHRA) announced its selection of 5 AI technologies for its pilot AI Airlock regulatory sandbox scheme, "where manufacturers can explore how best to collect evidence that could later be used to support the approval of their product" under MHRA supervision in a virtual or simulated setting. The chosen AI uses were: targeting at risk patients with Chronic Obstructive Pulmonary Disease (COPD); using LLMs to improve the efficiency and accuracy of radiology reporting; AI performance monitoring platforms in hospitals (for drift); improving cancer care efficent; facilitating clinician decision-making

3 Dec 24

Fundamental concepts: Dentons has published my video, initially intended only for internal training of lawyers generally on AI terminology and jargon
Misrepresentating AI, biometrics, bias: FTC enforcement action "for making false, misleading or unsubstantiated claims that its AI-powered facial recognition software was free of gender and racial bias and making other misleading claims about the technology"
Multimodal: Amazon announced its Nova generation of multimodal foundation models. "With the ability to process text, image, and video as prompts, customers can use Amazon Nova-powered generative AI applications to understand videos, charts, and documents, or generate videos and other multimedia content"

2 Dec 24

Sustainability, data centers: reportedly Amazon has partnered with AI startup Orbital to trial a new (AI-designed!) material in its data centres for carbon capture, removing carbon dioxide from the air (another article)

29 Nov 24

Tools: (algorithms for improved statistical models, not AI?) software to "equip individual insurance firms to assess probable liabilities arising from their specific mix of products and customers" and estimate needed cash reserves better for Solvency II purposes. Unlikely to be free/open source

28 Nov 24

GenAI, LinkedIn, detection, synthetic data: Wired article on Originality AI's analysis showing that "Over 54 percent of longer English-language posts on LinkedIn are likely AI-generated", "indicating the platform’s embrace of AI tools has been a success" (!)
Government policy: ODI's Global Policy Observatory for Data-centric AI report

"Analysing 512 policy documents from 64 countries, we find that a small group of typically wealthier nations with robust open data practices are more likely to focus on data-centric AI topics. In contrast, low and middle-income countries (LMICs), which generally lack a focus on data-centric AI topics, may find their ability to engage in global AI governance efforts hindered."
Recommends: international bodies supporting LMICs with guidance; investment in data-centric tools/toolkits; promoting equitable data sharing

Human rights, risks, assessment, governance: the Council of Europe's Committee on AI (CAI) adopted the HUDERIA methodology for the risk and impact assessment of AI systems from the point of view of human rights, democracy and the rule of law, with the involvement of Turing Institute researchers (Turing news with further links; "in 2025-2026 a more detailed model will be developed and piloted"). HUDERIA comprises a context-based risk analysis (COBRA), stakeholder engagement process (SEP), risk & impact assessment, and mitigation plan

EU: the EU was involved in negotiating HUDERIA as a non-binding instrument (NBI) "to support the Parties to the Council of Europe Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law [explan memo] in the implementation of the risk and impact management obligations included in Chapter V of the Convention", and HUDERIA took account of the EU AI Act. HUDERIA's approval by the EU is progressing. Current signatories to this Convention include the EU, UK and USA
EU AI Act: HUDERIA should be useful when considering fundamental rights impact assessments of high-risk AI systems under the AI Act. Indeed, the HUDERIA document itself states, "The HUDERIA can be used by both public and private actors to aid in identifying and addressing risks and impacts to human rights, democracy and the rule of law throughout the lifecycle of AI systems", and that its main objectives include "to promote compatibility and interoperability with existing and future guidance,standards and frameworks developed by relevant technical, professional and other organisations or bodies (such as ISO, IEC, ITU, CEN, CENELEC, IEEE, OECD, NIST), including the NIST AI Risk Management Framework and risk management and fundamental rights impact assessment under the EU AI Act"

28 Nov 24

EU AI Act: AI Office's AI Pact webinar on the AI Act's architecture - recording, slides

EU AI Pact (voluntary): including list of signatories

Government policy: the Open Data Institute (ODI) report analysing 512 policy documents from 64 countries - "we find that a small group of typically wealthier nations with robust open data practices are more likely to focus on data-centric AI topics. In contrast, low and middle-income countries (LMICs), which generally lack a focus on data-centric AI topics, may find their ability to engage in global AI governance efforts hindered". Recommendations:

Support LMICs via international organisations' guidance on digital infrastructure and accessing high quality data resources
Invest in data-centric tools/toolkits
Promote equitable data sharing

Transparency on AI use: reportedly the UK government was failing to list is use of AI on mandatory register (now see 17 Dec)

27 Nov 24

Data protection, training, enforcement, Italy: if "selling" personal data for AI training, GDPR still applies. Italy's Garante issued a warning to a news group that sharing its editorial content (including archive) with OpenAI could likely infringe GDPR's provisions on special category data, criminal offence data, privacy notices and data subject rights. (Machine translation) "all editorial content will be used by OpenAI to allow users [of the ChatGPT service, ed.] to carry out real-time searches for current news, with the simultaneous provision of a summary (generated by OpenAI artificial intelligence systems) and a direct link to the news item itself” and that “all editorial content will also be used by OpenAI to improve its services and train its artificial intelligence algorithms”.

From the associated news item of 29 Nov (machine translation): "Digital newspaper archives store the stories of millions of people, with information, details, even extremely sensitive personal data that cannot be licensed for use by third parties to train artificial intelligence, without due precautions... Based on the information received, the Authority believes that the processing activities are intended to involve a large volume of personal data, including sensitive and judicial data, and that the impact assessment [DPIA], carried out by the company and transmitted to Garante, does not sufficiently analyze the legal basis by virtue of which the publisher could transfer or license for use by third parties the personal data present in its archive to OpenAI, so that it can process them to train its algorithms [the DPIA cited legitimated interest]. Finally, the warning notice highlights how the information and transparency obligations towards the interested parties do not appear to have been sufficiently fulfilled and that [the controller] is not in a position to guarantee the latter the rights they are entitled to under European privacy legislation, in particular the right to object"

Human rights: are rights sufficiently human in the age of the machine? - speech by UK Master of the Rolls

26 Nov 24

Data centers, clean energy, sustainability: concerns that increasing power demands from data centres for AI processing will delay the world's transition to clean energy (some companies are going for nuclear power to reduce carbon emissions, even banks not just big tech: paywalled FT article)
Misrepresenting AI capabilities: US FTC action over allegations of false claims about the extent to which AI-powered security screening system can detect weapons and ignore harmless personal items, including in school settings
Public sector, government, AI usage: Google Cloud-commissioned report on genAI - news item, summary, PDF
Shadow AI: Strategy Insights survey - over a third of organisations struggled to monitor use by employees (even cybersecurity staff) of non-approved AI tools, particularly those integrated with legacy systems
Testing, red teaming: US Cybersecurity and Infrastructure Security Agency (CISA) on how AI red teaming (third-party safety and security evaluation of AI systems) must fit into the existing framework for AI Testing, Evaluation, Validation and Verification (TEVV) and into software TEVV

25 Nov 24

National security, defence, cybersecurity: the UK announced it will part-fund a new Laboratory for AI Security Research (LASR) that will "partner with world-leading experts from UK universities, the intelligence agencies and industry to boost Britain’s cyber resilience and support growth... to assess the impact of AI on our national security" including collaboration with Five Eyes countries and NATO allies
Open source, tools, models, data: Anthropic released its open source model context protocol (MCP), "an open standard that enables developers to build secure, two-way connections between their data sources and AI-powered tools"

22 Nov 24

Court judgments, genAI, Argentina: replacing a previous AI system PROMETEA, OpenAI's ChatGPT is now being used in Buenos Aires for contentious administrative and tax matters, reviewing uploaded case documents and drafting judgments (the first cut only, with human review, it seems): "20 rulings it has drafted have all been reviewed by a lawyer and approved by the deputy attorney"
Work, employees: OECD Global Deal group on social dialogue & workplace use of AI

21 Nov 24

AI PCs: IBM survey - AI PCs offered "potentially transformative impact on people’s lives, saving individuals roughly 240 minutes a week on routine digital tasks", but "current AI PC owners spend longer on tasks than their counterparts using traditional PCs", more consumer education needed
Financial services: third survey by UK Bank of England & Financial Conduct Authority on AI and machine learning in UK FS including use/adoption, third-party exposure, automated decision-making, materiality, understanding of AI systems, benefits/risks, constraints on use (data protection top), governance & accountability. The exec summary is worth a read if short on time!
Countries: Stanford University's Human-Centered Artificial Intelligence (HAI) center's 2024 Global AI Power Rankings: Stanford HAI Tool Ranks 36 Countries in AI using its Global Vibrancy Tool; "the U.S. is the global leader in artificial intelligence, followed by China and the United Kingdom. The ranking also highlights the rise of smaller nations such as Singapore when evaluated on both absolute and per capita bases..."
Red teaming: OpenAI published 2 papers on external and automated red teaming of AI models/systems
Religion!: Swiss church installs AI-powered Jesus instead of priests

20 Nov 24

Cybersecurity: EU Cyber Resilience Act (CRA) published in OJ, aiming to improve tge cyber security of "products with digital elements" by regulating the making available of such products on the EU market to ensure their cybersecurity, including "essential cybersecurity requirements" for their design, development and production, obligations for economic operators in relation to their cybersecurity, and cybersecurity requirements for vulnerability handling processes by their manufacturers during the time they are expected to be in use, and obligations for economic operators regarding those processes. Significance for AI? Art.12, Rec.51:

Products with digital elements classified as high-risk AI systems under the EU AI Act will be deemed to comply with the AI Act's Art.15 cybersecurity requirements if the product and the manufacturer's processes meet the CRA's essential cybersecurity requirements in CRA Annex I, but only to the extent the CRA declaration of conformity "demonstrates" achievement of the level of cybersecurity protection Art.15 requires. The assessment should take account of risks to an AI system's cyber resilience as regards attempts by unauthorised third parties to alter its use, behaviour or performance, including AI specific vulnerabilities such as data poisoning or adversarial attacks, as well as risks to fundamental rights, in accordance with the AI Act
There are nuances, e.g. "important products with digital elements" (Art.7, Ann.3) and "critical products with digital elements" (Art.8, Ann.4) that are high-risk AI systems will be subject to the CRA's conformity assessment procedures in so far as the CRA's essential cybersecurity requirements are concerned, but the AI Act's conformity assessment for all other aspects
Manufacturers of products with digital elements classified as high-risk AI systems under the AI Act may participate in AI regulatory sandboxes under the AI Act

EU AI Act: AI Office's updated GPAI models' Code of Practice FAQ / Q&A
GenAI, synthetic data, deepfakes, provenance, detection, risks, testing, management: NIST report Reducing Risks Posed by Synthetic Content An Overview of Technical Approaches to Digital Content Transparency, on existing standards, tools, methods, and practices, and potential development of further science-backed standards and techniques, for: authenticating content and tracking its provenance; labeling synthetic content, such as using watermarking; detecting synthetic content; preventing genAI from producing CSAM or non-consensual intimate imagery of real individuals (including intimate digital depictions of an identifiable individual's body/body parts); testing software used for the above purposes; and auditing and maintaining synthetic content.
Health, medical images: experts' alarm over people being encouraged to upload medical scans to Grok AI, including concerns on transparency, privacy, accuracy (reportedly it misidentified broken clavicle for dislocated shoulder, didn't recognise tuberculosis, mistook benign cyst for test*cles)
International cooperation: EU and Singapore signed an Administrative Arrangement on cooperation between the EU’s AI Office and Singapore’s AI Safety Institute, "to address the safety of general-purpose AI models through information exchanges and best practices, joint testing and evaluations, development of tools and benchmarks, standardisation activities, as well as research on how to advance safe and trustworthy AI" and to exchange views on trends and future technological developments in the field of AI
US, government policy, laws: the Computer & Communications Industry Association (CCIA) published its 2024 State Landscape Artificial Intelligence, outlining the major trends across the 50 US state legislatures, and highlighting key states expected to be active in the upcoming session

19 Nov 24

Models, testing: from the UK Artificial Intelligence Safety Institute (AISI) and US Artificial Intelligence Safety Institute, a joint pre-deployment evaluation of Anthropic's upgraded Claude 3.5 Sonnet model (see also 22 Oct 24)

18 Nov 24

AI usage, science, health, weather, maths, physics: Google highlighted some scientific breakthroughs enabled by AI and newly-developed AI models - protein structure prediction (Deepmind, free database) to assist developing new medicines, fight antibiotic resistance, tackle plastic pollution; mapping human brain in more detail to assist health research (public dataset); more accurate flood forecasting to help save lives; spotting wildfires earlier to help stop them faster; predicting weather more quickly and accurately (open source model code); advancing AI's mathematical & geometry reasoning (AlphaGeometry2, AlphaProof); more accurate predictions of chemical reactivity/kinetics using quantum computing for chemistry simulations; accelerating materials science which could help produce more sustainable solar cells, batteries, superconductors (some predictions in an open database); assisting nuclear fusion research
AI usage, potential: views of Google's Demis Hassabis & James Manyika - AI will help us understand the very fabric of reality
Data protection, LIAs: Information Accountability Foundation (IAF) published Assessments for an AI World - Legitimate Interest Assessment, with a draft model LIA on "how to demonstrate legitimate interest multi-dimensional balancing when AI is the processing activity"
Future, limits: view that genAI has "hit a dead end" and will stagnate...
Product liability: EU Product Liability Directive published in the OJ, applicable from 9 Dec 2026. Significance for AI? For the purposes of no-fault liability for defective products:

"Product" explicitly includes software like AI systems, including where supplied via SaaS
"developer or producer of software, including AI system providers" under the AI Act, "should be treated as a manufacturer" under this Directive
Where a substantial modification is made e.g. due to the continuous learning of an AI system, the substantially modified product should be considered to be made available on the market or put into service at the time that modification is actually made
National courts should presume the defectiveness of a product or the causal link between the damage and the defectiveness, or both, where, notwithstanding the defendant’s disclosure of information, it would be excessively difficult for the claimant, in particular due to the technical or scientific complexity of the case, to prove the defectiveness or the causal link, or both. Technical or scientific complexity should be determined by national courts on a case-by-case basis, taking into account various factors including the complex nature of the technology used such as machine learning, and the complex nature of the causal link, such as a link that, in order to be proven, would require the claimant to explain the inner workings of an AI system. While a claimant should provide arguments to demonstrate excessive difficulties, proof of such difficulties should not be required. For example, in a claim concerning an AI system, the claimant should, for the court to decide that excessive difficulties exist, neither be required to explain the AI system’s specific characteristics nor how those characteristics make it harder to establish the causal link.

17 Nov 24

Deepfakes: Sir David Attenborough against his voice being cloned by AI

14 Nov 24

AI usage, phone scams, fraud: an excellent beneficial use of AI is telecomms provider O2's AI "granny", Daisy, trained on "real scambaiter content" and designed to "answer calls in real time from fraudsters, keeping them on the phone and away from customers for as long as possible... [Daisy's] mission is to talk with fraudsters and waste as much of their time as possible with human-like rambling chat to keep them away from real people, while highlighting the need for consumers to stay vigilant as the UK faces a fraud epidemic... Able to interact with scammers in real time without any input from her creators, O2 has put Daisy to work around the clock answering dodgy calls. Daisy combines various AI models which work together to listen and respond to fraudulent calls instantaneously and is so lifelike it has successfully kept numerous fraudsters on calls for 40 minutes at a time... [Daisy] has told frustrated scammers meandering stories of her family, talked at length about her passion for knitting and provided exasperated callers with false personal information including made-up bank details. By tricking the criminals into thinking they were defrauding a real person and playing on scammers biases about older people, Daisy has prevented them from targeting real victims and, most importantly, has exposed the common tactics used so customers can better protect themselves"

Query: exactly how is Daisy triggered to answer a call? Can O2 customers opt in to use Daisy? The accompany video (see link above or Youtube) says, "If you want to help Daisy and O2 to ruin a scammer's day, you can report scam numbers to 7726". Does that mean that if calls are made to O2 customers' numbers from numbers reported to 7726 that O2 has determined are used by fraudsters, O2 intervenes and has Daisy answer the call instead of the customer?
As one customer said, do we really want O2 to be intercepting calls? More info on how Daisy works would be very helpful. I can imagine a troublemaker deliberately reporting a legitimate person's phone number to 7726, so that when that person tries to call an O2 number, they get Daisy instead of whoever they were trying to call!

Cybersecurity: from the UK AISI, Safety case template for ‘inability’ arguments - "How to write part of a safety case showing a system does not have offensive cyber capabilities" (paper)
EU AI Act: first draft of GPAI Code of Practice published

There's also a tentative future chronological timeline of next steps on the Code

GenAI poetry: research indicating that AI-generated poetry is indistinguishable from human-written poetry and even rated more favorably by the people involved! (Forbes article)
Frameworks, critical infrastructure: from the US DHS, consulting with AI Safety & Security Board: Roles and Responsibilities Framework for Artificial Intelligence in Critical Infrastructure, on deploying AI in critical infrastructure (but more generally useful) and the roles of cloud infrastructure providers, AI developers, critical infrastructure owners & operators, civil society and the public sector/government. Includes AI roles/responsibilities matrix and glossary.
Innovation, risks, red-teaming: from UK DSIT Responsible Technology Adoption Unit (RTA/RTAU), a Model for Responsible Innovation "to help teams across the public sector and beyond to innovate responsibly with data and AI" (but of interest to private organisations too) by:
- Setting out a vision for what responsible innovation in AI looks like, and the component Fundamentals and Conditions required to build trustworthy AI
- Providing a practical tool public sector teams can use to rapidly identify potential risks associated with AI development and deployment, and understand how to mitigate them - RTA uses this model in red-teaming workshops mapping data and AI projects against the model to rapidly identify where risks might arise and prioritise actions to ensure a trustworthy approach.
Product safety: unsurprisingly, AI and machine learning feature in the first horizon scan report by the UK Office for Product Safety & Standards (OPSS), and the scan's taxonomy of technologies included computational tools and platforms that collect, analyse or leverage data, including AI and machine learning (advanced analysis and algorithmic technologies that can interpret existing information and automate or support decision-making and action like AI, ML, neural networks, computer vision), cybersecurity and data platforms, and smart technology and internet of things (IoT)

13 Nov 24

Data protection, bias, Italy: Garante fined a food delivery company €2.6m and ordered various measures e.g. changing how it processes its riders' data through a digital platform and "verify that the algorithms used to book and assign orders for food and other products do not result into discrimination" including on automated decision-making (news item in English; another). Infringements noted included privacy notices/transparency, safeguards to ensure accuracy and fairness of algorithmic results used to rate riders' performance, lack of procedures to enforce the right to human interventionand contest the algorithms' decisions (which sometimes excluded riders from work assignments)
EU AI Act: the European Commission launched its consultation (ending 11 Dec 24) on AI Act prohibitions and the AI system definition
GenAI, media literacy: Ofcom paper on Future Technology and Media Literacy: Applications of Generative AI, covering the news sector & personalisation, personalisation & adaptation, content creation & education, and data protection concerns with genAI
Government policy, AI risks, AI benefits: OECD report Assessing Potential Future Artificial Intelligence Risks, Benefits nd Policy Imperatives
Justice, law enforcement: EU Council's draft conclusions on the use of AI in the field of justice (final conclusions from 4 Dec 24 meeting not yet available)

12 Nov 24

Bias, Denmark: Amnesty report Coded Injustice: Surveillance and Discrimination in Denmark’s Automated Welfare State, on "how the sweeping use of fraud detection algorithms, paired with mass surveillance practices, has led people to unwillingly –or even unknowingly– forfeit their right to privacy, and created an atmosphere of fear" (Fortune article)
Bias, Netherlands, ethnicity: reportedly the Dutch government is refunding >10k students "who were unjustly flagged for student finance fraud by an algorithm developed by the Education Executive Agency (DUO)" after an investigation found it to be "discriminatory, targeted students based on arbitrary risk factors that disproportionately affected those from immigrant backgrounds, particularly those of Turkish and Moroccan descent"

11 Nov 24

Automated decision-making (ADM): controversy over a UK Home Office tool proposing enforcement against migrants/asylum seekers. Government said its algorithms are rules-based not AI/ML and that humans remain responsible; objectors fear rubber-stamping of biased decisions, raised transparency regarding AI use.
Bias (age), health: does the UK’s liver transplant matching algorithm systematically exclude younger patients?
Ethics, sentience, feelings?: reportedly Anthropic previously hired an "AI welfare" researcher "to explore whether future AI models might deserve moral consideration and protection"
Spatial intelligence, physical world: Niantic's Large Geospatial Model is being built using player-contributed scans of public real-world geographic locations, as part of its Visual Positioning System (VPS), to enable computers to perceive and understand physical spaces e.g. for augmented reality, robots

8 Nov 24

GenAI, chatbots, LLMs: UK Ofcom's open letter to online service providers on how the UK Online Safety Act (OSA) will apply to generative AI and chatbots. E.g.

Sites enabling users to share AI-generated text, images, videos with other users such as via group chats; services letting users upload or create their own chatbots available to other users like chatbots mimicking real/fake people; any genAI content that's shared is considered "user-generated" including deepfake fraud material.
GenAI tools enabling searching of more than one site/database are OSA-regulated "search services" (e.g. tools using live search results)
Sites/apps with genAI tools that can generate p*rn material are also OSA-regulated (so, ensure guardrails that prevent this!)
Measures from draft Ofcom Codes to help services and users include having a named person accountable for OSA compliance, having an adequately resourced, well trained content moderation function for swift takedown of illegal content and child protection; using "highly effective age assurance" for child protection; having easily-accessible, usable reporting & complaints processes.

Regulatory cooperation, international: the UK Digital Regulation Cooperation Forum (DRCF) announced a joint OECD and The International Network for Digital Regulation Cooperation (INDRC) workshop to discuss the "interplay between digital regulatory frameworks – challenges and opportunities of structural collaboration", and published the resulting joint statement "to demonstrate INDRC members’ commitment to continued collaboration and dialogue on key matters of digital regulatory significance"

7 Nov 24

AI usage: Wendy's is using Palantir's software for supply chain management and anticipating/preventing ingredient shortages

6 Nov 24

Assurance, testing, frameworks, tools: UK DSIT report "surveys the state of the UK AI assurance market and sets out how DSIT will drive its future growth" (news release). Key actions planned:

Developing an AI assurance platform as a one-stop shop for developers/deployers, "bringing together existing assurance tools, services, frameworks and practices together in one place" including DSIT guidance/tools/resources, including an AI Essentials toolkit which, like the UK Cyber Essentials, will "distil key tenets of relevant governance frameworks and standards to make these comprehensible for industry"

The first tool was an AI Management Essentials self-assessment tool (draft tool, guidance) drawing on key principles from existing standards/frameworks including ISO/IEC 42001 (on AI management), the EU AI Act, and the NIST AI Risk Management Framework, to provide "a simple, free baseline of organisational good practice, supporting private sector organisations to engage in the development of ethical, robust and responsible AI". The consultation on this tool closes on 25 Jan 25.

Developing, with industry, a roadmap to trusted third-party AI assurance to increase the supply of independent, high-quality, trusted assurance
Collaborating with the UK Artificial Intelligence Safety Institute (AISI) to advance assurance research, development and adoption like new techniques for evaluating and assuring AI systems to ensure safe and responsible development/deployment. This includes:

Exploring how Privacy Enhancing Technologies (PETs) can enable data sharing with researchers to help them understand the capabilities and controllability of models while minimising risks to privacy or commercial confidentiality
Enabling/promoting the interoperability of AI assurance across jurisdictions internationally

Developing a Terminology Tool for Responsible AI, to define key terminology used in the UK and other jurisdictions and the relationships between them, to help industry and assurance service providers navigate key concepts and terms in different AI governance frameworks "to communicate effectively with consumers and trading partners within the UK and other jurisdictions, supporting the growth of the UK’s AI assurance market"

Work has already started on this with the US National Institute for Standards and Technology (NIST) and the UK’s National Physical Laboratory (NPL)
Sector-specific non-technical guidance on assurance good practice: already produced for employment (procuring and deploying AI for recruitment: see also below on ICO questions); guidance for other sectors including financial services to be published "in the near future"

Employment, recruitment, data protection: UK ICO's key data protection questions when procuring AI tools to help with recruitment (DPIA, lawful basis, documented responsibilities and clear processing instructions, bias mitigation, transparency re use, limiting unnecessary processing): ICO webinar 10 am, 22 Jan 25. Also see above on DSIT guidance
EU AI Act: EDPB letter to European Commission on role of data protection authorities

5 Nov 24

AI usage, chatbots, LLMs: the UK government released its experimental AI chatbot to help people set up small businesses and find support
EU: the Council approved conclusions on the European Court of Auditors' (ECA) report on strengthening EU "AI ambitions, notably by enhancing governance and ensuring an increased, more focused investment when moving forward in this field", including scaling up AI investments and facilitating access to digital infrastructure, noting the importance of AI systems' environmental impact, high-performance computing, possible solutions to increase energy efficiency, and securing a reliable hardware supply chain

4 Nov 24

Models, open source: Epoch AI's report comparing open and closed models: "The best open model today is on par with closed models in performance and training compute, but with a lag of about one year"

1 Nov 24

Gen AI, risks, management: Open Loop & Meta report, Generative AI Risk Management and the NIST Generative AI Profile (NIST AI 600-1)
On-device, LLMs: Apple explained "how to optimize and deploy an LLM to Apple silicon, achieving the performance required for real time use cases", using Meta's Llama-3.1-8B-Instruct, "a popular mid-size LLM", showing "how using Apple’s Core ML framework and the optimizations described here, this model can be run locally on a Mac with M1 Max with about ~33 tokens/s decoding speed"

30 Oct 24

Metrics, hallucinations: OpenAI announced its new open source SimpleQA benchmark that measures LMs' "ability to answer short, fact-seeking questions"
US EO14110: US DoC fact sheet on key accomplishments 1 year on from this Biden-Harris EO

Note: reportedly the new Trump government is likely to get get rid of EO14110

29 Oct 24

Employment, recruitment: LinkedIn introduced an AI-powered "Hiring Assistant to Help Recruiters Spend More Time On Their Most Impactful Work"

28 Oct 24

Cybersecurity, LLMs, open source, tools: paper Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks. "...We introduce Mantis, a defensive framework that exploits LLMs' susceptibility to adversarial inputs to undermine malicious operations. Upon detecting an automated cyberattack, Mantis plants carefully crafted inputs into system responses, leading the attacker's LLM to disrupt their own operations (passive defense) or even compromise the attacker's machine (active defense). By deploying purposefully vulnerable decoy services to attract the attacker and using dynamic prompt injections for the attacker's LLM, Mantis can autonomously hack back the attacker". Code
Data scraping, training data: following industry engagement, global privacy authorities issued their Concluding joint statement on data scraping and the protection of privacy
Facial recognition, data protection: the UK ICO applied for permission to appeal the First-tier Tribunal's judgment on Clearview AI
Intellectual property, competition, training: UK Prime Minister Keir Starmer's article includes: "Both artificial intelligence and the creative industries – which include news media – are central to this government’s driving mission on economic growth. To strike balance in our industrial policy, we are working closely with these sectors. We recognise the basic principle that publishers should have control over and seek payment for their work, including when thinking about the role of AI. Not only is it essential for a vibrant media landscape, in which the sector’s provision of trustworthy information is more vital than ever, it is also relevant to our ongoing work to roll out the Digital Markets, Competition and Consumers Act as swiftly as possible. This landmark legislation will help rebalance the relationship between online platforms and those, such as publishers, who rely on them."
Open source AI: the Open Source Initiative announced its release of the first Open Source AI Definition (OSAID) v1.0

Note critiques regarding the description/definition, e.g. security guru Bruce Schneier's 8 Nov post "AI Industry is Trying to Subvert the Definition of “Open Source AI” suggesting that the description should be "open weights", not "open source"

OSs, mobile: Apple released its Apple Intelligence AI capabilities for iPhone, iPad, Mac
Sustainability, US, water, data centers: J.P. Morgan and ERM report on water resilience in the US. Water is of course needed for cooling in data centers, which are increasingly being built for AI processing

25 Oct 24

AI usage, saving lives: UK announcement that Scotland-based SME Zelim (which benefited from DASA funding) had won a contract with the US Navy to deploy their innovative AI-enabled Person-in-Water detection and tracking technology ZOE. "Zelim’s detection and tracking system uses AI to scan the water surface to find people in the water much more accurately and consistently than human eyes and current systems can. Low-cost and easy to integrate, the software solution can be implemented in any camera or CCTV setup"

24 Oct 24

EU AI Act standards, intellectual property: Commission note on some of the key characteristics expected from upcoming standards for high-risk AI systems to support implementation of the AI Act (news article)

Note: the Commission previously requested EU standardisation organisations, led by CEN-CENELEC, to draft various standards for the AI Act, to cover: risk management systems, governance and quality of datasets used to build AI systems, record keeping through logging capabilities, transparency and information provisions for users, human oversight, accuracy specifications, robustness specifications, cybersecurity specifications, quality management systems for providers including post-market monitoring processes, conformity assessment.
In Case C-588/21P the CJEU stated that, while harmonised standards under Regulation 1049/2001 were protected by copyright, as CEN-CENELEC acknowledged, there was an overriding public interest in their disclosure, and annulled the Commission's refusal of access to those standards. So the new standards for high-risk AI should be publicly-available.

AI edits, transparency: Google is indicating when an image in Google Photos has been edited using Google's genAI (e.g. Magic Editor, Magic Eraser and Zoom Enhance), and using The International Press Telecommunications Council (IPTC) metadata to indicate when an image comprises elements from different photos using non-generative features
Health, life expectancy: UK NHS trials AI "that predicts when patients will die"
On-device ML, homomorphic encryption, PETs: Apple is combining machine learning and HE in its ecosystem for privacy while enriching on-device experiences with information privately retrieved from server databases
Testing, cybersecurity, tools: Google released its Secure AI Framework (SAIF) risk assessment tool "that can help others assess their security posture, apply these best practices and put SAIF principles into action". It's a questionnaire-based tool "that will generate an instant and tailored checklist to guide practitioners to secure their AI systems"
Testing, assurance, safety: by the UK Artificial Intelligence Safety Institute (AISI), Early lessons from evaluating frontier AI systems, discussing the evolving role of third-party evaluators in assessing AI safety, and how to design robust, impactful testing frameworks.

23 Oct 24

GenAI, synthetic data, identification: article by Google DeepMind researchers Scalable watermarking for identifying large language model outputs
GenAI, risks, management: Generative AI Training, Development, and Deployment Considerations, by the US Government Accountability Office (GAO)
Work, employment: will AI make work burnout worse? One small PR agency's experiences with ChatGPT's impact on employee productivity: "Staff reported that tasks were in fact taking longer as they had to create a brief and prompts for ChatGPT, while also having to double check its output for inaccuracies, of which there were many. And every time the platform was updated, they had to learn its new features, which also took extra time"

22 Oct 24

Agents, computer control: Anthropic announced that its experimental Claude 3.5 Sonnet model can (as a beta feature) interact with tools to manipulate a computer desktop environment, to "use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text" (see also 19 Nov)
Intellectual property, genAI, training: a statement on AI training "The unlicensed use of creative works for training generative AI is a major, unjust threat to the livelihoods of the people behind those works, and must not be permitted" was released. To date it's been signed by over 39.5k (and counting): musicians, writers, industry organisations and others including many famous names

21 Oct 24

Detection, genAI, synthetic data: good roundup of recently-reported errors made by claimed AI-detecting tools (no doubt many also using AI), falsely accusing students of plagiarism
Open source, LLMs: IBM released its open source Granite 3.0 models "High Performing AI Models Built for Business", and also announced its next generation of Granite-powered watsonx Code Assistant for general purpose coding, and new tools in watsonx.ai for building and deploying AI applications and agents
Transparency: summary of findings from a UK Digital Regulation Cooperation Forum (DRCF) workshop in Aug 2024 on why AI transparency is important, key considerations for participating regulators, and useful information from each regulator on existing guidance and their next steps related to AI transparency

18 Oct 24

Multimodal, open source, LLMs: Meta announced Spirit LM, its open source language modal that mixs both text and speech audio. Paper SPIRIT LM: Interleaved Spoken and Written Language Model (VB article)

17 Oct 24

Physical world, sensor data: Can AI Learn Physics from Sensor Data?

16 Oct 24

Employment, recruitment, bias, ethnicity: paper on using LLMs for resume / CV screening, Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval which tested whether various Massive Text Embedding (MTE) models are biased (bias regarding intersectionality involves intersecting attributes like gender and ethnicity). "We simulate this for nine occupations, using a collection of over 500 publicly available resumes and 500 job descriptions. We find that the MTEs are biased, significantly favoring White-associated names in 85.1% of cases and female-associated names in only 11.1% of cases, with a minority of cases showing no statistically significant differences. Further analyses show that Black males are disadvantaged in up to 100% of cases... We also find an impact of document length as well as the corpus frequency of names in the selection of resumes [such that increasing the ratio of signals that are proxies to race or gender information in a document by decreasing its length can increase the number of biased outcomes by 22.2%, and changing frequency matching strategies can alter whether Black names or White names are favored in a majority of cases]. These findings have implications for widely used AI tools that are automating employment, fairness, and tech policy"
EU AI Act, compliance, tools: Reuters reported on a new tool that "awards AI models a score between 0 and 1 across dozens of categories, including technical robustness and safety" with a leaderboard published of various models developed by big tech companies, and a "Large Language Model (LLM) Checker". According to the website, models can be evaluated locally and the JSON report file uploaded for a technical report. The tool provider states, "We have interpreted the high-level regulatory requirements of the EU AI Act as concrete technical requirements. We further group requirements within six EU AI Act principles and label them as GPAI, GPAI+SR (Systemic Risk), and HR (High-Risk)" (paper - on my "to read" list!). The technical aspects the tool seeks to evaluate are: Robustness and Predictability, Cyberattack resilience (cybersecurity), Training Data Suitability, No Copyright Infringement, User Privacy Protection, Capabilities, Performance, and Limitations, Interpretability, Disclosure of AI (transparency), Traceability, Explainability, Risks, Evaluations, Representation — Absence of Bias, Fairness — Absence of Discrimination, Harmful Content and Toxicity

Early Oct 24

LLMs, hallucinations: much ado (e.g. good article) about the Entropix project, on using uncertainty modelling to reduce LLM hallucinations

10 Oct 24

Roundup: excellent State of AI report 2024

7 Oct 24

LLMs, maths reasoning: paper by Apple researchers on limitations GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Wired article)

7 Oct 24

GenAI, transparency: Google joined the Coalition for Content Provenance and Authenticity (C2PA) as a steering committee member, and is incorporating C2PA's standard into its products

And just a few from before Oct 2024...

27 Sept 24

Employment, recruitment: half of a tech company's HR department was fired after their manager found their an application review system (not necessarily AI?) auto-rejected all applications, even the manager's own CV submitted under a fake name! "...HR had set up the system to search for developers with expertise in the wrong development software and one that no longer exists..." but also "[HR] always told [the manager] that they had some candidates that didn't pass the first screening processes (which was false)"

25 Sept 24

Misrepresentation: the US FTC announced 5 law enforcement actions "against operations that use AI hype or sell AI technology that can be used in deceptive and unfair ways"

24 Sept 24

Financial services, Canada, usage, risks, management: by the Office of the Superintendent of Financial Institutions (OSFI) and the Financial Consumer Agency of Canada (FCAC), OSFI-FCAC Risk Report - AI Uses and Risks at Federally Regulated Financial Institutions outlines key risks (from internal AI adoption or from AI use by external actors) that arise for financial institutions from AI, supported by findings from a previous questionnaire and insights from external publications. "It also presents certain practices that can help mitigate some of the risks. These are not meant to serve as guidance but can be positive steps in a financial institution's journey to manage the risks related to AI"

23 Sept 24

Testing: from the UK AISI, Early Insights from Developing Question-Answer Evaluations for Frontier AI. "A common technique for quickly assessing AI capabilities is prompting models to answer hundreds of questions, then automatically scoring the answers. We share insights from months of using this method"

18 Sept 24

GenAI, health, misrepresentation, accuracy: Texas AG settlement with a Dallas-based artificial intelligence healthcare technology company, resolving allegations that it deployed its products at several Texas hospitals after making false and misleading statements about the accuracy and safety of its products. At least four major Texas hospitals provided their patients’ healthcare data in real time for the generative AI product to “summarize” patients’ condition and treatment for hospital staff. The AG investigation found deceptive claims about the accuracy of its healthcare AI products, putting the public interest at risk; metrics used to claim accuracy, including advertising and marketing the accuracy of its products and services by claiming an error rate or “severe hallucination rate” of “<1 per 100,000”, were found likely inaccurate "and may have deceived hospitals about the accuracy and safety of the company’s products"

Older, but just because I think these are interesting!

EU AI Act, law enforcement: in Jul 2024, various EU Member States submitted queries to the Commission on the AI Act. Some related to definitions/concepts more broadly, not just in the crime/justice context. It would be good to know the answers!
Workplace, jobs: a 2023 report by Challenger, Gray noted that a CEO of an unnamed company [maybe Dictador, see below?] was "replaced by artificial intelligence... China-based gaming company NetDragon Websoft appointed an AI robot it calls Tang Yu as its CEO last August [i.e. 2022]. Legal software company Logikcull CEO and co-founder Andy Wilson has said he will replace himself with an AI bot named Andy Woofson by 2024"

In fact Logikcull was acquired in Aug 23 so that never happened...
Reportedly Polish drinks company Dictador appointed an AI-powered humanoid robot named Mika as its experimental CEO in 2023 (video interview)

AI usage: in 2021, NotCo launched 4 plant-based chicken varieties in Latin America. "NotCo utilizes a proprietary artificial intelligence technology, Giuseppe, which matches animal proteins to their ideal replacements among thousands of plant-based ingredients"

Proposed EU AI Liability Directive

2025-01-10T17:13:00.002+00:00

Some recent links on the progress with this proposed EU Directive

30 Oct 24 certain Member States' comments on the proposal
25 Oct 24 Council Presidency discussion paper on the proposal
19 Sept 24 Proposal for a directive on adapting non-contractual civil liability rules to artificial intelligence - complementary impact assessment by European Parliamentary Research Service (EPRS) for the European Parliament. Proposed that the AILD should extend its scope to include general-purpose and other 'high-impact AI systems', aswell as software. Discussed a mixed liability framework that balances fault-based and strict liability, recommending "transitioning from an AI-focused directive to a software liability regulation, to prevent market fragmentation and enhance clarity across the EU"

AI: legitimate interests, controller/processor questions - data protection/privacy

2024-11-18T08:03:00.007+00:00

Under GDPR, can personal data be processed based on legitimate interests for AI-related purposes, whether for training, deployment, or beyond? That was the key focus of the EDPB stakeholder event on AI models on 5 Nov 24 that I was registered to attend and was fortunate enough to get a place - many thanks to the EDPB for holding this event!

The event was intended to gather cross-stakeholder views to inform the EDPB's drafting of an Art.64(2) consistency opinion on AI models (defined quite broadly) requested by the Irish DPC. The EDPB said it will issue this opinion by the end of 2024 but, unlike EDPB guidelines, such consistency opinions can't be updated - which is concerning given how important this area is.

The specific questions were:

AI models and "personal data" - technical ways to evaluate whether an AI model trained using personal data still processes personal data? Any specific tools / methods to assess risks of regurgitation and extraction of personal data from AI models trained using personal data? Which measures (upstream or downstream) can help reduce risks of extracting personal data from such AI models trained using personal data? (including effectiveness, metrics, residual risk)
Can "legitimate interest” be relied on as a lawful basis for processing personal data in AI models?

When training AI models - and what measures to ensure an appropriate balance of interests, considering both first-party and third-party personal data?
In the post-training phase, like deployment or retraining - and what measures to ensure an appropriate balance, and what if the competent supervisory authority found the model's initial training involved unlawful processing?

There wasn't enough time for me to explain my planned input properly or to comment on some issues, given the number of attendees, so I am doing it here. I'll take the second set first.

Training AI models - legitimate interests

I strongly believe legitimate interest should be a valid legal basis for training AI with personal data. Particularly training AI to reduce the risk of bias or discrimination against people, when the AI is used in relation to them.

I had a negative experience with facial biometrics. The UK Passport Office's system kept insisting my eyes were shut, when they were wide open - they're just small East Asian eyes, white people's eyes are usually bigger. Others have suffered far worse from facial biometrics and facial recognition, including wrongful arrests, denial of food purchases, debanking (see my book and 23.5 of the free companion PDF under Facial recognition).

Had the AI concerned been trained on more, and enough, non-white faces, it would be much less likely to claim facial features that didn't match typical white facial features were "inappropriate" (like eye size, hair shape), or to misidentify the wrong non-white people leading to their wrongful arrests.

The EU AI Act is aware of this risk: Art.10(5) (and see Rec.70) specifically permits providers of high-risk AI systems to process special categories of personal data, subject to appropriate safeguards and meeting certain conditions:

the bias detection and correction cannot be effectively fulfilled by processing other data, including synthetic or anonymised data;
the special categories of personal data are subject to technical limitations on the re-use of the personal data, and state-of-the-art security and privacy-preserving measures, including pseudonymisation;
the special categories of personal data are subject to measures to ensure that the personal data processed are secured, protected, subject to suitable safeguards, including strict controls and documentation of the access, to avoid misuse and ensure that only authorised persons have access to those personal data with appropriate confidentiality obligations;
the special categories of personal data are not to be transmitted, transferred or otherwise accessed by other parties;
the special categories of personal data are deleted once the bias has been corrected or the personal data has reached the end of its retention period, whichever comes first;
(GDPR) records of processing activities include reasons why processing of special categories of personal data was strictly necessary to detect and correct biases, and why that objective could not be achieved by processing other data.

(Aside: I know that Article also mentions "appropriate safeguards", but I'd argue that meeting those conditions would provide the minimum required safeguards - although in some cases others could be considered necessary.)

The Act confines this permission to the use of special category data in high-risk AI systems, but I'd argue that legitimate interests should permit the use of non-special category personal data through meeting the above conditions (and any other appropriate safeguards).

Recall that personal data can be processed under GDPR's legitimate interests legal basis if "necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data, in particular where the data subject is a child." The EDPB's recent guidelines on processing based on Art.6(1)(f) note three cumulative conditions to enable processing based on legitimate interests:

the pursuit of a legitimate interest by the controller or by a third party;
the need to process personal data for the purposes of the legitimate interest(s) pursued; and
the interests or fundamental freedoms and rights of the concerned data subjects do not take precedence over the legitimate interest(s) of the controller or of a third party.

Let's review those in turn.

Detecting and correcting bias involves the pursuit of a legitimate interest of the controller, i.e. AI developer, and third parties. I'd argue that many, many third parties, being those in relation to whom the AI is to be used, have a legitimate interest in not being discriminated against due to biased AI. I've already mentioned biased AI resulting in wrongful arrests, denial of services important to life like food buying, and debanking (see 23.5 of my free PDF under Facial recognition).

It is indeed necessary to process personal data of people in certain groups in order to train AI models to reduce bias, as per the experiences noted above and much more.

Finally, the balancing test in the final limb must clearly consider the legitimate interests, not just of the controller, but also of "a third party" - in this case, the legitimate interest of third parties, in relation to whom the AI is to be used, not to be discriminated against. (While fairness is a core principle of the GDPR, this only concerns fairness to the individual whose personal data is being processed. Processing A's personal data to try to ensure fairness to B isn't a concept explicitly provided for in GDPR. There are mentions of "rights and freedoms of others" or other data subjects, but more in the sense of not adversely affecting their rights/freedoms, rather than positive obligations in their favour.)

I argue that, if the conditions in Art.10(5) AI Act are implemented as a minimum when training AI using personal data, that should tilt the balancing test in favour of the controller and those third parties, and enable legitimate interests to be used as the legal basis for the training - at least in the case of non-special category data - even when training non-high-risk AI. I really hope the EDPB will agree.

However, the problem remains of how to use special category data to train non-high-risk AI systems to detect and address bias. Some examples I mentioned could fall through the cracks.

The UK Passport Office's AI system, designed to reject photos with "inappropriate" facial features, is probably a high-risk AI system within Annex III para.5(a) (if the Act applied in the UK). Yet, para.5 (and Annex III more generally) does not protect anyone from being refused a private bank account or being debanked as a result of biased AI being applied to them.

And, a huge hole in the AI Act is this: Annex III para.1(a) excludes "AI systems intended to be used for biometric verification the sole purpose of which is to confirm that a specific natural person is the person he or she claims to be". What if an AI biometric verification system used by a bank mistakenly says someone is not who they claim to be, because it can't verify the identity of non-white people properly due to not having been trained on the faces of enough non-white people - and therefore the bank's systems automatically debanks that individual? How can such a biased AI biometric verification system be "fixed", if it can't be fully trained in this way?

Such an AI system is not classed as a high risk AI system, because of the biometric verification exclusion. Therefore, the developer isn't allowed to train the AI using special category data, because Art.10(5) AI Act only allows this for high-risk AI systems! (Yes, I know there's the odd situation where biometric data is "special category" data only when used for the purpose of uniquely identifying someone, so it could be argued that using non-white people's facial biometrics to train personal data isn't processing their special category data, because the processing purpose isn't to identify those specific people, and I'd certainly be putting that argument and pushing for being able to use legitimate interests for that training. But - really? Why should those arguments be necessary?)

It was argued that Art.9(2)(g) (necessary for reasons of substantial public interest etc) doesn’t allow processing of special category data to train AI, even though there is a substantial public interest in addressing bias. I agree there is a huge public interest there, but I also agree that, due to the wording of that provision, it can’t apply unless proportionate etc. EU or Member State law provides a basis for such processing. EU law in the form of AI Act Art.10* does provides a basis for processing special category data in high-risk AI systems - doesn’t provide such a basis in the case of non-high-risk AI, or non-special category data - hence the need to argue that biometric data isn’t special category when used for training! I guess it’ll have to be down to national laws to provide for this clearly enough. France, Germany or Ireland, perhaps?

(Consent isn’t feasible in practice here, given the volumes involved, and issues like having to repeat AI training after removing, from the training data, any personal data where consent has been withdrawn. It was argued that financial costs or training time for AI developers shouldn’t be relevant in data protection, but equally it was argued that environmental costs etc. of repeating training are relevant. I’ll only mention briefly practical workarounds, like not removing that data but preventing it from appearing in outputs using technical measures whose efficacy is debated)

If including my personal data in training datasets can help to reduce the risk of otherwise biased AI systems discriminating against you (should you be in the same ethnic or other grouping as me) when deployed, personally I'd be OK with that - partly informed by my own bad experiences with AI biometrics. Shouldn't such processing of data for AI training be permitted, even encouraged? But, currently, this issue is not properly or fully addressed, as I've shown above. So, there's a big data dilemma here, that still remains to be dealt with.

AI models and personal data

Does an AI model "contain" personal data, given that strictly it's not a database per se? Or is it just something that can be used to produce personal data when used in deployment, with personal data being processed only at the usage stage? Much debate, and diametrically opposing views (and difficult questions like, can a GPAI model developer be said to control the purposes and means for which deployers of the model use it?). [Added: I meant to expand on that the clarify that question, is the model developer controlling the purposes of processing personal data, particularly with general-purpose/foundation models, or is it merely providing part of the means of processing to others, i.e. is it really a "controller"?]

Rather than pinhead-dancing around that question, personally I think that use of a deployed AI system is the most relevant processing here, because that's the main point at which the LLMs/large language models (that the event focused on, pretty much exclusively) could regurgitate accurate or inaccurate personal data - whether through prompt injection attacks or similar in the case of LLMs, or because a model's guardrails weren't strong enough.

I feel the EDPB's query on technical ways to evaluate whether an AI model trained using personal data "still processes personal" data is really more one for technical AI experts to answer, and that what merits more attention is preventing training data's regurgitation/extraction at the deployment/use stage, whether personal data or otherwise. It's well known that attacks have successfully obtained training personal data from models - although with some limitations and caveats (paper & article; another article). This has been shown to be possible not only with open source models (where attackers obviously have access to more info about the model, its parameters etc., and indeed to the model itself), but even semi-open and closed source models like ChatGPT.

Again, my view is that assessing and reducing training data regurgitation/extraction risks are essentially questions for technical AI experts. Reducing such risks mainly involve technical measures, and this is an emerging area where much research continues to be conducted, so I feel it's premature to rule on such measures at this point in time (although organisational measures are also possible, and recommended, like deployers prohibiting their users from trying to extract personal data from any AI).

AI value chain: controllers, processors

More interesting, and difficult, from a GDPR perspective are the crucial questions of: who is a controller, who is a processor, who is liable for what, and at which stages in the AI lifecycle?

Unfortunately, these weren't really discussed at the event. To be fair, the focus of the event was meant to be legitimate interests, not the controller/processor position of AI model/system providers.

I still tried to raise them, but wasn't allowed to speak again to clarify my points, so I'll do that below in the form of some "exam questions". But, first, I want to spell out some issues with the AI supply chain that I couldn't expand on during the event.

If a developer organisation makes its own AI model available for customers to use, depending on the business model adopted by the organisation (and the following isn't comprehensive!), the supply chain can involve several alternative options:

The model could be accessed via the model developer's API, and/or
The model could be permitted to be:

Downloaded by customers as a standalone model, then

Embedded/integrated within an AI system developed by the customer (which the customer could use internally only, or offer to its own customers in turn), or
Accessed by a customer-developed AI system (which the customer could use internally only, or offer to its own customers in turn) via API, where the downloaded model is hosted

on-prem, or
(more likely) in-cloud, using the customer's IaaS/PaaS provider, but with all AI-related operations being self-managed by the customer, or

(Common nowadays) deployed and used by the customer for the customer's AI system (which the customer could use internally only, or offer to its own customers in turn), through the customer using a provider's cloud AI management platform with the benefit of tools/services available from the cloud provider to ease AI-related operations like fine-tuning models, building AI systems, using RAG, etc.

Note: the model used could be one of the cloud provider's own models (i.e. where the cloud provider is the model developer), or it could be a third-party model offered through the cloud provider's own AI marketplace or similar. Exactly what licence/contract terms apply to the customer in such a scenario, particularly with third-party models, let alone what the controller/processor position is there, is still clear as mud (see below).

Note that an AI system can use or integrate more than one AI model.

Also note that the above applies equally to how an AI system is accessed, i.e. via API, or by embedding the system within an AI product/solution/tool, or using a cloud AI management platform, and that an AI system can use or integrate more than one other AI system (i.e. rinse and repeat the above, on AI models, to AI systems). See my PDF that I'd previously uploaded to LinkedIn (with a small clarificatory update):

And I won't even mention the twists introduced by using RAG/retrieval-augmented generation in LLMs, at this point.

All that spelt out, now on to my exam questions!:

After an organisation deploys a third-party model in an AI system

If a user in the organisation deliberately extracts personal data from the AI without the deploying/employing organisation's authorisation

Is the rogue user a controller in their own right, so that the organisation is not responsible as a controller under data protection law (as with the Morrisons case in the UK)?
Does or should the AI model developer bear any responsibility or liability at the deployment and use stage as a controller in some way, if the guardrails they implemented against the extraction weren't appropriate? Or could it be a processor, particularly if the model is hosted by the model or system developer?

Even if the model is considered not to "contain" any personal data, so that the model developer is not a controller of the model itself, could the model developer be considered to have some responsibility if and when personal data is extracted from the AI at the deployment and use stage?
Remember, for security measures under GDPR, a security breach alone doesn't mean the security measures weren't appropriate; it's quite possible for an organisation that had implemented appropriate security measures to suffer a personal data breach nevertheless.
Also to reiterate, measures to reduce the risk of extracting training data from AI models are still being developed, this is very much a nascent research area.
Recall that a developer providing software for download/on-prem install is not generally considered a processor or controller, but when it offers software via the cloud as SaaS, it is at least a processor, even a controller to the extent it uses customer data for its own purposes. If a model developer makes available a model (software), but doesn't host it for customers, it seems the developer shouldn't even be a processor?

If a user in the organisation deliberately extracts personal data from an AI with the deploying/employing organisation's authorisation (e.g. for research, or for the organisation's own purposes)

Is the organisation a controller, responsible/liable for that extraction as "processing"? (and could the GDPR research exemption apply there, if for research?)
Could the AI model developer and/or AI system developer bear any responsibility or liability for this extraction as a controller in some way, if the guardrails they implemented against the extraction weren't appropriate, as above, or as a processor?

Note the same points/queries apply as in 1.1 above!

If a user in the organisation uses the AI in such a way that, without the user intending it, the AI regurgitates personal data, who is responsible as controller for the output, which is "processing"?

Remember, a user could process personal data by including it in the input provided to the AI (not discussed further here), but personal data could also be processed if it is included in the AI's output
Does or should the AI model developer and/or AI system developer bear any responsibility or liability as a controller in some way, if the guardrails they implemented against inadvertent regurgitation weren't appropriate, or could it be a processor, or neither?

Note the same points/queries apply as with deliberate extraction, 1.1 above.

What difference if any does it make if the personal data in the output is accurate, or inaccurate (e.g. defamatory of the individual concerned)?

If a person unrelated to the organisation, e.g. a third-party hacker, manages to access the deployed AI to extract training data such as personal data, is the deploying organisation responsible as controller? What about the model/system developer?

Do any of the above apply, are they relevant, when a AI developer makes its model available to customers via the developer's API only? Is the model developer/provider a processor for customers in that situation?

Again see 1.1 above. In particular, it seems the AI developer hosting the model offered to customers would at least be a processor here.

What if a customer uses a third-party AI model hosted by the customer's cloud provider? Is the cloud provider only a processor for the customer, or could it be a controller in any way?

Does it make a difference if the model used by the customer is the cloud provider's own model, or another party's model?
Does it make a difference if the model's use is completely self-managed by the customer, or if the customer is using a cloud provider's cloud AI management platform?
Do the license terms, cloud agreement terms and/or other terms applicable to the customer's use of the cloud service/AI platform affect the position (under GDPR it's the factual control of purposes and means that matters, and contract terms are not determinative, but nevertheless terms could influence the factual position in some cases, especially in what they permit or prohibit...).
Indeed, back to the AI Act, who is the model provider - the AI platform provider, or the model developer?

Rinse and repeat for AI system developers/providers - could they be responsible/liable as controllers and/or processors especially if a model provider hosts its model or AI systems using its model for customers in-cloud?

(There are many more questions and issues, these are just the key ones that spring to mind most immediately, believe it or not!)

Answers on a postcard...?

Cyber Security & Resilience Bill: consultation

2024-11-16T15:05:00.007+00:00

DSIT is seeking views on some measures planned under the UK Cyber Security and Resilience Bill, to be introduced in 2025 to update The Network and Information Systems Regulations 2018. I saw this a couple of days ago on the ICO's NIS webpage, then found more info on techUK's 8 Nov webpage.

Usefully, techUK has also listed all the consultation questions in one PDF, which is really helpful as, unlike EU consultations that usually offer a downloadable PDF listing the questions, sadly too many UK consultations expect respondents to go through a form page by page before they can see what the questions are, which wastes time for those wanting to provide considered responses to all questions holistically (some webpages don't even allow going back).

The deadline is soon according to ICO: 21 Nov 24, i.e. next Sunday!

As you'll know, the intention is to expand the NIS Regulations to catch even more types of organisations, and to reduce incident reporting deadlines (with staffing/costs implications for 24 hr reporting especially at the weekend). Some proposals resemble the changes under the EU's NIS2 Directive. Managed service providers will probably be brought into scope (proposed criteria below). Note the queries on the costs of rolling out MFA, and of password resets. DSIT is also asking competent authorities (but it seems not other stakeholders) whether data centres should be regulated. Interestingly, it also asks if any Competent Authorities currently review the supplier contracts of regulated entities for visibility into their supply chain, assurance of supplier cyber security and resilience measures, and/or have audit rights - familiar from GDPR, but could this be specifically required in future under NIS too?

Key excerpts:

Managed service providers (MSPs) to be brought within scope of Relevant Digital Service Provider (RDSP)

DSIT's proposed characteristics of a Managed Service Provider have 4 criteria:

1. The service is provided by one business to another business, and

2. The service is related to the provision of IT services, such as systems, infrastructure, networks, and/or security, and

3. The service relies on the use of network and information systems, whether this is the network and information systems of the provider, their customers or third parties, and

4. The service provides regular and ongoing management support, active administration and/or monitoring of IT systems, IT infrastructure, IT network, and/or the security thereof.

Incident reporting

Changes being considered "to ensure more incidents are reported and that incident information is communicated to relevant parties more quickly and clearly" include:

"1. A change to the definition of an incident under the existing NIS Regulations. To meet the current reporting threshold, an incident must have led to a significant or substantial disruption to service continuity. We are proposing to change the definition of a reportable incident to ensure that a wider range of incidents are captured, including incidents capable of resulting in a significant impact to service continuity and incidents that compromise the integrity of a network and information system.

2. A change to the amount of time an organisation has to report an incident from when it is detected. Currently, incidents must be reported without undue delay and no later than 72 hours after being made aware of the incident. We are assessing whether this time can be reduced to no later than 24 hours after being made aware of the incident.

3. New transparency requirements. We are considering introducing a transparency requirement which will ensure customers are notified of incidents which significantly compromise the integrity of a digital service upon which they rely."

On 24-hr reporting, DSIT wants to know:

1. Which members of staff are needed to develop and submit an NIS incident report?

2. Do you have the people required to submit an incident report already working weekend shifts?

3. Could you have staff on call as opposed to working weekend shifts in case there is the need to report an NIS incident? Could you save money by calling in members of staff when an incident is detected?

4. Is there a higher rate of pay for staff working weekends than those working during the week? If so, what overtime rate do staff get paid?

On transparency:

5. If an incident occurred which affected a service you provide, would you be able to

identify which customers have been affected? (‘Customers’ in this question should

be interpreted as businesses which rely on a digital service provider [cloud provider] for a service,

not individual clients.) If so, how long would it take to identify which customers have

been affected?

6. Do you have a plan in place for what to do if an incident occurs? [For RDSPs [i.e. cloud providers]]

MSPs:

7. [for OES] Do you use services provided by an MSP (or multiple MSPs) to deliver your essential service(s)? This would also include, for example, companies which provide IT outsourcing, BPO (business process outsourcing) where it is provided through IT networks, or cyber security services.

a. If yes, please provide examples of where these services provided by an MSP (or multiple MSPs) are critical to the provision of your essential service? (note: names of companies are not required)

8. [for RDSPs] Do you provide managed services? This would include, for example, providing IT outsourcing, Business Process Outsourcing (BPO) where it is provided through IT networks, or managed security services.

9. Do you provide Business Process Outsourcing (BPO) services that involve ongoing management of an IT system/ infrastructure/network and have a connection or access to the customer?

a. If yes, please provide examples of the BPO services provided by your organisation.

10. Do you provide managed IT services that secure or manage operational technology (OT)?

a. If yes, please provide examples. Detailed examples are welcome, particularly where these relate to critical national infrastructure (CNI).

11. Do you provide system integration?

a. If yes, is the system integration provided as part of a managed service? Please provide examples of the system integration you provide as part of a managed service.

12. Do you provide telecommunications services (e.g. WAN, LAN)?

- If yes, please provide examples of the telecommunications services you provide.

- If yes, do you consider that any of these telecommunication services constitute a ‘managed service’?

- If yes, are these telecommunications services regulated under the Communications Act 2003?

13. Is the cyber security of the services you provide (in the UK or overseas) currently regulated? Are you currently regulated for the cyber security for any of your services offered (in the UK or overseas)?

If yes, please provide details of these regulations.

[Questions about small and micro cloud or managed services in the supply chain]

Operational technology (OT):

15. Does your organisation use operational technology to manage any critical or essential services?

16. [if yes to 15] If you purchase operational technology (OT) from a vendor, do you maintain and operate it ‘in house’?

17. [if yes to 15] Do you outsource the management of operational technology (OT) to third party providers?

a. If yes, are these third party providers Managed Service Providers (MSPs)? (i.e., the same company that manages your IT systems/networks/Infrastructure)

b. If yes, please provide examples of operational technology (OT) that you outsource to third parties (note: a description of the company would suffice, names are not required)

Managing risks - costs impacts of serious incidents:

18. How much would it cost your organisation to conduct a full rollout of multi-factor authentication for all users?

19. How much would it cost your organisation to conduct a full organisation-wide reset of passwords?

20. What other actions do you anticipate you might need to take to protect your organisation in the event of a major cyber security attack or resilience incident?

[Some duplication: the next set of questions is for firms NOT regulated under NIS, including 24-hr reporting and staff costs, OT, managing risks, small/micro MSPs/cloud providers, MSPs]

25. If you purchase operational technology (OT) from a vendor, do you maintain and operate it ‘in house’?

26. Do you outsource the management of operational technology (OT) to third party providers?

a. If yes, are these third party providers Managed Service Providers (MSPs)? (i.e., the same company that manages your IT systems/networks/Infrastructure)

b. If yes, please provide examples of operational technology (OT) you outsource to third parties (note: a description of the company would suffice, company names are not required)

Plus questions to competent authorities (CAs) re 24-hr reporting, staff etc., private vs. public organisations regulated and their size from micro to large, and:

38. Do any Competent Authorities currently review the supplier contracts of regulated entities to ensure that appropriate measures are being taken to manage supply chain risk? E.g. that regulated entities have visibility of their suppliers’ supply chain, have some level of assurance of the cyber security and resilience measures followed by their supplier, and/or have the right to audit their supplier? If so, please share details

Data centres

39. How many standalone data centres are owned and operated by OES/RDSP/MSP businesses under your remit in the UK?

40. Do you include standalone data centres owned and operated (enterprise data centres) by OES/RDSP businesses under your remit in your supervisory activity?

a. If no under your current scope, have you previously considered or are you currently considering expanding your supervision to focus on your sector’s enterprise data centres?

b. If yes, what compliance obligations are applicable to and what assurance is required in relation to OES/RDSP owned-and-operated data centres? For example, appropriate and proportionate measures + CAF.

c. If yes, are there any measures or assurance designed for the data centre infrastructure that you apply and/or assess for your sector's data centres (or that guide your supervision) under the NIS? For example, standards designed for operational resilience of data centre infrastructure, the cyber security of operational technologies/industrial control systems, or levels of physical security of data centres.

41. To what extent do you agree with the following statements:

a. It would be beneficial to have standardised guidance on “appropriate and proportionate” measures in relation to the security and resilience of data centres / data centre infrastructure

(Strongly agree/Agree/Neither agree nor disagree/Disagree/Strongly disagree)

b. UK third-party operated data centres should be brought into the scope of the NIS under dedicated supervision with a view to protecting them as CNI and OES/RDSP supply chains?

(Same range from Strongly agree to Strongly disagree)

Things AI, Oct 2024

2024-10-19T20:56:00.005+01:00

AI tool for meeting recordings, taking notes, creating draft documents: ICO says if not used for new purpose, can rely on previous legal basis. For any new processing activity/purpose, identify lawful basis! NB. update privacy notice, accuracy, ADM, profiling, consider any data sharing with tool provider (ICO last updated date still says April but this Q&A is new since Sept).

EU AI Act contractual clauses drafted by SCL (I've not reviewed them myself). And, the Commission seeks feedback on a draft implementing regulation for scientific panel of AI experts to assist the AI Office.

EU algorithms regulation: don't forget the EU Platform Work Directive, just approved by the Council; 2-year transposition deadline. This aims to improve working conditions and protection of personal data in platform work (i.e. gig economy workers like drivers) by, among other things, promoting transparency, fairness, human oversight, safety and accountability in algorithmic management in "platform work". It will require measures on algorithmic management of people performing platform work in the EU, including those with no employment contract/relationship. Chapter III on algorithmic management limits certain processing of personal data by means of automated monitoring systems or automated decision-making systems, such as personal data on emotional or psychological state. Similarly where "digital labour platforms" use automated systems taking or supporting decisions that affect persons performing platform work; personal data processing by a digital labour platform by means of automated monitoring systems or automated decision-making systems is deemed high risk, requiring a DPIA under GDPR, and more, as well as detailed transparency requirements on automated monitoring systems and automated decision-making systems, and obligations regarding human oversight and human review, etc. There's certainly overlap with both GDPR and the AI Act.

US EO14110: NIST 1-pg summary of progress to date & next steps.

Open source AI: a draft definition 1.0-RC1 is open for comment. FAQs; and must all training data be made available for openness?

Federated learning: scalability challenges in privacy-preserving federated learning (UK RTAU & US NIST collaboration). (For an explanation of federated learning, please see my book)

UK AI Safety events: the Nov 2023 summit cost £27.7m; plus info on the Nov 2024 event incl. criteria for invites (names of invitees were withheld for data protection reasons, but names of their organisations were also withheld, not clear why): from FOI requests.

Financial services/finance/securities:

Artificial Intelligence: current and future usage within investment management, report of UK Technology Working Group (industry group for examining the impact of technology on the UK’s investment management sector, with the Investment Association), 10 Oct 24
AI Lab introduced by UK FCA, 17 Oct 24 (FCA approach to AI)
New York State Department of Financial Services (DFS) on Cybersecurity Risks Arising from Artificial Intelligence and Strategies to Combat Related Risks, 16 Oct 24
US SEC charges Rimar Capital Entities and owner Itai Liptz for defrauding investors by making false and misleading statements about use of AI (also charging one Rimar board member), 10 Oct 24

Training data collection, not just by web scraping!: certain robot vacuums were found to collect photos and audio to train AI, so big security and privacy risks with some robotic hoovers, though reportedly the privacy notice was suitably expansive (but who reads those?!, covering wholesale data collection for research including: device-generated 2D/3D map of user's houses, voice recordings, photos or videos! Talk about hoovering up data for AI training...😉🙄

LLMs: still can't do maths or reasoning (Apple researchers)

G7 Hiroshima AI Process (recall the Code of Conduct etc.) progresses:

G7 ministerial declaration
Overview of the OECD pilot of the Hiroshima artificial intelligence process reporting framework (for the international code of conduct for organizations developing advanced AI systems, like foundation models/GPAI) - summary by the Italian presidency (pilot phase); G7 joint statement
G7 toolkit for AI in the public sector - "a comprehensive guide designed to help policymakers and public sector leaders translate principles for safe, secure, and trustworthy Artificial Intelligence (AI) into actionable policies" - of interest/use to the private sector too. And see the Ada Lovelace Institute's Buying AI: s the public sector equipped to procure technology in the public interest?

Adtech: IAB Tech Lab's AI in advertising primer.

Recommender systems: seem to be particularly targeted, e.g. under the EU Digital Services Act (DSA) (and see ICO brief consultation re using children's data for recommender systems).

AI in healthcare: increasing focus e.g. by Google, Microsoft. See below on the new UK RIO.

LinkedIn & AI: LinkedIn may have agreed not to train AI using UK users' data, but it plans in its new user agreement to put all responsibility for AI-generated content on users - even though, when a user wants to start a new post, it encourages users to "try writing with AI"!

Fairness: evaluating first-person fairness in chatbots (PDF)

AI hype, costs cf productivity (is AI making work worse?) and environmental impact (is nuclear the answer?) vs. examples of AI uses: detecting that UK family court judges used victim-blaming language in domestic abuse cases; stymying mobile phone thieves; cancer detection (UKRI, gov news); pollen & allergies; UK Royal Navy like predictive maintenance; helping sustainable cities; fertilisation treatment

UK AI research programs: include wearable tech to help drug addicts; building resilience against AI risks like deepfakes, misinformation, and cyber-attacks.

UK Regulatory Innovation Office: the RIO promised in the Labour manifesto has been launched, within DSIT, "to reduce the burden of red tape and speed up access to new technologies... like AI training software for surgeons to deliver more accurate surgical treatments for patients and drones which can improve business efficiency", with the 4 initial areas including AI and digital in healthcare, and connected and autonomous technology. The RIO "it will support regulators to update regulation, speeding up approvals, and ensuring different regulatory bodies work together smoothly. It will work to continuously inform the government of regulatory barriers to innovation, set priorities for regulators which align with the government’s broader ambitions and support regulators to develop the capability they need to meet them and grow the economy... The new office will also bring regulators together and working to remove obstacles and outdated regulations to the benefit of businesses and the public, unlocking the power of innovation". But the RIO's first Chair has yet to be appointed, working 4-5 days a month (apply!). FT article (paywall).

(See also my blog on data protection & cyber security)

Data protection & cyber security, Oct 2024

2024-10-19T20:51:00.004+01:00

Cookies: consent or pay OK in UK? ICO says it's a business decision by the organisation, it holds no info! (FOI).

EU NIS2 Directive: applies from 18 Oct 2024 (news): see Commission implementing regulation on requirements for digital services incl. cloud, CDN, online marketplaces, social networks; too few Member States have transposed it into national law (published Commission list, so far just Belgium, Croatia, Italy, Latvia, Lithuania). Not listed doesn't mean "not implemented": a country might not have notified the Commission yet, or the Commission might not have added it to that list yet. But it's clear some Member States have missed the deadline, like Ireland (draft law heads of Bill). Microsoft has been quick off the mark to tout how Azure can help NIS2 compliance.

Added: ETNO-GSMA feedback on draft implementing act under the NIS2 Directive with general an detailed comments

EU Cyber Resilience Act (CRA): adopted by the Council in Oct 24, on security requirements for "products with digital elements" (software or hardware products and their remote data processing solutions, including software or hardware components being placed on the market separately). NB "remote data processing" as defined could catch some cloud servces. Applicable 36 months after CRA becomes effective (should be published in OJ in a few weeks), with some transitional provisions. Views that the CRA is an "accidental European alien torts statute"! Separately, the US CISA/FBI have published for consultation draft guidance on product security bad practices.

Revised EU Product Liability Directive: adopted by the Council in Oct 24, see some previous blog commentary on software/SaaS being caught, and defects including cybersecurity issues. Liability on repairers, compensation claims easier for claimants, importers/EU representatives can be liable for products of non-EU manufacturers. 2-year transposition period after it becomes effective (should be published in the OJ soon).

EU CSAM Regulation: recently revived by the Council's Hungarian presidency which suggested the amended compromise text. Remember, this would catch online service providers, such as providers of hosting services and interpersonal communications services. Currently this would apply 24 months from its effective date. (The previous temporary derogation from the ePrivacy Directive to allow scanning for CSAM was extended to 3 Apr 2026, in Apr 24.)

UK Product & Metrology Bill: the Delegated Powers and Regulatory Reform Committee has reservations, see my previous comments on LinkedIn including that things are mostly left to delegated legislation.

Backdoors?: but, note that any encryption/other backdoors into apps/products/networks, or special keys "only" for government access, will threaten everyone's security (as noted regarding Global Encryption Day, 21 Oct 2024!). Example: it seems Chinese hackers got into US broadband providers' networks and acquired information "from systems the federal government uses for court-authorized wiretapping".

Passkeys: more secure than passwords (see my book & free PDF!), it's great that this "passwordless" option is increasingly being adopted, and increasingly interoperable cross-platform: see passkeys on Windows, and Google's passkey syncing.

Ransomware, sanctions: individuals with links to Russian state and other prolific ransomware groups, including LockBit, have been found and sanctioned. NCA news; history of Evil Corp (not on technical matters)

Software bill of materials (SBOM): more from the US NIST e.g. on framing software component transparency (what's SBOM? CISA FAQ, resources, SBOM in SaaS/cloud, SBOM for assembled group of products. SBOM is explained in my book). I do feel contracts should include SBOM provisions.

IoT:

ETSI Cyber Security for Consumer Internet of Things: Baseline Requirements updated to v3.1.3, Sept 24 version.
Joint Guidance on Principles of OT Cybersecurity for Critical Infrastructure Organizations ASD’s ACSC, CISA, FBI, NSA, and International Partners published: Principles of operational technology cyber security. "The six principles outlined in this guide are intended to aid organizations in identifying how business decisions may adversely impact the cybersecurity of OT and the specific risks associated with those decisions."
If you use an iPhone/iPad: great tips from security expert Scott Helme

UK NCSC guidance:

for lawyers, cybersecurity tips (PDF), the usual basics - backups, software updates, encryption, strong passwords & 2FA/MFA, device access control, firewall, limit admin accounts, use antivirus, control/track/lock stolen/lost devices, app permissions
updated guidance on multi-factor authentication (MFA), emphasising that authenticating users to cloud-based corporate services using a password alone is not strong enough to protect any sensitive data; only promp for authentication or MFA when it makes a difference (tell that to law firms, please!), and anti-patterns that undermine or make MFA worse
how to talk to Boards about cyber
effective communications in a cyber incident

Microsoft, Cybersecurity and Infrastructure Security Agency (CISA) and the National Cybersecurity Alliance (NCA) Be Cybersmart Kit for Cybersecurity Awareness Month (which is October) also focuses on the basics: use strong passwords and consider a password manager; turn on MFA; learn to recognize and report phishing; keep software updated.

Quantum tech: ICO views; UK government response on regulating quantum applications; cybersecurity risks from quantum computing and steps for financial authorities and institutions (see the G7 Cyber Expert Group statement on planning for the opportunities and risks of quantum computing)

US & transfers: Commission's report on the first periodic review of the functioning of the adequacy decision on the EU-US Data Privacy Framework (DPF). Separately, industry body CCIA's comments on digital trade barriers affecting US companies include, for the EU (detailed PDF), data and infrastructure localization mandates and restrictions on cloud services (citing e.g. the EUCS, NIS2, Data Act), and restrictions on cross-border data flows (under not just GDPR but also the Data Act and Data Governance Act)

Other ICO:

Levales solicitors reprimand: "A threat actor accessed Levales’ cloud-based server using legitimate credentials and subsequently published data on the dark web". Levales "did not have Multi-Factor Authentication (MFA) in place for the affected domain account. Levales relied on computer prompts for the management and strength of password and did not have a password policy in place at the time of the incident. The threat actor was able to gain access to the administrator level account via compromised account credentials. Levales Solicitors LLP have not been able to confirm how these were obtained." And see above, NCSC and cybersecurity awareness month guidance reiterating the importance of using MFA, especially for cloud!
New data protection audit framework launched, including toolkits (on areas like security, personal data breach detection/prevention, AI), framework trackers (similar areas), resources, case studies
From 11 Oct 24, businesses must try online resources "Instead of first calling our phone line..." - will the expected increase in the data protection fee change this?
Children's data: ICO's further short consultation on its Children's Code (on use of children’s personal information in recommender systems, use of PD of children <13) has closed, sorry I didn't have time to blog it earlier this month
Cyber investigations/incidents: latest datasets, for Q1 24/25 published
ICO DPIA for its use of Canva - interestingly, here as in some other FOI responses, the ICO redacted internal tech info like, in this case, detailed links: "The disclosure of extended links reveals the ‘make up’ of our SharePoint system. Due to the nature of information this reveals, this information increases our vulnerability to cyber attacks."

Is security by obscurity really the best approach here? Previously, when asked for a "list of all the variable names in the database, together with any descriptive/user guides of the variable names in the database list of all the variable names in the database, together with any descriptive/user guides of the variable names in the database" for the ICO's database of data security incident trends, the ICO refused, saying "if disclosed, such information could be used by malicious actors seeking criminal access to our information and systems". It even took the view that "The size of our internal security team is exempt from disclosure to you under section 31(1)(a) of the FOIA, as it could make the ICO more vulnerable to crime".

Facial recognition:

Facewatch investigation FOI, unfortunately ICO's letter to Facewatch was heavily redacted, surely it would be useful for others to see what actions Facewatch took or would take, and further actions recommended, that resulted in the ICO not pursuing Facewatch?
ICO letter to Parliamentarians on facial recognition and biometrics
(Separately, UK National Crime Agency's facial recognition plans)

One court order for winding-up (liquidation) on ICO petition in Q2 24/25, wonder who?

Cyber Security Breaches Survey (UK, annual): how could this be developed and improved? DSIT call for views (survey questions), deadline 23:59, 4 Nov 24.

Cloud: NIST's A Data Protection Approach for Cloud-Native Applications (note: here "data protection" means protecting all types of data, not just personal data), and see NCSC on MFA and cloud

UN Cybercrime Convention: concerns continue to be raised (see other critiques summarised in my book and free PDF).

Adtech: the IAB has published its Repository of European IAB’s Initiatives for Responsible Digital Advertising with helpful links to its key docs on data protection, DSA etc. It also published, for consultation, a proposed privacy-centric Attribution Data Matching Protocol (ADMaP), a data clean room interoperability protocol for attribution measurement (tech specs) "that enables advertisers and publishers to measure attributions using Privacy Enhancing Technologies (PETs) in a Data Clean Room (DCR) and protecting their user’s Personal Identifiable Information".

GDPR non-material damage: CJEU case, reiterating that mere GDPR infringement isn't damage, but an apology could be sufficient compensation if previous position can't be restored, as long as it's full compensation; controller attitude/motivation irrelevant in awarding smaller compensation than the damage suffered. (I'd add, an apology is not full compensation without a binding promise not to do something similar again in future!)

GDPR Procedural Regulation: EDPB statement; the Council's Data Protection Working Party will be discussing the draft Regulation on 24 Oct 24.

Digital identity:

G7 mapping exercise of digital identity approaches (ministerial declaration)
Note proposed EU regulations on digital identity cards and passports/travel documents under the European Digital Identity Framework Regulation (EIDAS2), that amended the EIDAS Regulation; initiatives are proceeding to build EU digital identity/EUDI wallets (tech specs, but why not "digital identity purses/handbags", I ask?!), with EU security agency ENISA having been requested to support the certification of EUDI Wallets, including the development of a candidate European cybersecurity certification scheme under the Cybersecurity Act. ENISA is now calling for experts to join its ad hoc working group for such certification, deadline 18 Nov 24.
NIST is consulting on Attribute Validation Services for Identity Management: Architecture, Security, Privacy, and Operational Considerations (e.g. validating age>18), deadline 8 Nov 24
Biometrics, facial recognition - see Irish DPC's investigation into biometrics for customer verification.

Other EDPB:

Adopted a raft of docs including

Opinion 22/2024 on certain obligations following from the reliance on processor(s) and sub-processor(s), produced on the Danish SA's request (industry association BSA has raised concerns that these requirements are at odds with market practice, supply chain relationships, etc.)
For consultation, Guidelines 1/2024 on processing of personal data based on Article 6(1)(f) GDPR, deadline 20 Nov 24

Note: I've not read properly but there's at least one oddity. The cases the EDPB relied on to argue that personalised advertising is "direct marketing" don't actually say that. "However, CJEU case law suggests that personalised advertising could be considered a form of direct marketing" - well no, the para referenced stated processing for direct marketing may be for legitimate interests, not that personalised ads are direct marketing! Similarly, arguments about "communications" being for direct marketing skate over the case cited clearly being about "electronic mail" as defined in the ePrivacy Directive. I think we'd all agree that ads in emails are direct marketing, but the EDPB seems to be arguing that, under that case, all commercial communications like personalised ads are direct marketing. This can't follow from that case, which is clearly confined to "communications covered by Article 13(1)" of the ePrivacy Directive such as email.

Work programme 24-25
Granting Kosovan Information and Privacy Agency observer status for the EDPB's activities (contrast the polite No post-Brexit to the UK's then Information Commissioner, in a letter whose reference, coincidentally or not, was "OUT2020-0110"!)
Next coordinated enforcement action in 2025 will be on erasure (right to be forgotten, RTBF)

Final Guidelines 2/2023 on Technical Scope of Art. 5(3) of ePrivacy Directive i.e. "cookie" consent but much more; local processing, like on-device processing for AI/machine learning, is still caught according to the EDPB, if anything is sent to the "entity producing the client-side code". Small AI models that can "fit" on user devices are emerging, and may represent the only way forward for users who want AI applications on their phones, at this rate!
Response to the European Commission concerning the EDPB work on the interplay between EU data protection and competition law (DMA etc.: still working on it!)

For amusement value only: ICO FOI response, non!

(See also blog on AI and, just because, UK Attorney-General's speech on the rule of law in an age of populism, Commission webinars on development of model or standard contractual terms for data sharing and switching between data processing services i.e. cloud services under the EU Data Act, and EU Digital Services Act DSA transparency database researchers' workshop)

Things data protection / privacy (some AI), Sept/Oct 2024

2024-10-06T15:59:00.002+01:00

GDPR Procedural Regulation: the Council seems to be progressing this, in October 2024.

CJEU cases: there have been several lately that others have covered, such as on commercial interests possibly being legitimate interests, so I won't for now. I just want to highlight a case from a few months back, which is relevant to employee policies and training/awareness-raising, and possible strict liability to pay compensation to data subjects, at least for infringements arising from employee action/inaction.

Adtech: IAB Tech Lab has launched, for public consultation, its PAIR protocol 1.0 for a "privacy-centric approach for advertisers and publishers to match and activate their first-party audiences for advertising use cases without relying on third-party cookies". Initially donated by Google, PAIR has been developed into "an open standard that enables interoperability between data clean rooms and allows all DSPs to adopt the protocol for enhanced privacy-safe audience targeting".

Equality, AI: The public sector equality duty and data protection, Sept 2024, UK EHRC guidance (with ICO input), including helpful examples of proxy data for protected characteristics under the UK Equality Act 2010, and a short section on proxy analysis of AI models, with a case study on the Dutch benefit fraud scandal that led to unlawful discriminatinon (from using biased predictive algorithms).

Open-source AI: from UK ICO's previously-asked questions, this Q&A was added recently even though currently the "Last updated" date indicates 11 April 2024.
Q: We want to develop a speech transcription service for use in our organisation, using an open-source artificial intelligence (AI) model. Can we do this even though we don’t have detailed information about how the model was trained? (see the answer! It seems call transcription is a popular use of AI, see other Q&A on that webpage on that topic, e.g. this and this. Also, compare a Danish SA decision from June 2024 on the use of AI to analyse recordings of phone calls.)

Oral disclosures?: talking of contrasting approaches, compare a Polish SA decision holding that oral disclosure of personal data during a press conference was not in breach of GDPR, whereas an Icelandic SA decision ruled that oral disclosures by police under the Law Enforcement Directive infringed that Directive.Yes, different laws, but they ought to be interpreted consistently. And I don't get how oral statements amount to "processing" wholly or partly by automated means under EU data protection laws, just as I don't get how there have been so many fines in the EU/UK regarding paper records without first holding that they form part of a "filing system" as defined.

ICO big PSNI fine: well-known by now (news release, MPN), but it underlines the point that the many surnames can be unique, and indicate religion and/or ethnicity (see Equality above on proxy data).

ICO: selected recent ICO disclosures, that the ICO decided to publish following FOI requests to it:

How the ICO assesses incidents / possible personal data breaches: ICO internal guidance (request, PDB assessment methodology as of June 2023); seems to be based on ENISA's risk assessment for PDBs, which is unsurprising as that has been endorsed by both EDPB and ICO
Territorial scope under UK GDPR, DPA 2018: ICO internal guidance (request, copy)
What's a restricted transfer outside the UK: ICO internal guidance (request, copy); taking the outdated and misguided view that "transfer" is based on transfer of personal data's physical location, which is at odds with the ICO's own public guidance on transfers!
How does ICO decide whether to publicise its intention to fine (request, emails on decision, more info)? This was on one concrete situation, but it's helpful to know the factors, again unsurprising, which I summarise below:

ICO default posture of transparency, although it considers each circumstance.
This is consistent and fair with other similar cases where it has publicised the information at this stage.
For deterrence regarding perceived central provider issues: "We are seeing a pattern of central providers having security issues with consequences for patients, publishing this will act as a learning/ deterrent for other processors with large central contracts, including the provisional fine will help clarify the seriousness of these issues".
"The case has been extremely well reported and is well known, so this reduces the potential additional impact on the organisation and there is limited dispute about the facts of the attack."
"Publishing the NOI [notice of intention to fine] and the provisional fine will help improve information rights practice and compliance among those we regulate."
While it is possible that the fine value will change, as it is "provisional and subject to reps", this was balanced "the possible criticism of the ICO for changing the fine amount as the process concludes vs. the benefit of being transparent about the process... Idemonstrating that, if it does change, that is proof that the ICO does consider reps carefully and takes action based upon reps. This can serve to increase confidence in and awareness of our processes. I am comfortable that, subject to including suitable language to make clear it is provisional, that this risk is managed and the benefit is greater."
"in this case, I have decided that publicity at this point allows for improved public protection from threat and hence is overridingly in the public interest. It is also already in the public domain."

DRCF: UK regulators the Digital Regulation Cooperation Forum are seeking input on their 2025/26 workplan by 8 Nov 2024. Unsurprisingly, the work includes AI, but also bilateral work on data protection and online safety, competition and data protection and illegal online financial promotions, and risks and opportunites of emerging technologies like digital identity, digital assets and synthetic media.

Data protection fee: The consultation on increasing the UK data protection fee has closed. The ICO's own response supported the increase, but didn't advocate for any change in the bases for charging the fee, although the government was open to views on that, so it seems there will just be an increase in fee levels but no substantive changes to the bases.

Dark patterns: while not limited to data protection, see OECD dark patterns on online shopping: countdown timers, hidden information, nagging, subscription traps, forced registration and privacy intrusions, cancellation hurdles. Not dissimilar to the issues previously raised by UK regulators ICO and CMA on online choice architecture, control over personal data and harmful designs in digital markets.

Data transfers under the UN Digital Compact ("a comprehensive framework for global governance of digital technology and artificial intelligence"): the text is a bit vague and general on cross-border data flows, and 2030 is not exactly near-term!:

46. Cross-border data flows are a critical driver of the digital economy. We recognize the potential social, economic and development benefits of secure and trusted cross-border data flows, in particular for micro-, small and medium-sized enterprises. We will identify innovative, interoperable and inclusive mechanisms to enable data to flow with trust within and between countries to mutual benefit, while respecting relevant data protection and privacy safeguards and applicable legal frameworks (SDG 17).
47. We commit, by 2030, to advance consultations among all relevant stakeholders to better understand commonalities, complementarities, convergence and divergence between regulatory approaches on how to facilitate cross-border data flows with trust so as to develop publicly available knowledge and best practices (SDG 17)...

...We encourage the working group to report on its progress to the General Assembly, by no later than the eighty-first session, including on follow-up recommendations towards equitable and interoperable data governance arrangements, which may include fundamental principles of data governance at all levels as relevant for development; proposals to support interoperability between national, regional and international data systems; considerations of sharing the benefits of data; and options to facilitate safe, secure and trusted data flows, including cross-border data flows as relevant for development (all SDGs).

But on data protection more broadly, Objective 4. Advance responsible, equitable and interoperable data governance approaches, data privacy and security:

"We recognize that responsible and interoperable data governance is essential to advance development objectives, protect human rights, foster innovation and promote economic growth. The increasing collection, sharing and processing of data, including in artificial intelligence systems, may amplify risks in the absence of effective personal data protection and privacy norms...
...We commit, by 2030, to: (a) Draw on existing international and regional guidelines on the protection of privacy in the development of data governance frameworks (all SDGs); (b) Strengthen support to all countries to develop effective and interoperable national data governance frameworks (all SDGs); (c) Empower individuals and groups with the ability to consider, give and withdraw their consent to the use of their data and the ability to choose how those data are used, including through legally mandated protections for data privacy and intellectual property (SDGs 10 and 16); (d) Ensure that data collection, access, sharing, transfer, storage and processing practices are safe, secure and proportionate for necessary, explicit and legitimate purposes, in compliance with international law (all SDGs); (e) Develop skilled workforces capable of collecting, processing, analysing, storing and transferring data safely in ways that protect privacy (SDGs 8 and 9).

Survey on attitudes and awareness of emerging technologies, data protection, and digital products: There was a recent government survey of the UK public on the level of adoption and awareness of blockchain and immersive virtual worlds, attitudes towards pricing on digital platforms and behaviours regarding personal data control. But I can't yet find a summary of its outcomes, just the raw data.

Hungary: the Commission's decision to refer Hungary to the CJEU argues that Hungary's national law on the Defence of Sovereignty is in breach of EU law, including the e-Commerce Directive, the Services Directive, as well as EU Data protection legislation.

Canada: if attacker accesses and encrypts data without exfiltration for ransom purposes, that is still considered a breach that must be notified to affected individuals under Ontario’s Personal Health Information Protection Act (PHIPA), and the Child, Youth and Family Services Act (CYFSA).

Facial recognition & privacy / personal data: interesting and scary, students managed to adapt smart glasses to look up info on strangers in real-time, including parents' names!

(Also please see my blogs last week on security and AI: both have also been updated with more Sept links.)

GDPR compensation: strict liability? employee training / awareness

2024-10-06T15:42:00.003+01:00

Case C‑741/21, GP v juris GmbH is not a recent judgment, but it still bugs me. Yes, it clarifies that mere infringement of GDPR provisions giving data subjects rights doesn't in itself necessarily constitute non-material damage, and that factors for determining fines, including when the same processing infringes multiple provisions, don't apply when determining damages for Art.82 compensation purposes.

However, what concerns me is this: the court also said, "it is not sufficient for the controller, in order to be exempted from liability under paragraph 3 of that article [Art.82], to claim that the damage in question was caused by the failure of a person acting under his or her authority, within the meaning of Article 29 of that regulation." And:

"...it cannot be sufficient for him or her to demonstrate that he or she had given instructions to persons acting under its authority, within the meaning of Article 29 of that regulation, and that one of those persons failed in his or her obligation to follow those instructions, with the result that that person contributed to the occurrence of the damage in question.
53 If it were accepted that the controller may be exempted from liability merely by relying on the failure of a person acting under his or her authority, that would undermine the effectiveness of the right to compensation enshrined in Article 82(1) of the GDPR, as the referring court noted, in essence, and would not be consistent with the objective of that regulation, which is to ensure a high level of protection for individuals with regard to the processing of their personal data."

Where should the line be drawn, then? It seems that, at least in the UK, a controller is not responsible for the acts of a rogue employee, who clearly becomes a controller in their own right. But if, despite an employer giving clear instructions to its employees, providing them with training, and implementing awareness-raising measures, a careless, mistaken or ignorant employee does something they shouldn't have (or doesn't do something they should have), and that results in the employer infringing GDPR, the employer is now still liable to compensate affected data subjects for the damage, including non-material damage, that they suffer arising from the infringement.

It had generally been thought that proving the organisation conducted training and awareness-raising measures would help it, at least perhaps in relation to potential fines for security breaches or the amount of fines, and some national regulators have taken post-breach training/awareness-raising measures into account there. Indeed, regulators generally consider that employee training/awareness measures are essential to comply with Art.32. However, it looks like such measures will not help employers to reduce or avoid compensation claims, at least under the EU GDPR.

Hopefully, given that regulators expect employee training/awareness-raising, this case won't result in organisations deciding to stop providing clear instructions/policies and training and awareness-raising measures for their employees, whether on security or other GDPR requirements. But, it doesn't exactly incentivise such measures... though it will certainly incentivise data subjects to claim compensation, including perhaps collective action lawsuits directly or through representatives, in cases where infringements were caused by the controller's employee(s) not following instructions or their training. Proving that a controller "is not in any way responsible for the event giving rise to the damage" under Art.82(3) is a tough ask, but Art.82(3) says what it says. Effectively, this seems to create strict liability for compensation, unless the controller can disprove causation. Talk about rock and a hard place...

Things cyber security, summer / Sept 2024

2024-10-01T08:21:00.010+01:00

Software acquisition: procurement teams acquiring third-party software may find useful NIST's list of questions (PDF) to ask and security considerations relevant before, during and after procurement; e.g. some of those questions could be included in contractual warranties and/or due diligence questionnaires. See also CISA's related Software Acquisition Guide for Government Enterprise Consumers: Software Assurance in the Cyber-Supply Chain Risk Management (C-SCRM) Lifecycle (PDF, spreadsheet), again useful for private sector organisations too.

Personal data breaches/PDBs: an SA is not required to fine/enforce for a PDB if that's "not appropriate, necessary or proportionate to remedy the shortcoming found and to ensure that that regulation is fully enforced" (Case C‑768/21, TR v Land Hessen).

Revised EU Product Liability Directive: the new EU Parliament has approved the text (Eur-Lex), so it just remains for the Council to adopt it (although Estonia is against the procedural rules); when published in the OJ thereafter, it will become law. Significance? For the purposes of no-fault liability for defective products, "product" will explicitly include software including that supplied via SaaS. Note the emphasis on safety and cyber vulnerabilities.

Art.7(2): "In assessing the defectiveness of a product, all circumstances shall be taken into account, including... (f) relevant product safety requirements, including safety-relevant cybersecurity requirements..."
Also see the Recitals:"A product can also be found to be defective on account of its cybersecurity vulnerability, for example where the product does not fulfil safety-relevant cybersecurity requirements... relevant product safety requirements, including safety-relevant cybersecurity requirements, and interventions by competent authorities, such as issuing product recalls, or by economic operators themselves, should be taken into account in the assessment of defectiveness. Such interventions should, however, not in themselves create a presumption of defectiveness...The possibility for economic operators to avoid liability by proving that the defectiveness came into being after they placed the product on the market or put it into service should be restricted when a product’s defectiveness consists in the lack of software updates or upgrades necessary to address cybersecurity vulnerabilities and maintain the safety of the product... manufacturers should also not be exempted from liability for damage caused by their defective products when the defectiveness results from their failure to supply the software security updates or upgrades that are necessary to address those products’ vulnerabilities in response to evolving cybersecurity risks [unless not in their control e.g. owner fails to install it; yet, no obligation under this law to provide updates/upgrades but see CRA below]... a third party exploiting a cybersecurity vulnerability of a product. In the interests of consumer protection, where a product is defective, for example due to a vulnerability that makes the product less safe than the public at large is entitled to expect, the liability of the economic operator should not be reduced or disallowed as a result of such acts or omissions by a third party. However, it should be possible to reduce or disallow the economic operator’s liability where injured persons themselves have negligently contributed to the cause of the damage, for example where the injured person negligently failed to install updates or upgrades provided by the economic operator that would have mitigated or avoided the damage."

EU DORA Regulation, financial entities: there are corrections in the versions for FR, RO, SL [sic, SI?]

UK Cyber Security and Resilience Bill: while the UK recently designated data centres as Critical National Infrastructure (CNI), the CPNI list doesn't seem to have been updated accordingly yet. Note, this is not the same as extending the UK NIS Regulations to cover data centres (as the EU NIS2 Directive will do, though it's inapplicable in the UK post-Brexit). However, DSIT has indicated in its Sept newsletter (updated: now on gov.uk) that the Bill will strengthen UK’s cyber resilience and ensure the critical infrastructure and essential services are more secure, by "strengthening the UK’s only cross-sector cyber legislation – the Network and Information Systems (NIS) Regulations 2018. Measures will include expanding the remit of the regulation to protect more digital services and supply chains". And just out: a DSIT webpage on this Bill. Currently it says little more about the Bill that what was in the King's Speech background PDF, but it does indicate that this Bill will be introduced to Parliament in 2025. (On ransomware under the Bill, please see below.)

Ransomware: in late 2023, Interpol and 50 countries including the UK signed a Counter Ransomware Initiative (CRI) joint statement on ransomware payments (US press release). The European Commission has now been authorised to negotiate, on behalf of the EU, the International Counter Ransomware Initiative 2024 Joint Statement (background on CRI). UPDATED: now see the full CRI guidance for organisations during ransomware incidents (news release).

(In May 2024, the UK NCSC with insurance industry bodies had issued Guidance for organisations considering payment in ransomware incidents, and the King's Speech detailed PDF in July 2024 stated that the forthcoming Cyber Security and Resilience Bill will be, among other things, "mandating increased incident reporting to give government better data on cyber attacks, including where a company has been held to ransom".)

UK communications providers & security: Ofcom updated its Network and Service Resilience Guidance for Communications Providers for telcos in early Sept 2024, following consultation. Ofcom said, "Specifically, we are making clear that we expect them to: ensure networks are designed to avoid or reduce single points of failure; make sure key infrastructure points have automatic failover functionality built in, so traffic is immediately diverted to another device or site when equipment fails; and set out the processes, tools and training that should be considered to support the requirements on resilience".

Proposed EU CSAM Regulation: the Global Encryption Coalition is concerned about the Hungarian Presidency's 9 Sept 2024 compromise text, which would still require scanning of encrypted messaging services, undermining encryption and accordingly security and privacy. The Presidency is pushing for a partial general approach at the Council by as soon as 10 Oct 2024! (Good encryption FAQ).

Passwords: NIST's latest draft Digital Identity Guidelines: Authentication and Authenticator Management now states, among other things, that passwords:

Minimum - "shall" be required to be 8 characters minimum, and "should" be required to be 15 characters minimum
Maximum - "should" accept 64 characters (to enable passphrases)
Types of characters - "should" accept ASCII, space, Unicode; but "shall" NOT require other composition rules like a mix of different character types - unlike what most organisations currently require!
Change - "shall not" be required to be changed by users periodically (again unlike what too many organisations do), but change "shall" be required if there's evidence the "authenticator" was compromised (cf. that the password itself was compromised)
No storage of password hints accessible to unauthenticated people (e.g. not logged in), and no prompts for knowledge-based authentication (like first pet's name) or security questions when choosing passwords

(Added: security guru Bruce Schneier approves of these changes!)

Payment webpages: fines have been imposed on companies under GDPR because their payment webpages got hacked, directly or indirectly, enabling criminals to capture customers' payment card details for fraud. The recent Frame Watch feature of ReportURI, helmed by noted security expert Scott Helme (if you'll forgive the pun!) alongside its existing Script Watch and Data Watch features, looks helpful to monitor and provide alerts for suspicious activity on payment pages.

Cloud forensics: post-data breach forensics on cloud services isn't easy. NIST's Cloud Computing Forensic Reference Architecture document, from July 2024, suggests ways to implement cloud architecture to faciliate forensics.

Aligning US federal agencies' cyber defence: CISA's priority areas aren't surprising: asset management, vulnerability management, defensible architecture, cyber supply chain risk management, and incident detection and response. The tricky bit is, of course, aligning systems/processes accordingly, e.g. by increasing operational visibility of assets, managing the attack surface of Internet-accessible assets, securing cloud applications etc., under its Federal Civilian Executive Branch (FCEB) Operational Cybersecurity Alignment (FOCAL) Plan. Again, much of this is of use to the private sector too.

Also of interest:

The recently-published UK MoD's annual analysis of future global strategic trends 2024 mentions cyber and AI, unsurprisingly
NIST on Building a Cybersecurity and Privacy Learning Program
US proposed rule intended "to Secure Connected Vehicle Supply Chains from Foreign Adversary Threats"
US White House report, Roadmap to Enhancing Internet Routing Security
Many countries' agencies' Guidance on Detecting and Mitigating Active Directory Compromises (news release)
Initiatives on closing the cyber skills gap (what about encouraging career changers, part-timers, retired people etc., not just incentivising teens/younger people into cyber careers?)
UK NCSC on software bills of materials (SBOM)
Dissection of exactly what went wrong leading to the 2023 UK emergency services outage - this sort of detailed sequencing of events, root causes etc. is always helpful for others to learn from
Microsoft's Secure Future Initiative (for secure by design, secure by default, secure operations) - progress update and Sept 2024 report, initiated after attacks like the well-known successful attacks on Exchange for email. The SFI's 6 key pillars represent security 101 for all: protect identities and secrets, protect tenants and isolate production systems, protect networks, protect engineering systems, monitor and detect threats, accelerate response and remediation
ADDED Cloud is now quite mainstream, but the cloud adoption checklist in US CISA's Considerations for Public Safety Cloud Computing Adoption from Jul 24 is generally useful for all organisations, even those not in public safety, and covers much more than just security issues
ADDED Hybrid cloud risks: Microsoft has noted threat actor Storm-0501 compromising on-prem environments then moving laterally "from on-premises to cloud environment, leading to data exfiltration, credential theft, tampering, persistent backdoor access, and ransomware deployment", targeting many US public and private sector organisations
ADDED ENISA Threat Landscape 2024 identifies 7 prime cybersecurity threats, topped by threats against availability, then ransomware and threats against data, and analyses many publicly-reported incidents
ADDED Canada: if attacker accesses and encrypts data without exfiltration e.g. for ransom purposes, that is still considered a breach that must be notified to affected individuals under Ontario’s Personal Health Information Protection Act (PHIPA), and the Child, Youth and Family Services Act (CYFSA).
ADDED NIS2 Directive, very little time left before it's fully applicable! - draft national law implementations include:

Ireland's National Cyber Security Bill 2024 (from 30 Aug 24)
Italy

Things AI, Sept 2024

2024-09-29T21:18:00.008+01:00

Open-source AI models: from ICO's previously-asked questions, this Q&A was added recently even though currently the "Last updated" date indicates 11 April 2024.
Q: We want to develop a speech transcription service for use in our organisation, using an open-source artificial intelligence (AI) model. Can we do this even though we don’t have detailed information about how the model was trained? (see the answer!)

AI Act: from Deloitte's AI Act Survey 2024, not many companies surveyed have started prep, nearly half feel partially/poorly prepared, over half think the Act constrains their innovation capabilities in AI, there were mixed views on legal certainty and on the Act's impact on trust in AI, and almost half thought the Act's more of a hindrance to AI-based applications! But, over a 100 companies have signed the Commission's voluntary AI Pledge under its AI Pact, that seeks to encourage organisations to implement AI Act measures ahead of its formal applicable dates.

Beyond the AI Act, see more generally:

the European Commission's policy brief on Competition in Generative AI and Virtual Worlds, and
Europol's report on AI and policing and its benefits and challenges for law enforcement (summary).

Revised EU Product Liability Directive: the new EU Parliament has approved the text (Eur-Lex), so it just remains for the Council to adopt it (although Estonia is against the procedural rules); when published in the OJ thereafter it will become law. Significance? For the purposes of no-fault liability for defective products, "product" will explicitly include software including that supplied via SaaS. The text also mentions software as including AI systems. Also:

"A developer or producer of software, including AI system providers within [AI Act] should be treated as a manufacturer"... "Where a substantial modification is made through a software update or upgrade, or due to the continuous learning of an AI system, the substantially modified product should be considered to be made available on the market or put into service at the time that modification is actually made."

"National courts should presume the defectiveness of a product or the causal link between the damage and the defectiveness, or both, where, notwithstanding the defendant’s disclosure of information, it would be excessively difficult for the claimant, in particular due to the technical or scientific complexity of the case, to prove the defectiveness or the causal link, or both... Technical or scientific complexity should be determined by national courts on a case-by-case basis, taking into account various factors. Those factors should include...the complex nature of the causal link, such as... a link that, in order to be proven, would require the claimant to explain the inner workings of an AI system... ...in a claim concerning an AI system, the claimant should, for the court to decide that excessive difficulties exist, neither be required to explain the AI system’s specific characteristics nor how those characteristics make it harder to establish the causal link."

EU Cyber Resilience Act (CRA) on "horizontal cybersecurity requirements for products with digital elements": the new EU Parliament has approved the text (Eur-Lex), so it just remains for the Council to adopt it; when published in the OJ thereafter, it will become law. Note, this aims to "set the boundary conditions for the development of secure products with digital elements by ensuring that hardware and software products are placed on the market with fewer vulnerabilities and that manufacturers take security seriously throughout a product’s lifecycle". Also note, "Products with digital elements classified as high-risk AI systems pursuant to Article 6 of [AI Act] which fall within the scope of this Regulation should comply with the essential cybersecurity requirements set out in this Regulation..." (see much more in Art.12 and Rec.51 which specifically cover high-risk AI systems, and Art.52(14)). BTW, the Commission is inviting cybersecurity experts to apply to join its CRA Expert Group. Various criticisms of the CRA have been mentioned in my book/free companion PDF; here's another critique.

EU AI Liability Directive: added - Proposal for a directive on adapting non-contractual civil liability rules to artificial intelligence: Complementary impact assessment from the EPRS (as requested by a Europarl committee) "proposes that the AILD should extend its scope to include general-purpose and other 'high-impact AI systems', as well as software. It also discusses a mixed liability framework that balances fault-based and strict liability. Notably, the study recommends transitioning from an AI-focused directive to a software liability regulation, to prevent market fragmentation and enhance clarity across the EU" (PDF).

UK: the AI Act doesn't apply in the UK post-Brexit, so perhaps there are indeed more AI opportunities in the UK, on which Google has published a blog and fuller paper. The UK will make the AI Safety Institute (AISI) a statutory body as well as "identifying and realising the massive opportunities of AI" including for government/public services. (Here, the UK's not alone: a study for the European Commission emphasises AI's "significant potential" to improve EU public sector services.) AISI work includes assessing AI capabilities, e.g. Early Insights from Developing Question-Answer Evaluations for Frontier AI.

But, the GDPR still applies in the UK: ICO statement on LinkedIn's changes to its AI policy, so it is no longer training genAI models using UK users' data (opt-out link for others). There was separately an AI opt-out hoax that fooled a lot of people!

The recently-published UK MoD's annual analysis of future global strategic trends 2024 mentions cyber and AI, of course. UK civil servants (but not the rest of us!) are being offered free training AI-related courses, covering various aspects of AI, illustrating what's considered most important: Fundamentals, Understanding AI Ethics, The business value of AI, Gen AI Tools and Applications, Working with Large Language Models, Machine Learning and Deep Learning, Natural Language Processing and Speech Recognition, Computer Vision, and a Technical Curriculum.

Separately, case studies summarised based on the DSIT AI assurance techniques have been boosted by the addition of more products/platforms, on areas from governance, facial recognition e.g. for verification/identification, compliance management and bias assessment (even for NIST AI RMF, ISO, and NYC 144 bias audit with synthetic data!) to AI monitoring/audit. If you're planning to offer AI products to government (or beyond), it wouldn't be a bad idea to get your own products assured and listed similarly.

AI uses in the UK: a great use is autonomous robots to maintain fusion facilities. On health, a "novel ... AI tool, validated using NHS eye imaging datasets... could transform the efficiency of screening for Diabetic Retinopathy (DR)", while the MHRA is calling for applications for manufacturers and developers of AI medical devices to join its AI Airlock regulatory sandbox; and, Reflections on building the AI and Digital Regulations Service. Added: AI platform via QR code for citizen science info on bathing water quality in Devon and Cornwall.

Collaboration on cybersecurity and AI research announced between the UK, US and Canada, to support defence and security

United Nations: much activity on AI, such as the final Governing AI for Humanity report on global AI governance, gaps, and international cooperation.

The recently (and almost simultaneously) promulgated UN Digital Compact is "a comprehensive framework for global governance of digital technology and artificial intelligence":

Objectives agreed included: "Enhance international governance of artificial intelligence for the benefit of humanity"
Principles agreed included: "Safe, secure and trustworthy emerging technologies, including artificial intelligence, offer new opportunities to turbocharge development. Our cooperation will advance a responsible, accountable, transparent and human-centric approach to the life cycle of digital and emerging technologies, which includes the pre-design, design, development, evaluation, testing, deployment, use, sale, procurement, operation and decommissioning stages, with effective human oversight"
On Digital public goods and digital public infrastructure: "We recognize that digital public goods, which include open-source software, open data, open artificial intelligence models, open standards and open content that adhere to privacy and other applicable international laws, standards and best practices and do no harm, empower societies and individuals to direct digital technologies to their development needs and can facilitate digital cooperation and investment... ...We commit by, 2030, to: (a) Develop, disseminate and maintain, through multi-stakeholder cooperation, safe and secure open-source software, open data, open artificial intelligence models and open standards that benefit society as a whole (SDGs [Sustainable Development Goals] 8, 9 and 10)
On Objective 3. Foster an inclusive, open, safe and secure digital space that respects, protects and promotes human rights, they "urgently... Call on digital technology companies and developers to continue to develop solutions and publicly communicate actions to counter potential harms, including hate speech and discrimination, from artificial intelligence-enabled content. Such measures include incorporation of safeguards into artificial intelligence model training processes, identification of artificial intelligence-generated material, authenticity certification for content and origins, labelling, watermarking and other techniques (SDGs 10, 16 and 17).
On Objective 4. Advance responsible, equitable and interoperable data governance approaches, data privacy and security, "We recognize that responsible and interoperable data governance is essential to advance development objectives, protect human rights, foster innovation and promote economic growth. The increasing collection, sharing and processing of data, including in artificial intelligence systems, may amplify risks in the absence of effectivepersonal data protection and privacy norms...
...We commit, by 2030, to: (a) Draw on existing international and regional guidelines on the protection of privacy in the development of data governance frameworks (all SDGs); (b) Strengthen support to all countries to develop effective and interoperable national data governance frameworks (all SDGs); (c) Empower individuals and groups with the ability to consider, give and withdraw their consent to the use of their data and the ability to choose how those data are used, including through legally mandated protections for data privacy and intellectual property (SDGs 10 and 16); (d) Ensure that data collection, access, sharing, transfer, storage and processing practices are safe, secure and proportionate for necessary, explicit and legitimate purposes, in compliance with international law (all SDGs); (e) Develop skilled workforces capable of collecting, processing, analysing, storing and transferring data safely in ways that protect privacy (SDGs 8 and 9)
And Objective 5 was all about AI governance, not quoted in full here but
"We will: (a) Assess the future directions and implications of artificial intelligence systems and promote scientific understanding (all SDGs); (b) Support interoperability and compatibility of artificial intelligence governance approaches through sharing best practices and promoting common understanding (all SDGs); (c) Help to build capacities, especially in developing countries, to access, develop, use and govern artificial intelligence systems and direct them towards the pursuit of sustainable development (all SDGs); (d) Promote transparency, accountability and robust human oversight of artificial intelligence systems in compliance with international law (all SDGs). (Also see UNESCO's consultation from Aug-Sept 2024 with a policy brief summarising emerging regulatory approaches to AI.)
We therefore commit to: (a) Establish, within the United Nations, a multidisciplinary Independent International Scientific Panel on AI with balanced geographic representation to promote scientific understanding through evidence-based impact, risk and opportunity assessments, drawing on existing national, regional and international initiatives and research networks (SDG 17); (b) Initiate, within the United Nations, a Global Dialogue on AI Governance involving Governments and all relevant stakeholders which will take place in the margins of existing relevant United Nations conferences and meetings (SDG 17)."

US: global AI research agenda; proposed Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters (i.e. cloud providers); "This includes reporting about developmental activities, cybersecurity measures, and outcomes from red-teaming efforts, which involve testing for dangerous capabilities like the ability to assist in cyberattacks or lower the barriers to entry for non-experts to develop chemical, biological, radiological, or nuclear weapons." One I missed earlier: the IAF's paper on Risk/Data Protection Assessment (for AI) as Required by U.S. State Privacy Laws.

US FTC: action against "multiple companies that have relied on artificial intelligence as a way to supercharge deceptive or unfair conduct that harms consumers... include actions against a company promoting an AI tool that enabled its customers to create fake reviews, a company claiming to sell “AI Lawyer” services, and multiple companies claiming that they could use AI to help consumers make money through online storefronts."

And some miscellaneous things...

Hallucination issues with LLMs remain: a recent egregious example.

Comparing chatbots: interesting open-source tool to compare different (anonymized) chatbots by asking them the same questions, and do choose the best answer. See its leaderboard, currently OpenAI's o1-preview is top!

Cognitive bias: humans tend to think fluent content (e.g. LLM-generated) is more truthful/useful than less fluent content, which can produce systematic errors. Of course, this tendency is why even hallucinationary genAI output can be trusted and believed by humans!AWS scientists argue that "human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon insights from disciplines such as user experience research and human behavioral psychology".

AI users: apparently have a healthier relationship with work than colleagues who don't use AI! Although of course AI has been the reason for some job cuts.

Interesting article on AI hype and another on the importance of human thought and judgment when using AI.

ADDED:

D&TA Data Provenance Standards v1.0.0 (GitHub) from the Data & Trust Alliance (exec overview), partnering with OASIS. These are billed as "the first cross-industry metadata standards to bring transparency to the origin of datasets used for both traditional data and AI applications": use case scenarios, online metadata generator. Supporters include (from the tech sector) IBM, Meta and Salesforce, but also many from the manufacturing industry. (See also C2PA, Coalition for Content Provenance & Authenticity, on "An open technical standard providing publishers, creators, and consumers the ability to trace the origin of different types of media").
Facial recognition & privacy / personal data: interesting and scary, students managed to adapt smart glasses to look up info on strangers in real-time, including parents' names!

(Also see my separate blogs on privacy / data protection and on security: links now added.)

Browser cookie settings & consumer preferences - UK study

2024-09-16T17:01:00.004+01:00

"Evaluating browser-based cookie setting options to help the UK public optimise online privacy behaviours" (PDF), a study for the UK Department for Science, Innovation and Technology on consumer preferences conducted between Aug and Dec 2023, concluded that:

"...We recommend that any future cookie setting option should be interactive and detailed to a sufficient level that participants understand the real-world impact of accepting or declining a number of different options, e.g. that ‘functional’ cookies include login details, website preferences (language, currency), see Appendix 2, Figure 5. These setting designs secure stronger engagement by breaking participants out of the habit of automatically accepting all cookies purely for the sake of expedient access to the browser; furthermore, participants are satisfied after such a process of critical engagement...

...People remain divided over the idea of browser-based cookies. To improve sentiment, any future browser-based cookie settings should include features that will enhance web users’ feelings of control over their data (e.g. frequent prompts for updates, options to adapt preferences by types of websites, or for specific websites).
Participant engagement and satisfaction improved when they had access to more functionality details, an interactive interface to select their preferences, and timely prompting about privacy. As a result, should browser based cookie management systems replace the website level settings, we recommend that browser-based cookie setting design should attempt to disrupt users’ habits of automatically accepting through novel designs to create a dissonance with what they are used to seeing. Furthermore, any cookie settings that encourage participants to make a privacy-protective choice will lead to higher satisfaction regardless of initial preferences."

But if there's too much detail in the expanded info, users may just ignore them. And I'm not so sure about frequent prompts to users... that doesn't provide a great user experience. Strictly, when users have accepted cookies for a site, they should probably be told about their right to withdraw consent to cookies everytime they return to the site, but that doesn't really happen, at least not in a "disruptive" way, presumably because that's not great for UX too (popups continuing to appear even if you've accepted cookies previously?!).

AI and GPAI developments/info, Sept 2024

2024-09-16T09:00:00.037+01:00

Some AI-related links, which I hope will be of use:

✨Final text of AI Pact pledges, promulgated by the European Commission to get tech companies to comply voluntarily with (at least some of) the AI Act before its formal applicable date. So, the text is not dissimilar from that of the EU AI Act, and indeed the G7 Principles from the Hiroshima Process

✨Human oversight of ADM / automated decision-making: videos from the EDPS's workshop, 3 Sept 24

✨AI Act briefing for European Parliament, 2 Sept 24

✨BSA (The Software Alliance) Best Practices for Information Sharing Along the General Purpose AI Value Chain. There's some overlap with the EU AI Act's GPAI requirements, 3 Sept 24

✨Computer & Communications Industry Association's recommendations on GPAI code of practice, Aug 24

✨Note that the Council of Europe's AI Treaty, signed by the UK, EU and others, will come into force only on the first day of the month following the 3-month period after 5 signatories, including at least 3 Council of Europe member states, have ratified it. (On treaties/conventions, see the differences between signing versus ratification versus accession).

To support the Treaty's implementation, the COE's HUDERIA is a "legally non-binding methodology" for Risk & Impact Assessment of AI Systems for Human Rights, Democracy and Rule of Law (good summary). The UK's Alan Turing Institute is assisting on HUDERIA. The COE Committee on AI is considering HUDERIA soon. The European Commission & Council of the European Union are also involved. What's the betting as to how much HUDERIA will influence what is going to be required in Fundamental Rights Assessments for certain high-risk AI systems under the AI Act?

(Compare the US Department of State's risk management profile for AI and human rights, July 2024)

✨Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025. Certainly, genai risk disclosures in US SEC filings are increasing.

✨A data scientist has written a great outline of a practical approach in the face of the pressure to to AI-ify everything ASAP. Consider, is using AI always the best solution? Especially given that AI systems will increasingly be subject to more onerous obligations than non-AI systems (e.g. under the AI Act), is it always best to use AI when non-AI approaches/methods could work equally well or perhaps better?

✨What's up with the planned AI Liability Directive? It won't automatically be brought into force, but a June Parliamentary Committee report stated that "the legislative work under the leadership of the Committee on Legal Affairs... will continue under the new Parliament. In the meantime, the Committee... requested an additional impact assessment from the European Parliament Research Service (EPRS). It is to primarily deal with the compatibility of the three legal acts mentioned [AIAct, product liability etc] and the risk-based concept and is expected to be completed by the end of 2024". Note the date for the additional impact assessment - end of 2024. So it's very unlikely this Directive will be passed this year, perhaps ever.

(I had tried to post the above on LinkedIn last Thursday, please see this AI developments post, but LinkedIn saw fit to demote it and not show it in people's feeds, so I think it's best to blog here instead, and at least there's space for me to flesh things out a bit more here.)

Are AI principles like standards?

2024-04-14T17:45:00.010+01:00

AI principles, indeed AI standards, seem to be ever proliferating...

Age assurance/verification technologies & privacy/data protection

2023-08-23T11:25:00.011+01:00

Key ICO resources and UK info/standards on age checking/assurance & the Children's Code are below.

ICO work to date on children's privacy and age estimation/verification:

Measurement of age assurance technololgies, jointly commissioned with Ofcom under the Digital Regulation Cooperation Forum (DRCF), Aug 23 (previous report pt.1, Oct 22)
ICO-approved certifications under UK GDPR in 2021 include (news release)

Age Check Certification Scheme (ACCS) testing age assurance products work (tech requirements ACCS 2:2021)
Age Appropriate Design Certification Scheme (AADCS) criteria for age appropriatedesign of information society services (tech requirements ACCS 3:2021)
(I mention this first as many people don't seem to know about them)

Evaluation report on Children's Code & summary, Mar 23
Response to the Call for Evidence and roundtables on age assurance, Feb 23
Age assurance opinion, Oct 21
Seers Child Privacy Consent Management Platform (CPCMP) sandbox report, Oct 21
Yubo Age Appropriate Design Code Audit Report, Oct 22
Yoti age estimation tech for younger children sandbox report, Apr 22 & its AADC audit, Dec 21
Age-AppropriateDesign Code itself

Also relevant:

European Commission's age assurance tool, May 24
Ofcom's Quick guide to children’s access assessments, Protecting children from harms online Volume 5: What should services do to mitigate the risks of online harms to children?, section 15 Age assurance measures, and Ofcom's Annex 10 A10. Draft guidance on highly effective age assurance, May 24
Ofcom's Guidance on age assurance and other Part 5 duties for service providers publishing pornographic content on online services (under UK Online Safety Act 2023), Dec 23
UK's 2022 trials of age verification technology in alcohol sales - interestingly, several using Yoti's age estimation tech - mostly facial, some ID documents, one biometric finger vein
PAS 1296:2018 Online age checking. Provision and use of online age check services Code of Practice and tech requirements ACCS 4:2020 for checks based on that
IEEE Standard for an Age Appropriate Digital Services Framework Based on the 5Rights Principles for Children, Nov 21

Added May/June 2024:

European Commission-commissioned research report Mapping age assurance typologies and requirements Mapping age assurance typologies and requirements (news, exec summaryexec summary), Apr 24
ICO Age assurance for the Children's Code, Jan 24

Windows: try local LLMs easily

2023-07-23T13:10:00.000+01:00

1. Download kobold.cpp.exe from https://github.com/LostRuins/koboldcpp/releases (I picked the latest version)

2. Download the GGML BIN file for the model(s) you want to use - you can get Llama2 models from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML - check for which large language models/LLMs are compatible with Kobold, go to the Files tab to find and download the one(s) you want.

3. For command line avoiders, just doubleclick koboldcpp.exe. A command line interface window and GUI window open up

4. In the GUI window click Model, Browse, select one of the downloaded GGML BIN files then click Launch

5. In your default browser a new tab should open, if not just open a tab yourself and go to http://localhost:5001/ and prompt away!

6. The command line window stays open, with info on the input prompts, output, processing time etc. Just close it and the browser tab when done. All data stays local to your computer, inputs and outputs etc.

NB. you need a lot of RAM, especially for the bigger models.

Thanks to Autumn Skerritt's helpful blog (which also covers Mac & Linux and has other useful info) - I just added info on the GUI and other possible downloadable models I found.

Data Protection & Digital Information (No.2) Bill - key changes from 2022 Bill No.1; GDPR comparisons

2023-03-13T15:55:00.003+00:00

The UK Data Protection & Digital Information (No.2) Bill's key changes from the 2022 Bill, compared with the EU GDPR, are summarised in the table below.

After the table are some "But why didn't they do that?" questions, and "Will compliance with the EU GDPR really comply with the new Bill"?

Table of Key Changes

Only changes from the 2022 version are covered, and only those relating to GDPR (not law enforcement or intelligence services or the DVS trust framework).
Clarifications/typos/minor corrections and other minor textual changes are not covered.
The table below is also not a full comparison of the entire Bill against the EU GDPR.

Abbreviations

ADM
automated
decision-making

C
controller

ICO
UK Information
Commissioner's Office

P
processor

PD
personal data

S
UK Secretary of State

SRI
senior responsible individual

Issue	Cf 2022 version	Cf EU GDPR	Comments/Queries
Personal data	Tighter, as this specifically calls out the role of access protection measures. It’s PD if C/P knows/ought reasonably to know another person obtains/is likely to obtain info as result of C/P processing and the individual is identifiable/likely to be identifiable by that person at the time of processing, (added) including if an unauthorised person obtains info due to the C/P not implementing appropriate measures to mitigate the risk of their obtaining the info.	Clarifies: identifiability is assessed at the time of processing by C/P. Focuses on whether info is PD in the hands of whoever processes it (similar to the position under DPA 1998).	Time of processing - time of whose processing, processing by C, P, either, the other person? If an individual is identifiable to C but not P, or vice versa, does that make them identifiable to both? Why not also mention measures to mitigate the risk of unauthorised persons identifying individuals (e.g. strong encryption), vs. their obtaining the info? Surely such measures are equally important: focus on either/or, not just “obtaining”?
Legitimate interests	New Art.6(9) gives examples of types of processing that may be necessary for LI: - Necessary for direct marketing (defined in both versions as communication (by whatever means) of advertising or marketing material which is directed to particular individuals, and now also to be inserted into Art.4(1)(15A) UK GDPR), - Intragroup transmission necessary for internal admin, or - Necessary for security of network and info systems	Much has been made of this. But actually it’s just based on GDPR Recs.47 last sentence, 48 & 49, putting them into the operative text. Just without the “strictly necessary”, which in my view is very tight particularly in relation to ensuring security. However, "direct marketing" is defined more broadly than in say the European Commission and Council's approach in the draft ePrivacy Regulation - could it include targeted advertising on websites or mobile apps here?	Pity that necessity for preventing fraud Rec.47 wasn’t included, or necessity for the security of PD (not just systems). The scope of "direct marketing" would benefit from clarification, e.g. is "sent" intended or is displaying personalised ads on web/mobile enough to be "direct marketing"?
Scientific research	Clarified: - Even commercial activity can be scientific research - But activities only qualify if they can “reasonably described as scientific”	GDPR doesn’t define scientific research. The Bill just provides helpful clarifications, e.g. drawing on Rec.159 (GDPR doesn’t explicitly exclude commercial research and Art.89 of course requires safeguards there, which the Bill is changing).	Processing PD for studies in the area of public health are “scientific” only if conducted in the “public interest” – clarify “public interest” here? But generally that phrase isn’t defined anywhere… and see queries after this table.
Statistical purposes	Includes processing for statistical surveys or production of statistical results resulting in aggregate non-personal data, but (added) only if controller doesn’t use personal data processed or resulting information to support measures/ decisions regarding a particular data subject to whom the personal data relates	Just clarifications, reflecting Rec.162	-
ADM	Art.22A(2) no longer states that decisions include profiling. I consider this to now reflect the correct interpretation, rather than a relaxation - see the next cell. Instead, when considering whether there's meaningful human involvement, the extent to which the decision was reached by profiling must be considered among other things. That's one way to interpret the profiling reference in Art.22 and it makes some sense. S may make regulations stipulating that certain cases do, or don't, have meaningful human involvement.	Clarifies the debated issue of whether Art.22 only gives rights to data subjects to object to ADM, or positively prohibits ADM. Clarifies that decisions “based solely on automated processing” are those with “no meaningful human involvement”. Clarifies role of profiling, in the debate on whether Art.22 catches profiling per se, or only profiling that leads to ADM (I believe the latter). So, Art.22A(2) now reflects what I feel is the correct interpretation.	A positive prohibition usefully clarifies the position. Similarly with the meaning of automated decisions. Data subjects aren't deprived of rights regarding ADM, because the new Art.22C safeguards must enable data subjects to obtain human intervention and to contest decisions, and individuals can no doubt claim compensation for breach of this explicit prohibition. However, it's unclear why Sch.4 will omit s.14 DPA2018 altogether. Removing the notification requirement may reduce burdens on Cs, but retaining a positive obligation on Cs to consider requests to reconsider decisions could further help to show that data subjects do retain their ADM rights. Perhaps S regulations are intended to address this and other ADM-related issues?
ROPAs (records of processing activities)	Needed only for processing which, taking into account its nature, scope, context and purposes, is likely to result in a high risk to the rights and freedoms of individual - instead of 2022 exemption for <250 employees unless likely to result in high risk C records need include only categories of person with whom C shares PD, rather than named persons. However, there "recipients" has been changed to "persons" in third countries/ international organisations. Amends Art.57(1)(k) to require the ICO to produce and publish a document containing examples of types of processing which it considers are likely to result in a high risk to the rights and freedoms of individuals (for the purposes of Articles 27A, 30A and 35) - i.e., senior responsible individual, ROPAs and assessment of high-risk processing. This helps ensure a consistent view of what is considered "high-risk" across these different areas.	Required for all Cs and Ps with exemption for <250 employees unless processing is likely to result in a risk to rights and freedoms of data subjects, is not occasional, or the includes special category or criminal-related data. Changing "recipients" to "persons" actually goes broader than GDPR, as under GDPR Art.4(9) certain public authorities (again it's not entirely clear which) aren't considered "recipients", so this should be positive for UK adequacy as any sharing with public authorities must definitely be recorded.	Arguably, catching C/Ps even with <250 employees for high-risk processing would catch non-occcasional processing of special category or criminal-related data. While it's "high-risk" vs. "a risk", the latter catches most C/Ps; some might it's say is too strict given realistic risks, especially under EDPB's broad interpretation of Art.30.5's "or". So the Bill is less strict than GDPR, but hopefully that's not significant enough to prejudice UK adequacy. It's odd that the "categories" issue relates to C records (Cs will surely know those they share PD with), rather than DSARs/privacy notices - could the change have been intended for the latter, but inadvertently got inserted here instead?
DPIAs (assessment of high-risk processing)	Deleted ICO's Art.35(4)-(5) obligation to publish list of operations requiring DPIA and power to publish list of operations not requiring assessment. But, see above on the amended Art.57(1)(k) which effectively does the same thing, except that there's no longer power to publish lists of operations not requiring assessment.	No explicit requirement to consult DPO. However, arguably this is implicit in new Art.27B(2)(c), informing/advising of data protection obligations. No Art.35(3) criteria deeming certain types of processing always to be high risk (ADM, large-scale processing of special category/criminal-related data and large-scale systematic monitoring of publicly accessible areas!) The related Art.36 makes prior consultation with ICO optional, but see LinkedIn discussion in comments on whether this makes much difference in practice.	The legislative aim to require assessments for high-risk processing remains, in substance. I suspect the ICO's list of high-risk processing will include the Art.35(3) types! In which case, little difference in practice, but more flexibility. Oddly, there's no explicit power for the ICO to publish lists of activities that are not considered to require assessment as high-risk.
Senior responsible individual	To be designated by public body or likely high-risk processing, but note the amended Art.57(1)(k) regarding an ICO list to be published of what's high-risk processing for this purpose. (The ICO's "high-risk" lists could theoretically be different for SRI, high-risk assessments and ROPA purposes, but they may not be - consistency will be helpful here.)	No more Art.37(1)(b)-(c) criteria deeming certain types of processing always to require a DPO (core activities involve large-scale regular and systematic monitoring or processing special category/criminal-related data). The individual must be part of the organisation’s senior management which arguably goes beyond GDPR. Allowing job-sharing here is enlightened. SRI details must be notified to the ICO. However, there's no longer any "sharing" allowed ot the SRI across different public authorities or a related group.	Given the SRI must be designated in high-risk processing situations, and issues like resourcing and conflicts are clearly covered, is there much difference in practice? Again, I suspect the ICO's list of high-risk processing here will include the Art.37(1)(a) and (b) types! In which case, again, little difference in practice, but more flexibility. No SRI sharing could cause practical problems given the difficulties with recruiting people with data protection expertise! "Outsourcing" of SRI functions might perhaps still be possible as the SRI can alternatively "secure" that certain tasks are performed by another, taking into account expertise etc. Probably SRIs without sufficient privacy expertise (yet!) will have to secure another person (which doesn't seem limited to internal staff) to perform at least some tasks.
Transfers (data exports)	New transitional provisions to "grandfather" valid transfer mechanisms in place before the relevant Bill provisions take effect.	Comparing the transfers provisions generally, e.g. "not materially lower" vs "essentially equivalent", merits a note in itself, and will not be discussed here!	Not discussed here. And it will be up to the European Commission to assess the extent to which these and other changes may affect UK adequacy!

But why didn't they do that?

While the following are points where the 2022 and 2023 versions of the Bill don't differ, some queries spring to mind:

Research processing of special category/criminal-related data - under DPA2018 Sch.1 para.4, such processing is permitted if it's necessary for archiving purposes, scientific or historical research purposes or statistical purposes, is carried out in accordance with Article 89(1) [to be the new Art.84B i.e. safeguards], and is in the public interest. Here, the UK went beyond GDPR, because the "public interest" requirement doesn't appear in Art.9(2)(j). National law permitting such processing just has to be "proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject". Presumably it's a UK policy decision to require the "public interest" condition in addition? If so, giving examples or defining "public interest" here would be helpful as it's such a vague and broad term.
AI bias and anti-discrimination - the June 2022 consultation response intended to expand the DPA2018 sch.1 para.8 exemption, allowing processing of special category data and criminal offence-related data for equality of opportunity or treatment, to permit bias monitoring, detection and correction in AI systems. Surely this is a laudable aim that no one should object to, so it's not clear why this update didn't make it into the Bill?
PECR/cookies

Security - the Bill will allow storage/access to ensure security of the terminal equipment, but why not security of networks/data more broadly given the critical importance of security generally?
Analytics - the Bill would allow first party analytics, but it seems not the use of a third party analytics service, as sharing with third parties is allowed only to enable them to "assist with making improvements to the service or website" - why not also to enable them to assist with collecting that information? SMEs in particular won't have technical expertise to install their own on-prem inhouse analytics solutions, so not including "or collecting that information" there may undermine the legislative objective of easing web/mobile analytics for organisations.

BTW, on DSARs' change from "manifestly unfounded or excessive" to "vexatious or excessive" - the latter phrase has been much discussed (including at regulatory and judicial level), and therefore is well understood in the UK, in the FOI (freedom of information) context. See also the discussion on this in LinkedIn, in the comments section.

Interestingly, the first version of the press release said "Ministers have co-designed the Bill with key industry and privacy partners - including Which? and TechUK..." but the current press release no longer mentions Which?. Input from consumer organisations is obviously important in this context.

Will compliance with the EU GDPR really comply with the new Bill?

I spotted one minor example where strictly, it won't.

Privacy notices will have to include info about the right to complain to the controller, under the Bill. GDPR privacy notices needn't.

But, as per statements at the IAPP UK Intensive on 8 Mar 23, it's very unlikely that the ICO would fine or enforce against Cs lacking that one line (it'll just say, add that in)! And obviously including that extra info won't cause any issues under the EU GDPR.

Key points: EDPB transfers & territorial scope final guidance

2023-02-24T16:21:00.002+00:00

We now have the final version of the EDPB's Guidelines 05/2021 on the Interplay between the application of Article 3 and the provisions on international transfers as per Chapter V of the GDPR

1. Generally, it makes useful clarifications to draft guidance, rather than substantive changes. There are 5 extra examples and new Annex with diagrams for all examples. New Exec Summary. Maria and George remain the same (not Alice or Bob!), but specific third-country names were removed.

2. Most clarifications aren’t surprising e.g. remote viewing/access of/to EEA-hosted personal data from outside EEA whether for support/admin etc. is a “transfer”, including by a processor; EEA platform passing personal data to non-EEA controller is making a “transfer” (“controller” seems a misnomer if the non-EEA entity isn’t subject to GDPR, but the platform is making a transfer whether it is or isn’t)

3. Helpful: controller disclosing personal data to EEA-incorporated processor (with non-EEA parent) – not a “transfer”. If processor discloses to third-country authority, it does so as independent controller. So controllers must assess circumstances for sufficient guarantees before engaging such processors.

4. Also helpful:

when data subjects directly provide personal data to third country controller not subject to GDPR, that’s not a transfer
when data subjects directly provide personal data to third country controller that IS subject to GDPR under Art.3(2) offering/monitoring (added: “specifically targets the EU market”), that’s not a transfer but the controller must comply with GDPR (practical enforceability against it is a different issue of course)
when data subjects directly provide personal data to third country processor for third country controller, they don’t make transfers, but the controller “transfers” to the processor

5. Note: still not a transfer if EEA company employee travels to third country with laptop or remotely accesses EEA-hosted data – it’s within the same entity. New: if the employee in his capacity as such sends or makes available data to another entity in the third country, then that’s a transfer by the company.

6. Non-“transfers”:

New section on safeguards when processing personal data outside the EEA even if technically there’s no “transfer”. Pay “particular attention” to the third country’s legal framework, as there may still be “increased risks” because “it takes place outside the EU, for example due to conflicting national laws or disproportionate government access in a third country”. These risks must be considered for compliance e.g. Art.5 principles, 24 controller responsibility, 32 security, 35 DPIA, 48 transfers not authorised under EU law: “a controller may very well conclude that extensive security measures are needed – or even that it would not be lawful – to conduct or proceed with a specific processing operation in a third country although there is no transfer situation.”
Privacy notices for non-transfers outside EEA!: when a controller intends to process personal data outside the EU (although no transfer takes place), this information should as a rule be provided to individuals as part of the controller’s transparency obligations, e.g. to ensure compliance with the principle of transparency and fairness, which also requires controllers to inform individuals of the risks in relation to the processing”. Non-binding, strictly…

7. Still unaddressed:

Not a “transfer” if it’s within the same legal entity, so e.g. EEA branch of US corp sending personal data to HQ isn't making a transfer, but an EEA subsidiary sending to US parent IS. Obviously the EEA branch would be subject to GDPR, with easy enforceability due to its EEA presence.
Art.3(1) can apply directly to non-EEA “established” entities e.g. in the Costeja case, but EDPB focuses mainly on 3(2), mentioning 3(1) only in relation to processors used by EEA-established controllers. Presumably direct provision of personal data by data subjects to Art.3(1) non-EEA controllers would also not be “transfers”, but the controller is caught by GDPR? (practical enforceability…?)
EEA subprocessor to non-EEA processor – analogy with processor-to-controller transmissions, this must be a “transfer”, but no SCCs exist to allow this… (workaround – adapt P2C SCCs, hey we tried our best!)
The “conflicting laws” issue applies equally to EEA-established organizations that expand to third countries. Remember SWIFT, where using its own US data center was a “transfer”? Presumably now that use alone is not “transfer”, but disclosure to third-country entities would be.

8. My speculations about possible new options for non-EEA controllers:

will some non-EEA controllers just directly collect personal data from EEA data subjects now? They may still be subject to GDPR under Art.3(2) or even 3(1), but practical enforceability…
will some non-EEA groups set up non-EEA subsidiaries to operate branches in the EEA, that can send data “back” outside the EEA without making “transfers”? Of course, those subsidiaries are subject to GDPR, and their disclosure to non-EEA parents will be “onward transfers” that need SCCs etc, but that might be easier for some…

9. Puzzling: most of us share common views on what “made available” involves, but I didn’t follow “embedding a hard drive or submitting a password to a file” – what does that mean, how do they involve “making available” data?

Automated Decision Making (ADM) & GDPR - Flowchart

2022-10-16T22:02:00.001+01:00

ADM under GDPR - I produced this flowchart after noticing that my Imperial AI MSc students were struggling to parse Art.22. Admittedly it's been termed the worst-drafted of all GDPR provisions, rightly, by someone I used to work with, who knows who she is :) I hope it will be useful, and as always all comments are welcome!

UK NIS Regulations: enforcement, & future

2022-07-09T17:32:00.011+01:00

For both OESs and DSPs the UK NIS Regulations have barely been enforced, but change is coming, including to bring MSPs within scope. (OESs are operators of essential services, basically critical infrastructure service providers, while DSPs are "digital service providers": cloud computing service providers, online marketplaces or online search engines only, not other providers of digital services in the broad sense).

The Second Post-Implementation Review of the Network and Information Systems Regulations 2018 (PDF), 4 July 2022, revealed this and other interesting information:

NIS incident reporting hasn't actually been happening: “…the system does not appear to be working. As of this review, competent authorities have received little-to-no reports, despite other sources of information, such as the Breaches Survey, indicating a prevalence of incidents within the wider economy and society.”
NIS enforcement has been minimal; no NIS fines (penalty notices) have been imposed so far:

Only 2 competent authorities have enforced to date, "which raises the question of "is the enforcement regime appropriate?" But, “NCSC has also been informed of one very successful instance of a competent authority carrying out enforcement, which had very positive outcomes, suggesting that the enforcement regime may be appropriate."

Note: it's unclear if the UK ICO, which regulates DSPs under the NIS Regulations, was one of thise two authorities.

“…there is evidence from competent authorities to suggest that there are cases where enforcement activities were merited but no action was taken. The use of enforcement tools overall, is much lower than the reported need and so far competent authorities appear to have been less inclined to make use of their regulatory powers." Why, and why not? The reasons are not stated.
"There is also a reported concern from regulators that the grounds for enforcement (either via enforcement notices or penalty notices) is not clear enough”…
“NIS competent authorities... have additionally reported being very restrictive with their regulatory powers, relying more on regular engagements, inspections, and information notices rather than any binding provisions of the regulations, such as enforcement notices, civil proceedings, or penalty notices.”
"Of those who felt the enforcement regime wasn't proportionate, 44% gave other reasons including there is no clear link between the fine levied and the actions that operator of essential services took prior to the incident and the fact that fines result in double jeopardy as there is already a cost relating to a cyber breach."

Note: it's interesting that the double jeopardy cited was not the possibility of fines under both GDPR and NIS, which is the key double jeopardy risk in my view (to be addressed in the EU's NIS 2 Directive). The breach costs point is, of course, also relevant to GDPR fines too, but cited only sometimes (in conjunction with remediation costs) in GDPR supervisory authority decisions.

The only relevant DSPs who indicated the enforcement regime was not proportionate to the risk of disruption reported feeling that the Regulations were incorrectly applied to DSP organisations in general. (This I agree with, see later below.)
DCMS will aim to collect annual data from the competent authorities e.g. the number of incidents per year, the number of independent audits of the Cyber Assessment Framework, the number of improvement plans as a result of the Cyber Assessment Framework, the number of information notices issued by the competent authorities, the number and nature of enforcement notices issued by competent authorities, and the number of organisations regulated by sector and also the number of SMEs regulated by sector.

NIS Regs' Cyber Assessment Framework: this has allowed experts in competent authorities to review organisations' cyber security arrangements and ensure improvements are made. 67 known operators have received improvement plans (including updating legacy systems and software to reduce vulnerabilities), highlighting Regulations' role in improving cyber security.

Note: the reference was only to "operators". This suggests no DSPs were asked to make any improvements to their cybersecurity under NIS.

NIS Regs generally: effective to drive good cyber security behaviours; "...strong indication that without NIS, cyber security improvements across essential services in the UK would proceed at a much slower pace. ...added benefit of covering a large number of sectors, which is expected to address some of the inconsistencies of managing risks to networks and information systems across sectors...". But, areas of improvement remain, thought to be most appropriately tackled through regulatory intervention, to strengthen and future-proof the regulatory framework.

Other regulations or standards mentioned as drivers for improvements in cyber security included: UK General Data Protection Regulations (GDPR) (13 or 86% of relevant digital service providers, 68 or 78% of operators of essential services); ISO27001 (28% of operators of essential services); Cyber Essentials and Cyber Essentials Plus (11% of operators of essential services); as well as other industry standards (33% of operators of essential services).

Areas needing improvement, and future plans: Then-Minister Lopez's associated statement to Parliament on 4 July noted that recommended changes to the NIS Regs were included in the Department for Digital, Culture, Media & Sport's Jan 2022 consultation, Proposal for legislation to improve the UK’s cyber resilience (summarised in my Linkedin post). The outcome of that consultation is to be published "later this year", i.e. later in 2022. Recent UK political events, including her resignation on 6 July, may of course result in delays to the initially-planned timescale. The key areas are:

DSP registration and guidance: 54% of responding DSPs stated it was not easy to identify that their organisations are in scope (this deters registration, and ICO won't be aware of their activities to advise them!).

"Further work is required to ensure that the guidance makes it easy to identify whether firms are in or out of scope of the Regulations and to ensure that organisations that need to be included in the regulations are designated."
"Registration of digital service providers cannot be left to digital service providers alone... The Government will continue to support the ICO in the work it is already carrying out to identify firms that should be under the Regulations and support them in notifying those organisations of their responsibilities. Both the government and the Information Commissioner, should consider ways to increase awareness of the NIS Regulations with all potential digital service providers." The government should consider options to provide the Information Commissioner with increased information-seeking powers (similar to existing ones available to competent authorities of operators of essential services) to ascertain whether an organisation qualifies as a relevant DSP under the NIS Regulations.

Ensuring the right sectors are caught: managed service providers (MSPs) are not caught currently, but under the Jan 22 consultation they will be. (For other subsectors discussed e.g. BPO, SIEM, analytics & AI, see my Linkedin post, but it seems "While this Post-Implementation Review has not identified any other sectors that need to be included at this time, it has underlined a need for the government to maintain the powers to make such additions in the future.")
Supply chain security: OESs can't monitor supply chains due to lack of supplier cooperation and lack of resources. Action is needed to increase operators’ ability to manage security risks arising from supply chains, particularly suppliers critical to provision of essential services.

Proposed power to designate critical dependencies to identify, impose duties, and then regulate certain supply chain organisations that present systemic risks to OESs, due to their market concentration, reliance on those services, or other factors.

Comment: could IaaS/PaaS, perhaps even some SaaS providers, be caught both as DSP and as critical dependency? - highest common denominator of compliance required there. Also, could IaaS/PaaS providers that are critical enough, simply be designated as OESs themselves (legislative rules permitting)?

DCMS will consider options such as amending guidance to tackle supply chain security concerns, including using standards and certification, such as Cyber Essentials and Cyber Essentials +, to address this issue. But cross-government consultation is needed.
Note: see also the Government response to the call for views on supply chain cyber security, Nov 2021.

Capability & capacity of OESs, DSPS, competent authorities: lack of finance/funding or of general resources, more variable among authorities particularly lack of cyber regulator specific training or centralised NIS training (as opposed to GDPR training). Competent authorities also need more resources for effective enforcement. On authorities' resources:

DCMS will "commit to persuading those departments to ensure that they meet their legal obligations to fund their NIS oversight. For these, plus those regulators that are not central government departments, DCMS aims to ensure that competent authorities are able to recover the costs of regulation from those being regulated, in line with government policy."
Additional ways to improve resource-efficiency will be considered, e.g. promoting collaboration across authorities and with non-NIS authorities such as banking and financial services regulators (for designation of critical dependencies), exploring existing frameworks like CBEST and TBEST to test assumptions and highlight areas for further development.

Incident reporting: thresholds (in statutory guidance) are too high, and base criteria of a reportable incident is too narrow (disruption to the service, cf. impact on NIS) to capture the most high risk incidents risks. To ensure that the right incidents are captured:

Authorities should review reporting thresholds and lower if necessary.
OESs and DSPs will be required to report all incidents that have a material impact on the confidentiality, integrity, and availability of NIS [note: the well known CIA triad], and [note: I think "or" is intended here?] that have a potential impact on service continuity.

Enforcement: DCMS needs to conduct work to assess why the enforcement regime is not being utilised where it is merited.
Consistency and more robust oversight: greater consistency in regulatory implementation across sectors is required, alongside creation of performance metrics to better measure the impact and effectiveness of the Regulations.

DCMS should issue revised and updated guidance to competent authorities, setting out the requirement for a common approach to assessment and performance indicators; explore ways to make such guidance more binding on authorities; and establish a process by which competent authorities report against performance indicators and are held accountable for their performance (indicators could be linked to the delivery of the National Cyber Strategy and its performance framework).

Note also the related consultation on Data storage and processing infrastructure security and resilience - call for views (press release), including data centre infrastructure, cloud platform infrastructure and MSP infrastructure, which expires at the end of Sunday 24 July 2022.

The next UK NIS Regulations review isn't due for another 5 years.

Comments

Below are my personal views only, but they're based on my practical experience of advising clients on the UK NIS Regulations and EU NIS Directive: both their legal and technical/security teams.

Incident reporting:

"There is a lot of uncertainty around the incident response, and which incidents need to be reported...". In my view, this uncertainty is a contributing factor, and guidance is sorely needed, alongside the planned steps mentioned above regarding lowering reporting thresholds and requiring reporting of incidents materially affecting NIS CIA even if not affecting the service.
However, there's a risk of a tsunami of reports that regulators may not be able to cope with, if every incident "materially" impacting C, I or A has to be notified. It's important to bear this factor in mind when setting the reporting test/thresholds. Again, guidance on "materiality" will be vital.

Awareness, scope, DSPs and non-registration: I hope the government will take the opportunity, post-Brexit, to reconsider the scope of the NIS Regulations beyond just bringing MSPs into scope. In particular, please consider whether and to what extent SaaS providers should be caught by the NIS Regulations.

The NIS Regulations were binding from 10 May 2018. Guess what else there was in May 2018? Yep, the GDPR. No surprises then that most organisations focused their resources on GDPR rather than NIS compliance, especially with the huge publicity about GDPR fines and hardly anything being said about NIS.
It's understandable that IaaS/PaaS providers should be subject to the Regulations as DSPs, because many organisations build their own technology infrastructure or customer-facing services on top of those cloud services. I.e., many organisations create their own SaaS services based on third party IaaS/PaaS services, which do constitute technology infrastructure-type services.
However, automatically and unthinkingly copying out the NIST definition of cloud computing is not the right approach here. Applying NIS laws to SaaS is like applying certain laws to "all websites" when they should actually apply to "website hosting platforms/services". SaaS involves the provision of specific applications or services to end users (like a word processing application online, instead of via an application installed on a local computer). Those applications/services can vary hugely in their scope and purpose. The applicability of NIS requirements ought to depend on the specific type of application/service and its importance to the economy or society (e.g. is the service critical to the provision of an OES's essential service?) - and not just because of its general nature as SaaS. Currently, all SaaS services are technically caught, whether they're used for bill payments or as a forum for pet lovers to discuss their animals. To me, that doesn't seem to make sense.
As I've previously pointed out, SaaS providers don't always register with the ICO for various reasons.

Registering puts their heads firmly above the parapet for possible enforcement. Especially as, since Jan 2021, the top £17m tier of fines could be imposed based on serious service outages alone, whereas previously the top tier only applied if the service was important to the economy. If I provided a SaaS service for pet lovers' discussions, which no one could think would harm the economy or society if it went down, I wouldn't want to register and make my service known to the ICO either.
Saying that SaaS services are caught "only to the extent that they provide a scalable and elastic pool of resources to the customer" just parrots the definition without providing any useful guidance. All cloud services are, by definition, meant to be scalable and elastic. They're not infinitely scalable or elastic, of course; even IaaS/PaaS services impose practical commercial limits on customers' usage, so SaaS services' lack of infinite scalability/elasticity should be a non-point too. But some SaaS providers do argue they're not caught because their service doesn't enable access to a "scalable" and "flexible" pool of shareable computing resources. I have some sympathy here, not because the services really aren't scalable/flexible, but because (as above), given the legislative objective of NIS laws, I feel that it's simply not sensible to try to catch all SaaS services just because they're SaaS, regardless of the exact nature of their services or customers served. Business models are increasingly moving to SaaS, away from software licensing: but there's no legal requirement to have security measures or report vulnerabilities or security issues affecting all software applications regardless of their nature (although many might think that would be sensible). And I've always thought it odd that flexible/scalable services are subject to NIS, when inflexible, non-scalable "classic" hosting platforms are not, even though with the latter their customers are more at risk from availability issues (due to their inflexibility and non-scalability!). Surely it should be the other way round?
And making all SaaS services register is akin to making all software application manufacturers/distributors register their software. The ICO receives fees from controllers who register for data protection purposes, so there's a benefit to the ICO from that registration. But is the benefit of finding out about all online software applications of whatever type or importance worth the administration and other costs?
Would introducing a fine for non-registration help? I don't think so, because of the underlying issue I've emphasised regarding the inappropriateness and disproportionality of bringing all SaaS services within scope regardless of their importance to society or the economy (and see later below).
In my experience, SaaS providers may register if they provide important services to operators. Otherwise, they tend to keep their heads down, and I don't blame them.
The lack of publicly-reported enforcement of the Regulations is another reason for relative lack of awareness of NIS.

Capability and enforcement:

Certainly as regards DSPs, I've found that many ICO staff aren't familiar with NIS and need NIS training as well as more resources for NIS, e.g. those staffing the helpline number given on the ICO's NIS webpage. As flagged above, some DSPs consider the Regulations were incorrectly applied to DSPs in general, and I agree, possibly because of awareness and/or knowledge issues.
The reluctance of many SaaS providers to register, never mind report incidents, is fuelled by the factors I've outlined above, and fear of being subject to the maximum possible fine even though their service may be of minor importance to society or the economy. If they have to bear the costs of ICO investigations too, as is planned, that may drive even more SaaS providers to decide not to register.
The bigger risks for non-registering DSPs are monetary penalties for not reporting incidents when they should have, and/or not having the appropriate security measures in place. If they haven't registered and haven't notified incidents, that of course reduces those risks, because the ICO won't know about them! The main risk then is if they report a personal data breach under GDPR and the ICO says, "Aha! We will fine you under NIS too, because you should have reported the incident under NIS!". But, this depends on the ICO's NIS and GDPR enforcement divisions being sufficiently joined up and also trained up (again, the skills/knowledge issue flagged earlier).

Summary: personally, I would recommend:

Reconsidering the extent to which SaaS providers should be in scope under NIS, if at all. For example, consider introducing specific thresholds or criteria for SaaS providers to be in scope (Obviously if they are critical suppliers to OESs, or OESs themselves, they should be caught under those proposed changes and be exposed to possible designation as OESs, but that's a separate matter.)
Reconsidering the extent to which SaaS providers should be subject to the different tiers of NIS monetary penalties or other enforcement, if at all (with the same caveat). Again, consider if different types/tiers of fines or other enforcement should be applicable to SaaS providers or indeed DSPs that aren't OESs or critical suppliers.
These would help save the ICO's resources too, so they can be directed towards IaaS/PaaS and truly important SaaS providers.
If less radical changes are to be made, provide much clearer guidance on if/when SaaS providers will be caught by the Regulations and therefore need to register with the ICO.
Making publicly available the annual data DCMS aims to collect from regulators, particularly enforcement information and levels of fines imposed. This would help to raise awareness and incentivise compliance.
Requiring the ICO and other regulators to publish the full text of their NIS enforcement and monetary penalty etc notices, but redacted as necessary (including as to OES/DSP names), ideally also listing and linking to them on a centrally-maintained webpage of NIS enforcement action. That would also help raise awareness and incentivise compliance.

"Old fingers": digital exclusion, accessibility

2022-06-25T11:45:00.010+01:00

Song with serious message: tablets, smartphones & other touchscreens have built-in accessibility & usability issues. This is a real problem as we'll all get old eventually (& it's not just the elderly who may suffer from "zombie fingers"): see research; some user solutions are possible, but designing for lower skin conductivity would be ideal.

The lyrics below are original to me, but I don't provide any video of them being sung or indeed any backing music, to avoid any copyright issues (despite the parody exception). This seems to be the official YouTube video, so James Bond/Shirley Bassey fans please feel free to sing along!

Old fingers
Touchscreens weren’t designed for skin that’s dry
I want to cry!
Why?! my old fingers
Can’t control the same touchscreen anymore
Like once before?
And I press and I swipe all in vain
And I curse and I try it again
But a thousand times, won’t make a difference
It’s their **** design, conceived for
Young fingers
Supple skin, conducting the signals in
With no chagrin
You can press, you can swipe all in vain
You can curse and just try it again
Try a thousand times, won’t make a difference
It’s their **** design that beats my
Old fingers
Gaming gloves, or wet them, is what I’m told
Too bad you’re old
Can’t stop getting old
Getting old
We’ll be old
Who cares ‘bout the old
You'll be old
Just be old!

UK data protection reform post-Brexit: key points summary

2022-06-17T13:27:00.030+01:00

The UK government’s response to its data protection reform consultation is out (press release 17 June 2022).

Certain proposals will proceed under the Data Reform Bill announced in the 10 May 2022 Queen’s Speech (more info). Others won’t, while still others are to be be considered further. The devil’s always in the detail, of course, so when the Bill’s text is available the proposed changes will be clearer – it's still unknown exactly when it’s to be published (updated: TechUK says the Bill will be laid "this summer to undergo several rounds of amendments before it is formally passed into legislation". So, presumably June/July before the August summer holidays).

Some highlights below.

Anonymisation

To use Convention 108+ test para19: “Data is to be considered as anonymous only as long as it is impossible to re-identify the data subject or if such re-identification would require unreasonable time, effort or resources, taking into consideration the available technology at the time of the processing and technological developments. Data that appears to be anonymous because it is not accompanied by any obvious identifying element may, nevertheless in particular cases (not requiring unreasonable time, effort or resources), permit the identification of an individual. This is the case, for example, where it is possible for the controller or any person to identify the individual through the combination of different types of data, such as physical, physiological, genetic, economic, or social data (combination of data on the age, sex, occupation, geolocation, family status, etc.). Where this is the case, the data may not be considered anonymous and is covered by the provisions of the Convention”.
The test for anonymisation will be relative, i.e. will the individual remain identifiable by that controller, cf. a third party?

Artificial intelligence (AI) & machine learning (ML), and ADM

Anti-discrimination - the UK DPA sch1 para8 exemption allowing processing of special category data and criminal offence-related data for equality of opportunity or treatment will be expanded to allow bias monitoring, detection and correction in AI systems.
Fairness - the government will consider the role of UK GDPR “fairness” in wider AI governance in its forthcoming AI White Paper, but will not legislate here.
Art.22 automated decision-making (ADM) - will be retained, but with clarified limits & scope, including ADM as a right to specific safeguards, rather than a general prohibition on solely automated decision-making. The approach to ADM will be aligned with the broader approach to governing AI-powered ADM, which will be addressed as part of the upcoming UK White Paper on AI governance.
Explainability and intelligibility of AI-powered ADM, including the role of DP legislation in that context, will be considered in the White Paper on AI governance.
See also above on purpose limitation.

Accountability

Organisations must have a privacy management programme.
No need for DPO, but must designate a suitable individual to oversee data protection compliance
No more data protection impact assessments (DPIAs), or requirement for records of processing activities (ROPAs) as such.
Controllers must have simple, transparent complaint-handling processes for data subjects (but retaining clear pathway to complain to the ICO).

Legal basis - legitimate interests

No balancing test will be needed for a limited number of carefully-defined processing activities in the clear public interest based on legitimate interests, likely to include processing activities undertaken by controllers to prevent crime, report safeguarding concerns or that are necessary for other important reasons of public interest (the government will consider if any additional safeguards are needed for children’s data). Hopefully this should “encourage organisations to make the authorities aware of individuals who are at risk without delay”, including children and other vulnerable groups with protected characteristics. However, core principles like lawfulness, fairness & transparency, and further conditions for processing special category data, etc., would of course continue to apply.
Power to update the list of activities, subject to Parliamentary scrutiny.

Special category data, criminal offence-related data

The UK DPA 2018 sch1 part 2 exemptions for processing in the substantial public interest could be expanded to add certain activities, but “substantial public interest” will not be defined specifically.

Purpose limitation

Further processing or reuse by the same controller for incompatible purposes will be permitted “when based on a law that safeguards important public interest”, with “greater clarification on the rules and permissions of data re-use and the need for greater transparency”.
On consent-based processsing, “further processing cannot take place when the original legal basis is consent other than in very limited circumstances”. We’ll have to wait to see what those new circumstances will be.
Distinctions between further processing and new processing by a different controller to be clarified.

Transfers

Adequacy decisions - a risk-based approach will be taken; judicial or administrative redress are both acceptable. There will be ongoing review, cf 4-yr review of adequacy decisions.
The Secretary of State can recognise alternative transfer mechanisms (ATMs).
(But no repetitive derogations or reverse transfers etc.)

DSARs

No nominal fee to be introduced.
No cost ceiling, but controllers can refuse to deal with DSARs that are “vexatious or excessive” (cf. the current “manifestly unfounded or excessive”).

Research

No new lawful basis for research, but various changes will be made to assist and promote research.
E.g. a “scientific research” definition (hopefully making crystal clear the position on commercial scientific research, and what's research in the "public interest"?); and clarifying that broad consent is possible and can be relied on.
Privacy notices – the UK GDPR's Article 14(5)(b) “disproportionate effort” exemption will be replicated, but only for research purposes, to allow personal data being used for a research purpose differing from the original purpose to be exempt from re-providing information under Article 13(3) - but without exempting controllers who obtain personal data directly from data subjects from providing the required Article 13(1) & (2) information to them on collection. “Disproportionate effort” to be clarified by bringing in the GDPR's Rec.62 language into the operative text.

ePrivacy under PECR

Fines - to be increased to GDPR levels.
ICO powers - to include assessment notices etc.
Cookies and similar technologies (i.e. mobile apps, smart devices too)

Analytics will be considered “strictly necessary”.
Consent to be unnecessary in more situations: "a small number of other non-intrusive purposes" (e.g. website fault detection?), "where the controller can demonstrate legitimate interest for processing the data".
Websites must respect users’ browser preferences; the UK will move to no cookies banners for UK residents and an opt-out model for cookies once preferences management technology is widely available.

Direct marketing

Soft opt-in to be extended to political parties and non-commercial organisations like NGOs/charities.

Nuisance phone calls e.g. automated telephone marketing

The ICO will be able to take enforcement action against organisations based on the number of calls generated (cf. only the number that are connected, currently)
Communications service providers must report to the ICO “suspicious levels of traffic on their networks”.

ICO

New duties (e.g. to uphold data rights and to encourage trustworthy and responsible data use, have regard to economic growth and innovation, competition issues and public safety, to consult with relevant regulators and any other relevant bodies).
Structural changes e.g. independent Board and Chief Executive.
New powers for the DCMS Secretary of State, e.g. to prepare a statement of strategic priorities which the ICO must respond to; to approve statutory codes of practice and statutory guidance ahead of laying them in Parliament.
Legislative criteria for a more risk-based proportionate approach to complaints - ICO discretion to decide when/how to investigate complaints, including discretion not to investigate vexatious complaints, and complaints where the complainant has not first attempted to resolve the issue with the relevant data controller. "This will empower the ICO to exercise its discretion with confidence."
New ICO powers

To issue technical report notices where fair and reasonable, having regard to alternative investigatory tools, relevant knowledge and expertise available to the controller or processor and the impact of the cost of producing the report.
To compel witness interviews, without interfering with the right not to self-incriminate, rights to legal professional privilege and various procedural mechanisms to ensure proportionality & fairness of interview.

Must provide organisations with the expected timeline at the start of all investigations.

Note: on ICO resources and funding, the ICO announced, on 14 June 2022, its agreement with its sponsor department the Department for Digital, Culture, Media & Sport (DCMS) and with the Treasury (HMT) that the ICO will now able to retain some of the funds paid as a result of its civil monetary penalties i.e. fines to cover pre-agreed, specific and externally audited litigation costs. (Previously, all fines money went to the UK government’s central Consolidated Fund.)

Security training - review of Security Innovation's Cmd+Ctrl Shred cyber range & security training

2022-04-10T15:38:00.003+01:00

GDPR supervisory authorities (SAs) emphasise data protection training (e.g. the UK Information Commissioner's personal data breach notification form asks, "Had the staff member involved in this breach received data protection training in the last two years?", and "Please describe the data protection training you provide, including an outline of training content and frequency").

What about security? Security of personal data is of course important under GDPR, and organisations can be fined for not having appropriate security measures in place. While security training for developers is not specifically mentioned in GDPR as such, developers do also need training on application security issues that can lead to breaches of websites, online services and any databases or other data storage behind them (including personal data in systems). Most IT staff, developers and otherwise, are not necessarily cyber security (or even security) experts, and must be educated on what to look for and how to address, at least, the most common key security issues.

Many online training courses on cybersecurity for developers are now available. There are also "cyber ranges" offering users deliberately vulnerable systems, websites or online applications that users can attack and seek to exploit, to learn how hackers think and the kinds of the actions they take, and therefore be able to defend against them better.

As part of OWASP London CTF 2021, in Nov 2021 Security Innovation generously offered participants free access for a month to a fake e-commerce website "Shred Skateboards" on its CMD+Ctrl CTF (Capture the Flag) web application cyber range, and for 6 weeks to its Bootcamp Learning Path, a self-paced online training course incorporating 32 selected courses from its full catalog of training courses.

This blog reviews the Shred range, then the online training courses. These cover some of the issues referenced in the recently-finalised European Data Protection Board (EDPB) Guidelines 01/2021 on Examples regarding Personal Data Breach Notification, as those Guidelines include some recommended security measures as well as breach notification, and also mention OWASP for secure web application development.

Cmd+Ctrl Ranges and Shred

Cmd+Ctrl's ranges are generally available only to paying organisations to train their staff (but not to paying individuals, sadly. Missed trick there, as I think individuals wanting to improve their ethical hacking skills would pay a reasonable fee or sub for access). People who signed up for the event were however given free access to Shred for a month. Shred is meant to be one of the easy ranges.

The Cmd+Ctrl login page provides some sensible disclaimers and warnings:

After logging in, you need to click on the relevant range's name and wait a few minutes for it to start up (each user gets their own virtual machines I suspect on Amazon Web Services), as a real website available on the Internet with its own URL (hence the exhortation not to enter sensitive information on the website - I would expand that to real names, real email addresses and basically any real personal data, because real hackers can also access that website as much as you can!).

Then, basically explore the website and try different things to find vulnerabilities e.g. click the links, register user accounts, try different URLs, enter different things into the search or login forms, etc. I won't share screenshots of Shred so as not to give anything away, but it emulates an online shop for skateboards and related accessories and pages, with user accounts that can store user details including payment cards, the ability to purchase gift cards, etc. Each machine is up for I believe 48 hours, and each time you start it, it may have a different URL and IP address. If things go badly wrong you may have to reset the database (which loses your changes e.g. a fake user you registered) or even do a full reset, but you're not penalised for that, the system retains the record of scores you achieved for previous exploits.

When you successfully exploit a vulnerability, a banner slides in from the top of the webpage indicating what challenge was solved and how many points you gained for it. You can also see what broad types of other challenges remain unsolved.

Via the My Stats link, you can see a Challenges page, which also gives similar broad information about the types of challenges remaining unsolved. Unfortunately, only Category information was provided regarding unsolved challenges (see the Category column of the Solved table shown below for examples).

No detailed information about the exact nature of any challenge (i.e. the info under the Challenge column, such as "Unsafe File Upload" in the table above) was provided. It appeared only after you actually solved the challenge, whereupon it was listed in the Solved table (as well as the banner appearing). The "Get Hints" link was disabled for this event - but presumably hints are available in the paid versions of the ranges. However, Security Innovation provided a live online introduction on the first day of the CTF event, access to a one-page basic cheat sheet tutorial, with a guide to Burp Proxy for intercepting HTTP traffic, and weekly emails with some hints and links to helpful videos. A chat icon at the bottom right of every webpage allowed the user to ask questions of support staff. I tried to confine my range attempts to the afternoon/evening given that Cmd+Ctrl is US-based, but I was very impressed with how quickly responses were given to my chat queries, even though I was using the range as an unpaid user. The support staff did not give away any answers, but instead provided some hints, often very cryptic - I suspect similar to the tips that users for whom the Get Hints" link is enabled would receive.

Under My Stats there was also a Report Card link giving detailed information about your performance, also in comparison to others who had attempted the range, including the maximum score reached. Challenges were again shown here, broken down by category and percentage solved.

As well as repeating the solved challenges table further down on this page, there's also a time-based view of the user's stats. As you'll see, I had a go over the first weekend, solving a few basic and easy challenges, then left it until I realised that I would lose access to Shred soon, so I made a concerted effort over the last few days though I ran out of energy with an hour or two to spare!

I was rather chuffed that, as a mere lawyer and not cybersecurity professional, I managed to complete 25 out of the 35 challenges and reach the rank of 7, out of 54 people who at least attempted Shred (in the screenshots below I've redacted names and handles other than common ones like Mark or David). I admit I have attended some pen testing training, one excellent 2-day course with renowned web security expert Troy Hunt (yes, I was very lucky), and one terrible week-long course with someone whose name should never be mentioned again (but at least the food was great). However, those courses were several years ago, and this is the first time that I've attempted a range or CTF event. (I've signed up for other services with some similarities, Hack the Box and RangeForce Community Edition, but I haven't had time to try them properly yet.)

Prerequisites for trying these ranges

You do need some prior knowledge, particularly about HTML and how URLs, query parameters and web forms work, HTTP, cookies, databases and SQL etc, and concepts like base64 encoding and hashes. You also have to know how to use tools like Chrome developer tools, which is built into Chrome, to edit Shred webpages' HTML. I'd not used those developer tools before tackling Shred, but searched for how (I didn't resort to Burp for Shred, myself). I probably have a better foundation than most tech lawyers as I have computing science degrees as well as the pen testing training, coupled with a deep and abiding interest in computing and security since my childhood days. So I'd strongly recommend that those without such a foundation should take the courses before attempting any ranges (the courses are covered in more detail below).

Positives

The range provided an excellent assortment of different vulnerabilities to try to exploit, most of the type that exist in real life (indeed, recently I spotted a common one on one site I shop from, when I mistyped my order number into its order tracking form!). The chat support staff were very prompt, although I couldn't figure out some of their hints.

Negatives

Shred included 3 challenges (maybe more?) that involved the solving of certain puzzles (at least one of which scored quite a few points). However, I think the range would have been better if they had not been included, as you wouldn't find them on actual websites - they were simply puzzles to solve, not realistic website vulnerabilities. OK perhaps for some fun factor, not so much for learning about web vulnerabilities, particularly as access to the range is time-limited.

The biggest negative in my view is that no model answers are given at the end. If you haven't managed to solve some of the challenges, tough luck, they won't tell you how. A support person said they felt that these ranges could be devalued by "giving away too much", because customers pay to access its ranges. However, I think that view is misconceived.

It depends on how customers use these ranges internally. I believe they would be best used as hands-on training for tech staff (developers, security), but I can't see why previous users would give away the answers to colleagues or indeed people in other organisations, as it defeats the object of trying these ranges. If organisations required staff to achieve a minimum score on these ranges, then yes, that might incentivise "cheating" and disclosure of solutions. But it's not uncommon, and in fact often a good thing, to form teams to solve challenges together and share knowledge. For this and many other reasons, such a requirement would not make sense. And it would make no sense for one customer of Security Innovation to give the answers away to other customers, what would be the purpose of that?

Conversely, it would be very frustrating for someone who had paid to use the range to find out that they would not be told any outstanding answers at the end. If you haven't managed to teach yourself the solutions, you don't know what you don't know, how will you learn if they refuse to fill in the gaps? Security Innovation already impose a condition on the login page that users cannot post public write-ups or answer guides, which they could expand if they wish (though I don't think that's necessary or desirable).

In similar vein, I think they should at least give hints about the detailed challenges (e.g. "Unsafe file upload" as one challenge), not just categories of challenges. The cheat sheet mentioned a few types of vulnerabilities that I spent too many hours trying to find, and it was only on the last day or two before expiry that I asked on the chat, only to be told Shred didn't actually have those types of vulnerabilities! I appreciate Cmd+Ctrl doesn't want to give too much away, but knowing there's an unsafe file upload issue to try to exploit still doesn't tell you how to exploit it, and it would have saved me so much time particularly given that access to Shred was time-limited. Again, I think paying customers would appreciate more detailed hints so that they can be more targeted and productive in tackling the challenges during the limited time available (and perhaps "Get hints" would have done that, but access was disabled for this event).

Also, I'm not sure how time-limited access would be for the paid version, but organisations wanting to subscribe should of course check the details and ensure the time period is sufficient for their purposes, as staff also have to do their jobs! (I tried the range during my annual leave).

Final comments

I think it's definitely worth it for organisations to pay for their developers to try these ranges, subject to the negatives mentioned above (and see below for my review of the training courses). These ranges can be more interesting and fun for users, and certainly involve more active learning (looking into various issues in context as part of attempting to exploit those types of vulnerabilities), which research has shown improves understanding, absorption and retention. And of course, gamification is known to increase engagement. Attempting these ranges would help to consolidate knowledge gained during the security training.

But, as mentioned above, I believe the best way would be to give staff enough time to tackle the ranges, over a reasonable period over which the relevant range is open. Don't make staff do this exercise during their weekends or leave, or require each person to reach a minimum score; instead, hold a debrief at the end of the period, for staff to discuss the exercise and share their thoughts (and hopefully receive the answers to challenges none of them could solve, so that they can learn what they didn't know). I appreciate that leaderboards and rankings can bring out the competitive streak and make some people try harder, but I believe team members need to cooperate with each other, and staff shouldn't be appraised based on their leaderboard ranking (or be required to reach a minimum score) - the joint debrief and "howto" at the end is, I feel, the most critical aspect to getting developer teams to work together better in future to reduce or hopefully eliminate vulnerabilities in their online applications.

Cmd+Ctrl offers a good variety of ranges with the stats and other features covered above, which seem very up to date in their scope: banking (two), HR portal, social media, mobile/IoT (Android fitness tracker), cryptocurrency exchange, products marketplace, and cloud. I wish I'd had the chance to try the cloud ones! In fact, there now seem to be 3 separate cloud-focused ranges: cloud infrastructure, cloud file storage, and what seems to be a cloud mailing list management app, i.e. both IaaS and SaaS.

Wishlist

A range that actually allows the user to edit the application code to try to address each vulnerability, then test again for the vulnerability, would be great for developers!

Online training courses

Alongside access to Shred, for those who signed up to the Nov 2021 bootcamp, Security Innovation kindly offered access for 6 weeks to 32 online courses from its full catalog of training courses. I provide some comments on format and functionality first, then end with thoughts on the content.

I took the bootcamp courses, but the vast majority of them only after I'd finished the Shred range. The information in some of those courses would help with the Shred challenges, but not all of them, and they are aimed at developers, so to follow those courses you would also still need some prior computing and coding knowledge.

It was great that many courses were based on the Mitre CWE (common weakness enumeration) classifications often used in the security industry, e.g. incorrect authorization (CWE-863) and on the OWASP 2017 top 10 security risks, but I won't list them all here. The topics covered by the bootcamp: fundamentals of application security, secure software development, fundamentals of security testing, testing for execution with unnecessary privileges, testing for incorrect authorization, broken access control, broken authentication, database security fundamentals, testing for injection vulnerabilities, injection and SQL injection, testing for reliance on untrusted inputs in a security decision, testing for open redirect, security misconfiguration, cross site scripting (XSS), essential session management security, sensitive data exposure (e.g. encrypting), deserialization, use of components with known vulnerabilities, logging and monitoring and XML external entities.

Several courses were split logically into one course on the problem, and the next on mitigating it, or testing for it. Personally, I learn best by being told the point, then seeing practical concrete worked examples, and I would have liked to see more concrete examples of e.g. XSS attacks or SQL injection attacks. A couple were given occasionally, but not enough in my view. (I appreciate some examples can be found by searching online.)

The above shows Completed but a course's status could also be displayed as being in progress. You need to click against a particular course (where it shows Completed above) to enrol in the first place, an extra step whose purpose I couldn't fathom (why not just "Start"?). The 3 dots "action menu" enables you to copy the direct link to a particular course for sharing, or pin individual courses.

Clicking on a course name takes you to a launch page, from where you can also open a PDF of the text transcription of the audio.

You can leave a course part-completed, and resume later:

When you launch or resume a course, a video appears for playing. There are 3 icons on the top right, above the video, for a glossary (the book), help regarding how to use the video (the questionmark), and the text version of the course (printer icon).

Positives

This course caters for people with different learning styles, by providing both videos and PDF transcriptions. Personally, I scan text a zillion times faster than if I had to watch a video linearly at the slower pace at which people speak, so for learning I much prefer text over video (plus the ability to ask questions, but I didn't see a chat icon - I don't know if that's possible with the paid version?). So, I always clicked the printer icon to read the PDF (opens in another tab) rather than watch the video.

A TOC button on the bottom right brings up a table of contents on the left, where you can click to go straight to a particular section of the video. That it also shows progress, with a tick against the sections that you've watched.

Another positive, from an accessibility perspective: the CC (closed captions) button at the bottom right brings up the text transcript for the current part of the video, synchronised to the audio.

Negatives

The PDF didn't always show all the slides from the video, especially in the first few courses - not all the slides contained substantive content, but some slides with example URLs or code were missing from the PDF version. So, personally, I only played the videos to check for any useful slides missing from the PDFs.

If you play a video, it stops occasionally and you have to click the play button again to start the next section, which may not be obvious. Sometimes it stops to provide interactivity, i.e. the user has to click on one part of the slide to learn about that issue, click on another part to learn about another issue etc. I hate these types of features, myself. I would prefer videos to just play continuously, moving on from section and part to section and part, unless and until the user pauses it. Stopping a video to force the user to click on something just to get to the next portion seems popular, particularly with the periodic online staff training that many are compelled to undergo for regulatory compliance reasons, but really it's not the same as active learning, in my view! Forced stops like these just break the train of thought and get in the way, when the user wants to get a move on. But perhaps this is a matter of personal preference, so allow me my rant about "interactive" online training courses!

Exam

At the end of a video, you can take an exam (and there are also Knowledge Check quizzes to answer throughout the video). As I had scanned the PDFs rather than watch the videos, I generally went straight to the exam via the TOC or by dragging the position arrow.

If you pass an exam, you get a certificate of completion that you can download under the Transcripts section of the site, which also allows printing of the list of courses and marks (niggle: all certificate PDFs had the same filename, it would be great if certificate filenames followed the course name, and if you could download a single zipped file of all certificates in one go).

You're allowed to take the exam multiple times until you pass. Most exams comprise about 4-5 questions, although one had 3, a few 6-8, and another 12 questions. They estimate it takes about 5 mins per exam (10 mins sometimes), which I found was about right.

It doesn't seem possible to go back and amend your answer if you change your mind about a previous question - when I tried that to do that in one exam, it threw a fit and I ended up having to retake the exam (with the same answers) twice before it would register as completed.

At the end of the exam, your full results are shown (it doesn't show results per question as you go through):

Tips

The obvious answer is usually the right one, and if you think "Yes, but only if..", then the answer is probably "No"! I felt a few of the questions or multiple choice answers were unclearly or ambiguously phrased. I did think some of the answers were more about categorising vulnerabilities by type, e.g. broken authentication, or more about vulnerabilities than about how to mitigate them.

If you didn't pass, you can click Review Exam to see where you went wrong, which is helpful. I only had to retake one to pass (becase of the No answer above when I had answered Yes!), but didn't bother to retake a few others where I'd passed with less than 100%.

I discovered that I actually knew more than I thought I did, so the courses didn't actually help me with Shred (although the support staff tips did). But I still learned some useful things that I didn't already know, and I strongly recommend that those without the necessary foundation should take these courses before trying the ranges.

Final thoughts

Overall, I would recommend the Cmd+Ctrl ranges as an excellent way for developers and security staff to learn about online application vulnerabilities, subject to taking the courses first for those without the prior knowledge. They really are aimed at developers/programmers, so most lawyers may struggle, even tech lawyers. I do think it's helpful for lawyers to have a basic knowledge of the common vulnerabilities and how they are exploited and mitigated when discussing cybersecurity measures and breaches with clients that have suffered incidents, but you probably don't need to tackle the courses or ranges to gain that knowledge.

Thanks very much again to SecurityInnovation for making Shred and the courses available for the OWASP London CTF 2021 event!

(I wrote this back in Dec 2021 but for various reasons couldn't publish it till now.)