California reparations panel warns of ‘racially biased’ medical AI, calls for legislative action

California’s reparations task force is recommending as part of its set of proposals to make amends for slavery and anti-Black racism that state lawmakers address what it calls “racially biased” artificial intelligence used in health care.

The task force, created by state legislation signed by Gov. Gavin Newsom in 2020, formally approved last weekend its final recommendations to the California Legislature, which will decide whether to enact the measures and send them to the governor’s desk to be signed into law.

The recommendations include several proposals related to health care, including some concerning medical artificial intelligence (AI), which the task force describes as “racially biased” and contributing to alleged systemic racism against Black Californians. 

Specifically, the task force calls for the legislature to fund either state universities or government agencies to study the “potential for harmful biases” in medical AI.

CALIFORNIA REPARATIONS COMMITTEE CALLS FOR MANDATORY ‘ANTI-BIAS’ TRAINING TO GRADUATE MEDICAL SCHOOL

“The task force recommends that the legislature provide state funding to the California Department of Public Health, a University of California school, a California State University school, or another appropriate entity to study the potential for harmful biases in commercial algorithms and AI-enabled medical devices,” the committee writes in the final report outlining its proposals, adding that the study should also recommend how best to regulate medical AI tools in California.

The report additionally suggests the study should probe “‘evidence-based research into the use of devices and tools that recommend adjusting patients’ treatment or medication based on broad racial categories in the absence of information on genetics or socio-cultural risk factors.'”

The task force quotes from a recent American Civil Liberties Union (ACLU) paper that it cites several times. The paper provides examples of alleged racial bias in medical AI, such as a tool meant to decide how to best distribute the limited resource of extra care to new mothers at risk of postpartum depression that, according to the ACLU, directed care away from Black mothers and favoring White mothers. 

In California, meanwhile, the reparations committee recommends that the legislature require the state’s Department of Public Health to issue guidance to hospitals and other medical systems to ensure that AI-enabled medical devices “are not used for clinical applications without FDA approval or clearance, are not used on patient populations they were not intended for, and that cleared tools are not used outside of their intended use cases.” That recommendation is also in the ACLU paper.

The task force additionally wants the California Department of Public Health “to make and maintain a public list of software as a medical device (SaMD) products and provide demographic information about the subjects in which the devices were calibrated or trained.”

NEW AI TOOL HELPS DOCTORS STREAMLINE DOCUMENTATION AND FOCUS ON PATIENTS

A fourth proposal is to allocate positions and funding to the California Department of Justice to pursue claims against AI medical device manufacturers if their products have a “disparate impact” when providers use them according to manufacturers’ instructions or if the products “misleadingly promise fairness.”

Despite the task force’s claims, however, new AI tools have helped medical professionals treat patients in a variety of ways.

One such tool called RestoreU, for example, helps physicians create personalized care plans for patients with Alzheimer’s and other types of dementia. Another tool known as DAX Express streamlines the note-taking process, a benefit that has reportedly helped doctors improve patient outcomes, work more efficiently, and reduce costs.

Beyond AI, the California Reparations Task Force is pushing several controversial health-related proposals, such as mandating “anti-bias training” and an assessment based on that training as graduate requirements for medical school.

The task force is also pushing a universal, single-payer health care system as a way to achieve health “equity” for Black residents in California.