Bodycam footage shows Baltimore County, MD police handcuffing 4 students who were waiting to be picked up after football practice when they got an alert from AI software that detected a gun on the school’s CCTV system.

The supervisor asks “how did we get this call?”...“The [company name] alert”....“Ahhhhhhhh”.

When the officers stop and look at the image on their phone, the supervisor says “that looks like a chip bag”.

The officer then explains to the handcuffed kids, “basically the camera system picked up the way you guys were eating chips and thought it was a gun”. This was after the students were already forced to get on their knees at gunpoint, handcuffed, and had their bags searched. Baltimore County schools has a 3-year $2.6 million contract for the AI gun detection software. The salary of a Baltimore County teacher starts at $34,000 meaning the cost of this AI system is the same as 25.5 new fulltime teachers.

Here is what the student who was falsely detained had to say about it:

This brings up some very important legal questions:

If the system flagged a Doritos bag as a gun and then the officers found drugs, vapes, a knife, or even a gun inside a student’s bag, would the false pretense of the initial stop result in the entire case being thrown out?
Since the officers didn’t find a gun, can these 4 students sue the school district, police department, and AI company for being wrongfully detained and the emotional distress from being held at gunpoint?
If the students were arrested and charges were filed, how does the 6th Amendment work when the witness is an algorithm? Would discovery involve analyzing the proprietary software and accuracy rate to determine if AI was a credible witness?

Sixth Amendment:

In all criminal prosecutions, the accused shall enjoy the right to a speedy and public trial, by an impartial jury of the State and district wherein the crime shall have been committed, which district shall have been previously ascertained by law, and to be informed of the nature and cause of the accusation; to be confronted with the witnesses against him; to have compulsory process for obtaining witnesses in his favor, and to have the Assistance of Counsel for his defense.

The very end of the interaction between the officers and the students is the most troubling when the supervisor says: “Good thing you guys didn’t run [laughs]...that would have made a major problem.”

The biggest factor that kept this mix-up from escalating into a officer-involved shooting—or these teens being hit with felony charges for fleeing from police or resisting arrest—was that the students kept a cool head the entire time and didn’t panic when police pointed guns at them.

Inside the black box

What’s really going on with this software that classifies objects in CCTV images? (Note: the software being sold to schools for visual AI gun detection is different from the general purpose AI models like ChatGPT that can analyze the entire context of the scene. Sole-purpose gun detection software created in the late 2010s is based on the 2nd generation of AI software. For comparison, the newest LLMs like ChatGPT-5 are the 6th or 7th generation of AI tech. Read more: What generation of AI are you selling?)

Going back to sole-purpose image classification software from the 2010s, computer vision doesn’t see objects the way that humans do because software is based on math. Each pixel in the image is given a numeric value and then the program creates probability values that the proximity of clusters of pixels in training images are the same as a cluster of pixels in a new image.

How to Convert a Picture to Numbers - KDnuggets

While it sounds complicated, there are open-source tools that can analyze how deep learning algorithms work and what they are “seeing” in the images.

Without any special training, an existing image classification model can recognize attributes of a toy gun and the model will assign probabilities that the toy is different types of real firearms.

With a picture of a real handgun, the same open-source model can classify this handgun as a firearm but isn’t really sure if it’s a revolver (incorrect), assault rifle (incorrect), or rifle (incorrect). But the exact type of gun doesn’t really matter if you are just trying to detect any type of firearm. The software’s confusion does become a problem if you are trying to determine the difference between a toy gel blaster versus a real gun (or a shiny Doritos chip bag with edges that make a 90 degree angle versus a shiny pistol slide with edges that make a 90 degree angle).

Because a computer can’t see like a person, to understand how a model is classifying an image you need to calculate the gradients to measure the relationship between changes to a feature and changes in the model’s predictions. For comparing these two images of guns, the gradient shows which pixels have the strongest effect on the model’s predicted class probabilities.

The first step in analyzing how the software sees an image is using filtered gradients ranging from all black to the normal image to understand which features of the object (clusters of pixels) are being focused on.

The output from this analysis shows how transitioning from all black to clear features impacts the prediction probability of what the object is.

X-axis (alpha): Represents a parameter which scales the input from the baseline to the original input.
Y-axis (model p(target class)): Shows the predicted probability of the target class by the model.

Plot on the left is completely flat at 0 across all values of alpha. This indicates that the model’s predicted probability for the target class remains 0 regardless of the value of alpha. On the right, the plot shows how the average pixel gradients change as alpha varies from 0 to 1. The average pixel gradients start low, spike dramatically at a low value of alpha, then gradually decrease and stabilize as alpha increases. The spike in gradients show that there is a particular point where the input features have a significant impact on the model’s output.

The pink dots (attribution mask) show that this computer vision is classifying the object as a handgun based on pixels along the handgrip, trigger guard, and bottom of the barrel. When those are the primary features the software is looking for, a gun that is held sideways with these three features hidden won’t be classified correctly.

You can see exactly how gun detection software works in a demo posted on LinkedIn. When a handgun is held still against a contrasting background, the predictive algorithm assigns a 99.7% probability of a gun. When part of the weapon is obscured, the probability drops to 87%. The real world impact of these percentages is that when the alert sensitivity is set to 85% to detect the partially obscured gun, there are lots of non-gun objects that can score in the 80s. If the software has human review, the lower the threshold is set, the more images humans need to review and this creates more chances for human error and higher costs.

I tried the same set of tests with the toy gel blaster gun that looks like an AR-15.

Unlike the all black handgun that has straight lines and a uniform shape, the right plot shows the model’s sensitivity to input features.

These fluctuations in the gradients suggest that there are points where the input features have varying impacts on the model’s output, even though this does not translate into a higher predicted probability for the target class. With this much variation in the shape and colors, the software is generally confused where to look.

Unlike the handgun, the classification model is “looking” all over the place at this image to try to figure out what it is. This shows how many features and variations of a firearm, a toy, or a non-gun object impact accurate classification.

Basing student safety on ‘maybes’

In an article I wrote for CNN last summer, with school security, we want certainty. Is the person on CCTV holding a gun? We expect a “yes” or “no” answer. The problem is AI models provide “maybe” answers. This is because AI models are based on probability.

For AI classifying images as a weapon, an algorithm compares each new image to the patterns of weapons in training data. AI doesn’t know what a gun is because a computer program doesn’t know what anything is. When an AI model is shown millions of pictures of guns, the model will try to find that shape and pattern in future images. It’s up to the software vendor to decide the probability threshold between a gun and not a gun.

This is a messy process. An umbrella could score 90% while a handgun that’s partially obscured by clothing might only be 60%. Do you want to avoid a false alarm for every umbrella, or get an alert for every handgun?

Cameras generate poor-quality images in low light, bright light, rain, snow and fog. Should a school be using AI to make life-or-death decisions based on a dark, grainy image that an algorithm can’t accurately process?

For AI technology to be useful and successful at schools, companies need to address campuses’ greatest security challenges. In my studies of thousands of shootings, the most common situation that I see is a teenager who habitually carries a gun in their backpack, and they fire shots during a fight. Manually searching every student and bag is not a viable solution because students end up spending hours in security lines instead of classrooms. Searching bags is not an easy task and shootings still happen inside schools with metal detectors.

Neither image classification from CCTV nor retrofitted metal detectors address the systemic problem of teens freely carrying a gun at school each day. Solving this challenge requires better sensors with more advanced AI than any product available today.

Human review is not the solution

Relying on human review to verify every AI alert isn’t a realistic solution to correct these foundational flaws in image classification software. If the software is monitoring thousands of cameras, this can easily generate thousands of alerts each day (or even each hour) because many harmless objects like umbrellas, phones, or snack bags will flag an object for human review. If a company has a contract to monitor 50,000 cameras, they have about 8.6 BILLION images to review each day (analyzing 2 frames per second per camera). Aside from the enormous compute costs of running the AI software, even a system with 97% accuracy will require human review of about 1,500,000 images every single day.

This huge amount of data to process is why these companies are using 2010s image classification tech instead of more powerful general purpose AI models like ChatGPT. Analyzing an image with ChatGPT costs about 5k tokens. Based on OpenAIs enterprise pricing, analyzing 8.6 billion images per day would be $6.77M/day for low resolution images (1024×1024) and $26.55M/day for higher res (2048×2048). To identify a small object on a camera pointing at a courtyard by the front door of a school, this would require highest resolution possible.

These numbers get crazy fast. Analyzing 50,000 camera feeds with ChatGPT would be $9.67 billion per year which is 3.5x the entire Baltimore County schools operating budget. These companies are using outdated tech because models that could perform better aren’t even close to economically viable.

To avoid errors like what happened in Baltimore County, reviewing every frame would require a full-time monitoring team that effectively adds a second layer of costly surveillance personnel who are exclusively looking for guns. At a modest tech salary of $60,000 per reviewer per year, the human oversight would quickly exceed the cost of the AI software itself.

Texas high school goes into lockdown — Example of human reviewed false alert at a Texas high school

Human review doesn’t guarantee accuracy because people reviewing grainy, low-light footage are still prone to the same perceptual biases and fatigue that AI is meant to overcome. When a human reviewer’s sole task is to look for guns on CCTV images, reviewers can easily misinterpret ambiguous shapes, especially when they’re pressured to make decisions in seconds.

Adding humans to the loop may feel safer, but it mainly shifts the error from algorithmic probability to human judgment while still delaying response times, adding extra costs, and diverting even more resources away from education and prevention.

Bad tech makes for a messy future

What happened in Baltimore County is more than a minor technical glitch because four teens were forced to the ground at gunpoint because faulty software misinterpreted a bag of Doritos as a gun. The school district spent the equivalent of twenty-five teachers’ salaries on an AI system that can’t tell the difference between a snack and a threat. The outcome is students are even less safe when police with guns drawn show up on campus looking for a shooter who doesn’t exist.

This incident exposes the tension between safety and civil rights when decisions are driven by probability instead of proof. This AI system didn’t “see” a gun, it just takes a bunch of 2D camera images and tries to calculate a percentage. But the big problem (and potentially deadly consequence for these students) is that police officers acted on that percentage as if it were certainty.

That chain of false confidence created real fear, real trauma, and potentially real legal consequences. If AI initiates the stop, who is the witness in court? When a machine’s mistake leads to a human being handcuffed, we are no longer talking about hypothetical bias or future risks because this is the erosion of due process in real time.

Schools are right to want protection, but outdated tech built on “maybes” can’t deliver it. Before the next false alert escalates into tragedy, policymakers should ask a simple question: What level of uncertainty is acceptable when the cost of error is a gun pointed at a child?

Until AI can truly comprehend context rather than pixel clusters, basing student safety on probabilities will continue to cause avoidable disasters like this. When this class of software can’t tell the difference between a bag of chips versus a gun, it doesn’t belong on a school campus.

David Riedman, PhD is the creator of the K-12 School Shooting Database, Chief Data Officer at a global risk management firm, and a tenure-track professor. Listen to my weekly podcast—Back to School Shootings—or my recent interviews on Freakonomics Radio and the New England Journal of Medicine.

Riedman Report: Risk, AI, Education, & Security

AI classified Doritos chips as a gun and then…