COUNCIL OF THE CITY OF PHILADELPHIA COMMITTEE ON TECHNOLOGY AND INFORMATION SERVICES PhillyCAM 699 Ranstead Street Philadelphia, Pennsylvania Wednesday, October 15, 2025 10:15 a.m. PRESENT: COUNCILWOMAN RUE LANDAU, CHAIR COUNCILMAN MARK SQUILLA, VICE-CHAIR COUNCILMAN JIM HARRITY COUNCILMAN NICOLAS O'ROURKE COUNCILMAN ANTHONY PHILLIPS
Thank you all so much for being here. I now note that the hour has come. Mr. Serra, will you please call the roll to take attendance for the Committee on Technology and Information Services.
Thank you. A quorum of the committee is present and this hearing is now called to order. This is the public hearing of the Committee on Technology and Information Services regarding Resolution No. 240759. Mr. Serra, will you please read the title of the bill.
Authorizing the Committee on Technology and Information Services to hold hearings exploring the current status of artificial intelligence, artificial intelligence technologies and how Philadelphia can prepare for their impact on the public, private and nonprofit sectors.
Thank you again. Thank you all so much for being here. This is the first-of-its-kind hearing for Philadelphia. I'd like to acknowledge the presence of our colleague Dr. Anthony Phillips. So a few opening remarks for now. I want to thank by -- I want to begin by thanking my colleagues, advocates, academics, experts and members of the public who are joining us here today. I also want to thank PhillyCAM for hosting us in their space and broadcasting this. This hearing is for the public. For those of you don't know, PhillyCAM is the public access radio and TV station here in Philadelphia. And also as the Council President always has the opportunity to appoint us to many boards throughout the city, he appointed me to the Board of PhillyCAM, and I've been so proud to get to know them even better and so happy to represent them and so appreciative of you hosting us here today. Thank you also to the Parker Administration for joining us today to provide additional insight. I think we can all agree that artificial intelligence is becoming the new normal. AI allows computers to perform tasks that mimic human intelligence, enabling computers to learn from existing data and apply that to generate new content, seemingly overnight. AI is no longer an idea of the Sci-Fi future, but a part of our everyday lives. It is integrated into our search engines, our word processing software, our social media and even our phones. Whether you're talking to an AI chat bot when setting up your Wi-Fi or issued a traffic ticket based on AI-powered enforcement or using a generative AI app to create content for your social media, it's changing the status quo and bringing us into a new technological age. It's being used for writing, creating images and sound, analyzing data, recognizing images, and we don't always know when our peers or even when we ourselves are using it. I'll be the first to admit I'm not a tech expert. I don't have a background in software engineering. I don't know how to code and I am not always on the cutting edge of the apps that are available. As a City Councilmember and a longtime advocate for tenants, low-income people and the civil rights of marginalized communities, I'm holding this hearing today so that all of us can learn and make sure we're protecting people when implementing these new tools. We cannot repeat harms of the past with new technology. Throughout this hearing, we will be speaking with leaders in the tech and innovation industries, advocates for digital equity and tech justice and academics who have a deep background of research in emerging technology. We're going to explore the potential for technological innovation to improve the operations of our city. We will learn how upgrades to the systems already in place can help us achieve what I know. We all want from our government a modern, responsive, efficient city that can provide services to everyone. Today, we'll examine the current staffing and infrastructure to see if we garnered the expertise that will equip Philadelphia to meet modern challenges with modern solutions. As we hear from experts and advocates today about the possibilities, we will also hear some of the risks that we face as a city if AI is implemented without guardrails. Adopting new technology is guaranteed to impact not just our jobs but our economy, our labor market, our environment and our future. We're here to learn what AI is good at and what it isn't and we are here to get clarity on what AI is currently being used for by our city government and what it isn't. Transparency is essential, and that's why I'm holding this hearing today. City Council and the public deserve clarity on what the City's policies are for using AI and the potential guardrails we need to have in place. This hearing is setting the stage for an important conversation about data privacy, racial justice, tech equity and technology policy that includes those affected by the decisions that our government makes.
Philadelphia needs to show that we are committed to making sure that as we consider adopting and using emerging technology, our budget and our practices continue to align with our values. We need to remain centered on fairness, transparency, protecting the rights of our people and maintaining public trust. This is a forum for learning what's out there and what's working and what's still to come. As we have this conversation today and in the future hearings that I plan to hold on this and other related technology topics, I'm focused on learning how the City of Philadelphia plans to protect the best interests of the public as they improve their use of technology. I look forward to hearing from our guests today. Thank you all for joining. Any other members of the committee have any additional opening remarks? )
Mr. Serra, will you please call the first panel we have to testify this morning on Resolution No. 240759.
Kristin Bray, Chief Legal Counsel to the Mayor and Director of Philly Stat 360 and Melissa Scott, Director Office of Innovation and Technology. (Witnesses approached witness table.)
Good morning. Please state your names for the record and proceed with your testimony.
Good morning, Chairperson Landau and members of the Committee. My name is Kristin Bray and I serve as the Chief Legal Counsel to Mayor Parker and the Director of Philly Stat 360.
Melissa Scott, Chief Information Officer for the Office of Innovation and Technology for the City of Philadelphia.
Thank you. We're here to testify today on Resolution No. 240759. We both appreciate the opportunity to testify today before on the Committee on Technology and Information Services on how the City of Philadelphia is developing best practices for emerging technologies like artificial intelligence, AI. We're living in a remarkable moment. For the first time in history, an emerging and powerful technology tool, known as artificial intelligence, is available to anyone with a computer and internet access. It is a tool that can be used by anybody, not just computer scientists or major corporations. This powerful new technology opens numerous possibilities for a city like ours. At the same time, like any tool we use, we must use it responsibly and with full transparency to the public. Used well, AI tools can help the Administration serve Philadelphians better, deliver services more efficiently and make our neighborhoods safer. For example, artificial intelligence can help us by analyzing millions of data points to identify problems in our infrastructure before those problems become emergencies. It can streamline services so residents spend less time waiting for help and can have access to the very services they need in mere seconds at any time of the day. It can even help us identify patterns and data that will help us deliver city services more efficiently and to those communities that have the greatest needs. But those powerful tools must be used with human judgment. They must be used with an unceasing commitment to telling the public how we are using those tools transparently and they must be used accountably with a lens towards equity, constitutional legal frameworks and fairness. Powerful tools must be used responsibly or we risk deepening inequalities and weakening trust with the public. Nevertheless, to give Philadelphians the strong and responsive government they deserve, we must use these tools wisely. That's why we're building a framework for how we will use these emerging technologies like artificial intelligence guided by three principles: human-centered judgment, public trust and accountability and using smart tools responsibly. Just because the tools are new does not mean that the values that guide us every day should change. That is why our first principle in using these tools is centered on human judgment. Philadelphia's greatest strength has always been its people, the public servants who know this city, understand its residents and who bring compassion and expertise to their work every day. They are our best asset. Likewise, our residents and the people who work and play here bring their lived life experience to help make our vibrant community stronger. That's why our approach to artificial intelligence begins with this simple truth, technology should support human judgment, not replace it. Our people's experience, empathy and common sense cannot be automated. Artificial intelligence can help public employees do their jobs better but will never do their jobs for them. Likewise, artificial intelligence can help our residents access services quickly, but it will never replace the human interaction between city employees and the people they serve every day. Artificial intelligence is here to help enhance the Mayor's vision, to give Philadelphians a government they can see, touch and feel. Of course, powerful tools come with real risk. Artificial intelligence can amplify bias if it is not designed or used carefully. Automated systems can make mistakes that will then affect real people's lives. That's why the second principle that guides our framework is rooted in transparency, fairness and accountability, ensuring every use of artificial intelligence is responsible, explainable to the public, ethical and within constitutional and other legal parameters. In the coming weeks, the City will be releasing guidelines for all city employees on how to use artificial intelligence appropriately and responsibly.
These guidelines will help staff understand both the opportunities and limits of this technology so that every use reflects our ongoing commitment to fairness, trust, accountability and transparency. In addition to these guidelines, we will also be providing city employees with training on how to use generative artificial intelligence tools responsibly and ethically. Looking ahead in late winter or early spring, we will establish a cross- departmental artificial intelligence government's committee to build on this foundation. Its role will be to continuously review and refine our policies, track evolving technology, work with experts, ensure that the city's use of artificial intelligence remains aligned with our values, our legal obligations and the expectations of our residents and employees. This is how we lead with both innovation and accountability. Our aim is not to slow progress, but to guide it. Artificial intelligence, used responsibly, can make government more efficient, more transparent and more human-centered. In closing, artificial intelligence is a tool to help us serve Philadelphians better. It is not a replacement for our people. That is why our work is grounded in the three principles that will shape every decision we make: human judgment, public trust and accountability and using smart tools responsibly. Our people are the heart of this government and the city. Our continued responsibility is to make sure that the tools we use make this city stronger, position for an innovative future and give our public employees every opportunity to make Philadelphia the safest, cleanest, greenest city in America with economic opportunity for all. Thank you. We remain available for questions.
Thank you both very much for being here, for your testimony. We appreciate this partnership with you. I have some questions and then I'll pass it on to my colleagues. You testified a lot to kind of the parameters and the guiding principles that you have for AI when in use with the city of Philadelphia, but what I didn't hear about was how it's being used now. So there's two different issues that we see. One is policy development and the other is AI use. And as you know, I requested -- I had some questions for you and requested detailed information for your testimony before today. And what I've heard more about is policy development, not the current status of how AI is used today. So can we start with, has the City done an inventory of how and -- there we go. Has the City done inventory on how and where AI is being used across all city departments and agencies?
Yes, Chair. Melissa Scott, CIO. First, I would like to state that we have the same terminology. There are -- AI is not a new term. We always had machine learning versus generative AI. So I just want to give us some definitions so we understand, because I have two different lists of AI products and I want to make sure we understand what we're talking about here. So for decades, the public and private industries have used AI, which is called the machine learning version, for decades behind the scenes, to classify, predict and to detect patterns, forecasting demand, spotting fraud, routing 311 requests, flagging anomalies within the cyber security logs and improving search results. These systems are narrow. They're task- specific and trained on historical data sets to optimize a defined output. However, generative AI is different. It creates new content, on-demand text, code, images audio-based on a user's prompt. It relies on large scale models and is trained on bass corpora to produce plausible answers or drafts. And so, basically in short the machine learning predicts or classifies within a known task. However, generative AI produces new content and accelerates knowledge, work, summaries, drafts, code snippets, mock-ups and scenarios if it's used within guardrails. Okay. So what I also have here is a list of our generative AI tools, our SaaS solutions, which SaaS means that they're on the internet or they're a spy subscription. So I'll go through the list and explain to you what they're used for. First, we have GitHub Copilot uses, Open AI Code, text to generate code suggestions and completions. Microsoft Copilot, Microsoft 365 Copilot, which integrates generative AI in Word, Excel, Outlook, et cetera. Adobe Creative Cloud includes Firefly generative AI for image generation or editing. We also have Qualtrics, which uses generative AI for summarizing and analyzing open text responses. SalesForce and the license includes Einstein GPT. It supports generative AI for CRM tasks. Tableau enabled uses generative AI for natural languages, queries and insights. Service Now includes generative AI for ticket summarization and virtual agent responses. Zoom, it offers generative AI for meeting summaries, chat replies and email drafting. Myrow, Canva and Figma, these tools include generative design and context solutions. This is what most of our software engineers use. And Notion AI, ClickUp, Monday, if licensed, these platforms include generative AI for task automation and context generation. I also would like to add many of our vendors they claim to offer AI capabilities, but the term AI can refer to a wide range of functionalities which I just been explained earlier, which may vary significantly in scope and sophistication. So before I go any further, if you would like me to now discuss the machine learning tools, I can go through that list as well.
Okay. Splunk, it includes our anomaly detection and prediction analysis. So this is not generative AI. This is the machine learning. Rapid7 Insight uses AI for vulnerability management and application security. Abnormal Security is an AI-based email threat detection. SOCRadar, AI-driven threat intelligence and attack surface monitoring. This is all for cyber security. RunZero, Cyber Asset Attack Surface Management with AI driven-discovery. ServiceNow includes AI for items, automation and analysis. Articulate 360 includes AI tools for e-learning content creation. Relish Data Assure, AI based on validation and assurance for workday. Relish Data Assure for Workday, AI-based validation for assurance for workday -- sorry, I repeated that. CARTO uses AI for geo analysis and predictive modeling. SmartRecruiters, AI for candidate matching and recruitment automation, which is not currently in use, but it is available; and Vizio, an AI-driven performance management and strategic planning, Better Stack, AI power monitoring and incident response; and Penlink Cognitive Services include speech to text, text translation. I would also like to add again that a lot of the vendors include AI functionality, but the function needs to be turned on and we have to go through Project Health Check and a set of requirements and for departments to utilize them.
So let me -- I have some other members who have some questions. Well, let me follow up with a few questions on this. To start, are there currently any guidelines for City of Philadelphia employees on how they can use these products, especially the generative AI ones?
So in 2023, there was a use for generative AI published by the previous administration. Currently, we are working, as Kristin mentioned, on a governance group and we're currently working on a policy. She also mentioned, late early spring we will be publishing guidelines for city employees.
And I would like to add to that, that we're working on these guidelines to create a plain language document for every employee to understand and also for the public to understand how we are using this because it's not just about how our employees are using the technology, but it's also important to the Parker Administration that the public knows how we are using these technologies and the rules under which we are operating.
I want to dig down for a second into the vendors in particular. It's a little further in the questioning, but I'm very concerned that we will engage in contracts with vendors who may have AI products now or may include additional AI products in the future and how do we protect ourselves with contracting language or enforcement of what they're using our data for and the oversight and tracking that we can do to make sure that we know what they're doing and what they change. We've seen in national reports -- so around the country there's entities that are coming into cities with just license plate reader technology, but they're using that technology and sending the information to ICE. Is this something that the City of Philadelphia would enter into? And if not, how would we safeguard against that?
So I'll answer the first part of the question. So we currently have a process Project Health Check where any type of software when a department wants to procure a software, any solution, they would need to go through this process, whether it was AI or not. Currently, there are 6 questions that the particular 7 department would have to answer in 8 regards if there is an AI tool, our 9 team actually ensures that AI must 10 align with the city's data privacy 11 and accessibility record management 12 requirements to ensures 13 transparency, accountability and compliance with public records. So we work closely with law integrity and ensuring that any procured software goes through this process before we actually procure it, and this is a process that's been standing prior to generative AI becoming popular.
What if it changes while in the course of your contract? What oversight do you have with a company that says they want to do one thing -- what consistent oversight do you have to make sure that they are not changing the terms of what they're using with your data in the contract?
So I will add that part of our process -- what we're currently looking for right now we're looking at Copilot as the ability -- as to be the approved solution for -- and I'm just giving you this as an example. And we did a risk analysis. So we do a risk analysis on a particular software. And what we look at is how we can continue to generate reports on what the software is doing behind the scenes, who's using it, what are their prompts. So particularly with Copilot, we looked at chat reports. We pull out monthly monitoring, activity prompts, sensitive and appropriate data. We actually research that and pull these reports on a monthly basis in the background with or without the users knowledge, right. So it's really with the users knowledge, because this is within the contract but we ensure that these particular softwares have these capabilities. We also look for catch errors, language around output errors, removing biases, training users to detect incorrect information and outputs to ensure the review is done before utilizing the data. And we also do -- the chat gives warning prompts as well to reinforce training. We've also identified this as the top risk before utilizing the data. So we go through a series of design, security checks, risk assessments of the particular tool and research what can we put on the back end to continue the monitoring.
I'd like to add, Councilmember, as we're thinking about our AI Governance Committee, one of the things that we're thinking about is this is an emerging and evolving technology. So we can't develop a policy and put it on a shelf and congratulate ourselves on a job well done. We need to continuously evaluate that policy. As vendors, the public and our employees are using this new and emerging technology so that we're responsible or responsive to what is emerging, how vendors are using this, how the city is using this so that we are always aligning our actions with our values and the constitutional frameworks that we fought long and hard for and are still in place today. So we want to make sure that we're constantly evaluating what we're doing, that we're always aligned with our values.
So I want to get back to the inventory. Do you know which departments are using what or what employees and what departments?
My understanding is that -- I want to go back to who we is. You say we keep evaluating. Who's we?
Yes. From a security standpoint, yes, our tools allow us to evaluate and produce reports.
But are you -- is OIT the entity that's reviewing the vendor contracts and doing the spot checks of the vendors' work?
Law Integrity, our Office of Human Resources, our Chief Administrative Officer. We're making this a multi-faceted project because we know technology touches on every facet of city government.
And have you been doing this up until now or this will be the new --
My next question: My understanding is that, OIT does not necessarily have control over every technology unit in every department. So for example, I think the Department of Recreation, for example, has their own technology department. If they made a decision on a product, would that have to go through you as OIT? Do you oversee what Recreation does or any of the other departments with their independent technology units or do can they make those decisions?
Each department is different so it depends on the department. But with your example with Parks & Rec, yes, it would have to go through OIT.
What are the other departments that have their own technology department in the City of Philadelphia?
Yes. And I would also like to know if any of them make their own decisions outside of OIT? I'd also like to add now that we would like -- once we secure an oversight of some degree, at the oversight is there for every city department and agency and subcontractor, which is the next set of questions, to make sure that we have the same standards that we're using for everyone? Because if not, we won't have any control of this. With the subcontractors of the vendors now, do we do a review or have an understanding of what AI they use?
And that is with every department and agency in the city?
As I mentioned before, just the consolidated ones, as basically you pointed out. So I would need to get a list to you of the departments that are deconsolidated so you would have the separation between what OIT has governance over versus not.
Let me get a little closer to the microphone it seems. I'm going to ask a few more and then pass it on. There are many examples of use cases around the country of how cities are using AI. What would you and the City of Philadelphia see as an acceptable use case in Philadelphia, a way in which we would use it?
Sure. So my understanding is we use AI with Streets to make sure that it's evaluating potholes and obstructions in the road. We use AI with the Parking Authority for tickets. What do you as the City of Philadelphia see as acceptable use cases where the door is opening fast, jurisdictions around the country again are using facial recognition with the police, they're using license plate readers and sending the information to ICE? What are the boundaries that you see now for how we would use it? We could potentially use it in the City of Philadelphia and in what ways?
Part of what the committee is looking at and taking a look at for purposes of the guidelines and the ultimate guidance that we are going to issue out to the city in the late winter, early spring, what are those acceptable use cases that we can have? We are of course looking at what other cities are doing. Some cities have very strong AI policies. Some cities do not have strong AI policies. We want to be a leader in this field and we want to make sure that any policies that we're developing are taking into account our values and those constitutional parameters that we're well aware of in this city.
Who's going to be in the -- what's the makeup of the committee? Who is going to be on there? And are there people outside of the City of Philadelphia, experts who would be on there?
I think that's something that we're discussing about how we can bring in those outside expert voices into the committee. But currently we're working with OIT, the Law Department, OHR and Chief Administrative Officer.
Who do we see in the City of Philadelphia, who are our AI experts? No, it's not me. So who in the office of OIT or otherwise are experts in AI?
Interesting question, because if anyone in the world would say that they were an AI expert, they would be lying. Because it is continuously transforming. It's morphing into new technology. So you can be an expert on where AI is today, but three months from now it can be in another way. But I would say the Office of Innovation and Technology, we're most versed on AI products and working closely to continue to research and continue with vendors like Gardner and other professionals to continue to learn from one another. We also go to a lot of conferences. We talk to a lot of other professionals within other counties and cities just to educate ourselves and understand more about the evolvingness of the technologies of AI.
Thank you. I just want to ask just -- I have so many questions for you two. I'm going to ask one more and pass it along. But a very big concern of mine are people's privacy and ways in which AI can be used, whether it's for racial discrimination, whether it is for presumed identifying people who might be from other countries living here. I really want to dig down into the safeguards and the guardrails that you're going to do. I've heard a lot about the law. We know what the law is. This technology is moving very fast. We have a U.S. Supreme Court who doesn't seem very set on enforcing the current laws that we have. So I think it's really up to us to make sure that we're putting in the guardrails to make sure that we're not going to go years in the past of perpetuating discrimination, perpetuating inequities and making sure that we're taking care of people throughout Philadelphia. Can you just comment on that for a minute in which you think you can make that happen? I'll pass along to my colleagues and then come back.
We understand those concerns. And both Chief Information Officer Scott and I have been with the city for a very, very long time. We understand how we have gotten here and we understand the battles of the past. While we're developing this policy, we are going to be looking at those values and ensuring that we're incorporating those values into that policy. The policy is not written. We intend to have it by late winter, early spring, but those values are forefront in our mind about privacy and the Fourth Amendment. That's why we're bringing the Law Department into this committee because we believe that their privacy attorneys play a big role in this. We believe that their civil rights attorneys play a big role in this because we want to spot those issues and address them as best we can. That's also why we are intending to have this policy reviewed often and regularly to make sure as the technology evolves, we also evolve.
Thank you. I would like to pass it along to my colleague Councilmember Isaiah Thomas first.
Good morning. Thank you, Madam Chair, for your leadership as it relates to this particular issue. Clearly, this is an important conversation. You can see that by how many people are convened here. Thank you to the Administration for being here and for doing your due diligence as it relates to this particular issue. Just a couple questions in the midst of listening to Madam Chair. I didn't hear -- and I did hear Madam Chair talk about it towards the end right here, but I didn't hear much around what we're doing with AI and public safety. So I did hear a little bit around the Parking Authority, but I want to do a deeper dive into the police department. And so, when I think about the technology that was used around -- well, let's just ask: What type of AI is being used specifically to deal with and collect evidence when a hit-and-run is taking place?
I think we'd have to take that question back to the police department.
Okay. It's really important that we know that. In 2020, we've seen a record number of hit-and-runs. This legislative body decided to put money in the Managing Director's Office to give the Admin new dollars for new technology to be able to collect evidence of hit-and-runs. And it was agreed upon that those dollars will be spent. And I'm wondering if AI is a part of the new technology that's being used to collect evidence with hit-and-runs. Madam Chair talked about the license readers. Is the police department using AI or anything like that dealing with traffic stops, motor vehicle violations and things of that capacity?
So I can actually speak to the license plate reader. So to our knowledge, there are no AI capabilities currently being used for license plate readers. So I know that from where we are now, because I was actually involved as it was being implemented. But some of the things that you're mentioning is previous to my tenure and we don't have any knowledge currently documented or anything, so we would have to refer back to the police for additional information but we can get that back to the Chair, if that's okay.
It would just be good to confirm that we're not doing that, to invade people's privacy. I think that's the whole point of some of the work that we've tried to do. What about the speed cameras? We've been strong advocates for speed cameras, especially in school zones. And so, I'm wondering,the speed cameras that we're using to address some of those issues, what type of AI do they have in them because a lot of people are concerned about those speed cameras being used to do something else besides catch people for speeding? So can you communicate to the general public what type of technology that we use as it relates to speed cameras? I know Broad Street is something that a lot of folks are talking about right now. But I'm really excited about the collaboration we have with the Administration around putting speed cameras in school zones.
So I think the confusion could be the difference between machine learning and generative AI. So what I would like to do is go back to the police department and provide a list for you because it could be AI, but machine learning, not generative --
Can you explain to the general public what that is though so that folks can understand the difference, so that people can understand what it is that we're trying to do specifically with speeding, hit-and-run, issues like that?
So as far as hit-and-runs, and there's no 3 generative AI solutions to my knowledge today, but what I'm saying is that if there is an AI portion, it could be you're referring to machine learning AI, which as I mentioned, it predicts and classifies within a specific scope versus generative AI can actually accelerate and identify and go beyond the scope, the guardrails that you put in place. So we would actually need to get that back to you and we could get the answer back to the Chair. But again, I would like to state on the record that we don't have any knowledge of any generative AI being used within our speed cameras.
Follow-up: What company are you using for the license plate readers?
That's a good question. I don't -- let me see if I have that answer. One second.
This comes back to my control, once we enter into a contract with them.
Final two questions: Is there another area that AI might be being used that's specifically related to public safety, surveilling people, constituents across the city of Philadelphia? Is there something that might have slipped through the cracks that we didn't ask where we are using AI specifically around issues related to public safety?
Currently, we're not aware of any generative AI being used within public safety. So as far as slipping through the cracks, I don't believe that that's actually possible with our current process but we can definitely get a response --
No, I mean questions that we didn't ask thus far. I know we've asked some questions so far, but I'm wondering if there's an area where AI is being used, specifically in a realm of public safety that we haven't asked about so far today --
Okay. Amazing. Last question: I'm the Chair of the Education Committee so I got to take a minute to at least ask you is there anything in the working with the School District, the Mayor's Department of Education, to examine how AI is being involved in the academic journeys of our young people? And I'm not talking about teaching them how to use AI. I'm saying AI being used for things like tutoring, teaching and things of that capacity. Is there any plans or is there any vision, has there been any coordination or dialogue with the School District to assess how individual institutions are using AI, not teaching it, right, using AI as a supplement for a tutor, a teacher, a physician? I don't know, anything.
Okay. I'll close out with saying that's very important, right. And I think when I listen to our Chair talk about how AI is evolving, you know, last school year a teacher might have never thought to use AI to supplement some type of curriculum or lesson. And this summer, they might have had an amazing journey or a bad journey, I don't know, and they might have came back this school year looking to use AI. And I think part of the concern is that, number one, it doesn't always give accurate information. And I think another part of the concern is that you tend to because it's so personable now, you could develop a dependency, and we know a lot of our young people just basic human growth and development, could be facing mental issues, emotional, social issues, all types of trauma, and we don't want these things giving bad advice that create unintended consequences. So I think that's a really good homework assignment. I'll be looking forward to working with our Chair to figure out exactly how we move forward with that in the future. Thank you, Madam Chair. I'm done. Thank you.
Madam Chair, I do have an answer to your question about the license plate readers. The vendor is Johnson Controls, Inc, JCI.
Thank you. Next, we'll move on to Councilman Jimmy Harrity.
Thank you, Chair. I'm not the most tech savvy guy around as other people, but I'm learning. Whenever we talk about this kind of technology, I automatically go to the movie Minority Report. Everybody remember that, you know what I'm saying. This kind of technology really scares me. It's no secret that people in Philadelphia do not feel safe, even though they say crime is going down. The feeling is not safety. That being said, I do not want to give up my freedoms for the sake of somebody else's idea of safety. I would be curious to know, as we said, what we're using this technology for. I know all technology there are some good things that come of it. But for me, you know, movies have a funny way of predicting the future. I believe this is a slippery slope that we're going down right now and that we need to be very careful because, you know, there are consequences to the things we do now down the line. So technology may not be a way right now for them to do the kind of things that we don't want them to do, but technology grows, as you said yourself, and things evolve. I just want to make sure that we have a control of this, that we have people watching this, not just people that are in these departments that may actually have a stake in what is going on there, but regular citizens on the oversight of this stuff. Because again, I'm very leery of this kind of technology, and maybe that's because of my lack of technical skills when it comes to that. But for me, I'm definitely not giving up my freedom for a little suspected added safety. And, you know, I'm a safety guy. You know, I'm for policing. I'm a pro-police Democrat. You know, I want our citizens to be safe. But again, I don't want our citizens to feel like they're being watched all over the place. And this has already been bought up. This isn't -- I'm not bringing this up and it hasn't already been out there. The ACLU already has lawsuits in New York and other places where these technologies have gone, the drones too. We haven't really implemented a drone program now because we're still waiting to see what's going on in New York with the ACLU and their findings from the court cases. So I just want to make sure that we're aware. I also want us to be fiscally responsible because this stuff does lead to lawsuits and we can't afford that either in the City of Philadelphia. But some more of a little tech, more technical, the Administration consulted with city departments and quasi city aid entities about their upcoming guidelines for the city employees. So you're already talking about that. That was when your statement, right? When can we see some results from that? When will the guidelines be available? And then on top of that, will the community input be taken into regard across departmental AI governance? You know, when we're forming a committee and having recommendations, I believe that the citizens definitely need a say in that.
With regard to who is in the committee, we're still discussing who is going to be part of that, but we are definitely listening to the voices that we hear in the community here in Philadelphia, most especially. But we also look to other jurisdictions and what their experiences have been. We want to learn from that. We want to grow from that and we want to be a leader in this field to the extent that we can be.
We've all seen before when government tells us that they're going to take into account what the citizens say. That's a lot different than actually having a citizen with say. That's all I'm going to say. Thank you.
Thank you, Madam Chair. Real quick, and I know we mentioned about the concerns of using AI and how it's being used, but I think another concern is what are we doing now with the data that's being captured from that information? So even though we have the technology's being used there, obviously right now we have data and what is being used with that data because guidelines will be written in the future, but we have that data now of some things that we're using. So what do we do with that data? And how is that data being either shared with somebody else or used?
So part of the governance committee is to discuss just that, how to ensure that we keep the data secured. That is actually one of the major concerns of the generative AI tools that you can currently utilize online. We are very anxious to have a policy in place to ensure that city employees are not putting city data into those tools because it goes out into an ecosystem that's not protected. So we need to -- we are, as I mentioned earlier, pushing to have Copilot as our official tool where city employees can be able to use it to enhance their work. But what's most important about that is it is in a controlled ecosystem where it doesn't go out and then our data is then used back against us.
Oh, our current data is secured through our current structure as city government. But for AI, you're speaking specifically about AI. We don't currently have any AI tools, generative AI tools, that we procure that would allow our data to go elsewhere.
So we're not using that at this time. And so, we will be able to protect that in the future so that nobody else would have access to that?
Yes, but what we should also keep in mind is training city employees on how to use generative AI tools. There are a lot of tools that they can use and access on their own. So I have a list of them. So you have your ChatGPTs, your Copilots, your open AIs, your Gemini, those tools you could just go on right now and utilize that and upload city data in there on your own. So we are in the process, as Kristin mentioned, creating training for city employees so they understand that they too have a part in assisting us in protecting our data.
Yeah, but that is best behavior, right. So I can't go with you to your house and stop you from going to -- say, you're working on something for the city and you go home and you're working on it from home. I can't stop you from uploading that information into Gemini and then it goes out into the ecosystem of the world. We have to train you and then hold individuals accountable as far as letting them know that this is against the rules. You're not allowed to upload city data into these online tools. So that's a different conversation that we do need to have.
(Inaudible). Our Chief Information Officer points out we can control the environment here within the city and our city devices. If an employee goes home, they might not be aware, which is why we want to release guidelines and training while we work through the more complicated, complex issues that this panel has identified, like the privacy issues, the constitutional issues. They need -- we don't want to rush to put out a policy that doesn't take into account all those important, complicated issues.
But I think at the end of the day it's what are the consequences for somebody providing that information or sharing that information out and what challenges that presents to the city or other individuals, right. So I think that we really have to look at that and understanding how that data is protected and how we as a city, whether it's education or training but then also guidelines on, you know, what are those concerns and what happens if people share that. I mean, you have disgruntled employees that go out there and feed information out. We have other folks that could give information to AI and then compromise things that the city is doing as a whole. So again, technology is where it is and where it's going, but I think that's why we're so happy for the Chair to be able to bring this hearing too because these things have to happen now, like we can't wait further. And I know we're looking at other municipalities and what they're doing, but we have to have certain guidelines in place and then also rules and regulations on what we do with our captured data, right. We always hear about people selling information and using information. Not that I'm saying -- I'm not saying the city's doing that. I'm just saying. If you want to make people feel confident, and a lot of people, like I said, aren't even confident in the use, but then they're really not confident not knowing what happens to that data once we have it. So I think that is an important aspect of where we go from here. Thank you.
Thank you. Next up is Councilmember Anthony Phillips.
Thank you. I just want to thank the Chairwoman for her work on this. Thank you, Chairwoman Landau. This is an incredible hearing. I just want to say also thank you to the Administration for coming out today. I know this is a challenging thing to talk about. It's like the time when the world was introduced to credit cards and debit cards and cash, amongst some other things. This is a big structural change. I do want to say that I understand the public's sentiment around this as well because as you all probably understand, you don't want to lose jobs to AI systems. And so, that's why this is I guess a larger conversation issue. I just want to ask is there a way that you all -- have you thought about ways that Council can work closely with the Administration to create like a structure so that when residents do have these comments, we can find a way to facilitate them the best way we know how to assure that this was a collective reasoning around our initial piloting when it comes to AI. Is that something that's being thought about, just something that's helpful?
Yes, indeed. That is something that we're in the process of discussing of how we can collaborate with both partners in the community, with our partners at City Council and with our partners in other municipalities. You heard Chief Information Officer Scott talk about how we partner with other governments to see -- other municipal governments to see how they are working their policies and what they have encountered. But, yes, that is absolutely something that we are considering in this committee to make sure that we're getting voices at the table to receive that feedback.
No, I actually appreciate it because that was one of my other questions. I wanted to make sure that we were focused on it. There's other municipalities that have done this well or going through this community engagement process and that would be fairly helpful. The last question I just had is that you spoke about public trust, Kristin, about a government Philadelphia can see, touch and feel. Again, I just want to talk about ways in which when you come up with your plans for the winter, if there is a way in which you can have a public engagement plan around that. Of course, a lot of times the Administration make decisions that are in the best interest of belief because people have put you in the office to do the work. But this is, I think, a heavier change, such a big change that the public is probably going to want a lot more details about how this is working out. So, yeah, there is a level of public engagement, plans, that you could put together that be helpful. But thank you all for answering my other questions. I think we're all good to go. Thank you.
Thank you, Madam Chair. I appreciate your commitment to centering human judgment as well as public trust and accountability as you mentioned in your opening statements, which is particularly important to me because you cannot, as some folks have said earlier, I think automate certain sensitivities which is another reason why I think valuing the spaces where our ethics and morals are shaped is of paramount importance. I'm saying that to you as well as the listening public, in times like these making sure that we are prioritizing that sort of discernment, that sort of human sensitivity, those sorts of ethics. Because as someone taught me years ago, bad data in is bad data out. (Applause.)
And so, it's particularly important. But I think that's on top of also localities recognizing, as I think was also mentioned, our particular importance. We can't rely, and I felt this way -- at least it's been a growing feeling -- I felt for some time, but it's kind of come to afore, dealing with the whole SEPTA thing. We can't rely any more on big brother, state, federal government, what have you. We have to look out for ourselves. We got to protect us. And so, I think all this is particularly important. I want to look into and look forward to reviewing your guidelines as well as the findings around the AI Governance Committee on that. But just aside from the human judgment piece, I did not hear or I wanted to hear more about public trust and accountability, how you will ensure public trust and accountability as we move forward with AI in the city going forward.
So we're still discussing that and that's going to be coming out with our policy in the late winter, early spring. But one of the things that we really want to take a look at are those public voices. We will be listening to the additional committees and panels that testified before this committee today. We are looking at what has happened in other cities and we are listening to, as councilmember Phillips pointed out, some community engagement and Councilman Harrity pointed out community engagement. It is important here and we're going to take that back to our committee about how we can best incorporate those practices into our policy as we're rolling this out and moving this forward.
Both of you have mentioned looking at other cities and other states, other counties and what have you. Could you name actually which are the counties and cities that we are looking at as it relates to their AI policies?
Yeah, just curious as to who Philadelphia was looking at as it relates to our pattern or learning from as it relates to that policy?
Washington, D.C.; Seattle, Washington; Bellevue, Washington; San Jose, California; Spokane, Washington; San Francisco, California; New York, New York; Long Beach, California, Boston; Fort Worth, Texas; Salt Lake City, Utah and Baltimore, Maryland.
Got you. And I'm assuming they all have -- some of them I know have lengthy tech histories, but some of the other ones that are less known, they have proven AI policies, they have policies that people are saying we should look at or are we just kind of -- how are we deciding what cities are good to look at and what's not?
So we have evaluated them as far as the maturity level when it comes to the AI policy. So particularly the two that are very high, Washington, D.C., they have a comprehensive AI governance policy. Only IT-approved tools allowed, requires security review, document approval. And I'll share with you the next one, Bellevue which has a strong procurement governance, approved AI list, mandatory new tech request process and prohibits unapproved AI use. So those two are the highest. And then it goes from medium high to medium to medium low. We scored them.
Okay. Very good. Earlier you gave a list of some of the programs that were used, making distinctions between generative and machine learning. I think one of the programs that the city is using is Workday, correct?
So Workday is part of our ERP. So it's our Opal project which is currently still in development. So we mentioned it because it's procured, right, but it has not been implemented as yet.
Okay. Well, that's actually a good thing because it's been reported to me that Workday is known to embed bias into its programming. And so, it makes me ask the question what vetting processes do we have to ensure that we are not procuring programs that actually are furthering bias within our governance structure?
So as I mentioned before, we have a process that every department or that if consolidated will go through when they're procuring tools, particular softwares. But what we need to make sure is that they are -- if there is an AI portion and we're continually monitoring and producing reports looking at if any AI portion of those tools have been turned on, to assume that everyone won't just be as forthcoming, right. So, for instance, in your particular example even though Workday has not been implemented, us evaluating the current product and looking at what those AI tools are and then in the background producing reports to ensure if any of those have been turned on without our knowledge. We would flag those and address it with the particular department.
It sounds like we're kind of building things as we're going along. Maybe I'm mishearing that. But is there intentionality that's going to be implemented here to make sure that bias in particular is a thing that is not being perpetuated? Are we thinking along those lines about making sure that particular issue is being addressed as it relates to any particular program that we're using?
Absolutely. In part of the governance group of the AI governance group, we will continue to ensure that what we have to be conscious of here is that AI is continually evolving, right. So what biases of today and what biases there are tomorrow. They'll change. So as I mentioned before, as IT professionals it does sound like it's an evermoving wheel, because it is. Just as we implement, we have to then improve, implement, improve, because the technology is moving and emerging that fast. But the governance body will continue to be in place and we will continue to evaluate as new products come in, as the vendors see -- what happens is the vendors can create an AI tool and then put it available in the tool without anyone's knowledge. So the question is what are we going to do to ensure that we are running reports that we are evaluating in the background as these companies change their models without our knowledge. So that's a very important aspect. So if it sounds like it's ever continuing moving, because that's what the industry -- is currently going on in the industry and we have to stay up and ready at all times.
It sounds like we almost could compile a list of bad actor corporations or whatever that might help us as opposed to waiting a downstream look at the product. Maybe if there are or companies that are saying, okay, we know that they've done this before, they've slipped something in there, what have you, perhaps we can begin to compile a list of those sorts of companies and say, well, we can't trust you on the front end so we're going to not work with you as a place you're trying to get those sort of programs. But I think that's something we can address down the line, but I just want to suggest that it's something that we look out for as well, not just the programs but the companies themselves. Are you a bad actor, do you -- or maybe that's a bad phrase. Are you someone who's not trustworthy as it relates to providing a product, artificial intelligence, that we can ensure doesn't have certain little things put into it. But I thank you for being here today, presenting today. I understand that it is an ongoing rolling issue, but I am of the opinion that as it relates to what governing looks like now and going forward, we've got to figure out this situation around AI algorithms. It is literally shaping reality. And so, we got to get our arms around. And as I said earlier, we can't trust other places to do it anymore. I really feel like localities, municipalities, we got to beef up and really hold our own. So thank you again, Madam Chair.
Thank you. One, I wanted to follow up on just two things and then also wanted to say that a couple of our members have other obligations and are going to have to step out. But the first, I'm not sure if I heard you correctly but you talked about public safety, and I thought I heard you say that you're not using AI in the realm of public safety here in Philadelphia now?
To our knowledge, we're not. We would have to refer this to the police department. If there's anything else that's unknown, as you mentioned before, there is some form of deconsolidation, especially in regards to OIT. So to our knowledge, no. And I'm saying for me to our knowledge and my team's knowledge, no. 8
We already know that the city is using facial recognition software, ShotSpotter and also drones, and that is where my concern comes in about we know -- even in a hearing at City Council, talked about the police department's purchase of these drones through the outside Police Foundation, which goes outside of our budget process, which I was concerned about. We learned this week that the Trump Administration is launching a $500 million effort to help cities where World Cup and other 2026 events are happening to protect against drones in their airspace. Private military drones are often equipped with AI in their surveillance technology. Is the Parker Administration contemplating collecting, collaborating with the Trump Administration on this anti- drone initiative?
I think we can get back to you on that question. That was probably best directed to the police department.
Also used by AI of the federal government to further Trump's harmful agenda is a serious concern for all of us. For example, the U.S. Department of Homeland Security, DHS reported nearly 200 uses of AI across its departments, including approximately 20 active use cases by ICE, Immigration & Customs Enforcement. The use of AI to turbo charge, immigration enforcement is a serious and very present concern. Is the Parker Administration worried about that happening in our city. And will the Parker Administration also in any way collaborate with DHS and ICE?
Finally, with the makeup of your committee to oversee AI, I know we asked you already who would participate in that. Do you expect to include Labor, the city workforce, who both could be penalized for doing the wrong thing and/or have their job disappear because of AI, will Labor be at the table?
We do have our Office of Human Resources in our current committee format and that's part of the things that we are considering is how does this impact labor ultimately, any policy that we have.
Do you have any plan yet on how often this committee will meet?
Not yet, but we're going to meet routinely and regularly because the technology is evolving.
Follow-up to one of my colleague's questions, will you make those meetings public or will you allow an avenue for public input, especially because what we'll hear today is some pretty great experts on this subject that can really add to what we're doing here in Philadelphia?
Okay. Thank you. The Chair recognizes Councilmember O'ROURKE.
It's already kind of come through, but I want to get the exact wording on the record and get a response. As these technologies progress, the city will be collecting more and more data, especially from automated systems like the speed cameras and the $2 million new drone program that was already mentioned. How are we ensuring that this data isn't being used by outside law enforcement like ICE and other federal officials without a signed judicial warrant? (Applause.)
So I'm not an expert in policing and we'll get back to you on that, Councilmember.
Thank you so much. Thank you for being here. Thank you for staying I believe throughout the rest of the hearing or at least hopefully watching?
We will be watching and staff members are here too just in case anything comes up.
Thank you so much. We really appreciate you being here. Mr. Serra, will you please call the second panel.
Sorelle Friedler and Ravi Chawla. (Witnesses approached witness table.)
Good morning. Thank you so much for being here. Can you state your name please and proceed with your testimony.
Sure. Thank you, Councilmembers and Madam Chair. My name is Ravi Chawla. I'm the Vice-President and Chief Analytics Officer at Independence Blue Cross. Okay. Independence Blue Cross appreciates the opportunity to speak before the Philadelphia City Council Committee on Technology and Information Systems to share insights on the responsible use and transformative potential of artificial intelligence in our daily operations. As a health insurer serving the Philadelphia region for over 485 years, IBX has been on the forefront of AI innovation, using it to improve member experience, streamline operations and enhance health outcomes. We offer this testimony to support the city's exploration of AI technologies and to provide a framework for understanding how AI can be responsibly and effectively integrated into municipal services. AI, as you guys all know, is a multi-disciplinary field that combines computer science, mathematics, linguistics and other technologies to automate tasks, generate insights and support decision-making from chat bots and predictive analytics to generative AI tools. Its applications are vast and growing. At Independence Blue Cross, AI is used to predict health complications and reduce emergency room visits, conduct preliminary reviews of member data, to identify behavioral health needs, enhance customer service through chat tools and personalized wellness plans and support administrative tasks like drafting communications and presentations. These examples illustrate how AI can be used to improve service delivery, reduce cost and increase efficiency, goals that align with our mission to improve health care in the community. Independence AI work is guided by our internal Center of Excellence on AI and AI Ethics Committee which include experts in law, compliance and technology. We follow six core principles: accountability, compliance, privacy equity, reliability, safety and security. We also adhere to national frameworks such as those from the National Institute of Standards and Technology and participate in work groups that align with best practices. Importantly, human oversights remain central to our AI processes, especially in sensitive areas like clinical decision-making. This balance between automation and human judgment is critical for public trust and ethical governance. I would love to give an example. Consider the case of John Doe, an IBX member with congestive heart failure. After a routine check, his health data is updated in our system. Using AI, a nurse, a health coach, can identify potential risks and connects John with resources to prevent hospitalization. This proactive approach can lead to almost a 38% reduction in ER visits and 43% reduction in hospital admissions through timely interventions. These have been published, if you guys -- we would be happy to provide those papers. Cost savings of improved health outcomes and enhanced quality of life can really lead to cost savings of almost $716 per member over a six-month period, reflecting more efficient care, not reduced access. This model can demonstrate how AI can be used to anticipate needs, personalized services and improve outcomes. While AI is a powerful tool, its use must be accompanied by robust safeguards and IBX ensures that through our data privacy, through HIPAA and other legal standards, transparent processes with pre- and post- implementation testing, use of member-consented data and continuous monitoring and improvement of AI systems. These practices can serve as a model for any entity considering AI adoption. AI is not a replacement for human services. It is a powerful tool that when used responsibly can enhance the delivery, efficiency and equity of public service. At IBX, our commitment to responsible innovation is guided by these principles.
As Philadelphia explores the future of AI in government, IBX welcomes the opportunity to collaborate with the City Council and share insights from our experiences in health care, from improving (inaudible) engagement and streamlining operations to advance equity, we believe AI can offer transformative potential for city agencies. By embracing thoughtful governance, ethical safeguards and cross-sector partnerships, the city can harness AI to build a smarter, more inclusive future for all residents. We thank the Committee for its leadership and thoughtful consideration of this important topic.
Thank you so much for your testimony. We're going to hold questions until the end. So please state your name and deliver your testimony. Thank you.
As a Philly resident, I'm especially glad to see that you're taking seriously the way that AI can be used to make decisions that directly impact all of our lives. My name is Sorelle Friedler and the Shibulal Family Professor of Computer Science at Haverford College. S. Technology Policy Committee for the Association for Computing Machinery. That's the main professional organization for computer scientists. And I formerly served as the Assistant Director for Data and Democracy at the White House Office of Science and Technology Policy. During my time at the White House, I co-authored the AI Bill of Rights and helped develop policy governing AI use across the federal government. I've done research on responsible AI techniques for more than a decade, and I'm a co-founder of the Conference on Fairness, Accountability and Transparency, one of the main publication venues focusing on the ways that AI impacts people. Before becoming a professor, I was a software engineer at Google. I've been asked to talk to you today about what AI is. As policymakers, you don't need to understand the detailed formula or specific algorithms behind these systems. What you do need are analogies that you can rely on, that build on your existing expertise as you make decisions for our city. I want to give you a few of those analogies today. AI is like the past. Broadly, we can think of AI as taking data about people and historical events and fighting patterns that predict and allow the replication of past actions. These patterns of the past do not perfectly predict the future. If there's one thing you take away from my testimony here today, let it be this: AI is not designed to work all the time. The guarantees that we make about AI as computer scientists are statistical. We might say, this system achieves 98% accuracy. That sounds pretty good. 98% is pretty good. It also means we got it wrong 2% of the time. A lot of AI governance is about making sure there are systems in place to handle that 2%. Given our city's population, if an AI system with a 2% error rate makes a decision about everyone in the city, that's more than 31,000 people who got the wrong results. We need plans in place to determine who will suffer from these errors and staffing to help people fix them. This also means there are times when the right answer will be to prohibit AI use. In cases where errors would be damaging or hard to catch, AI can't be the answer. For example, we should be cautious about AI tutors in our classrooms. They may confidently assert falsehoods spreading lies in our schools. In order for you all to have control and make sure AI systems used by the Administration actually do the job, we need careful guardrails, including testing requirements, human review processes and transparency into how AI is being used in our city. AI is like redlining. Since AI is like the past, it will replicate the biases of the past. In my own work and work of colleagues, we've found that predictive policing systems have this problem. Instead of identifying locations where crime will take place in the future, they repeatedly send police back to the same neighborhoods where they've made arrests in the past. If historical policing and arrests were more likely to occur in predominantly Black neighborhoods, this will ensure future policing and arrests match that bias pattern, regardless of the actual locations of high crime areas. AI is like a line of best fit. To better understand how an AI system works behind the scenes, I think it's useful to think of a line of best fit. You take data with two attributes, X and Y, and plot it, then you draw a line through the data. This is a way to identify a relationship between two attributes. Say, home values five years ago and home values now. It can then be used to predict resulting data in the future, say, home values in five years.
Modern AI systems are built on a large number of building blocks that work essentially like this: AI is like a flow chart. To handle more complex patterns, aI systems operate like a giant flow chart with the results of a computation, like a line of best fit, determining which flow chart step comes next. This could allow us to make different home value predictions for different neighborhoods. The specific computations performed at each step are determined based on training, which is the word for taking data from the past and finding patterns in it. AI is like the web remixed. Often when we're talking about AI these days, we're talking about generative AI and chat bots like ChatGPT. I think it's useful to focus on the data used to create these systems. AI companies gathered all the data from the web they could get their hands on, old newspaper articles, Reddit posts, YouTube comments, porn sites, conspiracy theories, everything. We can think of the results as the web remixed with prompts to the chat bot providing information about which parts of the web to remix and show to the user. AI is like a factory. One final analogy. AI is like the modernization of a factory. It replaces people's jobs with automation. The data centers that run AI take a huge physical footprint. They spew pollution into the surrounding neighborhoods. And as we've already seen in Philly, take so much energy that they increase all of our electricity costs. Yet there's one important way AI is not like a factory. It doesn't create long- term jobs here. After construction, data centers are mostly empty of people with just a few security guards. I hope as you provide needed oversight to city use of AI you remember that this technology is not magic. You can expect to be able to have your questions about it answered, have transparency into how these systems are being used and take control over whether AI is working for us. Thanks again for inviting me to speak and for this important work you're doing for the city. )
Thank you both so much for being here and for providing your expertise, especially in specific areas because I think that's what we were looking for, a little bit more of the specifics. My questions are to both of you, but particularly to Professor Friedler. In what experience, what kinds of use cases are particularly appropriate for a city government to use AI? In other words, what is AI good at and what problems can it help solve for the City of Philadelphia?
Yeah. So when I was in the federal government, one of the agencies that I really appreciated working with was the VA. They've been using AI in a lot of really useful ways to try to reduce the backlogs of veteran services that we all know they have suffered from for years, and they've been doing a really great job at that. So what that means is they've been trying to proactively identify -- as people sort of file 21 for requests for benefits they've been trying to figure out, okay, what are the really obvious requests using AI that we can just streamline for approval, right. You know, we can help these people faster because their case is really obvious and we know that they deserve benefits they're eligible for these, right. And so, they've been trying to use AI to sort of identify those places where they can give people services faster, right, with less paperwork, with sort of more streamlined paperwork. I think that's a great use case at the VA but also for the city, right, what are those city services that people should have access to, how can we get people access to those services faster.
This is what the City's Chief OIT Officer was telling us: It's machine learning AI though, it's the AI of the information you already have contained at the VA and using that in order to streamline its own services. Is there any way in which you've seen generative AI useful in city government?
That is a great question. Most of the cases where I've seen AI be really effective are these sort of more narrowly-tailored cases. That's not to say that there couldn't necessarily be a use case, but I think that there is a reason why it's mostly been these narrowly- tailored use cases that I've seen be successful, which is that those are really the places where you can test to make sure the system is actually working, right. As I said in my testimony earlier, I do think it's important to understand that a lot of generative AI cases are just trained on the entire internet. That means they're not necessarily trained to do what the city needs them to do. They're trained to give answers that are pleasing to people. That doesn't necessarily mean correct answers, right. So that means they're good at wordsmithing things to sound pleasing, but that is not quite the same as delivering city services, right. Those are I think clearly distinct use cases.
And I think that some of these cases could be is primarily in improving productivity and that is for the administration staff itself. I know that there are cities -- and I'm giving you an example from somewhere in India. They are using gen AI primarily to get rid of the backlog of legal cases. And that is not to replace lawyers, but it is more around, you know, can they be able to fast move some of the legal cases which are sitting there probably for 15, 20, years. And there are people who are incarcerated for that much time because the legal cases have not moved fast enough. So there are things like that at Independence Blue Cross, you know, we are using it more from a productivity perspective. You know, the human in the loop is always going to be there whenever we make any kind of decisions. But when it comes down to productivity, for people who are even using it for -- I think the City CIO mentioned that you guys also have GitHub Copilot. GitHub Copilot is a technology which can assist developers in IT space in order to do coding much faster. The kind of efficiency that we have seen in that is almost 30% to 40% improvement in productivity. Things like that can be used on our day-to-day activities. And I think that where the gen AI component comes into place. Even though the answers, you know, you guys all have heard about hallucinations, but in terms of using it for improving your own productivity, I would think of it like an intern. Everybody has an intern at their hands by having something like Copilot so that you can at least fast-track some of the work that you, yourself, are doing.
Thank you. I'm going to get on to my next question, except to tell you that the scenario from India was frightening to me. I don't know how you would fast-track a case of humans with generative AI. Even if you're using the law, you need humans in there.
Oh, absolutely. I think the humans are absolutely needed. I think it's more around how quickly can they go through the cases like just to even read them. The volume is that high. The population is so much more.
Thank you. So can you tell us a little bit -- just want to go back to our basic questions that we were asking the City of Philadelphia. What would a public inventory of AI and algorithmic tools look like? And do any other cities maintain this? And what are the benefits for it and how often would this be updated?
I worked with the Biden administration to put in place some of the guidelines for the AI use case inventory for the federal government, so not at the city level but at the federal level. And I believe you were actually quoting some of the numbers from that earlier when you were describing the number of DHS use cases. I think that that can serve as a model for the city, right. You know, those same procedures are all -- all the procedures that they use to collect that inventory are also all public, and I think that your ability to quote from it exactly demonstrates its usefulness, right. It's important that we know what systems are being used and that we have a public inventory and transparency into the goals of those systems and also the risks.
Very similar response. I would also say that I think you guys initially were discussing the vendor management part of the tools and the technologies in terms of, you know, how do we know what the vendors are using either the data for or even the kind of Gen AI components that they're adding. I think that it definitely needs additional amendments that should be added to the vendor contract because every vendor is exploring this as we go. You know, like the vendors like Adobe was one of them which was quoted. They have added gen AI in their tool sets. Our contracts might not reflect that. So we are going down that policy as well with our compliance committee to make sure that we have amended some of those contracts which have been in the past as we see solutions come by.
And you enter into a contract with a subcontractor, an entity now, and it changes going forward and that's not what you entered into. It's the same as --
At the federal government we were working on procurement guidance that includes in the contract clauses about updating those contracts and requirements for transparency on the part of the vendor, right. And so, I think that the city can and should build in contract clauses that require that type of transparency and require updates and notification as the system changes.
Yeah. How often should that happen for the review of the contracts or to go back through and check it?
So I guess I would say I think that it's important that the city maintains an inventory of the AI systems that are being used that is reasonably up-to-date, I would think certainly at least once a year, right. And that the contract clauses with the companies require that if they're going to have rolled out some new AI component for something, right, that they work with the city on incorporating a contract that oversees that.
Do you know if most cities procure their AI software through their procurement office or through their IT office or through their law department or is it a mix of one or more?
So in some places, cities and states and the federal government obviously all have sort of different structures for these types of things and different levels of staffing, right. I think what's important is to make sure that there is both an expert on the procurement side but also an expert on the technology side in the room 7 to make sure that the right questions are being asked.
Thank you. What kind of audit does it look great for the city? I know we talked about just the contracts, but just the audit of the city infrastructure and the workforce and all of it, what does the best audit practice look like?
So I think that part of what makes understanding what's going on within the city possible, right, is this inventory that you have, I think, rightfully been asking for. I think additionally depending on the system it may make sense to have an outside expert perform an audit. There are some practices that states and the federal government have been thinking about where any time, especially high risk, right, for example, a policing system is procured that relates to AI, that simultaneously they might procure a contract for an external evaluator to make sure that that system is both audited when it is first brought onboard but also in an ongoing manner to make sure that it continues to work for the city.
One entity I didn't talk about with what this was -- and I think the CIO talked about it a bit -- also that an ethics board would be involved as well. Can you just give us your experience on whether or not an ethics board would be involved, whether they are experts in AI and technology? Our ethics boards tell us a lot of things we're supposed to do and not do as employees and we appreciate that, but we don't hear much about AI.
So I think ethics boards are useful, but I think more important is the community input that you all have been rightfully asking for, right. I think that when systems are implemented that directly impact people in our city, right, those people should have a say.
Just imagine a typical city. What are the departments or areas which you think are most at risk for carrying the highest errors for AI, whether it's the police department, you know, Licenses & Inspections, Human Services? And if there's any best practices you can think of how to mitigate those harms?
So I think the way that you asked that question was exactly right. I think that we should be thinking about what are the places where when the AI system gets it wrong, which it will, those errors are the most damaging, right, you know, where are mistakes especially damaging. Certainly, that's policing. That's also education, right. It's also health care. And I think as you -- I guess I would just encourage you again sort of as you think about what are those really high risk cases, think about what happens when the system makes a mistake, right. In the example that I gave you earlier about the VA granting services, right, if the system makes a mistake, then somebody gets services who shouldn't have, that seems like you don't want to make too many of those mistakes because that's going to cost you money. But in the scheme of things, right, somebody getting health benefits when they maybe weren't eligible is a good kind of mistake to make, right. On the other hand, somebody being arrested or surveilled when they maybe shouldn't have been under the planned use of the system, that's a much harsher mistake to make. And in terms of what sort of risk mitigation strategies can be put into place, I would say that this goes hand-in-hand with these questions of transparency and auditing, right. We need to know that the systems work, right. We need to know what the systems are, what they're meant to do, whether they've been tested to see that they actually work at doing those things, whether those tests have occurred actually in our city, right. From speaking to an attorney, I learned about a system that was being rolled out in some of our prisons that had actually been originally created in rural Ohio. You know, so that there might be a mismatch between the types of data that we might expect in our city versus rural Ohio, right. And so, we need to think about has this system actually been created and tested for here. And we also need to think about bias testing, right, as you were bringing up earlier.
I think the use case has to be very thoughtfully designed first, number one. And then I completely agree bias testing is huge. We have to do bias testing and then also human in the loop. You cannot be rolling out any kind of programs without having somebody take a look at the output of some of these gen AI components before somebody is doing any kind of an outreach or a program. You know, at Independence Blue Cross, we do care management. And from for care management perspective, we have used machine learning and AI to see which of our members need an extra call from our nurses. There was an article probably about four or five years ago, it was in the Journal of Science, which spoke about bias, where the bias was based on access of care and people were being outreached who should not. There were people who were not being outreached who should have been outreach too. Things like that happen if people do not carefully design a program. So it's very important that anything that is rolled out has a human in the loop. Yes, it creates a lot more protocol. But at the same time, it's exactly to safeguard citizens as well as the members of the community.
Just want to -- I suppose I'm adding commentary. There'll be a little question at the end of this, but for health care providers, it's essential also -- you talked about HIPAA. There's a law that says that you can't disclose the information of your patients, but we have a federal administration and a president who does not respect HIPAA and is asking for data from health care providers, whether it's for gender-affirming care for patients and their families or other essential services and health care that people needs. We've got a president who doesn't respect that and is saying, give us your data, which is I believe essential for IBX and hospitals to keep safe. And what kind of conversations are you having within for that and how can you imagine our Philadelphia Health Department responding to that as well?
Yes. So I will have to get back to you on that particular piece. I can definitely talk to our compliance department on that. We will make sure that you guys get a response on that. But at the same time, IBX does not share vendor data, our data with any vendors. Other than for providing medical management services to us, there is never -- like we are not selling our data. We are not using our data beyond the HIPAA guidelines. And even when there is an ask from the federal government or even the Pennsylvania Insurance Department, we are making sure that we have people in the government affairs team, they are looking at it from a legal perspective and only with very careful thought. We even decide what the outcome is. But as of what I recall, I do not know this conversation with the federal government, but we'll get back to you for sure.
That's still great. It's that kind of affirmative thought process that we need in this, being able to think ahead and being able to make sure that you know who you want to protect first and baking that in as we bake the cake.
Just want to ask a little bit more about the human interaction in this. We're all scared of this, as you've heard about what's happening next. And you've heard from our city that we're still working on it. They're going to put out a policy. They're going to put out a committee. Talk more about the best practices of what human interaction looks like in this. I just want to make sure that if there are best practices that it's loud and clear for our city of Philadelphia.
Yeah. So I think that it is especially important in these sort of high risk systems that we were talking about, right, where the decision matters, that that be a person making that decision, right. We shouldn't be, you know, denying people services based off of some algorithmic decision. That should always be based off of policy and based off of human review. There may also need to be sort of a human appeals process in place, right, sort of a phone number you can call where an actual person responds, right. We need that both for general communication with the city, but also for response in case the system has gotten it wrong, right. There needs to be some process to correct mistakes that get made.
Very similar answer. But I will say one thing that we as a company -- I'm going to give Independence Blue Cross' perspective -- as a company we have not exposed our members to any of our chat bots and, that is, the technology is moving fast. Use cases are moving fast. As you see, that every single day there's a new technology out there. But you have to be slow and thoughtful and make sure that is only used for the right cases and that's what we have gone with. The technology has been out since 2023. We do not have a single use case as of today where the members, our community members, are interacting with chat bots regarding their health coverage.
Excellent. Thank you. Do any other members have questions? Chair recognizes Councilmember Nic O'Rourke.
I'll just say that I appreciate your testimony and giving us the analogies that will help us to kind of process this and understand it in layman's terms. One of them that stood out to me in particular, I actually prepared questions for, related to your analogy around AI is like a modernization of a factory data center. Now, Madam Chair, as I looked it up this particular hearing is not about data centers. There will be another time where we will go into that in deeper detail. So I'll save those questions. But I did just want to nitpick for a second -- not nitpick, but for my own knowing, thinking about how people understand, how people talk about this. In the previous panel, there was comment made about AI producing new, the term is new content, and I appreciated your point around AI is like the web remixed. Would you challenge the idea that AI produces new content? And, in fact, something or a mash- up of something else?
Yeah, I would. I mean, I think mash-ups can be fun and in music, can feel new, right. So in that sense, sure, it's new. But it's new only in the way that a remix is new.
And I also appreciate the analogy around AI is like a flow chart. The comment that you made was that to handle more complex patterns, AI systems operate like a giant flow chart with the results of computation, like the line of best fit determining which flow chart comes next. This could allow us to make different home value predictions for different neighborhoods, specific computation performed at each step or determined based on training, which is the word for taking data for the past and finding patterns in it. This Council banned I guess late last year, early this year algorithm rental price fixing. Not only our home values, right being addressed, rental prices as well are often see themselves going higher, they're predicting that they should be higher for whatever reason. In a city like Philadelphia, you were looking at almost every other person who rents. Nearly 50% of the city rents. So you're talking about folks who are combating AI as it were, would have before we banned it as it relates to their prices. So again, I just find these to be particularly helpful, relevant and really, really appreciate your commentary around crime and the importance of paying attention to the 2% that you said. That was super helpful for me as well. Yes, it may be 98% well, but 2% can literally mean 31,000 people are impacted by an incorrect output coming from artificial intelligence. That is not a thing to sniff at. That's something that we should be extremely -- I don't like using the word scared -- but we ought to be concerned, and then move to action to make sure that we address it. So I believe that this particular panel has been super helpful towards that end. Thank you, Madam Chair.
Okay. Thank you so much. Really appreciate you and your testimony. Appreciate you being here. (Applause.)
Andrew Jennison and Alli Finn. (Witnesses approached witness table.)
Thank you. Can you state your name and proceed with your testimony.
Andrew Jennison, CGI Technologies. Thank you for the opportunity to testify regarding the governance and responsible deployment of artificial intelligence in the City of Philadelphia. CGI Technologies, founded in 1966 is one of the -- '76, is one of the largest IT and business consulting firms globally with over 93,000 professionals operating across 400 locations. As AI technology has become increasingly embedded in public services, infrastructure and decision-making, it is imperative that we adopt a responsible our AI framework, one that ensures transparency, fairness, privacy and accountability. The CGI 2024 ESG Responsible AI Report outlines a comprehensive approach to our AI that aligns with Philadelphia's values of equity, innovation and sustainability. AI has the power to transform how we deliver services, manage resources and engage with residents. However, without ethical guardrails it can also amplify bias, compromise privacy and erode public trust. Responsible AI is not just a technical necessity. It is an imperative. CGI's RAI framework emphasizes human in the loop, designed to ensure AI supports, not replace human judgment, privacy and security protocols to protect sensitive data, transparency and AI outputs, including citations and source tagging, environmental sustainability, social equity. Philadelphia can leverage responsible AI across multiple domains. AI can help identify early signs of addiction, mental health crisis, social car e needs. AI tools can support law enforcement in identifying patterns preventing crime and protecting vulnerable populations while ensuring fairness and avoiding bias. Tools like CGI's Data Twin 360 and Environment Monitor 360 use AI to stimulate energy use. The New Jersey DEP utilizes CGI's S data platform to provide residents with localized answers on air, water quality, flood risk and permits. Open grid forums and our renewable management systems is used by our utility clients to optimize renewable energy and grid stability. To ensure ethical and effective AI adoption, I respectfully recommend the City of Philadelphia establish a responsible AI Task Force to guide policy and implementation, adopt a citywide RAI framework based on CGI's principles and engage communities in AI decision-making. To build trust and transparency, partner with academia and industry to stay ahead of regulatory and technological developments and prioritize privacy, security and inclusivity in all AI deployments. Thank you very much for your leadership for a smarter and fair future.
Please state your name for the record and proceed with your testimony.
My name is Clarence Okoh, Senior Attorney for Civil Rights and Technology at TechTonic Justice. As we gather, over 92 million poor and working class people in the United States will have some basic human need regulated by artificial intelligence and automated decision-making systems. This represents one of the most consequential transformations and democratic governance in modern American history. The results thus far have been catastrophic. Faulty algorithms have cut lifesaving care for Medicaid services. Students have been placed on secret police databases due to discriminatory predictive policing models. Federal data sets have been exploited to surveil immigrant families and to suppress political dissent. Automated policing technologies are being used to track pregnant people across state lines, and major American cities are experimenting with mass biometric surveillance to track our faces across networks of public and private cameras. This is not an Orwellian fantasy. This is a civil and human rights crisis enabled by the rise of artificial intelligence. And the City of Philadelphia must meet this critical moment and set a new national standard by placing civil and human rights at the foundation of AI and digital policy. As I mentioned, my name is Clarence Okoh. I'm a Senior Attorney at TechTonic Justice and a co-founder of the NOTICE Coalition, No Tech Criminalization In Education. I worked across academia, government and civil society to effectively, successfully challenge the use of artificial intelligence and other emerging technologies to reproduce social and racial inequality. I appreciate the opportunity to testify today, especially at this critically urgent moment. I'll focus the remainder of my remarks on three topics: One of the emerging threats presented by federal authorities that seek to weaponize artificial intelligence against vulnerable communities; number two, the real world consequences of AI injustice in the lives of vulnerable people, including children; and three, recommendations for local policy make makers to protect our community's rights against AI and justice. Beginning with the federal context, it's been mentioned earlier this year, the federal policymakers have essentially reversed course on all key civil and human rights protections as it relates to artificial intelligence. During the Biden administration, we saw some of the first guidance from the Departments of Justice, Education and Health and Human Services as it relates to enforcing civil and human rights standards against emerging technologies. That guidance is no longer in place. There is no federal enforcement of civil rights standards against emerging technologies. And as members of the Council have already noted, it is going to be incumbent upon localities and states to pick up where the federal government has left off. R. 1 the massive federal tax and spending bill enacted earlier this year that's going to cut $1 trillion for Medicaid services, costing as many as million 6 people to lose their health 7 insurance. 8 Key changes to Medicaid 9 and SNAP imposing new work 10 requirements and shorter 11 eligibility periods are going to 12 create new administrative burdens 13 on state and local agencies. That 14 burden is going to create an 15 incentive to adopt and use artificial intelligence and other automated decision-making systems to a much greater degree, and we've already seen the devastating consequences of this in places like Arkansas that have in recent years attempted to use these technologies in ways that have resulted in thousands of people losing their Medicaid services. But to really understand this issue, I think it's important to ground ourselves at what this actually looks like in community settings.
So I would like to briefly share a story from one of the most egregious uses of algorithmic systems to violate individual rights in the United States, a predictive policing program that was used in school district to target vulnerable children and their families. This is a case I know well. I helped lead a local coalition of national organizations and community members to end this program. So in 2021, the Tampa Bay Times uncovered that the Pasco County School District entered into a secret data-sharing agreement with the local sheriff's office. The agreement granted police access to confidential student data, including their grades, their GPAs, credit completion, attendance records and student discipline. These data were combined with other sensitive data, including records of child abuse and whether or not those children have been victims of crime themselves, even information about parental custody disputes were considered by this predictive policing program. " At the program's height, the sheriff's office had developed risk scores for over 18,000 children in this school district representing in students in that community. Students with the highest scores were placed on secret lists that were shared with school-based police officers and those officers were instructed to interrogate these students and capture "actionable intelligence and secret dossiers" that were reviewed by intelligence officers and allegedly shared with state and federal law enforcement. Those students with high scores experienced higher rates of exclusionary discipline, including suspensions, expulsions, school- based arrests and even involuntary psychiatric detention. Students were also subject to a variety of arbitrary home visitations by law enforcement who exploited every opportunity to issue civil citation fines, fees and other code violations. According to one officer who spoke under oath, the predictive policing program was designed to be so intense that families would feel pressure to move or sue the county. Again, another direct quote. So what took place in Pasco County should be a cautionary tale to policymakers, and the lesson is very simple, that government use of AI and algorithmic technologies can easily result in systematic civil and human rights abuses for the most vulnerable members of our community, even our children. To protect our communities from AI injustice, policymakers should, number one, develop critical AI competencies. At a minimum, policymakers must develop those competencies that allow them to be able to identify and assess these technologies for civil and human rights implications. Number two, policymakers need to adopt greater public accountability and oversight measures, as we've already spoken about today. Number three, policymakers must protect basic civil and human rights. You already have the legal authority under existing local, state and federal law, and it's important that you all prioritize enforcement activities for those who are using these technologies in unlawful ways. And furthermore, use that power to engage in agency guidance and use your rule-making authority to clarify how those existing laws apply to AI and digital context. And then most importantly, policymakers must ban and decommission harmful and unlawful AI systems. And I love the suggestion earlier today about banning vendors that are using these technologies in unlawful ways. Policymakers must create simple, accessible procedures for the public, especially for those individuals that are directly impacted by AI injustice. To request that these systems be suspended, reviewed, terminated and/or banned, decommissioning harmful AI is a policy best practice that has not really been adopted in the United States, but it's something that is necessary to safeguard our communities. I want to thank you all again for the opportunity to share this testimony and I look forward to your questions. )
Thank you. Please state your name for the record and proceed with your testimony.
Thank you so much to Chair Landau and the members of the Committee. My name is Alli Finn and I am the Senior Partnerships and Strategy Lead at the AI Now Institute, an independent organization producing policy research on artificial intelligence. We focus on policy strategies to shape AI in the public interest and redirect the industry away from its current path, unbridled commercial surveillance, the concentration of power in a few dominant tech companies and the deployment of AI on the public instead of with and by us. Our work includes advising local, state and federal, federal officials across the country on the reality of the AI industry and best practices in policymaking. We are at a critical inflection point in both the trajectory of AI technology deployment and policymaking attempts to regulate the power structures behind it and the downstream impacts on everyday people. The wealthiest companies on the planet are aggressively selling narratives that AI will transform the world and our lives for the better, as they rapidly accelerate AI use across all sectors of society. In particular, we are repeatedly told that AI can be used to make our governments and public institutions more efficient and effective. But under this logic of efficiency, governments are funneling millions of dollars into private contracts to develop AI technology presumably for our public benefit, with little attention to the well-documented risks. The industry hype around AI obscures that this acceleration is often to the detriment of everyday people. We just heard many of those examples. For one or two more, AI tools are consistently used to devalue workers' labor and increase extreme monitoring and surveillance over their every move. AI's associated infrastructure also increases inequity. Last year alone Pennsylvanian households increased electricity bills paid for over $491 million in transmission costs for the data center build-out, not the wealthy companies driving this expansion. My testimony will cover policies to protect the public and immediate actions that City Council and the Committee can take to limit the most egregious harms. And in considering these areas, I want to highlight the critical role that this Committee holds in not only taking strong and protective action, but in shaping the way that these decisions are made and the potential to lead the way, not only in Philadelphia, but in the state and as a model for jurisdictions across the country. The core conversations around AI in this moment are not questions of computer science expertise, but how social and economic policy is implemented and whose actions these serve -- or and whose these actions serve. I respectfully disagree with the Administration. This room is full of experts on AI. (Applause.)
Communities will always be the most qualified experts in what they need, whether that's workplace protections, high quality education and health care and how their representatives should invest our taxpayer dollars. In questions about how, when and critically if to use AI, we need to keep these fundamental needs as our core focus. And we need to remember that at the core, technology is never, will never be neutral. Algorithms are embedded with the intentions, biases and norms of their creators and AI is never acting on its own. It is being deployed by people and by companies with agendas. I'm skipping my section around what AI is and how to understand the industry, but I'm happy to also speak to the supply chain in questions. I want to briefly highlight several principles that we can follow in policymaking on AI for the public good and to ground this in saying that the stakes are incredibly high. AI must not be used to justify and entrench big tech corporate interests in our public infrastructure while creating a centralized surveillance system that violates Philadelphian's privacy and autonomy. So here are the three principles: First, start with a zero trust policy agenda for AI. Tech companies need to prove every step of the way, including before any implementation, that their product is not causing harm, rather than putting the responsibility of accountability on the people upon whom the tech is deployed. Years of experience show that blanket trust in AI firms benevolence is not a smart, informed or credible option. We need to ensure that the industry that currently creates and profits from AI is not left to regulate and evaluate itself. Second, establish transparency as a critical step in the process, but understand that transparency alone will never be enough and it cannot stop the most egregious uses. We've seen this similarly in surveillance ordinances passed in a number of cities like Boston and New York, where required city reporting on technology and how it's used is often delayed, incomplete and has not at all curtailed the technology's deployment and harms. Third, simply start with the problem, not the perceived solution. Tech companies aggressively pitch products to city agencies, resulting often in immense pressure to adopt AI to be more efficient. This approach obscures and bypasses the fundamental problem supposedly being fixed most of the time and further distances us from addressing the core challenges of limited resources that city agencies face. Instead we need to ask from the beginning what is the problem that we're trying to address, what will help us do that. And if someone is telling us it's AI or another technology, is that the correct intervention. Investing in the foundations of our communities, more schools, public parks and libraries, job training programs and deeply affordable housing is frequently the smarter, more durable investment than the promised quick fix of a tech product. (Applause.)
To conclude, I just want to highlight a few immediate actions that the Council can take, and to emphasize that the push for deregulation -- with the push for deregulation on AI at the federal level, it is local governments such as this one that are on the front lines of protecting constituents. And there are multiple immediate, meaningful actions that the Committee and the Council can take. These are drawn from examples across the country and consultation with community groups in Philadelphia in the room today. First is to protect the public from AI use in high-stakes agency decisions. More than a decade of research has shown how algorithms encode bias, harm low-income communities and meaningfully take away people's opportunities with little recourse. We know that when governments implement algorithmic decision-making systems and benefits administration or ostensibly to reduce fraud, they often discriminate and produce incorrect results with life- altering consequences, as we have heard. The most scrutiny needs to be at the beginning where we evaluate whether AI is the right tool for the job and assess the value proposition of substituting human decisions with AI versus other tools or mechanisms, and where a significant percentage of AI tools are being provided by external vendors, measures can also be taken to reform procurement processes to ensure that other forms of AI use within city agencies do not exacerbate documented harms. And I'm happy to speak to this and follow up as well. Second, we need to pass bright line bans on the most egregious applications of AI. This has been spoken to quite a bit. I want to emphasize that Philly is already a leader here. Thank you to Councilmember O'Rourke with the ban on rental price fixing. We need to prohibit or limit the expansion of power, hungry, thirsty AI data centers. This is intimately linked to what the technology is doing and how it's being deployed, and I can speak to mechanisms here as well. Finally to conclude, in the event that A city agency must purchase AI technology, the city can attach strong conditions onto public funds to ensure that tax dollars work directly for the city and that the purchase technology abides by strong accountability measures and where possible, that agencies are investing in their own tech deployment and people power, rather than big tech vendor lock- in. Transparency and procurement contracting and AI use is essential alongside robust mechanisms for community input, accountability and public control of deployed technologies. There are additional strong actions to take, protecting city jobs, democratizing procurement and protecting residents' most intimate data, and the city can work with community members and advocates to build out a robust suite of protections and ensure that any deployment of technology and AI prioritizes residents over corporate wallets and interests. Thank you. (Applause.)
Thank you very much. Your testimony was so thorough. I actually don't have many questions and we'll open it up to my colleagues, if there's any. Councilman O'Rourke.
I'm mindful of time, but I do really feel like this is one of the most important conversations we should be having right now. I'm going to ask and share what I can share. We've heard from multiple panelists about the dangers of AI reproducing racial biases and social inequities, especially if not given guardrails and including human beings in the loop or I think the term is human loop, somebody said, so I'm learning as we go. We heard from Clarence Okoh about some of the devastating effects it can have on the most vulnerable among us, including children. That testimony was riveting. I'm wondering if any of our panelists can share any specific success stories where AI has been used by local governments to promote equity or access people?
I think the silence is indicative of the answer, which is unfortunately what we've seen too often is -- and Alli was getting at this in their testimony -- is this kind of what I would call a fire-ready-aim approach, where we're kind of identifying the technology first and then figuring out how to use it. And so, that unfortunately has meant that many of the most kind of prominent uses of these technologies, even at times where it's been introduced to try and mitigate bias, has actually done the exact opposite. So I'll give you an example: In five or six years ago with the passage of the First Step Act, one of the provisions of that federal law required the use of an algorithm to basically help determine individuals that would be eligible for parole at the federal level. The algorithm was supposed to introduce greater fairness. Independent evaluations of it found that it actually exacerbated racial bias, particularly against Black folks, and that has been the case across a number of these systems. The story that I shared in Pasco when you actually asked the sheriff's office, they were on the record saying this, what the program was designed to be, they said this was a mentorship program. This was a mentorship program that was supposed to be helping Black and Brown young people, and yet they're being arrested and yet they're being spied upon and yet they're being pushed out the community. So I struggle to think of those examples where there's a kind of equity expansion as a result of artificial intelligence. And quite frankly, it's because that often times is not the intention brought to these technologies. In most instances in which they're being used, particularly in the lives of low-income and working-class people and people of color, these technologies are being used to expand social inequality rather than reduce it.
Wow. Well, I guess it was important to raise that point, huh. Duly noted again. Thank you for your testimony. What you shared today I think this is absolutely essential that we figure out how to get this thing right. Thank you, Madam Chair.
Thank you. I just want to add one quick question here and then we'll move on to Councilman Phillips. When AI gets it wrong in localities, what are the best kind of appeals or remediation programs to right the wrong?
That's a great question and I appreciate that you said when it gets it wrong and not if. I think we need to keep that in mind in any application, that no matter what promises companies make around accuracy rates, it's a slippery slope and bias and discriminatory results are going to be encoded. In terms of remediation, I would encourage us to think not only about after a harm is perpetrated, but before any technology is introduced. There are a number of very strong reports on the robust protective measures that can be taken to reform technology and AI procurement. And those involve deep evaluation, not by companies, not by industry, but by the city, by residents, right, decision-making with people, meaningful decision-making with people before anything is put into place, and making sure that people have access to information about when AI is used, redress mechanisms as well so that you're not only trying to fix harms after the fact, but prevent them, including by not deploying the technology at all, understanding that a significant amount of AI deployment currently is not fit for purpose.
I'll briefly add, so under the Biden administration we saw the Federal Trade Commission really pioneer some really expansionary and creative approaches to accountability for the misuse of AI systems, strategies like algorithmic disgorgement which means that a company that's actually found to be using an algorithm in a harmful way is no 22 longer allowed to use that algorithm. The algorithm is literally taken away and they're prohibited from using it again. There are other strategies like data purges, so we know that there's been an unlawful collection of data actually requiring that that data be purged and that you use strategies like secure deletion to make sure that that data can't be used again. What we've seen as a best practice emerging in states and other localities are ensuring that there are private rights of action for any type of legislation that actually creates new data privacy rights and civil rights to ensure that we can actually get into court to make sure that folks' rights are being protected. And then as I mentioned in my remarks, decommissioning AI. We've seen this as a strategy being used in places like the EU but essentially allowing for individuals, civil society groups or others, that are identifying these harms to be able to actually make requests of city agencies or whoever the authority is in that particular instance to stop using those technologies. But what's really important here to understand is -- one quick point I wanted to make is sometimes accountability can be very difficult, particularly for local policymakers, because there's a lot of interagency interactions that happen, particularly in the law enforcement and schooling context. Almost every example that I've worked on, there's been some type of work around for traditional procurement processes, traditional like processes coming to the City Council asking for funding for these technologies. Things like using funding from philanthropy to be able to procure these technologies. Some policing technologies, for example, do these things called freemiums, where they essentially allow for officers to do a free trial of their product without the kind of larger city being able to know what's happening, and their workarounds also with federal grant programs where agencies will go off on their own initiative to actually apply for these grants and again avoid accountability, and that average disastrous results. So I'll also think about how we can control those interagency interactions as well to make sure that we're having accountability.
And if I can add just one very brief point, a word of caution to the Administration's characterization of AI and different types of AI. It was implied that machine learning is fine and generative AI is what we should have concern about, and that just simply doesn't reflect the reality on the ground. There is no one -- (Applause.)
There is no 11 one good type of AI and one bad type of AI. There are risks in all applications and there are many, many examples. We heard some of them from Clarence of machine learning perpetuating irrevocable harms.
Yeah. I'll just really quickly -- I just want to ask you how can Council require -- no, I want to ask you more about the community structure. We are interested in learning about how do we structure community members and comment so residents help decide an AI system should be used, right? In your experience how do you structure that?
So I can kick us off by saying I think a redundant approach makes the most sense. So as soon as we are considering using this technology, there should be public notice. Once we're selecting vendors, there should be public notice and opportunities for comment. Once the folks have actually been selected, right, putting these kind of redundant opportunities for public engagement is really important. Because at each stage, we have different information that's going to be available to us to be able to determine the impacts of these technologies. And then I think using as many different forms of public engagement and oversight as possible, putting together and bringing in different stakeholders from across the city to be able to offer perspectives from the civil rights community, from the data privacy community, from community organizations, from frontline folks who are working and providing direct services to vulnerable communities, public defenders, legal aid attorneys, so on and so forth. But I think trying to incorporate as many redundancies as possible to allow for there to be robust stakeholder engagement across the life cycle of that AI system is really important.
I fully agree with everything that was shared. We see a lot of examples of symbolic participation, right, and symbolic input, but we also have strong examples of what it would look like to fully engage the public in decision-making. And I just really want to emphasize ensuring that public engagement starts at the outset. I agree with redundancy, but making sure that it starts at the outset of any process before a technological solution is identified. So not, hey, we got this pitch from this company about how to improve health care via AI, but we're trying to address health care needs in the city, what should we do, right, what is your experience. So not starting with the assumption that technology is always our friend or is always the answer, and making sure that these methods for engagement are accessible, right, include multi- lingual resources and language also that I thought Professor Friedler did a great job, right, making the issue accessible to people across a wide range of backgrounds and experiences, building longstanding partnerships with community groups and advocacy organizations that can also help engage people who might not be able to participate in mechanisms that the City Council or city agencies put forth. And then really involve the public in deep assessment, collective evaluation, decision-making and continue decisions about whether the technology is working for people and if it should continue to be used. And in procurement and in public participation, always making sure that the city has the opportunity to stop that use and that deployment. (Applause.)
And to elaborate further -- great job. Consistent re-evaluation, right, are you doing the right things, do you need to reset, do you need to frame re-look at the framework, see if it needs to be updated, compliance, governance processes, et cetera.
Thank you. The Chair recognizes Councilman Nic O'Rourke.
Thank you very much, Madam Chair. The point you just made about constant updates earlier, I think it was suggested at least once a year that that's something, that revision, that sort of re-evaluation should happen. Do you have a particular suggestion about how often a regularity around that should be?
Meeting, engaging right. What is right for the client, what is right internally for our framework and structure. And I think I can't remember who alluded to it earlier. The technology is constantly evolving, so setting a defined date that you have to meet may not be the best course for the city.
And you had mentioned several pieces of things to use to keep things accountable. One them was data purging, as I understand it just means delete data wherever it needs to go. But there was another term that you used, I think it was algorithmic disgorgement. Is that --
So to make sure I'm understanding this correctly, it's basically that the entity can no longer use the algorithm that's been used once it's been deemed harmful in some way or another; is that correct?
That's absolutely correct. So to give you an example, going back to that Pasco example I shared earlier, the idea would be that they are no 13 longer allowed to use that algorithm to make any kind of decision moving forward. So it's not just enough to hold them accountable for the discriminatory effects. It's not just enough to purge the data, but to make sure that they actually cannot use that model again to do what they've done before. Because we've seen a lot of times happen is the company will say, okay, well, this technology produced a disparate impact, right. And so, what we're going to do is fine-tune our model and make our model a little bit better so that it will reduce those disparities. But algorithmic disgorgement says if this model is producing some type of unlawful or unethical outcome or if it's designed in a way that is unlawful or unethical, then that model, that model should not be used moving into the future. So disgorgement is the thing that's been happening for some time. Yet the FTC has been using it under the Biden administration.
Is there a database of like here's all the algorithms that have now been retired out of (inaudible)?
That's a good question. I would have to go and look at kind of the web, the archives of the FTC, on the internet. But as far as I know, I don't think that there's kind of a complete listing of that, but that's something that I'm happy to work with you all to figure that out.
We've just learned that our wonderful stenographer has to be at City Hall by 2 o'clock, so I do want to -- if there's no more questions, we're going to move on to public comment I believe, right. Thank you very much. Your testimony was really wonderful. (Applause.)
So now, we are going to move into the public comment section of our hearing. If everybody -- so in order -- we have a long list of folks for public comment. If you can keep your comment brief, your testimony brief. If it's written, please summarize it. And we're looking at tops two minutes each person in order to conclude on time. Some of the commenters are potentially coming in from the overflow room. Thank you. Wonderful. Great. If you could state your name for the record and please proceed. (Witnesses approached witness table.)
My name is Devin Washington. I am the Organizing Director at People's Tech and one of the founders of the Philly Tech Justice Collective, and I appreciate the conversation and the hearing that we've had today. We see this as the first step that the city is taking and a lot of our movements are taking in ensuring that the way in which technology is used is accountable to the communities who are directly impacted. And what we've been seeing over the past couple of years where we've been doing this project is that some of the biggest voices in how technology is deployed in the city are the billionaires and corporations who are invested in making sure that they get access to our public dollars, right. Their priority is not necessarily the common good, but enriching their own pockets and their shareholders. Meanwhile the machine learning, the AI that the city is currently using is empowering a network of 50,000 surveillance cameras in some of the poorest areas of the city, under the guise of this is what makes our community safer. When we know that investment in our community, investment in our schools, investment in our rec centers, making sure that folks have access to a social safety net that actually treats them as a human being is what makes our community safer, right. So instead what we have is 50,000 surveillance cameras owned by the city, SEPTA, the universities that are watching our communities be poor. The promise of jobs brought by technology from data centers in other parts of the state, et cetera, that never actually pan out for everyday working-class Philadelphians, I can't tell you how many friends I have who have gotten degrees in computer science who can only get an IT job, cannot get a robotics job. And I can guarantee you in places where there are data centers, they are not getting those jobs either because they are not vehicles for long-term job growth for cities, for the communities that they're in, in addition to the fact that the data centers themselves are deeply impacting the communities they exist in there. You know, we've talked about it already where utilities bills are going up, air quality is being harmed, and Black, Brown and working-class communities are left holding the bag for infrastructure that billionaires and big tech are enriching themselves off of. So for those of us at Philly Tech Justice, we are going to be laser-focused going forward and ensuring that not just generative AI and machine learning, but all forms of tech that the city plans on using, that these corporations are trying to sell to our government, are accountable to our communities because we are fighting for a vision of community ownership and accountability of how technology is used. Thank you. (Applause.)
Oh, it do. My name is Shawmar Pitts and Co-managing Director at Philly Thrive and I represent the marginalized community that has been spoken about a little bit up here today. And I just feel like I try to be as nice as I can in my descent, but I really -- because I don't know much about AI and things like that, right, I'm learning about it. But just from this hearing, from the beginning to right now from everybody that testified and from my past experience in dealing with our legislators, man, I feel like we about to get played. Man, I feel like we about to get played. I swear I feel it because when the Administration came up and gave this is what AI is, this is how we can use it, and we going to do and we going to do, and then when the questions come, they say, you know, the ones that I was paying attention to, that I can understand, the ones that's right to the point, we got to get back to you on that, we got to get with the police on that and all that kind of stuff. I just feel like we get ready -- they trying to play us. This is the stuff that they do. So now, I'm only left to ask the people who care about the people that I represent in South Philadelphia and at-large the poor people, right. Don't let them play us, man. Don't let them play us. (Applause.)
Because we know when they talk about data centers, where they gonna want to build it at? They gonna want to build it right next to my house. That's where they gonna want it. They gonna want to build it right there. We know that. We not silly. We know when the adverse effects come, right, my people gonna be in that 2%. 99% of the 2% is gonna be my people. We the ones that's going to have the adverse effects of AI. So please I just ask don't let them play us. Don't let them play us. And, you know, like my brother just said right here, we going to be laser-focused and we going to be on it and we going to bring the information and we just want everybody to act in the best interest of the people of this city, because a lot of times that's not what's being considered when the Administration brings these things. That's not the interest that they're representing. They representing the big tech companies and the billionaires and all of them. And these are the people that they are representing when they come and they bring things, they just don't say. But that's what they doing because they don't represent me. They don't represent me. And they don't represent my people when they come with the H.O.M.E. plan. That H.O.M.E. plan, that's not representing my people. When they come with all these other things, that's not representing my people. And this is just more of it. I just feel it and I see it and I know I ain't going crazy. They try to make me think I'm going crazy, but it just happens over and over again. So I know it's not me, you know. So we just need someone to stand up for us. Because when they want to build a data center, they want to raise the electric bills and they want to mess up the water quality and they want to mess up the air quality, they don't want to do it in my neighborhood. That's just been the way it's been done for the past 150 years so they not going to stop. So I'm asking our Councilmembers, you know, the ones that represent us, you know, don't let them do this to us. Don't let them do it. Thank you.
Thank you. Can you please state your name for the record and proceed with your testimony.
My name is Rodney Gardner. I am a member with YEAH Philly and also an instructor at Slash (inaudible) with the Energy Collaborative. We run a solar agriculture program where youth and young adults can come and learn about the importance of environmental justice, how to properly care for the environment and the benefits of urban farming. We also teach how solar energy works and how to install solar panels. And lastly, we discuss air pollution, its causes and its harmful effects on the environment. My involvement in this work, I have learned that AI consumes a vast amount of electricity, which contributes to the burning of fossil fuels and results in air pollution. This pollution releases nitrogen oxide, sulfur dioxide and carbon monoxide into the atmosphere. Air pollution is detrimental to plants, animals and of course humans, and it is one of the leading causes of lung cancer, heart disease and asthma in minority communities. AI is more harmful than it is helpful and we must put an end to it now. (Applause.)
Good afternoon. My name is Jordan Holbert. Councilmembers, I want to say thank you for the opportunity to speak. I'm a coach, a consultant, a community advocate who believes in building systems that help people and communities grow together. Today I want to challenge all of us to be bold and brave enough to make decisions, not just for the convenience of the future, but for the well-being of the future us. Only a handful of tools have had the impact rivaling the influence artificial intelligence is already having on humanity. The printing press made written word accessible. The wheel made people and objects transportable. The Internet made the world interconnected. Each of these innovations brought people closer together. If we're not careful, AI's impact could divide us beyond repair. Right now AI is still in its infancy. Yet we already have examples of it defying human instruction and behaving based on bias, a bias that like most in America impacts Black and Brown people before the implications spread through the rest of the nation. No other tool in history has shared the ability to "create something new without a human." That means we must be strategic, intentional and guided not by fear or greed, but by values of health, humanity and fairness. We must also avoid making emotional decisions in the name of safety. Applying AI to police our streets or to surveil our neighborhoods may sound like safety, but it risks creating deeper distrust and unintended harm. Amazon's Ring cameras already accessible to law enforcement, offer a glimpse of this reality, while emotionally Some believe this increases safety. Logically and historically, we know that more policing does not necessarily create safer communities. Implementing AI won't change that. It will only magnify the injustices. We know that education predicts outcomes, so how we educate our children, our workforce and our communities about AI will determine who benefits from it and who's left behind. I'm here today understanding the opportunities of AI and as someone who believes in its power as a technological tool, we are at an inflection point however because AI can help us build a more efficient, creative and connected future, but that future must be equitable, transparent and sustainable. If we lead with humanity now, we can ensure AI becomes a tool that unites us rather than replaces us or divides us.
Hello. My name is (inaudible) Albert. I'm a proud Philadelphia resident. I live in District 2. I'm also a nationally recognized attorney and scholar whose work focuses on algorithmic justice and more important questions of how technology affects people's lives. I submitted comments which were mostly about trade secret and transparency which you could read, but I wanted to just take the time that I have to actually echo my colleague Shawmar Pitts and respond to the first panel that we heard where -- what I heard from city officials as an expert in the space who has been working on AI for a very long time, was embarrassing. It was the suggestion that it was time to form a committee and develop a policy with no timeline, no commitment to who would be on it, and a long list of cities, some of which bear very little resemblance to Philadelphia, to look to as examples suggest that this issue is seen as new, novel and sort of suddenly something that landed on the lap of our city officials. Those claims are wildly incorrect and deeply concerning as someone who wants our city to actually develop a policy that it can implement with the things it's already doing, rather than with this idea that generative AI is something new that we now need to worry about because city officials might use ChatGPT to be more productive at work. So I want to really just echo this idea that you have some tremendous experts who came and spoke before you in this room in the city of Philadelphia and that they have the expertise and the knowledge to help this Council make decisions and the city make decisions, and that I hope that whatever committee is formed will use that expertise rather than starting from scratch. And maybe producing something like some voluntary guidelines in like five years. Thank you. (Applause.)
Thank you very much and thanks for submitting your testimony as well. Mr. Serra.
Hi, I'm Olivia. And I'm here to give testimony speaking as an educator and an advocate for the children of Philadelphia. So my first year of teaching was 2019-2020 so during the pandemic. And it was also the first year that my school had one-to-one technology with students. So that kind of worked out well when the pandemic happened. But since then, I've seen this constant push for more tech, more innovation in our schools, more subscriptions to these programs that are going to improve outcomes and raise test scores and all of these things. And the reality that I've seen as a teacher is more students watching anime on their Chromebooks in class, and more students who are so distracted to the point where they can't even regulate their emotions when you redirect them during class, because their attention spans are short because of algorithms that have been designed by these tech companies to, you know, take as much of our attention as possible. So I just speak to that to say that the decision to invest in technology in terms of the education sector was made under a lot of pressure and this, like, COVID, is happening, we need to move in this direction. No matter what it will be helpful in this situation. We have time to stop and wait and put limitations on the way that AI is being used while we figure out how and if it could be beneficial for the citizens of Philadelphia. So I just encourage you to think about what innovation actually means. Because from what I've heard of all the testimony today, innovation moving towards more technology and AI inclusion is going to make a worse society for us all to live in when we know that investing in real things, buildings, parks, recreation centers, places where students can go and gather, we already know the answers to the questions that we are trying to use technology to solve. We know how to stop crime. It's to provide other alternatives. We know how to -- I don't know. I just encourage us to use our brains and think also looking at history. If AI is looking through all of this, like historic data and what happened, we can also do that. We can make innovation be wow, like, Philly is the city where we actually have dialogues like this and we include the perspective of the public, and the public helps shape policy, not AI. Thank you.
Thanks. Mr. Serra, can you call the next few public testifiers.
Thank you. Can you please state your name for the record and proceed with your testimony.
Absolutely. My name is Sierra Freeman. I am more or less, right, the cultural organizer and strategist for the Brewerytown Sharswood Neighborhood Coalition. But more importantly than all of that, I am a team member of BSNC and a neighbor of the 29th Ward that sits within the 5th District of the city of Philadelphia. Irresponsible tech and AI usage separates us from our neighbors, from our humanity, our health and wellness and our future's well-being. Instead of neighbor-created and led town watches, we had Ring cameras directly accessible to the police, now accessible to them if you fill out their online form. PPD wants to expand its usage of drones and make them first responders in some instances and hire more pilots directly from high school. Automation of tasks, automated decision-making and direct access to minds of our youngest adult neighbors isn't something we need in the hands of AI law enforcement. It takes humanization, which is the exact opposite of AI, to make decisions at any given moment when people's lives and livelihoods are at stake. Neighbors taught to be fearful of each other with biased AI detecting threats, watching from behind technology instead of standing on their stoops getting to know each other. There's no more watching out for each other as neighbors. There's no more watching out for our children as a community. We're just watching with integrations like Alexa and Echo, feeding us back our words and phrases with products for us to buy, keeping our already financially vulnerable neighbors in an endless capitalist consumer salute. We're paying with our children. The Stanford School of Medicine's Brainstorm Lab for Mental Health Innovation partnered with Common Sense Media to create test accounts simulating 14-year- olds in order to evaluate how AI companions from three chat bot developers interact with young users. Their findings revealed that with minimal prompting, chatbot from Character AI Nome and Replika engaged in conversations that could be harmful to mental health. Nonprofits or quasi government institutions given access to us by way of contracts voted on and implemented by city administration with their mundane usage of AI for notes, scheduling, info, synthesizing and many other things. These things that may not be and most likely wasn't listed in any clauses and/or use or contracts with our city or our neighbors directly, while as of October 2025 Pennsylvania still does not have a comprehensive consumer data privacy law. We are grateful that in October 2024 Philadelphia passed a ban on the use of algorithmic rent setting software by landlords addressing AI use that was harmful. Thank you again, Councilman O'Rourke. We're not happy that AI is still a major player in raising the cost of everyday living. More access to AI has meant proposing data centers that are harmful to the environment being built in our backyard. And we don't just pay for them with our health and wellness. We're charged a premium on our PECO bill to add another financial hardship that subtracts years from our lives, not to mention surge pricing driven by algorithmic AI and facial recognition at supermarkets like Walmart and Kroger. As BSNC is a part of the Philly Tech Justice Coalition, we're paying attention as a collective deeply rooted in community and we're watching. As Brewerytown Sharswood Neighborhood Coalition entrusted to be an advocate for our neighbors in this time, we're doing what we can in our neighborhood with and for our neighbors. It starts here. In all of what I listed, they're only the things that we know so far. We're demanding transparency, accountability, oversight and community collaboration. Philadelphia is a city of neighborhoods. Our neighbors are the lifeblood of this city. Responsible AI oversight means a duty to preserve and protect our neighbors humanity. Thank you. (Applause.)
My name is Thomas Noah. You can probably tell from my outfit that I'm an academic. So I'm a lecturer for the School of Social Policy and Practice over at Penn, teaching a course right now, Democratizing Data? Critical Data Studies in Algorithmic Governance. So I want to reiterate the idea that there are no experts on this subject matter is an absurdity, because I assume that Penn is not instrumentally irrational in paying my salary. I'm not here to speak on behalf of Penn. I do not speak on behalf of Penn. I do not speak on behalf of the School of Social Policy and Practice. I speak as a community member who has a house in Point Breeze at the intersection of Pierce and Point Breeze Avenue, right across from the mosque. And just in keeping with the theme from my class, I hope we take seriously what my brother is saying here, that we appreciate these efforts to try to democratize data, but it can't be Kabuki democracy. It has to listen to the community members. It has to serve their interests and their needs and the attempts by the speakers from the Admin to continue to shield certain aspects of their behavior and practice don't lend themselves to democracy and community control and community-led efforts. People in the community know what they need. My sister right here knows what her community members need. There's a real danger in social, economic, political elites like us coming in and telling other people how they should live their lives, doing interventions, putting technologies on people that they don't understand, that they haven't consented to, that they haven't authorized. And so, I just ask that, you know, going forward with what you intend on doing, we remain mindful about this and we take our commitment to democracy seriously because it's under attack domestically and internationally, and we have the opportunity to be the change that we hope to see. Thank you.
Yes. Good afternoon. My name is Miguel Willis. First, thank you for having us. I'm also a lecturer at UPenn as evidenced for my outfit, and the Director of the Responsible Computing for Just Futures Initiative, which aims to equip the next generation of technologists, lawyers and policymakers to think critically about how technology is designed, deployed and governed, so that it promotes equity rather than entrench inequality. My words are my own today and not the official position of the university. I'm speaking today as a resident, a new father, as someone who cares deeply about the future of the city.
You know, I don't need to reiterate what's been said already in terms of what we're seeing amongst the growing adoption of AI across all systems and data- driven tools across city agencies. We heard comments and comments on how the lack of accountability, public transparency oversight and how these tools risk replicating the same structural inequities that we've been working so hard to dismantle. But two critical areas I want to highlight: First is the secondary use of open data that the city collects for one purpose, but it's later repurposed by private AI firms or shady data brokers for another, right. When residents consent to their collection of their data, for example, through 311 complaints, transit cards or school enrollment systems, as someone testified earlier it's often with the understanding that this data serves a specific purpose. But shady data brokers routinely retain, aggregate and repurpose this data indefinitely, using it to train predictive algorithms that the public never consented to and cannot meaningfully challenge. This practice erodes the foundation of public trust, undermines informed consent. So we want to confirm -- curb the secondary use of Philadelphia's -- to help ensure and protect Philadelphia's privacy, assured this data is not weaponized against the very communities that it's meant to serve. And equally important, and which has been reiterated, is the need for transparency and community accountability of how AI is designed, procured and deployed. Historically, decisions about what technologies to build have been made by experts like myself, city officials, academics, researchers, without significant input from the communities most impacted, again most impacted, those are the experts by these technologies. To rectify this, it's essential that we elevate community voices and ensure they are central in the decision-making process through participatory design and community-led processes where residents, community organizations and advocates are central and involved before the technologies are purchased, designed, thought of. Many of the groups here today like the Philly Tech Coalition, 215 People's Alliance and all the other community advocates are already modeling what this looks like, grounding conversations about AI and lived experience in equity and in democratic governance. The city should invest and learn from these efforts, rather than outsourcing decision- making to unidentified governance systems or opaque corporate systems. Ultimately, the question is not whether Philly will use AI, but who will it ultimately serve and how will that be humane. So I'll close by saying let's just make Philly a model and not replicate the mistakes from the past. So thank you all. (Applause.)
Good afternoon. Thank you for doing this. I'm not entirely sure how my name ended up on the list, but that being the case I don't have prepared remarks. I came to listen, but I'll speak to the issue. So my name is Karl Morris. I am a computer science professor at Temple University and I showed up today because I was of course deeply interested in understanding what the city government was doing with regards to its use of artificial intelligence, both in terms of its own operations and how it intends to use it to disseminate its services to the residents of Philadelphia. I think we need to contextualize so many things in this conversation, but we can start by looking at the history, understanding the demographics of this city, recognizing how different groups historically have been -- how they've been treated overall, whether it's how they've been ignored and put aside when we look at things like the digital divide or access to high speed internet, whether we look at the ways that technology has harmed certain communities, very recent examples, for example, during COVID where we saw things like pulse oxometers giving poor readings or incorrect readings for folks with darker skin, leading to deaths and harm or as we see again, just looking at emerging technologies generally. We see self-driving cars are now in the city and we recognize that while these vehicles have lower accident rates than vehicles driven by humans. When they do have accidents, they tend to be or they tend to involve folks again with darker skin. So we know that technologies, emerging technologies, can in fact harm certain communities while helping others and ultimately, their use can benefit one group while exploiting other groups. We heard earlier testimony about, for example, when data centers are being built or the power plants that operate them where are they placed. It's often times in communities where folks are, whether they're low on the socioeconomic ladder, whether they are racial minorities, are ultimately people who don't have a voice. They're not able to speak up. So what we need to ensure is that the Council in providing oversight in the operations of the city, they consider these things, they look at the historical context, they look at the groups that tend to be served and groups that tend to be harmed and ensure that with input from residents, with input from stakeholders and the community, we ensure that while, yes, the city will utilize and benefit from artificial intelligence on the whole, we want to make sure that no 9 one is left behind and no group is exploited. So thank you. (Applause.)
Thank you. Mr. Serra, is there anyone else for public comment?
Last speaker. Thank you very much. Please state your name for the record and proceed with your testimony.
Hello and thank you so much for allowing this forum to take place. My name is Roselle Kenney and I'm 67 years old. Picture this: A 72-year-old woman finished her shopping at 23rd and Walnut Street. She walks outside and suddenly has no idea where she parked. minutes later, she 13 was still wandering and embarrassed 14 and anxious. Finally, she has to 15 call her daughter. But here's the 16 thing, her phone could have found 17 her car in 10 seconds through AI, 18 whether she had Bluetooth, Google 19 Maps, Google Play, but she didn't 20 know, or how about this: We've all experienced this if you have parents. A senior leaves their doctor's office clutching a medical report. He sits down to read it. It might as well been written in Mandarin, hypertension, glucose levels, contraindications. And when the doctor did explain it, it sounded like the teacher on Charlie Brown, womp, womp, womp, womp. He's too embarrassed to call back, but AI, AI could translate that report into plain English in seconds. Answer his questions as many times as he needs, no 13 judgment. This happens every day in Philadelphia. Not because seniors can't learn, because nobody thought of including them. And I noticed in this whole entire thing no one mentioned senior citizens. Now, let me be clear, can seniors learn AI? Can we master new technology? Of course, we can. We lived through rotary phones, ATMs, cassette tapes, 8 track tapes, VCRs, personal computers, cell phones and Uber. We adapted to every technology revolution for centuries. So don't tell me we can't handle AI. We just need a seat at the table. Philadelphia has over 250,000 residents over 60 years old. That's 1 in 6 people. When we plan for AI without them, we're not just leaving people behind. We're leaving our parents, our grandparents, our neighbors, the people who actually built Philadelphia. And here's what's at stake: AI is already changing health care, banking, transportation, government services. Seniors who don't understand these tools lose access. They get isolated. They fall for scams. But when we do this right, amazing things can happen. Tools like sit-and-pee apps help seniors to monitor their health, prevent infections, find a bathroom. AI can read prescription labels out loud, find bus routes, provide companionships. One of the biggest problems with seniors now is loneliness. It can detect falls, call for immediate help. So here's what I'm asking: One, put senior representatives on your AI planning committee; two, fund training programs and libraries and seniors and senior centers; three, require that AI tools are designed with older adults in mind, not as an -- I hate to say here -- afterthought; four, protect seniors from AI-powered scams. Our elders built Philadelphia. They deserve to benefit from this technology, not to be left behind by it. Let's make this city a model for how we bring every generation into the future together. We do have wisdom. Thank you.
Thank you very much. Thank you for stepping up. We really appreciated your testimony. Okay. Seeing there are no additional people for --
Okay. Thank you. Please state your name for the record and proceed with your testimony.
Sure. My name is Vincent Perez. I was raised in an underserved community. Although I was given an opportunity in high school and that led to a career in investment banking, later becoming an entrepreneur, strategic advisor and have been a professor for over 20 years. I'm also a founder of Longview Partners where I select who I want to help, what are venture institutions, I designed a strategy and increase their impact, public, private, academia. I've taught business and innovation for over years, most recently at 8 Penn State where I serve on the 9 Teaching and Learning with 10 Technology Faculty Advisory 11 Committee and I champion Invent 12 Penn State's AI initiatives. 13 I'm also part of the 14 Philadelphia AI Strike Force and 15 the Vice-chair of the Robotics 16 Coalition here in Philadelphia, 17 we're expanding AI and robotics 18 access for students from cradle to 19 career. There are no true experts 20 in AI right now. The field is evolving faster than anyone can imagine. But those of us on the Strike Force, we're on the cutting edge across academia, industry, government. We just had a meeting last night with two dozen folks and we're making sure that we want to keep an eye on Philadelphia and make us a hub of AI and robotics. With our talent, university, civic partnerships, Philadelphia can become a natural model for responsible and an inclusive AI innovation that is built on trust, opportunity and growth. Thank you for your time.
Is there anybody -- thank you very much. Appreciate all of your comments. Anybody else here for public comment? (No response.)
Hearing none, that concludes our hearing. I want to take just one last second to tell people again I'm Councilmember Rue Landau. I am the Chair of the Technology and Information Services Committee and I deeply appreciate everybody who came to speak to us today. It is more clear than ever to me and to my colleagues that we have all the experts we need right here in Philadelphia, and we need to tap you to make sure that you are providing your wisdom, your insight in this process as we move forward. So this will not be -- we are going to have a series of AI hearings through this Committee and look forward to hearing your input going forward as well. Thank you. (Applause.)
I have a paragraph I have to read everyone. One second. While I'm looking for my official paragraph, I will tell you that this is the first time Philadelphia has ever had a hearing on something like AI. We haven't been using our Technology Committee very much in the past year. So really look forward to opening it up to all of the new and pressing concerns as we go forward. This concludes the public hearing of the Committee of Technology and Information Systems. This committee will stand in recess until the call of the Chair. Thank you all very much. (Applause.) (Committee on Technology and Information Services concluded at 1:19 p.m.) C E R T I F I C A T I O N I, hereby certify that the proceedings and evidence noted are contained fully and accurately in the stenographic notes taken by me in the foregoing matter, and that this is a correct transcript of the same. __________________________________ TANEHA CARROLL