COUNCIL OF THE CITY OF PHILADELPHIA COMMITTEE ON TECHNOLOGY AND INFORMATION SERVICES PhillyCAM 699 Ranstead Street Philadelphia, Pennsylvania Wednesday, October 15, 2025 10:15 a.m. PRESENT: COUNCILWOMAN RUE LANDAU, CHAIR COUNCILMAN MARK SQUILLA, VICE-CHAIR COUNCILMAN JIM HARRITY COUNCILMAN NICOLAS O'ROURKE COUNCILMAN ANTHONY PHILLIPS
Thank you all so much for being here. I now note that the hour has come. Mr. Serra, will you please call the roll to take attendance for the Committee on Technology and Information Services.
Thank you. A quorum of the committee is present and this hearing is now called to order. This is the public hearing of the Committee on Technology and Information Services regarding Resolution No. 240759. Mr. Serra, will you please read the title of the bill.
Authorizing the Committee on Technology and Information Services to hold hearings exploring the current status of artificial intelligence, artificial intelligence technologies and how Philadelphia can prepare for their impact on the public, private and nonprofit sectors.
Thank you again. Thank you all so much for being here. This is the first-of-its-kind hearing for Philadelphia. I'd like to acknowledge the presence of our colleague Dr. Anthony Phillips. So a few opening remarks for now. I want to thank by -- I want to begin by thanking my colleagues, advocates, academics, experts and members of the public who are joining us here today. I also want to thank PhillyCAM for hosting us in their space and broadcasting this. This hearing is for the public. For those of you don't know, PhillyCAM is the public access radio and TV station here in Philadelphia. And also as the Council President always has the opportunity to appoint us to many boards throughout the city, he appointed me to the Board of PhillyCAM, and I've been so proud to get to know them even better and so happy to represent them and so appreciative of you hosting us here today. Thank you also to the Parker Administration for joining us today to provide additional insight. I think we can all agree that artificial intelligence is becoming the new normal. AI allows computers to perform tasks that mimic human intelligence, enabling computers to learn from existing data and apply that to generate new content, seemingly overnight. AI is no longer an idea of the Sci-Fi future, but a part of our everyday lives. It is integrated into our search engines, our word processing software, our social media and even our phones. Whether you're talking to an AI chat bot when setting up your Wi-Fi or issued a traffic ticket based on AI-powered enforcement or using a generative AI app to create content for your social media, it's changing the status quo and bringing us into a new technological age. It's being used for writing, creating images and sound, analyzing data, recognizing images, and we don't always know when our peers or even when we ourselves are using it. I'll be the first to admit I'm not a tech expert. I don't have a background in software engineering. I don't know how to code and I am not always on the cutting edge of the apps that are available. As a City Councilmember and a longtime advocate for tenants, low-income people and the civil rights of marginalized communities, I'm holding this hearing today so that all of us can learn and make sure we're protecting people when implementing these new tools. We cannot repeat harms of the past with new technology. Throughout this hearing, we will be speaking with leaders in the tech and innovation industries, advocates for digital equity and tech justice and academics who have a deep background of research in emerging technology. We're going to explore the potential for technological innovation to improve the operations of our city. We will learn how upgrades to the systems already in place can help us achieve what I know. We all want from our government a modern, responsive, efficient city that can provide services to everyone. Today, we'll examine the current staffing and infrastructure to see if we garnered the expertise that will equip Philadelphia to meet modern challenges with modern solutions. As we hear from experts and advocates today about the possibilities, we will also hear some of the risks that we face as a city if AI is implemented without guardrails. Adopting new technology is guaranteed to impact not just our jobs but our economy, our labor market, our environment and our future. We're here to learn what AI is good at and what it isn't and we are here to get clarity on what AI is currently being used for by our city government and what it isn't. Transparency is essential, and that's why I'm holding this hearing today. City Council and the public deserve clarity on what the City's policies are for using AI and the potential guardrails we need to have in place. This hearing is setting the stage for an important conversation about data privacy, racial justice, tech equity and technology policy that includes those affected by the decisions that our government makes.
Philadelphia needs to show that we are committed to making sure that as we consider adopting and using emerging technology, our budget and our practices continue to align with our values. We need to remain centered on fairness, transparency, protecting the rights of our people and maintaining public trust. This is a forum for learning what's out there and what's working and what's still to come. As we have this conversation today and in the future hearings that I plan to hold on this and other related technology topics, I'm focused on learning how the City of Philadelphia plans to protect the best interests of the public as they improve their use of technology. I look forward to hearing from our guests today. Thank you all for joining. Any other members of the committee have any additional opening remarks? )
Mr. Serra, will you please call the first panel we have to testify this morning on Resolution No. 240759.
Kristin Bray, Chief Legal Counsel to the Mayor and Director of Philly Stat 360 and Melissa Scott, Director Office of Innovation and Technology. (Witnesses approached witness table.)
Good morning. Please state your names for the record and proceed with your testimony.
Good morning, Chairperson Landau and members of the Committee. My name is Kristin Bray and I serve as the Chief Legal Counsel to Mayor Parker and the Director of Philly Stat 360.
Melissa Scott, Chief Information Officer for the Office of Innovation and Technology for the City of Philadelphia.
Thank you. We're here to testify today on Resolution No. 240759. We both appreciate the opportunity to testify today before on the Committee on Technology and Information Services on how the City of Philadelphia is developing best practices for emerging technologies like artificial intelligence, AI. We're living in a remarkable moment. For the first time in history, an emerging and powerful technology tool, known as artificial intelligence, is available to anyone with a computer and internet access. It is a tool that can be used by anybody, not just computer scientists or major corporations. This powerful new technology opens numerous possibilities for a city like ours. At the same time, like any tool we use, we must use it responsibly and with full transparency to the public. Used well, AI tools can help the Administration serve Philadelphians better, deliver services more efficiently and make our neighborhoods safer. For example, artificial intelligence can help us by analyzing millions of data points to identify problems in our infrastructure before those problems become emergencies. It can streamline services so residents spend less time waiting for help and can have access to the very services they need in mere seconds at any time of the day. It can even help us identify patterns and data that will help us deliver city services more efficiently and to those communities that have the greatest needs. But those powerful tools must be used with human judgment. They must be used with an unceasing commitment to telling the public how we are using those tools transparently and they must be used accountably with a lens towards equity, constitutional legal frameworks and fairness. Powerful tools must be used responsibly or we risk deepening inequalities and weakening trust with the public. Nevertheless, to give Philadelphians the strong and responsive government they deserve, we must use these tools wisely. That's why we're building a framework for how we will use these emerging technologies like artificial intelligence guided by three principles: human-centered judgment, public trust and accountability and using smart tools responsibly. Just because the tools are new does not mean that the values that guide us every day should change. That is why our first principle in using these tools is centered on human judgment. Philadelphia's greatest strength has always been its people, the public servants who know this city, understand its residents and who bring compassion and expertise to their work every day. They are our best asset. Likewise, our residents and the people who work and play here bring their lived life experience to help make our vibrant community stronger. That's why our approach to artificial intelligence begins with this simple truth, technology should support human judgment, not replace it. Our people's experience, empathy and common sense cannot be automated. Artificial intelligence can help public employees do their jobs better but will never do their jobs for them. Likewise, artificial intelligence can help our residents access services quickly, but it will never replace the human interaction between city employees and the people they serve every day. Artificial intelligence is here to help enhance the Mayor's vision, to give Philadelphians a government they can see, touch and feel. Of course, powerful tools come with real risk. Artificial intelligence can amplify bias if it is not designed or used carefully. Automated systems can make mistakes that will then affect real people's lives. That's why the second principle that guides our framework is rooted in transparency, fairness and accountability, ensuring every use of artificial intelligence is responsible, explainable to the public, ethical and within constitutional and other legal parameters. In the coming weeks, the City will be releasing guidelines for all city employees on how to use artificial intelligence appropriately and responsibly.
These guidelines will help staff understand both the opportunities and limits of this technology so that every use reflects our ongoing commitment to fairness, trust, accountability and transparency. In addition to these guidelines, we will also be providing city employees with training on how to use generative artificial intelligence tools responsibly and ethically. Looking ahead in late winter or early spring, we will establish a cross- departmental artificial intelligence government's committee to build on this foundation. Its role will be to continuously review and refine our policies, track evolving technology, work with experts, ensure that the city's use of artificial intelligence remains aligned with our values, our legal obligations and the expectations of our residents and employees. This is how we lead with both innovation and accountability. Our aim is not to slow progress, but to guide it. Artificial intelligence, used responsibly, can make government more efficient, more transparent and more human-centered. In closing, artificial intelligence is a tool to help us serve Philadelphians better. It is not a replacement for our people. That is why our work is grounded in the three principles that will shape every decision we make: human judgment, public trust and accountability and using smart tools responsibly. Our people are the heart of this government and the city. Our continued responsibility is to make sure that the tools we use make this city stronger, position for an innovative future and give our public employees every opportunity to make Philadelphia the safest, cleanest, greenest city in America with economic opportunity for all. Thank you. We remain available for questions.
Thank you both very much for being here, for your testimony. We appreciate this partnership with you. I have some questions and then I'll pass it on to my colleagues. You testified a lot to kind of the parameters and the guiding principles that you have for AI when in use with the city of Philadelphia, but what I didn't hear about was how it's being used now. So there's two different issues that we see. One is policy development and the other is AI use. And as you know, I requested -- I had some questions for you and requested detailed information for your testimony before today. And what I've heard more about is policy development, not the current status of how AI is used today. So can we start with, has the City done an inventory of how and -- there we go. Has the City done inventory on how and where AI is being used across all city departments and agencies?
Yes, Chair. Melissa Scott, CIO. First, I would like to state that we have the same terminology. There are -- AI is not a new term. We always had machine learning versus generative AI. So I just want to give us some definitions so we understand, because I have two different lists of AI products and I want to make sure we understand what we're talking about here. So for decades, the public and private industries have used AI, which is called the machine learning version, for decades behind the scenes, to classify, predict and to detect patterns, forecasting demand, spotting fraud, routing 311 requests, flagging anomalies within the cyber security logs and improving search results. These systems are narrow. They're task- specific and trained on historical data sets to optimize a defined output. However, generative AI is different. It creates new content, on-demand text, code, images audio-based on a user's prompt. It relies on large scale models and is trained on bass corpora to produce plausible answers or drafts. And so, basically in short the machine learning predicts or classifies within a known task. However, generative AI produces new content and accelerates knowledge, work, summaries, drafts, code snippets, mock-ups and scenarios if it's used within guardrails. Okay. So what I also have here is a list of our generative AI tools, our SaaS solutions, which SaaS means that they're on the internet or they're a spy subscription. So I'll go through the list and explain to you what they're used for. First, we have GitHub Copilot uses, Open AI Code, text to generate code suggestions and completions. Microsoft Copilot, Microsoft 365 Copilot, which integrates generative AI in Word, Excel, Outlook, et cetera. Adobe Creative Cloud includes Firefly generative AI for image generation or editing. We also have Qualtrics, which uses generative AI for summarizing and analyzing open text responses. SalesForce and the license includes Einstein GPT. It supports generative AI for CRM tasks. Tableau enabled uses generative AI for natural languages, queries and insights. Service Now includes generative AI for ticket summarization and virtual agent responses. Zoom, it offers generative AI for meeting summaries, chat replies and email drafting. Myrow, Canva and Figma, these tools include generative design and context solutions. This is what most of our software engineers use. And Notion AI, ClickUp, Monday, if licensed, these platforms include generative AI for task automation and context generation. I also would like to add many of our vendors they claim to offer AI capabilities, but the term AI can refer to a wide range of functionalities which I just been explained earlier, which may vary significantly in scope and sophistication. So before I go any further, if you would like me to now discuss the machine learning tools, I can go through that list as well.
Okay. Splunk, it includes our anomaly detection and prediction analysis. So this is not generative AI. This is the machine learning. Rapid7 Insight uses AI for vulnerability management and application security. Abnormal Security is an AI-based email threat detection. SOCRadar, AI-driven threat intelligence and attack surface monitoring. This is all for cyber security. RunZero, Cyber Asset Attack Surface Management with AI driven-discovery. ServiceNow includes AI for items, automation and analysis. Articulate 360 includes AI tools for e-learning content creation. Relish Data Assure, AI based on validation and assurance for workday. Relish Data Assure for Workday, AI-based validation for assurance for workday -- sorry, I repeated that. CARTO uses AI for geo analysis and predictive modeling. SmartRecruiters, AI for candidate matching and recruitment automation, which is not currently in use, but it is available; and Vizio, an AI-driven performance management and strategic planning, Better Stack, AI power monitoring and incident response; and Penlink Cognitive Services include speech to text, text translation. I would also like to add again that a lot of the vendors include AI functionality, but the function needs to be turned on and we have to go through Project Health Check and a set of requirements and for departments to utilize them.
So let me -- I have some other members who have some questions. Well, let me follow up with a few questions on this. To start, are there currently any guidelines for City of Philadelphia employees on how they can use these products, especially the generative AI ones?
So in 2023, there was a use for generative AI published by the previous administration. Currently, we are working, as Kristin mentioned, on a governance group and we're currently working on a policy. She also mentioned, late early spring we will be publishing guidelines for city employees.
And I would like to add to that, that we're working on these guidelines to create a plain language document for every employee to understand and also for the public to understand how we are using this because it's not just about how our employees are using the technology, but it's also important to the Parker Administration that the public knows how we are using these technologies and the rules under which we are operating.
I want to dig down for a second into the vendors in particular. It's a little further in the questioning, but I'm very concerned that we will engage in contracts with vendors who may have AI products now or may include additional AI products in the future and how do we protect ourselves with contracting language or enforcement of what they're using our data for and the oversight and tracking that we can do to make sure that we know what they're doing and what they change. We've seen in national reports -- so around the country there's entities that are coming into cities with just license plate reader technology, but they're using that technology and sending the information to ICE. Is this something that the City of Philadelphia would enter into? And if not, how would we safeguard against that?
So I'll answer the first part of the question. So we currently have a process Project Health Check where any type of software when a department wants to procure a software, any solution, they would need to go through this process, whether it was AI or not. Currently, there are 6 questions that the particular 7 department would have to answer in 8 regards if there is an AI tool, our 9 team actually ensures that AI must 10 align with the city's data privacy 11 and accessibility record management 12 requirements to ensures 13 transparency, accountability and compliance with public records. So we work closely with law integrity and ensuring that any procured software goes through this process before we actually procure it, and this is a process that's been standing prior to generative AI becoming popular.
What if it changes while in the course of your contract? What oversight do you have with a company that says they want to do one thing -- what consistent oversight do you have to make sure that they are not changing the terms of what they're using with your data in the contract?
So I will add that part of our process -- what we're currently looking for right now we're looking at Copilot as the ability -- as to be the approved solution for -- and I'm just giving you this as an example. And we did a risk analysis. So we do a risk analysis on a particular software. And what we look at is how we can continue to generate reports on what the software is doing behind the scenes, who's using it, what are their prompts. So particularly with Copilot, we looked at chat reports. We pull out monthly monitoring, activity prompts, sensitive and appropriate data. We actually research that and pull these reports on a monthly basis in the background with or without the users knowledge, right. So it's really with the users knowledge, because this is within the contract but we ensure that these particular softwares have these capabilities. We also look for catch errors, language around output errors, removing biases, training users to detect incorrect information and outputs to ensure the review is done before utilizing the data. And we also do -- the chat gives warning prompts as well to reinforce training. We've also identified this as the top risk before utilizing the data. So we go through a series of design, security checks, risk assessments of the particular tool and research what can we put on the back end to continue the monitoring.
I'd like to add, Councilmember, as we're thinking about our AI Governance Committee, one of the things that we're thinking about is this is an emerging and evolving technology. So we can't develop a policy and put it on a shelf and congratulate ourselves on a job well done. We need to continuously evaluate that policy. As vendors, the public and our employees are using this new and emerging technology so that we're responsible or responsive to what is emerging, how vendors are using this, how the city is using this so that we are always aligning our actions with our values and the constitutional frameworks that we fought long and hard for and are still in place today. So we want to make sure that we're constantly evaluating what we're doing, that we're always aligned with our values.
So I want to get back to the inventory. Do you know which departments are using what or what employees and what departments?
My understanding is that -- I want to go back to who we is. You say we keep evaluating. Who's we?
Yes. From a security standpoint, yes, our tools allow us to evaluate and produce reports.
But are you -- is OIT the entity that's reviewing the vendor contracts and doing the spot checks of the vendors' work?
Law Integrity, our Office of Human Resources, our Chief Administrative Officer. We're making this a multi-faceted project because we know technology touches on every facet of city government.
And have you been doing this up until now or this will be the new --
My next question: My understanding is that, OIT does not necessarily have control over every technology unit in every department. So for example, I think the Department of Recreation, for example, has their own technology department. If they made a decision on a product, would that have to go through you as OIT? Do you oversee what Recreation does or any of the other departments with their independent technology units or do can they make those decisions?
Each department is different so it depends on the department. But with your example with Parks & Rec, yes, it would have to go through OIT.
What are the other departments that have their own technology department in the City of Philadelphia?
Yes. And I would also like to know if any of them make their own decisions outside of OIT? I'd also like to add now that we would like -- once we secure an oversight of some degree, at the oversight is there for every city department and agency and subcontractor, which is the next set of questions, to make sure that we have the same standards that we're using for everyone? Because if not, we won't have any control of this. With the subcontractors of the vendors now, do we do a review or have an understanding of what AI they use?
And that is with every department and agency in the city?
As I mentioned before, just the consolidated ones, as basically you pointed out. So I would need to get a list to you of the departments that are deconsolidated so you would have the separation between what OIT has governance over versus not.
Let me get a little closer to the microphone it seems. I'm going to ask a few more and then pass it on. There are many examples of use cases around the country of how cities are using AI. What would you and the City of Philadelphia see as an acceptable use case in Philadelphia, a way in which we would use it?
Sure. So my understanding is we use AI with Streets to make sure that it's evaluating potholes and obstructions in the road. We use AI with the Parking Authority for tickets. What do you as the City of Philadelphia see as acceptable use cases where the door is opening fast, jurisdictions around the country again are using facial recognition with the police, they're using license plate readers and sending the information to ICE? What are the boundaries that you see now for how we would use it? We could potentially use it in the City of Philadelphia and in what ways?
Part of what the committee is looking at and taking a look at for purposes of the guidelines and the ultimate guidance that we are going to issue out to the city in the late winter, early spring, what are those acceptable use cases that we can have? We are of course looking at what other cities are doing. Some cities have very strong AI policies. Some cities do not have strong AI policies. We want to be a leader in this field and we want to make sure that any policies that we're developing are taking into account our values and those constitutional parameters that we're well aware of in this city.
Who's going to be in the -- what's the makeup of the committee? Who is going to be on there? And are there people outside of the City of Philadelphia, experts who would be on there?
I think that's something that we're discussing about how we can bring in those outside expert voices into the committee. But currently we're working with OIT, the Law Department, OHR and Chief Administrative Officer.
Who do we see in the City of Philadelphia, who are our AI experts? No, it's not me. So who in the office of OIT or otherwise are experts in AI?
Interesting question, because if anyone in the world would say that they were an AI expert, they would be lying. Because it is continuously transforming. It's morphing into new technology. So you can be an expert on where AI is today, but three months from now it can be in another way. But I would say the Office of Innovation and Technology, we're most versed on AI products and working closely to continue to research and continue with vendors like Gardner and other professionals to continue to learn from one another. We also go to a lot of conferences. We talk to a lot of other professionals within other counties and cities just to educate ourselves and understand more about the evolvingness of the technologies of AI.
Thank you. I just want to ask just -- I have so many questions for you two. I'm going to ask one more and pass it along. But a very big concern of mine are people's privacy and ways in which AI can be used, whether it's for racial discrimination, whether it is for presumed identifying people who might be from other countries living here. I really want to dig down into the safeguards and the guardrails that you're going to do. I've heard a lot about the law. We know what the law is. This technology is moving very fast. We have a U.S. Supreme Court who doesn't seem very set on enforcing the current laws that we have. So I think it's really up to us to make sure that we're putting in the guardrails to make sure that we're not going to go years in the past of perpetuating discrimination, perpetuating inequities and making sure that we're taking care of people throughout Philadelphia. Can you just comment on that for a minute in which you think you can make that happen? I'll pass along to my colleagues and then come back.
We understand those concerns. And both Chief Information Officer Scott and I have been with the city for a very, very long time. We understand how we have gotten here and we understand the battles of the past. While we're developing this policy, we are going to be looking at those values and ensuring that we're incorporating those values into that policy. The policy is not written. We intend to have it by late winter, early spring, but those values are forefront in our mind about privacy and the Fourth Amendment. That's why we're bringing the Law Department into this committee because we believe that their privacy attorneys play a big role in this. We believe that their civil rights attorneys play a big role in this because we want to spot those issues and address them as best we can. That's also why we are intending to have this policy reviewed often and regularly to make sure as the technology evolves, we also evolve.
Thank you. I would like to pass it along to my colleague Councilmember Isaiah Thomas first.
Good morning. Thank you, Madam Chair, for your leadership as it relates to this particular issue. Clearly, this is an important conversation. You can see that by how many people are convened here. Thank you to the Administration for being here and for doing your due diligence as it relates to this particular issue. Just a couple questions in the midst of listening to Madam Chair. I didn't hear -- and I did hear Madam Chair talk about it towards the end right here, but I didn't hear much around what we're doing with AI and public safety. So I did hear a little bit around the Parking Authority, but I want to do a deeper dive into the police department. And so, when I think about the technology that was used around -- well, let's just ask: What type of AI is being used specifically to deal with and collect evidence when a hit-and-run is taking place?
I think we'd have to take that question back to the police department.
Okay. It's really important that we know that. In 2020, we've seen a record number of hit-and-runs. This legislative body decided to put money in the Managing Director's Office to give the Admin new dollars for new technology to be able to collect evidence of hit-and-runs. And it was agreed upon that those dollars will be spent. And I'm wondering if AI is a part of the new technology that's being used to collect evidence with hit-and-runs. Madam Chair talked about the license readers. Is the police department using AI or anything like that dealing with traffic stops, motor vehicle violations and things of that capacity?
So I can actually speak to the license plate reader. So to our knowledge, there are no AI capabilities currently being used for license plate readers. So I know that from where we are now, because I was actually involved as it was being implemented. But some of the things that you're mentioning is previous to my tenure and we don't have any knowledge currently documented or anything, so we would have to refer back to the police for additional information but we can get that back to the Chair, if that's okay.
It would just be good to confirm that we're not doing that, to invade people's privacy. I think that's the whole point of some of the work that we've tried to do. What about the speed cameras? We've been strong advocates for speed cameras, especially in school zones. And so, I'm wondering,the speed cameras that we're using to address some of those issues, what type of AI do they have in them because a lot of people are concerned about those speed cameras being used to do something else besides catch people for speeding? So can you communicate to the general public what type of technology that we use as it relates to speed cameras? I know Broad Street is something that a lot of folks are talking about right now. But I'm really excited about the collaboration we have with the Administration around putting speed cameras in school zones.
So I think the confusion could be the difference between machine learning and generative AI. So what I would like to do is go back to the police department and provide a list for you because it could be AI, but machine learning, not generative --
Can you explain to the general public what that is though so that folks can understand the difference, so that people can understand what it is that we're trying to do specifically with speeding, hit-and-run, issues like that?
So as far as hit-and-runs, and there's no 3 generative AI solutions to my knowledge today, but what I'm saying is that if there is an AI portion, it could be you're referring to machine learning AI, which as I mentioned, it predicts and classifies within a specific scope versus generative AI can actually accelerate and identify and go beyond the scope, the guardrails that you put in place. So we would actually need to get that back to you and we could get the answer back to the Chair. But again, I would like to state on the record that we don't have any knowledge of any generative AI being used within our speed cameras.
Follow-up: What company are you using for the license plate readers?
That's a good question. I don't -- let me see if I have that answer. One second.
This comes back to my control, once we enter into a contract with them.
Final two questions: Is there another area that AI might be being used that's specifically related to public safety, surveilling people, constituents across the city of Philadelphia? Is there something that might have slipped through the cracks that we didn't ask where we are using AI specifically around issues related to public safety?
Currently, we're not aware of any generative AI being used within public safety. So as far as slipping through the cracks, I don't believe that that's actually possible with our current process but we can definitely get a response --
No, I mean questions that we didn't ask thus far. I know we've asked some questions so far, but I'm wondering if there's an area where AI is being used, specifically in a realm of public safety that we haven't asked about so far today --
Okay. Amazing. Last question: I'm the Chair of the Education Committee so I got to take a minute to at least ask you is there anything in the working with the School District, the Mayor's Department of Education, to examine how AI is being involved in the academic journeys of our young people? And I'm not talking about teaching them how to use AI. I'm saying AI being used for things like tutoring, teaching and things of that capacity. Is there any plans or is there any vision, has there been any coordination or dialogue with the School District to assess how individual institutions are using AI, not teaching it, right, using AI as a supplement for a tutor, a teacher, a physician? I don't know, anything.
Okay. I'll close out with saying that's very important, right. And I think when I listen to our Chair talk about how AI is evolving, you know, last school year a teacher might have never thought to use AI to supplement some type of curriculum or lesson. And this summer, they might have had an amazing journey or a bad journey, I don't know, and they might have came back this school year looking to use AI. And I think part of the concern is that, number one, it doesn't always give accurate information. And I think another part of the concern is that you tend to because it's so personable now, you could develop a dependency, and we know a lot of our young people just basic human growth and development, could be facing mental issues, emotional, social issues, all types of trauma, and we don't want these things giving bad advice that create unintended consequences. So I think that's a really good homework assignment. I'll be looking forward to working with our Chair to figure out exactly how we move forward with that in the future. Thank you, Madam Chair. I'm done. Thank you.
Madam Chair, I do have an answer to your question about the license plate readers. The vendor is Johnson Controls, Inc, JCI.
Thank you. Next, we'll move on to Councilman Jimmy Harrity.
Thank you, Chair. I'm not the most tech savvy guy around as other people, but I'm learning. Whenever we talk about this kind of technology, I automatically go to the movie Minority Report. Everybody remember that, you know what I'm saying. This kind of technology really scares me. It's no secret that people in Philadelphia do not feel safe, even though they say crime is going down. The feeling is not safety. That being said, I do not want to give up my freedoms for the sake of somebody else's idea of safety. I would be curious to know, as we said, what we're using this technology for. I know all technology there are some good things that come of it. But for me, you know, movies have a funny way of predicting the future. I believe this is a slippery slope that we're going down right now and that we need to be very careful because, you know, there are consequences to the things we do now down the line. So technology may not be a way right now for them to do the kind of things that we don't want them to do, but technology grows, as you said yourself, and things evolve. I just want to make sure that we have a control of this, that we have people watching this, not just people that are in these departments that may actually have a stake in what is going on there, but regular citizens on the oversight of this stuff. Because again, I'm very leery of this kind of technology, and maybe that's because of my lack of technical skills when it comes to that. But for me, I'm definitely not giving up my freedom for a little suspected added safety. And, you know, I'm a safety guy. You know, I'm for policing. I'm a pro-police Democrat. You know, I want our citizens to be safe. But again, I don't want our citizens to feel like they're being watched all over the place. And this has already been bought up. This isn't -- I'm not bringing this up and it hasn't already been out there. The ACLU already has lawsuits in New York and other places where these technologies have gone, the drones too. We haven't really implemented a drone program now because we're still waiting to see what's going on in New York with the ACLU and their findings from the court cases. So I just want to make sure that we're aware. I also want us to be fiscally responsible because this stuff does lead to lawsuits and we can't afford that either in the City of Philadelphia. But some more of a little tech, more technical, the Administration consulted with city departments and quasi city aid entities about their upcoming guidelines for the city employees. So you're already talking about that. That was when your statement, right? When can we see some results from that? When will the guidelines be available? And then on top of that, will the community input be taken into regard across departmental AI governance? You know, when we're forming a committee and having recommendations, I believe that the citizens definitely need a say in that.
With regard to who is in the committee, we're still discussing who is going to be part of that, but we are definitely listening to the voices that we hear in the community here in Philadelphia, most especially. But we also look to other jurisdictions and what their experiences have been. We want to learn from that. We want to grow from that and we want to be a leader in this field to the extent that we can be.
We've all seen before when government tells us that they're going to take into account what the citizens say. That's a lot different than actually having a citizen with say. That's all I'm going to say. Thank you.
Thank you, Madam Chair. Real quick, and I know we mentioned about the concerns of using AI and how it's being used, but I think another concern is what are we doing now with the data that's being captured from that information? So even though we have the technology's being used there, obviously right now we have data and what is being used with that data because guidelines will be written in the future, but we have that data now of some things that we're using. So what do we do with that data? And how is that data being either shared with somebody else or used?
So part of the governance committee is to discuss just that, how to ensure that we keep the data secured. That is actually one of the major concerns of the generative AI tools that you can currently utilize online. We are very anxious to have a policy in place to ensure that city employees are not putting city data into those tools because it goes out into an ecosystem that's not protected. So we need to -- we are, as I mentioned earlier, pushing to have Copilot as our official tool where city employees can be able to use it to enhance their work. But what's most important about that is it is in a controlled ecosystem where it doesn't go out and then our data is then used back against us.
Oh, our current data is secured through our current structure as city government. But for AI, you're speaking specifically about AI. We don't currently have any AI tools, generative AI tools, that we procure that would allow our data to go elsewhere.
So we're not using that at this time. And so, we will be able to protect that in the future so that nobody else would have access to that?
Yes, but what we should also keep in mind is training city employees on how to use generative AI tools. There are a lot of tools that they can use and access on their own. So I have a list of them. So you have your ChatGPTs, your Copilots, your open AIs, your Gemini, those tools you could just go on right now and utilize that and upload city data in there on your own. So we are in the process, as Kristin mentioned, creating training for city employees so they understand that they too have a part in assisting us in protecting our data.
Yeah, but that is best behavior, right. So I can't go with you to your house and stop you from going to -- say, you're working on something for the city and you go home and you're working on it from home. I can't stop you from uploading that information into Gemini and then it goes out into the ecosystem of the world. We have to train you and then hold individuals accountable as far as letting them know that this is against the rules. You're not allowed to upload city data into these online tools. So that's a different conversation that we do need to have.
(Inaudible). Our Chief Information Officer points out we can control the environment here within the city and our city devices. If an employee goes home, they might not be aware, which is why we want to release guidelines and training while we work through the more complicated, complex issues that this panel has identified, like the privacy issues, the constitutional issues. They need -- we don't want to rush to put out a policy that doesn't take into account all those important, complicated issues.
But I think at the end of the day it's what are the consequences for somebody providing that information or sharing that information out and what challenges that presents to the city or other individuals, right. So I think that we really have to look at that and understanding how that data is protected and how we as a city, whether it's education or training but then also guidelines on, you know, what are those concerns and what happens if people share that. I mean, you have disgruntled employees that go out there and feed information out. We have other folks that could give information to AI and then compromise things that the city is doing as a whole. So again, technology is where it is and where it's going, but I think that's why we're so happy for the Chair to be able to bring this hearing too because these things have to happen now, like we can't wait further. And I know we're looking at other municipalities and what they're doing, but we have to have certain guidelines in place and then also rules and regulations on what we do with our captured data, right. We always hear about people selling information and using information. Not that I'm saying -- I'm not saying the city's doing that. I'm just saying. If you want to make people feel confident, and a lot of people, like I said, aren't even confident in the use, but then they're really not confident not knowing what happens to that data once we have it. So I think that is an important aspect of where we go from here. Thank you.
Thank you. Next up is Councilmember Anthony Phillips.
Thank you. I just want to thank the Chairwoman for her work on this. Thank you, Chairwoman Landau. This is an incredible hearing. I just want to say also thank you to the Administration for coming out today. I know this is a challenging thing to talk about. It's like the time when the world was introduced to credit cards and debit cards and cash, amongst some other things. This is a big structural change. I do want to say that I understand the public's sentiment around this as well because as you all probably understand, you don't want to lose jobs to AI systems. And so, that's why this is I guess a larger conversation issue. I just want to ask is there a way that you all -- have you thought about ways that Council can work closely with the Administration to create like a structure so that when residents do have these comments, we can find a way to facilitate them the best way we know how to assure that this was a collective reasoning around our initial piloting when it comes to AI. Is that something that's being thought about, just something that's helpful?
Yes, indeed. That is something that we're in the process of discussing of how we can collaborate with both partners in the community, with our partners at City Council and with our partners in other municipalities. You heard Chief Information Officer Scott talk about how we partner with other governments to see -- other municipal governments to see how they are working their policies and what they have encountered. But, yes, that is absolutely something that we are considering in this committee to make sure that we're getting voices at the table to receive that feedback.
No, I actually appreciate it because that was one of my other questions. I wanted to make sure that we were focused on it. There's other municipalities that have done this well or going through this community engagement process and that would be fairly helpful. The last question I just had is that you spoke about public trust, Kristin, about a government Philadelphia can see, touch and feel. Again, I just want to talk about ways in which when you come up with your plans for the winter, if there is a way in which you can have a public engagement plan around that. Of course, a lot of times the Administration make decisions that are in the best interest of belief because people have put you in the office to do the work. But this is, I think, a heavier change, such a big change that the public is probably going to want a lot more details about how this is working out. So, yeah, there is a level of public engagement, plans, that you could put together that be helpful. But thank you all for answering my other questions. I think we're all good to go. Thank you.
Thank you, Madam Chair. I appreciate your commitment to centering human judgment as well as public trust and accountability as you mentioned in your opening statements, which is particularly important to me because you cannot, as some folks have said earlier, I think automate certain sensitivities which is another reason why I think valuing the spaces where our ethics and morals are shaped is of paramount importance. I'm saying that to you as well as the listening public, in times like these making sure that we are prioritizing that sort of discernment, that sort of human sensitivity, those sorts of ethics. Because as someone taught me years ago, bad data in is bad data out. (Applause.)
And so, it's particularly important. But I think that's on top of also localities recognizing, as I think was also mentioned, our particular importance. We can't rely, and I felt this way -- at least it's been a growing feeling -- I felt for some time, but it's kind of come to afore, dealing with the whole SEPTA thing. We can't rely any more on big brother, state, federal government, what have you. We have to look out for ourselves. We got to protect us. And so, I think all this is particularly important. I want to look into and look forward to reviewing your guidelines as well as the findings around the AI Governance Committee on that. But just aside from the human judgment piece, I did not hear or I wanted to hear more about public trust and accountability, how you will ensure public trust and accountability as we move forward with AI in the city going forward.
So we're still discussing that and that's going to be coming out with our policy in the late winter, early spring. But one of the things that we really want to take a look at are those public voices. We will be listening to the additional committees and panels that testified before this committee today. We are looking at what has happened in other cities and we are listening to, as councilmember Phillips pointed out, some community engagement and Councilman Harrity pointed out community engagement. It is important here and we're going to take that back to our committee about how we can best incorporate those practices into our policy as we're rolling this out and moving this forward.
Both of you have mentioned looking at other cities and other states, other counties and what have you. Could you name actually which are the counties and cities that we are looking at as it relates to their AI policies?
Yeah, just curious as to who Philadelphia was looking at as it relates to our pattern or learning from as it relates to that policy?
Washington, D.C.; Seattle, Washington; Bellevue, Washington; San Jose, California; Spokane, Washington; San Francisco, California; New York, New York; Long Beach, California, Boston; Fort Worth, Texas; Salt Lake City, Utah and Baltimore, Maryland.
Got you. And I'm assuming they all have -- some of them I know have lengthy tech histories, but some of the other ones that are less known, they have proven AI policies, they have policies that people are saying we should look at or are we just kind of -- how are we deciding what cities are good to look at and what's not?
So we have evaluated them as far as the maturity level when it comes to the AI policy. So particularly the two that are very high, Washington, D.C., they have a comprehensive AI governance policy. Only IT-approved tools allowed, requires security review, document approval. And I'll share with you the next one, Bellevue which has a strong procurement governance, approved AI list, mandatory new tech request process and prohibits unapproved AI use. So those two are the highest. And then it goes from medium high to medium to medium low. We scored them.
Okay. Very good. Earlier you gave a list of some of the programs that were used, making distinctions between generative and machine learning. I think one of the programs that the city is using is Workday, correct?
So Workday is part of our ERP. So it's our Opal project which is currently still in development. So we mentioned it because it's procured, right, but it has not been implemented as yet.
Okay. Well, that's actually a good thing because it's been reported to me that Workday is known to embed bias into its programming. And so, it makes me ask the question what vetting processes do we have to ensure that we are not procuring programs that actually are furthering bias within our governance structure?
So as I mentioned before, we have a process that every department or that if consolidated will go through when they're procuring tools, particular softwares. But what we need to make sure is that they are -- if there is an AI portion and we're continually monitoring and producing reports looking at if any AI portion of those tools have been turned on, to assume that everyone won't just be as forthcoming, right. So, for instance, in your particular example even though Workday has not been implemented, us evaluating the current product and looking at what those AI tools are and then in the background producing reports to ensure if any of those have been turned on without our knowledge. We would flag those and address it with the particular department.
It sounds like we're kind of building things as we're going along. Maybe I'm mishearing that. But is there intentionality that's going to be implemented here to make sure that bias in particular is a thing that is not being perpetuated? Are we thinking along those lines about making sure that particular issue is being addressed as it relates to any particular program that we're using?
Absolutely. In part of the governance group of the AI governance group, we will continue to ensure that what we have to be conscious of here is that AI is continually evolving, right. So what biases of today and what biases there are tomorrow. They'll change. So as I mentioned before, as IT professionals it does sound like it's an evermoving wheel, because it is. Just as we implement, we have to then improve, implement, improve, because the technology is moving and emerging that fast. But the governance body will continue to be in place and we will continue to evaluate as new products come in, as the vendors see -- what happens is the vendors can create an AI tool and then put it available in the tool without anyone's knowledge. So the question is what are we going to do to ensure that we are running reports that we are evaluating in the background as these companies change their models without our knowledge. So that's a very important aspect. So if it sounds like it's ever continuing moving, because that's what the industry -- is currently going on in the industry and we have to stay up and ready at all times.
It sounds like we almost could compile a list of bad actor corporations or whatever that might help us as opposed to waiting a downstream look at the product. Maybe if there are or companies that are saying, okay, we know that they've done this before, they've slipped something in there, what have you, perhaps we can begin to compile a list of those sorts of companies and say, well, we can't trust you on the front end so we're going to not work with you as a place you're trying to get those sort of programs. But I think that's something we can address down the line, but I just want to suggest that it's something that we look out for as well, not just the programs but the companies themselves. Are you a bad actor, do you -- or maybe that's a bad phrase. Are you someone who's not trustworthy as it relates to providing a product, artificial intelligence, that we can ensure doesn't have certain little things put into it. But I thank you for being here today, presenting today. I understand that it is an ongoing rolling issue, but I am of the opinion that as it relates to what governing looks like now and going forward, we've got to figure out this situation around AI algorithms. It is literally shaping reality. And so, we got to get our arms around. And as I said earlier, we can't trust other places to do it anymore. I really feel like localities, municipalities, we got to beef up and really hold our own. So thank you again, Madam Chair.
Thank you. One, I wanted to follow up on just two things and then also wanted to say that a couple of our members have other obligations and are going to have to step out. But the first, I'm not sure if I heard you correctly but you talked about public safety, and I thought I heard you say that you're not using AI in the realm of public safety here in Philadelphia now?
To our knowledge, we're not. We would have to refer this to the police department. If there's anything else that's unknown, as you mentioned before, there is some form of deconsolidation, especially in regards to OIT. So to our knowledge, no. And I'm saying for me to our knowledge and my team's knowledge, no. 8
We already know that the city is using facial recognition software, ShotSpotter and also drones, and that is where my concern comes in about we know -- even in a hearing at City Council, talked about the police department's purchase of these drones through the outside Police Foundation, which goes outside of our budget process, which I was concerned about. We learned this week that the Trump Administration is launching a $500 million effort to help cities where World Cup and other 2026 events are happening to protect against drones in their airspace. Private military drones are often equipped with AI in their surveillance technology. Is the Parker Administration contemplating collecting, collaborating with the Trump Administration on this anti- drone initiative?
I think we can get back to you on that question. That was probably best directed to the police department.
Also used by AI of the federal government to further Trump's harmful agenda is a serious concern for all of us. For example, the U.S. Department of Homeland Security, DHS reported nearly 200 uses of AI across its departments, including approximately 20 active use cases by ICE, Immigration & Customs Enforcement. The use of AI to turbo charge, immigration enforcement is a serious and very present concern. Is the Parker Administration worried about that happening in our city. And will the Parker Administration also in any way collaborate with DHS and ICE?
Finally, with the makeup of your committee to oversee AI, I know we asked you already who would participate in that. Do you expect to include Labor, the city workforce, who both could be penalized for doing the wrong thing and/or have their job disappear because of AI, will Labor be at the table?
We do have our Office of Human Resources in our current committee format and that's part of the things that we are considering is how does this impact labor ultimately, any policy that we have.
Do you have any plan yet on how often this committee will meet?
Not yet, but we're going to meet routinely and regularly because the technology is evolving.
Follow-up to one of my colleague's questions, will you make those meetings public or will you allow an avenue for public input, especially because what we'll hear today is some pretty great experts on this subject that can really add to what we're doing here in Philadelphia?
Okay. Thank you. The Chair recognizes Councilmember O'ROURKE.
It's already kind of come through, but I want to get the exact wording on the record and get a response. As these technologies progress, the city will be collecting more and more data, especially from automated systems like the speed cameras and the $2 million new drone program that was already mentioned. How are we ensuring that this data isn't being used by outside law enforcement like ICE and other federal officials without a signed judicial warrant? (Applause.)
So I'm not an expert in policing and we'll get back to you on that, Councilmember.
Thank you so much. Thank you for being here. Thank you for staying I believe throughout the rest of the hearing or at least hopefully watching?
We will be watching and staff members are here too just in case anything comes up.
Thank you so much. We really appreciate you being here. Mr. Serra, will you please call the second panel.
Sorelle Friedler and Ravi Chawla. (Witnesses approached witness table.)
Good morning. Thank you so much for being here. Can you state your name please and proceed with your testimony.
Sure. Thank you, Councilmembers and Madam Chair. My name is Ravi Chawla. I'm the Vice-President and Chief Analytics Officer at Independence Blue Cross. Okay. Independence Blue Cross appreciates the opportunity to speak before the Philadelphia City Council Committee on Technology and Information Systems to share insights on the responsible use and transformative potential of artificial intelligence in our daily operations. As a health insurer serving the Philadelphia region for over 485 years, IBX has been on the forefront of AI innovation, using it to improve member experience, streamline operations and enhance health outcomes. We offer this testimony to support the city's exploration of AI technologies and to provide a framework for understanding how AI can be responsibly and effectively integrated into municipal services. AI, as you guys all know, is a multi-disciplinary field that combines computer science, mathematics, linguistics and other technologies to automate tasks, generate insights and support decision-making from chat bots and predictive analytics to generative AI tools. Its applications are vast and growing. At Independence Blue Cross, AI is used to predict health complications and reduce emergency room visits, conduct preliminary reviews of member data, to identify behavioral health needs, enhance customer service through chat tools and personalized wellness plans and support administrative tasks like drafting communications and presentations. These examples illustrate how AI can be used to improve service delivery, reduce cost and increase efficiency, goals that align with our mission to improve health care in the community. Independence AI work is guided by our internal Center of Excellence on AI and AI Ethics Committee which include experts in law, compliance and technology. We follow six core principles: accountability, compliance, privacy equity, reliability, safety and security. We also adhere to national frameworks such as those from the National Institute of Standards and Technology and participate in work groups that align with best practices. Importantly, human oversights remain central to our AI processes, especially in sensitive areas like clinical decision-making. This balance between automation and human judgment is critical for public trust and ethical governance. I would love to give an example. Consider the case of John Doe, an IBX member with congestive heart failure. After a routine check, his health data is updated in our system. Using AI, a nurse, a health coach, can identify potential risks and connects John with resources to prevent hospitalization. This proactive approach can lead to almost a 38% reduction in ER visits and 43% reduction in hospital admissions through timely interventions. These have been published, if you guys -- we would be happy to provide those papers. Cost savings of improved health outcomes and enhanced quality of life can really lead to cost savings of almost $716 per member over a six-month period, reflecting more efficient care, not reduced access. This model can demonstrate how AI can be used to anticipate needs, personalized services and improve outcomes. While AI is a powerful tool, its use must be accompanied by robust safeguards and IBX ensures that through our data privacy, through HIPAA and other legal standards, transparent processes with pre- and post- implementation testing, use of member-consented data and continuous monitoring and improvement of AI systems. These practices can serve as a model for any entity considering AI adoption. AI is not a replacement for human services. It is a powerful tool that when used responsibly can enhance the delivery, efficiency and equity of public service. At IBX, our commitment to responsible innovation is guided by these principles.
As Philadelphia explores the future of AI in government, IBX welcomes the opportunity to collaborate with the City Council and share insights from our experiences in health care, from improving (inaudible) engagement and streamlining operations to advance equity, we believe AI can offer transformative potential for city agencies. By embracing thoughtful governance, ethical safeguards and cross-sector partnerships, the city can harness AI to build a smarter, more inclusive future for all residents. We thank the Committee for its leadership and thoughtful consideration of this important topic.
Thank you so much for your testimony. We're going to hold questions until the end. So please state your name and deliver your testimony. Thank you.
As a Philly resident, I'm especially glad to see that you're taking seriously the way that AI can be used to make decisions that directly impact all of our lives. My name is Sorelle Friedler and the Shibulal Family Professor of Computer Science at Haverford College. S. Technology Policy Committee for the Association for Computing Machinery. That's the main professional organization for computer scientists. And I formerly served as the Assistant Director for Data and Democracy at the White House Office of Science and Technology Policy. During my time at the White House, I co-authored the AI Bill of Rights and helped develop policy governing AI use across the federal government. I've done research on responsible AI techniques for more than a decade, and I'm a co-founder of the Conference on Fairness, Accountability and Transparency, one of the main publication venues focusing on the ways that AI impacts people. Before becoming a professor, I was a software engineer at Google. I've been asked to talk to you today about what AI is. As policymakers, you don't need to understand the detailed formula or specific algorithms behind these systems. What you do need are analogies that you can rely on, that build on your existing expertise as you make decisions for our city. I want to give you a few of those analogies today. AI is like the past. Broadly, we can think of AI as taking data about people and historical events and fighting patterns that predict and allow the replication of past actions. These patterns of the past do not perfectly predict the future. If there's one thing you take away from my testimony here today, let it be this: AI is not designed to work all the time. The guarantees that we make about AI as computer scientists are statistical. We might say, this system achieves 98% accuracy. That sounds pretty good. 98% is pretty good. It also means we got it wrong 2% of the time. A lot of AI governance is about making sure there are systems in place to handle that 2%. Given our city's population, if an AI system with a 2% error rate makes a decision about everyone in the city, that's more than 31,000 people who got the wrong results. We need plans in place to determine who will suffer from these errors and staffing to help people fix them. This also means there are times when the right answer will be to prohibit AI use. In cases where errors would be damaging or hard to catch, AI can't be the answer. For example, we should be cautious about AI tutors in our classrooms. They may confidently assert falsehoods spreading lies in our schools. In order for you all to have control and make sure AI systems used by the Administration actually do the job, we need careful guardrails, including testing requirements, human review processes and transparency into how AI is being used in our city. AI is like redlining. Since AI is like the past, it will replicate the biases of the past. In my own work and work of colleagues, we've found that predictive policing systems have this problem. Instead of identifying locations where crime will take place in the future, they repeatedly send police back to the same neighborhoods where they've made arrests in the past. If historical policing and arrests were more likely to occur in predominantly Black neighborhoods, this will ensure future policing and arrests match that bias pattern, regardless of the actual locations of high crime areas. AI is like a line of best fit. To better understand how an AI system works behind the scenes, I think it's useful to think of a line of best fit. You take data with two attributes, X and Y, and plot it, then you draw a line through the data. This is a way to identify a relationship between two attributes. Say, home values five years ago and home values now. It can then be used to predict resulting data in the future, say, home values in five years.
Modern AI systems are built on a large number of building blocks that work essentially like this: AI is like a flow chart. To handle more complex patterns, aI systems operate like a giant flow chart with the results of a computation, like a line of best fit, determining which flow chart step comes next. This could allow us to make different home value predictions for different neighborhoods. The specific computations performed at each step are determined based on training, which is the word for taking data from the past and finding patterns in it. AI is like the web remixed. Often when we're talking about AI these days, we're talking about generative AI and chat bots like ChatGPT. I think it's useful to focus on the data used to create these systems. AI companies gathered all the data from the web they could get their hands on, old newspaper articles, Reddit posts, YouTube comments, porn sites, conspiracy theories, everything. We can think of the results as the web remixed with prompts to the chat bot providing information about which parts of the web to remix and show to the user. AI is like a factory. One final analogy. AI is like the modernization of a factory. It replaces people's jobs with automation. The data centers that run AI take a huge physical footprint. They spew pollution into the surrounding neighborhoods. And as we've already seen in Philly, take so much energy that they increase all of our electricity costs. Yet there's one important way AI is not like a factory. It doesn't create long- term jobs here. After construction, data centers are mostly empty of people with just a few security guards. I hope as you provide needed oversight to city use of AI you remember that this technology is not magic. You can expect to be able to have your questions about it answered, have transparency into how these systems are being used and take control over whether AI is working for us. Thanks again for inviting me to speak and for this important work you're doing for the city. )
Thank you both so much for being here and for providing your expertise, especially in specific areas because I think that's what we were looking for, a little bit more of the specifics. My questions are to both of you, but particularly to Professor Friedler. In what experience, what kinds of use cases are particularly appropriate for a city government to use AI? In other words, what is AI good at and what problems can it help solve for the City of Philadelphia?
Yeah. So when I was in the federal government, one of the agencies that I really appreciated working with was the VA. They've been using AI in a lot of really useful ways to try to reduce the backlogs of veteran services that we all know they have suffered from for years, and they've been doing a really great job at that. So what that means is they've been trying to proactively identify -- as people sort of file 21 for requests for benefits they've been trying to figure out, okay, what are the really obvious requests using AI that we can just streamline for approval, right. You know, we can help these people faster because their case is really obvious and we know that they deserve benefits they're eligible for these, right. And so, they've been trying to use AI to sort of identify those places where they can give people services faster, right, with less paperwork, with sort of more streamlined paperwork. I think that's a great use case at the VA but also for the city, right, what are those city services that people should have access to, how can we get people access to those services faster.
This is what the City's Chief OIT Officer was telling us: It's machine learning AI though, it's the AI of the information you already have contained at the VA and using that in order to streamline its own services. Is there any way in which you've seen generative AI useful in city government?
That is a great question. Most of the cases where I've seen AI be really effective are these sort of more narrowly-tailored cases. That's not to say that there couldn't necessarily be a use case, but I think that there is a reason why it's mostly been these narrowly- tailored use cases that I've seen be successful, which is that those are really the places where you can test to make sure the system is actually working, right. As I said in my testimony earlier, I do think it's important to understand that a lot of generative AI cases are just trained on the entire internet. That means they're not necessarily trained to do what the city needs them to do. They're trained to give answers that are pleasing to people. That doesn't necessarily mean correct answers, right. So that means they're good at wordsmithing things to sound pleasing, but that is not quite the same as delivering city services, right. Those are I think clearly distinct use cases.
And I think that some of these cases could be is primarily in improving productivity and that is for the administration staff itself. I know that there are cities -- and I'm giving you an example from somewhere in India. They are using gen AI primarily to get rid of the backlog of legal cases. And that is not to replace lawyers, but it is more around, you know, can they be able to fast move some of the legal cases which are sitting there probably for 15, 20, years. And there are people who are incarcerated for that much time because the legal cases have not moved fast enough. So there are things like that at Independence Blue Cross, you know, we are using it more from a productivity perspective. You know, the human in the loop is always going to be there whenever we make any kind of decisions. But when it comes down to productivity, for people who are even using it for -- I think the City CIO mentioned that you guys also have GitHub Copilot. GitHub Copilot is a technology which can assist developers in IT space in order to do coding much faster. The kind of efficiency that we have seen in that is almost 30% to 40% improvement in productivity. Things like that can be used on our day-to-day activities. And I think that where the gen AI component comes into place. Even though the answers, you know, you guys all have heard about hallucinations, but in terms of using it for improving your own productivity, I would think of it like an intern. Everybody has an intern at their hands by having something like Copilot so that you can at least fast-track some of the work that you, yourself, are doing.
Thank you. I'm going to get on to my next question, except to tell you that the scenario from India was frightening to me. I don't know how you would fast-track a case of humans with generative AI. Even if you're using the law, you need humans in there.
Oh, absolutely. I think the humans are absolutely needed. I think it's more around how quickly can they go through the cases like just to even read them. The volume is that high. The population is so much more.
Thank you. So can you tell us a little bit -- just want to go back to our basic questions that we were asking the City of Philadelphia. What would a public inventory of AI and algorithmic tools look like? And do any other cities maintain this? And what are the benefits for it and how often would this be updated?
I worked with the Biden administration to put in place some of the guidelines for the AI use case inventory for the federal government, so not at the city level but at the federal level. And I believe you were actually quoting some of the numbers from that earlier when you were describing the number of DHS use cases. I think that that can serve as a model for the city, right. You know, those same procedures are all -- all the procedures that they use to collect that inventory are also all public, and I think that your ability to quote from it exactly demonstrates its usefulness, right. It's important that we know what systems are being used and that we have a public inventory and transparency into the goals of those systems and also the risks.
Very similar response. I would also say that I think you guys initially were discussing the vendor management part of the tools and the technologies in terms of, you know, how do we know what the vendors are using either the data for or even the kind of Gen AI components that they're adding. I think that it definitely needs additional amendments that should be added to the vendor contract because every vendor is exploring this as we go. You know, like the vendors like Adobe was one of them which was quoted. They have added gen AI in their tool sets. Our contracts might not reflect that. So we are going down that policy as well with our compliance committee to make sure that we have amended some of those contracts which have been in the past as we see solutions come by.
And you enter into a contract with a subcontractor, an entity now, and it changes going forward and that's not what you entered into. It's the same as --
At the federal government we were working on procurement guidance that includes in the contract clauses about updating those contracts and requirements for transparency on the part of the vendor, right. And so, I think that the city can and should build in contract clauses that require that type of transparency and require updates and notification as the system changes.
Yeah. How often should that happen for the review of the contracts or to go back through and check it?
So I guess I would say I think that it's important that the city maintains an inventory of the AI systems that are being used that is reasonably up-to-date, I would think certainly at least once a year, right. And that the contract clauses with the companies require that if they're going to have rolled out some new AI component for something, right, that they work with the city on incorporating a contract that oversees that.
Do you know if most cities procure their AI software through their procurement office or through their IT office or through their law department or is it a mix of one or more?
So in some places, cities and states and the federal government obviously all have sort of different structures for these types of things and different levels of staffing, right. I think what's important is to make sure that there is both an expert on the procurement side but also an expert on the technology side in the room 7 to make sure that the right questions are being asked.
Thank you. What kind of audit does it look great for the city? I know we talked about just the contracts, but just the audit of the city infrastructure and the workforce and all of it, what does the best audit practice look like?
So I think that part of what makes understanding what's going on within the city possible, right, is this inventory that you have, I think, rightfully been asking for. I think additionally depending on the system it may make sense to have an outside expert perform an audit. There are some practices that states and the federal government have been thinking about where any time, especially high risk, right, for example, a policing system is procured that relates to AI, that simultaneously they might procure a contract for an external evaluator to make sure that that system is both audited when it is first brought onboard but also in an ongoing manner to make sure that it continues to work for the city.
One entity I didn't talk about with what this was -- and I think the CIO talked about it a bit -- also that an ethics board would be involved as well. Can you just give us your experience on whether or not an ethics board would be involved, whether they are experts in AI and technology? Our ethics boards tell us a lot of things we're supposed to do and not do as employees and we appreciate that, but we don't hear much about AI.
So I think ethics boards are useful, but I think more important is the community input that you all have been rightfully asking for, right. I think that when systems are implemented that directly impact people in our city, right, those people should have a say.
Just imagine a typical city. What are the departments or areas which you think are most at risk for carrying the highest errors for AI, whether it's the police department, you know, Licenses & Inspections, Human Services? And if there's any best practices you can think of how to mitigate those harms?
So I think the way that you asked that question was exactly right. I think that we should be thinking about what are the places where when the AI system gets it wrong, which it will, those errors are the most damaging, right, you know, where are mistakes especially damaging. Certainly, that's policing. That's also education, right. It's also health care. And I think as you -- I guess I would just encourage you again sort of as you think about what are those really high risk cases, think about what happens when the system makes a mistake, right. In the example that I gave you earlier about the VA granting services, right, if the system makes a mistake, then somebody gets services who shouldn't have, that seems like you don't want to make too many of those mistakes because that's going to cost you money. But in the scheme of things, right, somebody getting health benefits when they maybe weren't eligible is a good kind of mistake to make, right. On the other hand, somebody being arrested or surveilled when they maybe shouldn't have been under the planned use of the system, that's a much harsher mistake to make. And in terms of what sort of risk mitigation strategies can be put into place, I would say that this goes hand-in-hand with these questions of transparency and auditing, right. We need to know that the systems work, right. We need to know what the systems are, what they're meant to do, whether they've been tested to see that they actually work at doing those things, whether those tests have occurred actually in our city, right. From speaking to an attorney, I learned about a system that was being rolled out in some of our prisons that had actually been originally created in rural Ohio. You know, so that there might be a mismatch between the types of data that we might expect in our city versus rural Ohio, right. And so, we need to think about has this system actually been created and tested for here. And we also need to think about bias testing, right, as you were bringing up earlier.
I think the use case has to be very thoughtfully designed first, number one. And then I completely agree bias testing is huge. We have to do bias testing and then also human in the loop. You cannot be rolling out any kind of programs without having somebody take a look at the output of some of these gen AI components before somebody is doing any kind of an outreach or a program. You know, at Independence Blue Cross, we do care management. And from for care management perspective, we have used machine learning and AI to see which of our members need an extra call from our nurses. There was an article probably about four or five years ago, it was in the Journal of Science, which spoke about bias, where the bias was based on access of care and people were being outreached who should not. There were people who were not being outreached who should have been outreach too. Things like that happen if people do not carefully design a program. So it's very important that anything that is rolled out has a human in the loop. Yes, it creates a lot more protocol. But at the same time, it's exactly to safeguard citizens as well as the members of the community.
Just want to -- I suppose I'm adding commentary. There'll be a little question at the end of this, but for health care providers, it's essential also -- you talked about HIPAA. There's a law that says that you can't disclose the information of your patients, but we have a federal administration and a president who does not respect HIPAA and is asking for data from health care providers, whether it's for gender-affirming care for patients and their families or other essential services and health care that people needs. We've got a president who doesn't respect that and is saying, give us your data, which is I believe essential for IBX and hospitals to keep safe. And what kind of conversations are you having within for that and how can you imagine our Philadelphia Health Department responding to that as well?
Yes. So I will have to get back to you on that particular piece. I can definitely talk to our compliance department on that. We will make sure that you guys get a response on that. But at the same time, IBX does not share vendor data, our data with any vendors. Other than for providing medical management services to us, there is never -- like we are not selling our data. We are not using our data beyond the HIPAA guidelines. And even when there is an ask from the federal government or even the Pennsylvania Insurance Department, we are making sure that we have people in the government affairs team, they are looking at it from a legal perspective and only with very careful thought. We even decide what the outcome is. But as of what I recall, I do not know this conversation with the federal government, but we'll get back to you for sure.
That's still great. It's that kind of affirmative thought process that we need in this, being able to think ahead and being able to make sure that you know who you want to protect first and baking that in as we bake the cake.
Just want to ask a little bit more about the human interaction in this. We're all scared of this, as you've heard about what's happening next. And you've heard from our city that we're still working on it. They're going to put out a policy. They're going to put out a committee. Talk more about the best practices of what human interaction looks like in this. I just want to make sure that if there are best practices that it's loud and clear for our city of Philadelphia.
Yeah. So I think that it is especially important in these sort of high risk systems that we were talking about, right, where the decision matters, that that be a person making that decision, right. We shouldn't be, you know, denying people services based off of some algorithmic decision. That should always be based off of policy and based off of human review. There may also need to be sort of a human appeals process in place, right, sort of a phone number you can call where an actual person responds, right. We need that both for general communication with the city, but also for response in case the system has gotten it wrong, right. There needs to be some process to correct mistakes that get made.
Very similar answer. But I will say one thing that we as a company -- I'm going to give Independence Blue Cross' perspective -- as a company we have not exposed our members to any of our chat bots and, that is, the technology is moving fast. Use cases are moving fast. As you see, that every single day there's a new technology out there. But you have to be slow and thoughtful and make sure that is only used for the right cases and that's what we have gone with. The technology has been out since 2023. We do not have a single use case as of today where the members, our community members, are interacting with chat bots regarding their health coverage.
Excellent. Thank you. Do any other members have questions? Chair recognizes Councilmember Nic O'Rourke.