I'll just say that I appreciate your testimony and giving us the analogies that will help us to kind of process this and understand it in layman's terms. One of them that stood out to me in particular, I actually prepared questions for, related to your analogy around AI is like a modernization of a factory data center. Now, Madam Chair, as I looked it up this particular hearing is not about data centers. There will be another time where we will go into that in deeper detail. So I'll save those questions. But I did just want to nitpick for a second -- not nitpick, but for my own knowing, thinking about how people understand, how people talk about this. In the previous panel, there was comment made about AI producing new, the term is new content, and I appreciated your point around AI is like the web remixed. Would you challenge the idea that AI produces new content? And, in fact, something or a mash- up of something else?
Yeah, I would. I mean, I think mash-ups can be fun and in music, can feel new, right. So in that sense, sure, it's new. But it's new only in the way that a remix is new.
And I also appreciate the analogy around AI is like a flow chart. The comment that you made was that to handle more complex patterns, AI systems operate like a giant flow chart with the results of computation, like the line of best fit determining which flow chart comes next. This could allow us to make different home value predictions for different neighborhoods, specific computation performed at each step or determined based on training, which is the word for taking data for the past and finding patterns in it. This Council banned I guess late last year, early this year algorithm rental price fixing. Not only our home values, right being addressed, rental prices as well are often see themselves going higher, they're predicting that they should be higher for whatever reason. In a city like Philadelphia, you were looking at almost every other person who rents. Nearly 50% of the city rents. So you're talking about folks who are combating AI as it were, would have before we banned it as it relates to their prices. So again, I just find these to be particularly helpful, relevant and really, really appreciate your commentary around crime and the importance of paying attention to the 2% that you said. That was super helpful for me as well. Yes, it may be 98% well, but 2% can literally mean 31,000 people are impacted by an incorrect output coming from artificial intelligence. That is not a thing to sniff at. That's something that we should be extremely -- I don't like using the word scared -- but we ought to be concerned, and then move to action to make sure that we address it. So I believe that this particular panel has been super helpful towards that end. Thank you, Madam Chair.
Okay. Thank you so much. Really appreciate you and your testimony. Appreciate you being here. (Applause.)
Andrew Jennison and Alli Finn. (Witnesses approached witness table.)
Thank you. Can you state your name and proceed with your testimony.
Andrew Jennison, CGI Technologies. Thank you for the opportunity to testify regarding the governance and responsible deployment of artificial intelligence in the City of Philadelphia. CGI Technologies, founded in 1966 is one of the -- '76, is one of the largest IT and business consulting firms globally with over 93,000 professionals operating across 400 locations. As AI technology has become increasingly embedded in public services, infrastructure and decision-making, it is imperative that we adopt a responsible our AI framework, one that ensures transparency, fairness, privacy and accountability. The CGI 2024 ESG Responsible AI Report outlines a comprehensive approach to our AI that aligns with Philadelphia's values of equity, innovation and sustainability. AI has the power to transform how we deliver services, manage resources and engage with residents. However, without ethical guardrails it can also amplify bias, compromise privacy and erode public trust. Responsible AI is not just a technical necessity. It is an imperative. CGI's RAI framework emphasizes human in the loop, designed to ensure AI supports, not replace human judgment, privacy and security protocols to protect sensitive data, transparency and AI outputs, including citations and source tagging, environmental sustainability, social equity. Philadelphia can leverage responsible AI across multiple domains. AI can help identify early signs of addiction, mental health crisis, social car e needs. AI tools can support law enforcement in identifying patterns preventing crime and protecting vulnerable populations while ensuring fairness and avoiding bias. Tools like CGI's Data Twin 360 and Environment Monitor 360 use AI to stimulate energy use. The New Jersey DEP utilizes CGI's S data platform to provide residents with localized answers on air, water quality, flood risk and permits. Open grid forums and our renewable management systems is used by our utility clients to optimize renewable energy and grid stability. To ensure ethical and effective AI adoption, I respectfully recommend the City of Philadelphia establish a responsible AI Task Force to guide policy and implementation, adopt a citywide RAI framework based on CGI's principles and engage communities in AI decision-making. To build trust and transparency, partner with academia and industry to stay ahead of regulatory and technological developments and prioritize privacy, security and inclusivity in all AI deployments. Thank you very much for your leadership for a smarter and fair future.
Please state your name for the record and proceed with your testimony.
My name is Clarence Okoh, Senior Attorney for Civil Rights and Technology at TechTonic Justice. As we gather, over 92 million poor and working class people in the United States will have some basic human need regulated by artificial intelligence and automated decision-making systems. This represents one of the most consequential transformations and democratic governance in modern American history. The results thus far have been catastrophic. Faulty algorithms have cut lifesaving care for Medicaid services. Students have been placed on secret police databases due to discriminatory predictive policing models. Federal data sets have been exploited to surveil immigrant families and to suppress political dissent. Automated policing technologies are being used to track pregnant people across state lines, and major American cities are experimenting with mass biometric surveillance to track our faces across networks of public and private cameras. This is not an Orwellian fantasy. This is a civil and human rights crisis enabled by the rise of artificial intelligence. And the City of Philadelphia must meet this critical moment and set a new national standard by placing civil and human rights at the foundation of AI and digital policy. As I mentioned, my name is Clarence Okoh. I'm a Senior Attorney at TechTonic Justice and a co-founder of the NOTICE Coalition, No Tech Criminalization In Education. I worked across academia, government and civil society to effectively, successfully challenge the use of artificial intelligence and other emerging technologies to reproduce social and racial inequality. I appreciate the opportunity to testify today, especially at this critically urgent moment. I'll focus the remainder of my remarks on three topics: One of the emerging threats presented by federal authorities that seek to weaponize artificial intelligence against vulnerable communities; number two, the real world consequences of AI injustice in the lives of vulnerable people, including children; and three, recommendations for local policy make makers to protect our community's rights against AI and justice. Beginning with the federal context, it's been mentioned earlier this year, the federal policymakers have essentially reversed course on all key civil and human rights protections as it relates to artificial intelligence. During the Biden administration, we saw some of the first guidance from the Departments of Justice, Education and Health and Human Services as it relates to enforcing civil and human rights standards against emerging technologies. That guidance is no longer in place. There is no federal enforcement of civil rights standards against emerging technologies. And as members of the Council have already noted, it is going to be incumbent upon localities and states to pick up where the federal government has left off. R. 1 the massive federal tax and spending bill enacted earlier this year that's going to cut $1 trillion for Medicaid services, costing as many as million 6 people to lose their health 7 insurance. 8 Key changes to Medicaid 9 and SNAP imposing new work 10 requirements and shorter 11 eligibility periods are going to 12 create new administrative burdens 13 on state and local agencies. That 14 burden is going to create an 15 incentive to adopt and use artificial intelligence and other automated decision-making systems to a much greater degree, and we've already seen the devastating consequences of this in places like Arkansas that have in recent years attempted to use these technologies in ways that have resulted in thousands of people losing their Medicaid services. But to really understand this issue, I think it's important to ground ourselves at what this actually looks like in community settings.
So I would like to briefly share a story from one of the most egregious uses of algorithmic systems to violate individual rights in the United States, a predictive policing program that was used in school district to target vulnerable children and their families. This is a case I know well. I helped lead a local coalition of national organizations and community members to end this program. So in 2021, the Tampa Bay Times uncovered that the Pasco County School District entered into a secret data-sharing agreement with the local sheriff's office. The agreement granted police access to confidential student data, including their grades, their GPAs, credit completion, attendance records and student discipline. These data were combined with other sensitive data, including records of child abuse and whether or not those children have been victims of crime themselves, even information about parental custody disputes were considered by this predictive policing program. " At the program's height, the sheriff's office had developed risk scores for over 18,000 children in this school district representing in students in that community. Students with the highest scores were placed on secret lists that were shared with school-based police officers and those officers were instructed to interrogate these students and capture "actionable intelligence and secret dossiers" that were reviewed by intelligence officers and allegedly shared with state and federal law enforcement. Those students with high scores experienced higher rates of exclusionary discipline, including suspensions, expulsions, school- based arrests and even involuntary psychiatric detention. Students were also subject to a variety of arbitrary home visitations by law enforcement who exploited every opportunity to issue civil citation fines, fees and other code violations. According to one officer who spoke under oath, the predictive policing program was designed to be so intense that families would feel pressure to move or sue the county. Again, another direct quote. So what took place in Pasco County should be a cautionary tale to policymakers, and the lesson is very simple, that government use of AI and algorithmic technologies can easily result in systematic civil and human rights abuses for the most vulnerable members of our community, even our children. To protect our communities from AI injustice, policymakers should, number one, develop critical AI competencies. At a minimum, policymakers must develop those competencies that allow them to be able to identify and assess these technologies for civil and human rights implications. Number two, policymakers need to adopt greater public accountability and oversight measures, as we've already spoken about today. Number three, policymakers must protect basic civil and human rights. You already have the legal authority under existing local, state and federal law, and it's important that you all prioritize enforcement activities for those who are using these technologies in unlawful ways. And furthermore, use that power to engage in agency guidance and use your rule-making authority to clarify how those existing laws apply to AI and digital context. And then most importantly, policymakers must ban and decommission harmful and unlawful AI systems. And I love the suggestion earlier today about banning vendors that are using these technologies in unlawful ways. Policymakers must create simple, accessible procedures for the public, especially for those individuals that are directly impacted by AI injustice. To request that these systems be suspended, reviewed, terminated and/or banned, decommissioning harmful AI is a policy best practice that has not really been adopted in the United States, but it's something that is necessary to safeguard our communities. I want to thank you all again for the opportunity to share this testimony and I look forward to your questions. )
Thank you. Please state your name for the record and proceed with your testimony.
Thank you so much to Chair Landau and the members of the Committee. My name is Alli Finn and I am the Senior Partnerships and Strategy Lead at the AI Now Institute, an independent organization producing policy research on artificial intelligence. We focus on policy strategies to shape AI in the public interest and redirect the industry away from its current path, unbridled commercial surveillance, the concentration of power in a few dominant tech companies and the deployment of AI on the public instead of with and by us. Our work includes advising local, state and federal, federal officials across the country on the reality of the AI industry and best practices in policymaking. We are at a critical inflection point in both the trajectory of AI technology deployment and policymaking attempts to regulate the power structures behind it and the downstream impacts on everyday people. The wealthiest companies on the planet are aggressively selling narratives that AI will transform the world and our lives for the better, as they rapidly accelerate AI use across all sectors of society. In particular, we are repeatedly told that AI can be used to make our governments and public institutions more efficient and effective. But under this logic of efficiency, governments are funneling millions of dollars into private contracts to develop AI technology presumably for our public benefit, with little attention to the well-documented risks. The industry hype around AI obscures that this acceleration is often to the detriment of everyday people. We just heard many of those examples. For one or two more, AI tools are consistently used to devalue workers' labor and increase extreme monitoring and surveillance over their every move. AI's associated infrastructure also increases inequity. Last year alone Pennsylvanian households increased electricity bills paid for over $491 million in transmission costs for the data center build-out, not the wealthy companies driving this expansion. My testimony will cover policies to protect the public and immediate actions that City Council and the Committee can take to limit the most egregious harms. And in considering these areas, I want to highlight the critical role that this Committee holds in not only taking strong and protective action, but in shaping the way that these decisions are made and the potential to lead the way, not only in Philadelphia, but in the state and as a model for jurisdictions across the country. The core conversations around AI in this moment are not questions of computer science expertise, but how social and economic policy is implemented and whose actions these serve -- or and whose these actions serve. I respectfully disagree with the Administration. This room is full of experts on AI. (Applause.)
Communities will always be the most qualified experts in what they need, whether that's workplace protections, high quality education and health care and how their representatives should invest our taxpayer dollars. In questions about how, when and critically if to use AI, we need to keep these fundamental needs as our core focus. And we need to remember that at the core, technology is never, will never be neutral. Algorithms are embedded with the intentions, biases and norms of their creators and AI is never acting on its own. It is being deployed by people and by companies with agendas. I'm skipping my section around what AI is and how to understand the industry, but I'm happy to also speak to the supply chain in questions. I want to briefly highlight several principles that we can follow in policymaking on AI for the public good and to ground this in saying that the stakes are incredibly high. AI must not be used to justify and entrench big tech corporate interests in our public infrastructure while creating a centralized surveillance system that violates Philadelphian's privacy and autonomy. So here are the three principles: First, start with a zero trust policy agenda for AI. Tech companies need to prove every step of the way, including before any implementation, that their product is not causing harm, rather than putting the responsibility of accountability on the people upon whom the tech is deployed. Years of experience show that blanket trust in AI firms benevolence is not a smart, informed or credible option. We need to ensure that the industry that currently creates and profits from AI is not left to regulate and evaluate itself. Second, establish transparency as a critical step in the process, but understand that transparency alone will never be enough and it cannot stop the most egregious uses. We've seen this similarly in surveillance ordinances passed in a number of cities like Boston and New York, where required city reporting on technology and how it's used is often delayed, incomplete and has not at all curtailed the technology's deployment and harms. Third, simply start with the problem, not the perceived solution. Tech companies aggressively pitch products to city agencies, resulting often in immense pressure to adopt AI to be more efficient. This approach obscures and bypasses the fundamental problem supposedly being fixed most of the time and further distances us from addressing the core challenges of limited resources that city agencies face. Instead we need to ask from the beginning what is the problem that we're trying to address, what will help us do that. And if someone is telling us it's AI or another technology, is that the correct intervention. Investing in the foundations of our communities, more schools, public parks and libraries, job training programs and deeply affordable housing is frequently the smarter, more durable investment than the promised quick fix of a tech product. (Applause.)
To conclude, I just want to highlight a few immediate actions that the Council can take, and to emphasize that the push for deregulation -- with the push for deregulation on AI at the federal level, it is local governments such as this one that are on the front lines of protecting constituents. And there are multiple immediate, meaningful actions that the Committee and the Council can take. These are drawn from examples across the country and consultation with community groups in Philadelphia in the room today. First is to protect the public from AI use in high-stakes agency decisions. More than a decade of research has shown how algorithms encode bias, harm low-income communities and meaningfully take away people's opportunities with little recourse. We know that when governments implement algorithmic decision-making systems and benefits administration or ostensibly to reduce fraud, they often discriminate and produce incorrect results with life- altering consequences, as we have heard. The most scrutiny needs to be at the beginning where we evaluate whether AI is the right tool for the job and assess the value proposition of substituting human decisions with AI versus other tools or mechanisms, and where a significant percentage of AI tools are being provided by external vendors, measures can also be taken to reform procurement processes to ensure that other forms of AI use within city agencies do not exacerbate documented harms. And I'm happy to speak to this and follow up as well. Second, we need to pass bright line bans on the most egregious applications of AI. This has been spoken to quite a bit. I want to emphasize that Philly is already a leader here. Thank you to Councilmember O'Rourke with the ban on rental price fixing. We need to prohibit or limit the expansion of power, hungry, thirsty AI data centers. This is intimately linked to what the technology is doing and how it's being deployed, and I can speak to mechanisms here as well. Finally to conclude, in the event that A city agency must purchase AI technology, the city can attach strong conditions onto public funds to ensure that tax dollars work directly for the city and that the purchase technology abides by strong accountability measures and where possible, that agencies are investing in their own tech deployment and people power, rather than big tech vendor lock- in. Transparency and procurement contracting and AI use is essential alongside robust mechanisms for community input, accountability and public control of deployed technologies. There are additional strong actions to take, protecting city jobs, democratizing procurement and protecting residents' most intimate data, and the city can work with community members and advocates to build out a robust suite of protections and ensure that any deployment of technology and AI prioritizes residents over corporate wallets and interests. Thank you. (Applause.)
Thank you very much. Your testimony was so thorough. I actually don't have many questions and we'll open it up to my colleagues, if there's any. Councilman O'Rourke.
I'm mindful of time, but I do really feel like this is one of the most important conversations we should be having right now. I'm going to ask and share what I can share. We've heard from multiple panelists about the dangers of AI reproducing racial biases and social inequities, especially if not given guardrails and including human beings in the loop or I think the term is human loop, somebody said, so I'm learning as we go. We heard from Clarence Okoh about some of the devastating effects it can have on the most vulnerable among us, including children. That testimony was riveting. I'm wondering if any of our panelists can share any specific success stories where AI has been used by local governments to promote equity or access people?
I think the silence is indicative of the answer, which is unfortunately what we've seen too often is -- and Alli was getting at this in their testimony -- is this kind of what I would call a fire-ready-aim approach, where we're kind of identifying the technology first and then figuring out how to use it. And so, that unfortunately has meant that many of the most kind of prominent uses of these technologies, even at times where it's been introduced to try and mitigate bias, has actually done the exact opposite. So I'll give you an example: In five or six years ago with the passage of the First Step Act, one of the provisions of that federal law required the use of an algorithm to basically help determine individuals that would be eligible for parole at the federal level. The algorithm was supposed to introduce greater fairness. Independent evaluations of it found that it actually exacerbated racial bias, particularly against Black folks, and that has been the case across a number of these systems. The story that I shared in Pasco when you actually asked the sheriff's office, they were on the record saying this, what the program was designed to be, they said this was a mentorship program. This was a mentorship program that was supposed to be helping Black and Brown young people, and yet they're being arrested and yet they're being spied upon and yet they're being pushed out the community. So I struggle to think of those examples where there's a kind of equity expansion as a result of artificial intelligence. And quite frankly, it's because that often times is not the intention brought to these technologies. In most instances in which they're being used, particularly in the lives of low-income and working-class people and people of color, these technologies are being used to expand social inequality rather than reduce it.
Wow. Well, I guess it was important to raise that point, huh. Duly noted again. Thank you for your testimony. What you shared today I think this is absolutely essential that we figure out how to get this thing right. Thank you, Madam Chair.
Thank you. I just want to add one quick question here and then we'll move on to Councilman Phillips. When AI gets it wrong in localities, what are the best kind of appeals or remediation programs to right the wrong?
That's a great question and I appreciate that you said when it gets it wrong and not if. I think we need to keep that in mind in any application, that no matter what promises companies make around accuracy rates, it's a slippery slope and bias and discriminatory results are going to be encoded. In terms of remediation, I would encourage us to think not only about after a harm is perpetrated, but before any technology is introduced. There are a number of very strong reports on the robust protective measures that can be taken to reform technology and AI procurement. And those involve deep evaluation, not by companies, not by industry, but by the city, by residents, right, decision-making with people, meaningful decision-making with people before anything is put into place, and making sure that people have access to information about when AI is used, redress mechanisms as well so that you're not only trying to fix harms after the fact, but prevent them, including by not deploying the technology at all, understanding that a significant amount of AI deployment currently is not fit for purpose.
I'll briefly add, so under the Biden administration we saw the Federal Trade Commission really pioneer some really expansionary and creative approaches to accountability for the misuse of AI systems, strategies like algorithmic disgorgement which means that a company that's actually found to be using an algorithm in a harmful way is no 22 longer allowed to use that algorithm. The algorithm is literally taken away and they're prohibited from using it again. There are other strategies like data purges, so we know that there's been an unlawful collection of data actually requiring that that data be purged and that you use strategies like secure deletion to make sure that that data can't be used again. What we've seen as a best practice emerging in states and other localities are ensuring that there are private rights of action for any type of legislation that actually creates new data privacy rights and civil rights to ensure that we can actually get into court to make sure that folks' rights are being protected. And then as I mentioned in my remarks, decommissioning AI. We've seen this as a strategy being used in places like the EU but essentially allowing for individuals, civil society groups or others, that are identifying these harms to be able to actually make requests of city agencies or whoever the authority is in that particular instance to stop using those technologies. But what's really important here to understand is -- one quick point I wanted to make is sometimes accountability can be very difficult, particularly for local policymakers, because there's a lot of interagency interactions that happen, particularly in the law enforcement and schooling context. Almost every example that I've worked on, there's been some type of work around for traditional procurement processes, traditional like processes coming to the City Council asking for funding for these technologies. Things like using funding from philanthropy to be able to procure these technologies. Some policing technologies, for example, do these things called freemiums, where they essentially allow for officers to do a free trial of their product without the kind of larger city being able to know what's happening, and their workarounds also with federal grant programs where agencies will go off on their own initiative to actually apply for these grants and again avoid accountability, and that average disastrous results. So I'll also think about how we can control those interagency interactions as well to make sure that we're having accountability.
And if I can add just one very brief point, a word of caution to the Administration's characterization of AI and different types of AI. It was implied that machine learning is fine and generative AI is what we should have concern about, and that just simply doesn't reflect the reality on the ground. There is no one -- (Applause.)
There is no 11 one good type of AI and one bad type of AI. There are risks in all applications and there are many, many examples. We heard some of them from Clarence of machine learning perpetuating irrevocable harms.
Yeah. I'll just really quickly -- I just want to ask you how can Council require -- no, I want to ask you more about the community structure. We are interested in learning about how do we structure community members and comment so residents help decide an AI system should be used, right? In your experience how do you structure that?
So I can kick us off by saying I think a redundant approach makes the most sense. So as soon as we are considering using this technology, there should be public notice. Once we're selecting vendors, there should be public notice and opportunities for comment. Once the folks have actually been selected, right, putting these kind of redundant opportunities for public engagement is really important. Because at each stage, we have different information that's going to be available to us to be able to determine the impacts of these technologies. And then I think using as many different forms of public engagement and oversight as possible, putting together and bringing in different stakeholders from across the city to be able to offer perspectives from the civil rights community, from the data privacy community, from community organizations, from frontline folks who are working and providing direct services to vulnerable communities, public defenders, legal aid attorneys, so on and so forth. But I think trying to incorporate as many redundancies as possible to allow for there to be robust stakeholder engagement across the life cycle of that AI system is really important.
I fully agree with everything that was shared. We see a lot of examples of symbolic participation, right, and symbolic input, but we also have strong examples of what it would look like to fully engage the public in decision-making. And I just really want to emphasize ensuring that public engagement starts at the outset. I agree with redundancy, but making sure that it starts at the outset of any process before a technological solution is identified. So not, hey, we got this pitch from this company about how to improve health care via AI, but we're trying to address health care needs in the city, what should we do, right, what is your experience. So not starting with the assumption that technology is always our friend or is always the answer, and making sure that these methods for engagement are accessible, right, include multi- lingual resources and language also that I thought Professor Friedler did a great job, right, making the issue accessible to people across a wide range of backgrounds and experiences, building longstanding partnerships with community groups and advocacy organizations that can also help engage people who might not be able to participate in mechanisms that the City Council or city agencies put forth. And then really involve the public in deep assessment, collective evaluation, decision-making and continue decisions about whether the technology is working for people and if it should continue to be used. And in procurement and in public participation, always making sure that the city has the opportunity to stop that use and that deployment. (Applause.)
And to elaborate further -- great job. Consistent re-evaluation, right, are you doing the right things, do you need to reset, do you need to frame re-look at the framework, see if it needs to be updated, compliance, governance processes, et cetera.
Thank you. The Chair recognizes Councilman Nic O'Rourke.
Thank you very much, Madam Chair. The point you just made about constant updates earlier, I think it was suggested at least once a year that that's something, that revision, that sort of re-evaluation should happen. Do you have a particular suggestion about how often a regularity around that should be?
Meeting, engaging right. What is right for the client, what is right internally for our framework and structure. And I think I can't remember who alluded to it earlier. The technology is constantly evolving, so setting a defined date that you have to meet may not be the best course for the city.
And you had mentioned several pieces of things to use to keep things accountable. One them was data purging, as I understand it just means delete data wherever it needs to go. But there was another term that you used, I think it was algorithmic disgorgement. Is that --
So to make sure I'm understanding this correctly, it's basically that the entity can no longer use the algorithm that's been used once it's been deemed harmful in some way or another; is that correct?
That's absolutely correct. So to give you an example, going back to that Pasco example I shared earlier, the idea would be that they are no 13 longer allowed to use that algorithm to make any kind of decision moving forward. So it's not just enough to hold them accountable for the discriminatory effects. It's not just enough to purge the data, but to make sure that they actually cannot use that model again to do what they've done before. Because we've seen a lot of times happen is the company will say, okay, well, this technology produced a disparate impact, right. And so, what we're going to do is fine-tune our model and make our model a little bit better so that it will reduce those disparities. But algorithmic disgorgement says if this model is producing some type of unlawful or unethical outcome or if it's designed in a way that is unlawful or unethical, then that model, that model should not be used moving into the future. So disgorgement is the thing that's been happening for some time. Yet the FTC has been using it under the Biden administration.
Is there a database of like here's all the algorithms that have now been retired out of (inaudible)?
That's a good question. I would have to go and look at kind of the web, the archives of the FTC, on the internet. But as far as I know, I don't think that there's kind of a complete listing of that, but that's something that I'm happy to work with you all to figure that out.
We've just learned that our wonderful stenographer has to be at City Hall by 2 o'clock, so I do want to -- if there's no more questions, we're going to move on to public comment I believe, right. Thank you very much. Your testimony was really wonderful. (Applause.)
So now, we are going to move into the public comment section of our hearing. If everybody -- so in order -- we have a long list of folks for public comment. If you can keep your comment brief, your testimony brief. If it's written, please summarize it. And we're looking at tops two minutes each person in order to conclude on time. Some of the commenters are potentially coming in from the overflow room. Thank you. Wonderful. Great. If you could state your name for the record and please proceed. (Witnesses approached witness table.)
My name is Devin Washington. I am the Organizing Director at People's Tech and one of the founders of the Philly Tech Justice Collective, and I appreciate the conversation and the hearing that we've had today. We see this as the first step that the city is taking and a lot of our movements are taking in ensuring that the way in which technology is used is accountable to the communities who are directly impacted. And what we've been seeing over the past couple of years where we've been doing this project is that some of the biggest voices in how technology is deployed in the city are the billionaires and corporations who are invested in making sure that they get access to our public dollars, right. Their priority is not necessarily the common good, but enriching their own pockets and their shareholders. Meanwhile the machine learning, the AI that the city is currently using is empowering a network of 50,000 surveillance cameras in some of the poorest areas of the city, under the guise of this is what makes our community safer. When we know that investment in our community, investment in our schools, investment in our rec centers, making sure that folks have access to a social safety net that actually treats them as a human being is what makes our community safer, right. So instead what we have is 50,000 surveillance cameras owned by the city, SEPTA, the universities that are watching our communities be poor. The promise of jobs brought by technology from data centers in other parts of the state, et cetera, that never actually pan out for everyday working-class Philadelphians, I can't tell you how many friends I have who have gotten degrees in computer science who can only get an IT job, cannot get a robotics job. And I can guarantee you in places where there are data centers, they are not getting those jobs either because they are not vehicles for long-term job growth for cities, for the communities that they're in, in addition to the fact that the data centers themselves are deeply impacting the communities they exist in there. You know, we've talked about it already where utilities bills are going up, air quality is being harmed, and Black, Brown and working-class communities are left holding the bag for infrastructure that billionaires and big tech are enriching themselves off of. So for those of us at Philly Tech Justice, we are going to be laser-focused going forward and ensuring that not just generative AI and machine learning, but all forms of tech that the city plans on using, that these corporations are trying to sell to our government, are accountable to our communities because we are fighting for a vision of community ownership and accountability of how technology is used. Thank you. (Applause.)
Oh, it do. My name is Shawmar Pitts and Co-managing Director at Philly Thrive and I represent the marginalized community that has been spoken about a little bit up here today. And I just feel like I try to be as nice as I can in my descent, but I really -- because I don't know much about AI and things like that, right, I'm learning about it. But just from this hearing, from the beginning to right now from everybody that testified and from my past experience in dealing with our legislators, man, I feel like we about to get played. Man, I feel like we about to get played. I swear I feel it because when the Administration came up and gave this is what AI is, this is how we can use it, and we going to do and we going to do, and then when the questions come, they say, you know, the ones that I was paying attention to, that I can understand, the ones that's right to the point, we got to get back to you on that, we got to get with the police on that and all that kind of stuff. I just feel like we get ready -- they trying to play us. This is the stuff that they do. So now, I'm only left to ask the people who care about the people that I represent in South Philadelphia and at-large the poor people, right. Don't let them play us, man. Don't let them play us. (Applause.)
Because we know when they talk about data centers, where they gonna want to build it at? They gonna want to build it right next to my house. That's where they gonna want it. They gonna want to build it right there. We know that. We not silly. We know when the adverse effects come, right, my people gonna be in that 2%. 99% of the 2% is gonna be my people. We the ones that's going to have the adverse effects of AI. So please I just ask don't let them play us. Don't let them play us. And, you know, like my brother just said right here, we going to be laser-focused and we going to be on it and we going to bring the information and we just want everybody to act in the best interest of the people of this city, because a lot of times that's not what's being considered when the Administration brings these things. That's not the interest that they're representing. They representing the big tech companies and the billionaires and all of them. And these are the people that they are representing when they come and they bring things, they just don't say. But that's what they doing because they don't represent me. They don't represent me. And they don't represent my people when they come with the H.O.M.E. plan. That H.O.M.E. plan, that's not representing my people. When they come with all these other things, that's not representing my people. And this is just more of it. I just feel it and I see it and I know I ain't going crazy. They try to make me think I'm going crazy, but it just happens over and over again. So I know it's not me, you know. So we just need someone to stand up for us. Because when they want to build a data center, they want to raise the electric bills and they want to mess up the water quality and they want to mess up the air quality, they don't want to do it in my neighborhood. That's just been the way it's been done for the past 150 years so they not going to stop. So I'm asking our Councilmembers, you know, the ones that represent us, you know, don't let them do this to us. Don't let them do it. Thank you.
Thank you. Can you please state your name for the record and proceed with your testimony.
My name is Rodney Gardner. I am a member with YEAH Philly and also an instructor at Slash (inaudible) with the Energy Collaborative. We run a solar agriculture program where youth and young adults can come and learn about the importance of environmental justice, how to properly care for the environment and the benefits of urban farming. We also teach how solar energy works and how to install solar panels. And lastly, we discuss air pollution, its causes and its harmful effects on the environment. My involvement in this work, I have learned that AI consumes a vast amount of electricity, which contributes to the burning of fossil fuels and results in air pollution. This pollution releases nitrogen oxide, sulfur dioxide and carbon monoxide into the atmosphere. Air pollution is detrimental to plants, animals and of course humans, and it is one of the leading causes of lung cancer, heart disease and asthma in minority communities. AI is more harmful than it is helpful and we must put an end to it now. (Applause.)
Good afternoon. My name is Jordan Holbert. Councilmembers, I want to say thank you for the opportunity to speak. I'm a coach, a consultant, a community advocate who believes in building systems that help people and communities grow together. Today I want to challenge all of us to be bold and brave enough to make decisions, not just for the convenience of the future, but for the well-being of the future us. Only a handful of tools have had the impact rivaling the influence artificial intelligence is already having on humanity. The printing press made written word accessible. The wheel made people and objects transportable. The Internet made the world interconnected. Each of these innovations brought people closer together. If we're not careful, AI's impact could divide us beyond repair. Right now AI is still in its infancy. Yet we already have examples of it defying human instruction and behaving based on bias, a bias that like most in America impacts Black and Brown people before the implications spread through the rest of the nation. No other tool in history has shared the ability to "create something new without a human." That means we must be strategic, intentional and guided not by fear or greed, but by values of health, humanity and fairness. We must also avoid making emotional decisions in the name of safety. Applying AI to police our streets or to surveil our neighborhoods may sound like safety, but it risks creating deeper distrust and unintended harm. Amazon's Ring cameras already accessible to law enforcement, offer a glimpse of this reality, while emotionally Some believe this increases safety. Logically and historically, we know that more policing does not necessarily create safer communities. Implementing AI won't change that. It will only magnify the injustices. We know that education predicts outcomes, so how we educate our children, our workforce and our communities about AI will determine who benefits from it and who's left behind. I'm here today understanding the opportunities of AI and as someone who believes in its power as a technological tool, we are at an inflection point however because AI can help us build a more efficient, creative and connected future, but that future must be equitable, transparent and sustainable. If we lead with humanity now, we can ensure AI becomes a tool that unites us rather than replaces us or divides us.
Hello. My name is (inaudible) Albert. I'm a proud Philadelphia resident. I live in District 2. I'm also a nationally recognized attorney and scholar whose work focuses on algorithmic justice and more important questions of how technology affects people's lives. I submitted comments which were mostly about trade secret and transparency which you could read, but I wanted to just take the time that I have to actually echo my colleague Shawmar Pitts and respond to the first panel that we heard where -- what I heard from city officials as an expert in the space who has been working on AI for a very long time, was embarrassing. It was the suggestion that it was time to form a committee and develop a policy with no timeline, no commitment to who would be on it, and a long list of cities, some of which bear very little resemblance to Philadelphia, to look to as examples suggest that this issue is seen as new, novel and sort of suddenly something that landed on the lap of our city officials. Those claims are wildly incorrect and deeply concerning as someone who wants our city to actually develop a policy that it can implement with the things it's already doing, rather than with this idea that generative AI is something new that we now need to worry about because city officials might use ChatGPT to be more productive at work. So I want to really just echo this idea that you have some tremendous experts who came and spoke before you in this room in the city of Philadelphia and that they have the expertise and the knowledge to help this Council make decisions and the city make decisions, and that I hope that whatever committee is formed will use that expertise rather than starting from scratch. And maybe producing something like some voluntary guidelines in like five years. Thank you. (Applause.)
Thank you very much and thanks for submitting your testimony as well. Mr. Serra.
Hi, I'm Olivia. And I'm here to give testimony speaking as an educator and an advocate for the children of Philadelphia. So my first year of teaching was 2019-2020 so during the pandemic. And it was also the first year that my school had one-to-one technology with students. So that kind of worked out well when the pandemic happened. But since then, I've seen this constant push for more tech, more innovation in our schools, more subscriptions to these programs that are going to improve outcomes and raise test scores and all of these things. And the reality that I've seen as a teacher is more students watching anime on their Chromebooks in class, and more students who are so distracted to the point where they can't even regulate their emotions when you redirect them during class, because their attention spans are short because of algorithms that have been designed by these tech companies to, you know, take as much of our attention as possible. So I just speak to that to say that the decision to invest in technology in terms of the education sector was made under a lot of pressure and this, like, COVID, is happening, we need to move in this direction. No matter what it will be helpful in this situation. We have time to stop and wait and put limitations on the way that AI is being used while we figure out how and if it could be beneficial for the citizens of Philadelphia. So I just encourage you to think about what innovation actually means. Because from what I've heard of all the testimony today, innovation moving towards more technology and AI inclusion is going to make a worse society for us all to live in when we know that investing in real things, buildings, parks, recreation centers, places where students can go and gather, we already know the answers to the questions that we are trying to use technology to solve. We know how to stop crime. It's to provide other alternatives. We know how to -- I don't know. I just encourage us to use our brains and think also looking at history. If AI is looking through all of this, like historic data and what happened, we can also do that. We can make innovation be wow, like, Philly is the city where we actually have dialogues like this and we include the perspective of the public, and the public helps shape policy, not AI. Thank you.
Thanks. Mr. Serra, can you call the next few public testifiers.
Thank you. Can you please state your name for the record and proceed with your testimony.
Absolutely. My name is Sierra Freeman. I am more or less, right, the cultural organizer and strategist for the Brewerytown Sharswood Neighborhood Coalition. But more importantly than all of that, I am a team member of BSNC and a neighbor of the 29th Ward that sits within the 5th District of the city of Philadelphia. Irresponsible tech and AI usage separates us from our neighbors, from our humanity, our health and wellness and our future's well-being. Instead of neighbor-created and led town watches, we had Ring cameras directly accessible to the police, now accessible to them if you fill out their online form. PPD wants to expand its usage of drones and make them first responders in some instances and hire more pilots directly from high school. Automation of tasks, automated decision-making and direct access to minds of our youngest adult neighbors isn't something we need in the hands of AI law enforcement. It takes humanization, which is the exact opposite of AI, to make decisions at any given moment when people's lives and livelihoods are at stake. Neighbors taught to be fearful of each other with biased AI detecting threats, watching from behind technology instead of standing on their stoops getting to know each other. There's no more watching out for each other as neighbors. There's no more watching out for our children as a community. We're just watching with integrations like Alexa and Echo, feeding us back our words and phrases with products for us to buy, keeping our already financially vulnerable neighbors in an endless capitalist consumer salute. We're paying with our children. The Stanford School of Medicine's Brainstorm Lab for Mental Health Innovation partnered with Common Sense Media to create test accounts simulating 14-year- olds in order to evaluate how AI companions from three chat bot developers interact with young users. Their findings revealed that with minimal prompting, chatbot from Character AI Nome and Replika engaged in conversations that could be harmful to mental health. Nonprofits or quasi government institutions given access to us by way of contracts voted on and implemented by city administration with their mundane usage of AI for notes, scheduling, info, synthesizing and many other things. These things that may not be and most likely wasn't listed in any clauses and/or use or contracts with our city or our neighbors directly, while as of October 2025 Pennsylvania still does not have a comprehensive consumer data privacy law. We are grateful that in October 2024 Philadelphia passed a ban on the use of algorithmic rent setting software by landlords addressing AI use that was harmful. Thank you again, Councilman O'Rourke. We're not happy that AI is still a major player in raising the cost of everyday living. More access to AI has meant proposing data centers that are harmful to the environment being built in our backyard. And we don't just pay for them with our health and wellness. We're charged a premium on our PECO bill to add another financial hardship that subtracts years from our lives, not to mention surge pricing driven by algorithmic AI and facial recognition at supermarkets like Walmart and Kroger. As BSNC is a part of the Philly Tech Justice Coalition, we're paying attention as a collective deeply rooted in community and we're watching. As Brewerytown Sharswood Neighborhood Coalition entrusted to be an advocate for our neighbors in this time, we're doing what we can in our neighborhood with and for our neighbors. It starts here. In all of what I listed, they're only the things that we know so far. We're demanding transparency, accountability, oversight and community collaboration. Philadelphia is a city of neighborhoods. Our neighbors are the lifeblood of this city. Responsible AI oversight means a duty to preserve and protect our neighbors humanity. Thank you. (Applause.)
My name is Thomas Noah. You can probably tell from my outfit that I'm an academic. So I'm a lecturer for the School of Social Policy and Practice over at Penn, teaching a course right now, Democratizing Data? Critical Data Studies in Algorithmic Governance. So I want to reiterate the idea that there are no experts on this subject matter is an absurdity, because I assume that Penn is not instrumentally irrational in paying my salary. I'm not here to speak on behalf of Penn. I do not speak on behalf of Penn. I do not speak on behalf of the School of Social Policy and Practice. I speak as a community member who has a house in Point Breeze at the intersection of Pierce and Point Breeze Avenue, right across from the mosque. And just in keeping with the theme from my class, I hope we take seriously what my brother is saying here, that we appreciate these efforts to try to democratize data, but it can't be Kabuki democracy. It has to listen to the community members. It has to serve their interests and their needs and the attempts by the speakers from the Admin to continue to shield certain aspects of their behavior and practice don't lend themselves to democracy and community control and community-led efforts. People in the community know what they need. My sister right here knows what her community members need. There's a real danger in social, economic, political elites like us coming in and telling other people how they should live their lives, doing interventions, putting technologies on people that they don't understand, that they haven't consented to, that they haven't authorized. And so, I just ask that, you know, going forward with what you intend on doing, we remain mindful about this and we take our commitment to democracy seriously because it's under attack domestically and internationally, and we have the opportunity to be the change that we hope to see. Thank you.
Yes. Good afternoon. My name is Miguel Willis. First, thank you for having us. I'm also a lecturer at UPenn as evidenced for my outfit, and the Director of the Responsible Computing for Just Futures Initiative, which aims to equip the next generation of technologists, lawyers and policymakers to think critically about how technology is designed, deployed and governed, so that it promotes equity rather than entrench inequality. My words are my own today and not the official position of the university. I'm speaking today as a resident, a new father, as someone who cares deeply about the future of the city.
You know, I don't need to reiterate what's been said already in terms of what we're seeing amongst the growing adoption of AI across all systems and data- driven tools across city agencies. We heard comments and comments on how the lack of accountability, public transparency oversight and how these tools risk replicating the same structural inequities that we've been working so hard to dismantle. But two critical areas I want to highlight: First is the secondary use of open data that the city collects for one purpose, but it's later repurposed by private AI firms or shady data brokers for another, right. When residents consent to their collection of their data, for example, through 311 complaints, transit cards or school enrollment systems, as someone testified earlier it's often with the understanding that this data serves a specific purpose. But shady data brokers routinely retain, aggregate and repurpose this data indefinitely, using it to train predictive algorithms that the public never consented to and cannot meaningfully challenge. This practice erodes the foundation of public trust, undermines informed consent. So we want to confirm -- curb the secondary use of Philadelphia's -- to help ensure and protect Philadelphia's privacy, assured this data is not weaponized against the very communities that it's meant to serve. And equally important, and which has been reiterated, is the need for transparency and community accountability of how AI is designed, procured and deployed. Historically, decisions about what technologies to build have been made by experts like myself, city officials, academics, researchers, without significant input from the communities most impacted, again most impacted, those are the experts by these technologies. To rectify this, it's essential that we elevate community voices and ensure they are central in the decision-making process through participatory design and community-led processes where residents, community organizations and advocates are central and involved before the technologies are purchased, designed, thought of. Many of the groups here today like the Philly Tech Coalition, 215 People's Alliance and all the other community advocates are already modeling what this looks like, grounding conversations about AI and lived experience in equity and in democratic governance. The city should invest and learn from these efforts, rather than outsourcing decision- making to unidentified governance systems or opaque corporate systems. Ultimately, the question is not whether Philly will use AI, but who will it ultimately serve and how will that be humane. So I'll close by saying let's just make Philly a model and not replicate the mistakes from the past. So thank you all. (Applause.)
Good afternoon. Thank you for doing this. I'm not entirely sure how my name ended up on the list, but that being the case I don't have prepared remarks. I came to listen, but I'll speak to the issue. So my name is Karl Morris. I am a computer science professor at Temple University and I showed up today because I was of course deeply interested in understanding what the city government was doing with regards to its use of artificial intelligence, both in terms of its own operations and how it intends to use it to disseminate its services to the residents of Philadelphia. I think we need to contextualize so many things in this conversation, but we can start by looking at the history, understanding the demographics of this city, recognizing how different groups historically have been -- how they've been treated overall, whether it's how they've been ignored and put aside when we look at things like the digital divide or access to high speed internet, whether we look at the ways that technology has harmed certain communities, very recent examples, for example, during COVID where we saw things like pulse oxometers giving poor readings or incorrect readings for folks with darker skin, leading to deaths and harm or as we see again, just looking at emerging technologies generally. We see self-driving cars are now in the city and we recognize that while these vehicles have lower accident rates than vehicles driven by humans. When they do have accidents, they tend to be or they tend to involve folks again with darker skin. So we know that technologies, emerging technologies, can in fact harm certain communities while helping others and ultimately, their use can benefit one group while exploiting other groups. We heard earlier testimony about, for example, when data centers are being built or the power plants that operate them where are they placed. It's often times in communities where folks are, whether they're low on the socioeconomic ladder, whether they are racial minorities, are ultimately people who don't have a voice. They're not able to speak up. So what we need to ensure is that the Council in providing oversight in the operations of the city, they consider these things, they look at the historical context, they look at the groups that tend to be served and groups that tend to be harmed and ensure that with input from residents, with input from stakeholders and the community, we ensure that while, yes, the city will utilize and benefit from artificial intelligence on the whole, we want to make sure that no 9 one is left behind and no group is exploited. So thank you. (Applause.)
Thank you. Mr. Serra, is there anyone else for public comment?
Last speaker. Thank you very much. Please state your name for the record and proceed with your testimony.
Hello and thank you so much for allowing this forum to take place. My name is Roselle Kenney and I'm 67 years old. Picture this: A 72-year-old woman finished her shopping at 23rd and Walnut Street. She walks outside and suddenly has no idea where she parked. minutes later, she 13 was still wandering and embarrassed 14 and anxious. Finally, she has to 15 call her daughter. But here's the 16 thing, her phone could have found 17 her car in 10 seconds through AI, 18 whether she had Bluetooth, Google 19 Maps, Google Play, but she didn't 20 know, or how about this: We've all experienced this if you have parents. A senior leaves their doctor's office clutching a medical report. He sits down to read it. It might as well been written in Mandarin, hypertension, glucose levels, contraindications. And when the doctor did explain it, it sounded like the teacher on Charlie Brown, womp, womp, womp, womp. He's too embarrassed to call back, but AI, AI could translate that report into plain English in seconds. Answer his questions as many times as he needs, no 13 judgment. This happens every day in Philadelphia. Not because seniors can't learn, because nobody thought of including them. And I noticed in this whole entire thing no one mentioned senior citizens. Now, let me be clear, can seniors learn AI? Can we master new technology? Of course, we can. We lived through rotary phones, ATMs, cassette tapes, 8 track tapes, VCRs, personal computers, cell phones and Uber. We adapted to every technology revolution for centuries. So don't tell me we can't handle AI. We just need a seat at the table. Philadelphia has over 250,000 residents over 60 years old. That's 1 in 6 people. When we plan for AI without them, we're not just leaving people behind. We're leaving our parents, our grandparents, our neighbors, the people who actually built Philadelphia. And here's what's at stake: AI is already changing health care, banking, transportation, government services. Seniors who don't understand these tools lose access. They get isolated. They fall for scams. But when we do this right, amazing things can happen. Tools like sit-and-pee apps help seniors to monitor their health, prevent infections, find a bathroom. AI can read prescription labels out loud, find bus routes, provide companionships. One of the biggest problems with seniors now is loneliness. It can detect falls, call for immediate help. So here's what I'm asking: One, put senior representatives on your AI planning committee; two, fund training programs and libraries and seniors and senior centers; three, require that AI tools are designed with older adults in mind, not as an -- I hate to say here -- afterthought; four, protect seniors from AI-powered scams. Our elders built Philadelphia. They deserve to benefit from this technology, not to be left behind by it. Let's make this city a model for how we bring every generation into the future together. We do have wisdom. Thank you.
Thank you very much. Thank you for stepping up. We really appreciated your testimony. Okay. Seeing there are no additional people for --
Okay. Thank you. Please state your name for the record and proceed with your testimony.
Sure. My name is Vincent Perez. I was raised in an underserved community. Although I was given an opportunity in high school and that led to a career in investment banking, later becoming an entrepreneur, strategic advisor and have been a professor for over 20 years. I'm also a founder of Longview Partners where I select who I want to help, what are venture institutions, I designed a strategy and increase their impact, public, private, academia. I've taught business and innovation for over years, most recently at 8 Penn State where I serve on the 9 Teaching and Learning with 10 Technology Faculty Advisory 11 Committee and I champion Invent 12 Penn State's AI initiatives. 13 I'm also part of the 14 Philadelphia AI Strike Force and 15 the Vice-chair of the Robotics 16 Coalition here in Philadelphia, 17 we're expanding AI and robotics 18 access for students from cradle to 19 career. There are no true experts 20 in AI right now. The field is evolving faster than anyone can imagine. But those of us on the Strike Force, we're on the cutting edge across academia, industry, government. We just had a meeting last night with two dozen folks and we're making sure that we want to keep an eye on Philadelphia and make us a hub of AI and robotics. With our talent, university, civic partnerships, Philadelphia can become a natural model for responsible and an inclusive AI innovation that is built on trust, opportunity and growth. Thank you for your time.
Is there anybody -- thank you very much. Appreciate all of your comments. Anybody else here for public comment? (No response.)
Hearing none, that concludes our hearing. I want to take just one last second to tell people again I'm Councilmember Rue Landau. I am the Chair of the Technology and Information Services Committee and I deeply appreciate everybody who came to speak to us today. It is more clear than ever to me and to my colleagues that we have all the experts we need right here in Philadelphia, and we need to tap you to make sure that you are providing your wisdom, your insight in this process as we move forward. So this will not be -- we are going to have a series of AI hearings through this Committee and look forward to hearing your input going forward as well. Thank you. (Applause.)
I have a paragraph I have to read everyone. One second. While I'm looking for my official paragraph, I will tell you that this is the first time Philadelphia has ever had a hearing on something like AI. We haven't been using our Technology Committee very much in the past year. So really look forward to opening it up to all of the new and pressing concerns as we go forward. This concludes the public hearing of the Committee of Technology and Information Systems. This committee will stand in recess until the call of the Chair. Thank you all very much. (Applause.) (Committee on Technology and Information Services concluded at 1:19 p.m.) C E R T I F I C A T I O N I, hereby certify that the proceedings and evidence noted are contained fully and accurately in the stenographic notes taken by me in the foregoing matter, and that this is a correct transcript of the same. __________________________________ TANEHA CARROLL