COUNCIL OF THE CITY OF PHILADELPHIA COMMITTEE ON PUBLIC SAFETY Room 400, City Hall Philadelphia, Pennsylvania Tuesday, June 8, 2010 1:50 p.m. PRESENT: COUNCILWOMAN DONNA REED MILLER, CHAIR COUNCILMAN DARRELL CLARKE COUNCILMAN WILLIAM K. GREENLEE COUNCILMAN JAMES KENNEY COUNCILMAN FRANK RIZZO, JR. RESOLUTION 100263 - Resolution authorizing City Council's Committee on Public Safety to hold a public hearing to investigate and determine the cause of the repeated failures of the City of Philadelphia's Police Department Radio System. RESOLUTION 100272 - Resolution authorizing Council's Committee on Public Safety to hold public hearings for applicants seeking nomination to serve on the Police Advisory Commission... - - - 2
Hi. Good afternoon, everyone. The City Council Committee on Public Safety is in a public hearing to testify on Resolution 100263 concerning the police radio system. Before we begin, however, we are going to recess briefly and go into a public meeting so that we may consider the Committee's recommendations with respect to Resolution 100272 concerning the Police Advisory Commission. Joining us today in this afternoon's meeting are Councilman Frank Rizzo to my left, Councilman Bill 16 Greenlee to my left, Councilman Darrell Clarke to my left, and I am Councilwoman Donna Reed Miller, Chair of this Committee. We will now ask the Clerk to read the title of Resolution 100272.
Resolution 23 authorizing Council's Committee on Public Safety to hold public hearings for applicants seeking nomination to serve on 3 6/8/10 - PUBLIC SAFETY - RES. 100263 the Police Advisory Commission, and further authorizing the Committee to provide a report to the Committee of the Whole setting forth its recommended list of nominations for submission to the Mayor.
Thank you. We have circulated to members of this Committee a report setting forth recommended nominees for eventual 13 submission to the Mayor. Before I ask 14 the Clerk to read the report into the 15 record, I would like the record to 16 reflect that the pool of applicants was 17 very, very impressive. There were 50 18 applicants seeking 21 positions. We were 19 blessed with a variety of talents, 20 experiences and remarkable diversity, and 21 with applicants who showed a great willingness to contribute to this exceptionally important commission. I was honored to participate in this process. I believe that my 4 6/8/10 - PUBLIC SAFETY - RES. 100263 colleagues share that feeling. I particularly enjoyed personally welcoming and hearing from many of the applicants at our public hearing in this Chamber last week on June 1st. So after due deliberation and this Committee having had the opportunity to review the initial applicants, along with the written testimony and letters of recommendations, I would ask the Clerk to read the report into the record.
Committee on Public Safety report on Resolution 100272. We hereby submit to the Committee of the Whole the following recommended list of nominations for the Police Advisory Commission for submission to the Mayor pursuant to Executive Order 8-93: Terrance H. Barnes; Rachel E. Branson; Jamira Burley; Ronald A. Burton; Devon Cade; Robert S. Cavanaugh; Mujeeb Chaudhary; James C. Crumlish, III; Rodney Lee Dunn; Cheryl L. Haynes; Cassandra C. Hines; Edward James, Jr.; Keith E. Kunz; 5 6/8/10 - PUBLIC SAFETY - RES. 100263 Elizabeth Mell; Andrea Nicole Phillips; Curtis M. Pontz; Oscar Rosario; Robert P. Shine, Sr.; Joi C. Spraggins; Joseph T. Stapleton; and Louis Troup, Jr.
Okay. Councilmember Greenlee, could I please have a motion pursuant to Resolution 9 100272 reporting the Committee's recommendations for the Police Advisory Commission.
Thank you, Madam Chair. I move the adoption of the recommendations.
Thank you. The report has been adopted with a favorable recommendation. 6 6/8/10 - PUBLIC SAFETY - RES. 100263 Councilmember Greenlee, may I please have an additional motion instructing the Chief Clerk to distribute copies of this report to all members of City Council at the close of this meeting.
The motion has been adopted. The next step is that our recommendations will be provided to Council's Committee of the Whole, which will consider our report. If the Committee of the Whole concurs, it will then recommend that the full Council 7 6/8/10 - PUBLIC SAFETY - RES. 100263 submit these names to the Mayor. If the Council concurs, the Mayor will then select seven Commission members and two alternatives from that list. This concludes our public meeting, and we will now return to our public hearing on Resolution 100263 concerning the Police Department Radio System. Will the Clerk please read the title of the resolution.
Resolution 100263, resolution authorizing City Council's Committee on Public Safety to hold a public hearing to investigate and determine the cause of the repeated failures of the City of Philadelphia's Police Department Radio System.
Thank you. Will you please call the first panel of witnesses forward.
Panel No. 1: Joseph James, for Verizon Douglas Sullivan and Douglas Smith, and for 8 6/8/10 - PUBLIC SAFETY - RES. 100263 Motorola Jacqueline Wasni. (Witnesses approached witness table.)
Hi. Have a seat. Councilman Rizzo has some comments before we start your testimony, but don't forget to pull the mike up close so we can hear you and state your name and any other identifying information you'd like to state. Councilman Rizzo.
Thank you, Madam Chair. And, Verizon, Motorola, thank you for being here today. Good afternoon, Madam Chair and fellow Committee members. The purpose of this hearing this afternoon is to investigate and determine the cause of the recent problems associated with the police, fire and municipal radio system. The incidents in question occurred on March 24th, April 12th and April 13th, respectively. 9 6/8/10 - PUBLIC SAFETY - RES. 100263 Even a single incident is unacceptable. The safety and well-being of residents and visitors alike, as well as the brave police officers and firefighters who place themselves in harm's way each and every day are paramount. After a history of such radio communication problems going back a number of years, members of both Police and the Fire Departments have had their confidence in the radio communication eroded. It is critical that this is resolved. We look forward to hearing from all relevant parties today who will explain the cause of the incidents but, most importantly, what is being done to prevent any such occurrences in the future. Welcome, Allan Frank, Chief Technology Information Officer, Joe James, one of his deputies, and Deputy Mayor Gillison, our Public Safety Deputy 10 6/8/10 - PUBLIC SAFETY - RES. 100263 Mayor overseeing Fire and Police. I appreciate your presence here today. Mr. James.
We appreciate your presence. So all three of you guys want to testify or you're there to answer questions?
I'm here to answer questions. Joe James will be giving testimony.
Okay. All right. Thank you. Proceed with your testimony. DEPUTY COMMISSIONER JAMES: Good afternoon and thank you. My name is Joseph James, Deputy Chief Information Officer for the Division of Technology. Also with me is Allan Frank, the Chief Technology Officer, and Deputy Mayor Everett Gillison and George Westerman, the 800 megahertz system manager. I'm here today to discuss the City of Philadelphia's 800 megahertz digital radio system, the circumstances 11 6/8/10 - PUBLIC SAFETY - RES. 100263 that affected the Philadelphia Police and Fire Department March 24th, 2010, April 12th and April 13th, 2010. The City's current 800 megahertz digital radio system was implemented in 2000, and last year began the process to upgrade the system to enhance radio channel capacity, ensure reliable operation and system processing speed. The incidents that occurred are of very serious concerns to the City, Verizon and Motorola. Since the last incident on July 22nd, 2008, Motorola has deployed significant resources to the City to identify the root cause of trouble and implement a solution. The 800 megahertz digital radio system is designed and constructed as a very complex radio system and is dependent upon the Motorola radio infrastructure and the Verizon telecommunications network. Since the last incident, there has not been a similar incident or a 12 6/8/10 - PUBLIC SAFETY - RES. 100263 repeat of circumstances that occurred, and the incidents in March and April have no relationship to the past issue. The incident on March 24th has been identified as a failure of Verizon network equipment which affected both the Philadelphia Police and Fire Department. The equipment failure caused eight of the City's ten radio tower sites to go off the air, disrupting the ability of users to communicate. The maximum amount of downtime was minutes. 14 The incident on April 12th 15 appears to be a shared outage of Verizon 16 and Motorola equipment that affected the 17 Fire Department North Fire console for 18 approximately 24 minutes. The fire 19 dispatch was hearing tones that impaired 20 their ability to use the primary dispatch console and required the dispatcher to move to a back-up console to communicate with units in the field. Finally, the third incident on April 13th was a Motorola-specific issue 13 6/8/10 - PUBLIC SAFETY - RES. 100263 caused by a damaged radio antenna that caused the Victory Garden site to be off air, so when Motorola technicians attempted to do a preventative maintenance task at that 31st and Grays Ferry site, it affected police units' radio coverage in the southern part of the City. The Grays Ferry site needed to be taken off the air to reprogram and the Motorola team expected the Victory Garden site to provide overlapping coverage, but they were unaware of the damage to the antenna at the Victory Garden site. The incident caused a 13-minute loss of radio coverage for only police units dependent upon those two sites for coverage. Once the Motorola team was made aware of the loss of coverage, the Grays Ferry site was restored and subsequently repair was done to the Victory Garden antenna to ensure complete overlapping radio coverage. In conclusion, as in the past, what we have learned is the need to 14 6/8/10 - PUBLIC SAFETY - RES. 100263 constantly train all users in the fail-safe measures built into the design of the system and conduct specific training for police and fire personnel on the radio systems back-up features. Both Verizon and Motorola responded quickly in reacting to the incidents, and years of experience working together were evident in the short duration of the incidents. Since the incident occurred, the City team is reviewing all aspects of the current and new system in an effort to anticipate, prevent and react to incidents that may occur in the future. The Division, Verizon, along with Motorola, do not believe the radio system poses a risk to City users. I'm now available to answer any questions you may have.
I think one of the points that has to be made -- and I think Deputy Mayor Gillison is the 15 6/8/10 - PUBLIC SAFETY - RES. 100263 appropriate person or Commissioner Gaittens -- some of the things that happen behind the scenes when there's problems with the system, I think it's important that we for the record indicated that it costs the City a lot of inconvenience, not just the danger associated with the lack of the communication system working properly, but if you could just explain for the record what happens and what the Police Department and Fire Department do when the system is acting up. DEPUTY
I'll be glad to, and you are absolutely right, Councilman. Again, thank you very much. Everett Gillison, Deputy Mayor for Public Safety. As I have made clear, whenever there is a problem with the radios, the Police Department especially immediately recalls all single-person units and immediately begins to double up. That causes us an operational impact that is 16 6/8/10 - PUBLIC SAFETY - RES. 100263 well in excess of whatever time the radio is down. So even though directly the radio might be out for ten minutes, 5 minutes, by the time we recall all 6 officers from the field, put them in 7 two-person cars and then go through our 8 back-up procedures, we could actually be 9 in that mode for up to eight hours, a 10 full shift. 11 I'll actually defer the fuller 12 explanation to Deputy Commissioner 13 Gaittens, because I'm sure he can 14 actually give you the hands-on of what 15 the procedures are, but I can tell you, operationally it is a nightmare.
And also I don't think we need to get into the real detail of it, but I think that, from what I've learned, that it slows down the 9-1-1 response -- DEPUTY
-- because you're down to half as -- 17 6/8/10 - PUBLIC SAFETY - RES. 100263 DEPUTY
Two officers have to show up when normally it would be one, and then the backlog starts. It's a significant problem. Like you used the word "nightmare." People don't understand why when they call 9-1-1 there's a delay in the response. DEPUTY
That's correct, and one of the things, as we all are aware -- and I know that we have made it very plain and very simple. When these things happen, we go by our protocols. Our protocols call for us to double up, to make sure that the safety of the officers is paramount, the safety of the firefighters are paramount. We do whatever we can to remedy this, and we try to make sure that the communications are clear so that when we do get the "all clear" sign, that we don't have to keep the officers in. But as you're aware, there has 18 6/8/10 - PUBLIC SAFETY - RES. 100263 been a history, so there is this confidence issue that we're trying to build up again. But it does take the Commissioner some time to remedy and now a double-up situation. And it's not just turning it over and saying that we're going to -- Okay, all clear, let's now stop the bringing them in, let's stop the double-up. It takes time to bring everyone in, re-get them back out and push them back out on another time.
And plus the fact that you want to be confident that whatever the problem was -- DEPUTY
-- has been resolved and back to normal operations. Mr. James, I know that we're in the midst right now of an upgrade and I'd like to know, in our meetings you explained to me, which I understand, but, again, for the record, that you believe that this upgrade will be very helpful in 19 6/8/10 - PUBLIC SAFETY - RES. 100263 resolving some of these issues that we have experienced, and that's a good thing and I think people need to hear that. DEPUTY COMMISSIONER JAMES: Thank you very much, Councilman. Yes. The intent of the upgrade is to upgrade a ten-year-old radio system that was designed with a certain amount of technology and infrastructure, which today with the newer systems is going to provide the kind of reliability that we expect the radio system to have. Some of the inherent features in the system which are not features that are seen by many of the users happen in the network behind the system, and some of the problems that we've experienced back in 2003 and 2004 with the controller rolling will be avoided with the redundancy and the reliability of the newer equipment, as well as we have a ten-year infrastructure that's time for it to be replaced. And the technology has advanced, so it's more along an Internet-based protocol, which 20 6/8/10 - PUBLIC SAFETY - RES. 100263 provides a certain amount of efficiency of the ability. It can route those messages and communications, which in the past would be reflected in officers getting busy signals or bonks because they're not able to communicate. The system is much faster. It's much more reliable, and I believe it's going to provide them the types of system that we can look to to get another ten years out of.
Well, I think that this time around I'm seeing a different relationship between the people that make this system work and to also support it. And I understand that there's even the technology -- Mr. Frank might want to elaborate on it, but I understand that we might even go into a new area, if you permit me to discuss that, where the T1, the telephone company lines, will be the secondary, the backup, and microwave will be the primary. And I think that the users need to hear this, 21 6/8/10 - PUBLIC SAFETY - RES. 100263 because you're doing everything humanly possible to avoid the problems of the past by using some of the technologies that have developed over the past ten years that will make this a better system. DEPUTY COMMISSIONER JAMES: That's correct.
This is about making the people that are using the system, our police and our firefighters and other municipal workers, have confidence, like Mr. Gillison just said, again in their communication system, and I believe that from what you've told me privately and now publicly, that even some of the additional enhancements will do just that. Mr. Frank.
Yeah. This is Allan Frank, Chief Technology Officer. Yeah. I independently looked -- I wanted to also hear the 22 6/8/10 - PUBLIC SAFETY - RES. 100263 explanations and feel that we're solving the right problems with correcting and the replacement of the system. From my perspective, when we talk, for instance, about these T1 lines and the need for secondary connections between antennas and our sites, this really, if you think about it, we're replacing the guts of a radio, and the things that are falling apart that cause, in my view, the situations last month were pieces not associated directly with the radio itself that's being upgraded. Lines go down. And so one of the things that we did differently is besides just relying on what Motorola is doing or Verizon is doing in terms of the new upgrade, we, both Deputy Mayor Gillison and I, wanted to look at what were all the possible weak links that could cause any sort of disruption, not just the radio itself, and what we in fact found was the need to have these secondary back-up lines from the Verizon 23 6/8/10 - PUBLIC SAFETY - RES. 100263 T1 lines to microwave. And, again, that is the weak link. And I know there's been a lot of technical jargon about antennas. The bottom line is, it looks like because of the winter, one of the antennas at the base cracked, effectively, and because we have redundancy, frankly, with how our antennas work, we didn't know this one was actually down at the time. So my point is is that I think the big difference since I've come into the City, not only have I seen obviously everybody's working together, is that we now also have focused on not just the radio as the weak link but all the crazy parts around it and making sure immediately that we're remediating that. So with the turn-on of the new radio system, we're also making sure the antennas, the connections and anything else that wasn't directly a part of the Motorola radio itself.
Well, thank 24 6/8/10 - PUBLIC SAFETY - RES. 100263 you very much for that, and we're looking forward to having those enhancements. Thanks, Madam Chair.
Thank you, Councilman Rizzo. I do want to note that Councilman Jim Kenney, a member of this Committee, has joined us in the hearing. Since we purchased EarthLink's equipment or wires or fiberoptic, whatever we did, how will this help or will it not even have an impact on police radio?
Well, we obviously haven't technically purchased it yet, but over a period of time, there's a couple of pieces that we look at. First of all, there is a redundancy. It's a data communication network, so that we would have any number of options on how we could use that data communication network to directly have a conversation between any two points. I think that in itself -- we obviously have some work to 25 6/8/10 - PUBLIC SAFETY - RES. 100263 do, but, I mean, that in itself gives us a level of redundancy. And anybody that even at home, you have people that are able to listen to the radio over the Internet or voice, we can use it much the same way. So for us, it is another form of communication, very critical communication, that the police and other public safety entities would be able to use.
And I just want to reemphasize, I pretty much feel that when we go to use our phone, we want it to work and particularly in a situation of emergency. So I just agree that the Police Department and the Fire Department needs to have that confidence that when they need their radio, it's going to work. You did say that the weather caused structural failure at one of these yards or one of the towers. How routinely are they inspected? 6/8/10 - PUBLIC SAFETY - RES. 100263 DEPUTY COMMISSIONER JAMES: The antennas, based on the experience that we've had, they're inspected twice a year, and there's also monitoring equipment that's at the site to monitor the antennas to determine its ability to work properly. This particular incident was one of those unique situations where even the monitoring equipment that was in place to monitor the usability of that antenna did not detect that there was a structural failure. And so we sweep the antennas twice a year, and just in our discussions when we talked about it being preventive and being more proactive, based on this kind of weather that we had this past summer (sic), we're probably going to change that depending on the weather conditions so if we have the kind of storms that we had this year, we'll probably even do more preventive maintenance on those antennas just to make sure the condition of them are 27 6/8/10 - PUBLIC SAFETY - RES. 100263 operating.
If I could add one point. That was a question that we asked. Twice a year. And then I asked, Well, what dates. And we got hit with an incredible winter. We ended up doing it in December, which is kind of at the front end of the winter. So even these little things make a difference, as Deputy James said, is that going forward, we will still continue to have these multiple times per year tests, but in the case of a major event like we've had, we'll go out and do more checks.
Probably like if we have a snowstorm or a series of snowstorms like we did this past winter, probably like right after a snowstorm. I mean, the snowstorms cause a lot of damage to lots of things. One year I had an ice storm actually crack a swimming pool and caused about $750,000 worth of damage. So I guess we learn as 28 6/8/10 - PUBLIC SAFETY - RES. 100263 we go along, but prevention is best, if we could set up a system where we're out there checking to see what's going on.
I think it's also important to note that the life of these antennas -- we also said, Well, how long do they last. Guess what? The answer is ten years. So the fact is is that we happen to be at the point where we are replacing radio and now we got to replace everything else.
So when you say you're going to upgrade, that sounds to me -- and you may have answered this in your testimony, but when you're going to upgrade, does that mean that you're replacing the entire system or you're going to, what, just replace all the parts that need to be replaced? DEPUTY COMMISSIONER JAMES: It was largely replacing the internal parts from the controllers and most of the equipment that is out at the tower sites. The parts that we're talking about right 29 6/8/10 - PUBLIC SAFETY - RES. 100263 now were not envisioned as part of that upgrade as far as the antennas. So that's one of those things that kind of has grown out of this discussion, is that the original plan was to replace everything. That really did not fall within the realm of what we were going to need or afford. And then the second thing was, well, now that we've had this kind of experience, let's look at, as Mr. Allan said, all of the other pieces that don't appear to be that critical as a failure point right now, but could be a potential failure point and now we got to put a plan together to replace them.
So I think for the Council -- I mean, this is a big point. I think we've all been focusing on the guts of the radio, and we'll get that level of security and stability. When in fact it's kind of like I had a V-8, what's the life of the antenna, where are these other pieces. Even the air conditioning that requires -- and even 30 6/8/10 - PUBLIC SAFETY - RES. 100263 the battery back-ups have a life of ten years. So what Deputy Mayor Gillison and I have done is gone through all of these line of questioning and basically made the decision that we really want to make sure that all the antennas and all these other kind of ancillary pieces that are also at end of life are dealt with concurrently now with the radio upgrade, because, I mean, clearly these are all mechanical parts that are at end of life.
All right. Thank you. Any other questions for these witnesses? (No response.)
Panel 2: Verizon, Douglas Sullivan and Douglas Smith, and also from Motorola, Jacquelyn, will you please come up. The three of you can 31 6/8/10 - PUBLIC SAFETY - RES. 100263 come up together. (Witnesses approached witness table.)
Hi. Good afternoon. Please state your name for the record and speak directly into the mike.
Welcome. Thank you for coming. You can proceed with your testimony, and we can get Verizon to go first and then we'll have Motorola. Thank you.
Thank you, Madam Chair. My name is Doug Sullivan. My business address is 7550 Teague Road, Second Floor, Hanover, Maryland. I am a Regional Vice-President for Customer 32 6/8/10 - PUBLIC SAFETY - RES. 100263 Service for Verizon Business, and I'm responsible for providing customer service for over 1,000 large business and institutional customers throughout North and South America, including the City of Philadelphia. My team's responsibilities include service management, technical service management, and project management, and service ordering. I have years' experience with Verizon and 12 assumed the current responsibilities in 13 January of 2008. 14 I've been asked to appear here to address recent incidents that have impacted the operation of the City of Philadelphia's 800 megahertz emergency dispatch radio system. The radio system itself, as you know, is provided by Motorola. Verizon provides the City with a high-capacity and fiberoptic telecommunication services that support the radio system. I'd like to first provide you with a high-level description of the 33 6/8/10 - PUBLIC SAFETY - RES. 100263 Verizon support network design and configuration. The Motorola equipment utilized for the City's 800 megahertz system is connected throughout the City via Verizon's high-capacity circuits. These circuits, commonly referred to as T1s, feed off of fiber-based technology backbone network known as SONET rings. This SONET fiber backbone is engineered in a ring configuration in order to provide survivability against service interruptions in the event of a fiber cut or other type of impediment. If a cut or other problem occurs which prevents traffic from routing in one direction of the ring, the system is designed to automatically reroute traffic in the other direction so that the transmission of the communication is not interrupted. In addition, SONET rings guard against service interruptions which might otherwise result from a transmission failure at one location or node of the ring. Therefore, if one node goes down 34 6/8/10 - PUBLIC SAFETY - RES. 100263 and does not take down the whole ring, the SONET ring will retain its transmission functionality despite the loss of node. The Verizon network architecture and equipment providing connectivity for the City's 800 megahertz radio system are comprised of two SONET rings and 90 T1s. Thirty of these 90 T1s utilize public network infrastructure in addition to the SONET. Verizon has deployed the same SONET technology for many of today's fortune 1000 networks with great success and proven operational reliability. The SONET fiber technology used for the 800 megahertz system was originally put in place by Verizon to provide data connectivity to hundreds of City of Philadelphia sites and has been in operation for many years. The City's SONET fiber infrastructure was expanded to incorporate the T1 circuit requirements for the Motorola-provided 35 6/8/10 - PUBLIC SAFETY - RES. 100263 800 megahertz system. Verizon's SONET networks are proactively monitored hours a day, 5 seven days a week. Our service 6 performance for these types of networks, 7 including the one provided to the City, 8 is excellent and reflects the fact that 9 we are extremely vigilant and proactive 10 in quickly investigating and fixing even 11 the smallest anomalies that are detected 12 in the network, even ones that are not 13 service impacting. 9 percent 19 for the past six months. 20 With this information as a 21 backdrop, I will now address the two 22 recent 800 megahertz system service 23 incidents that involved Verizon. Verizon 24 has provided the City with detailed root cause analyses that explain each incident 36 6/8/10 - PUBLIC SAFETY - RES. 100263 and include chronologies of all pertinent events. I will provide a brief overview of each incident, and then we'll be happy to address any follow-up questions that you may have.
, multiple T1 circuits supporting the 800 megahertz radio system were reported as being out of service, causing an initial outage of approximately ten minutes. Ultimately, of the radio system's remote tower 14 T1s were affected. Verizon isolated the 15 trouble to a network timing distribution 16 module card in the Verizon Locust Central Office K34 Electrical Digital Access Cross Connect System, or DACS. The National Transport Network Operations Center, the Locust Central Office and Advanced Technical Support were all engaged to diagnose the issue. While the card that failed did restore, Verizon nonetheless identified a spare card and had it sent to the Locust Central Office 37 6/8/10 - PUBLIC SAFETY - RES. 100263 for replacement. m. that same evening. The card was replaced approximately one hour later and all associated alarms were resolved. On April 12th, two T1 circuits between the City and the Fire Administration Building were reported as out of service. Verizon technicians were dispatched to both sites, and their investigation found a possible timing issue with the T1 circuits. On these circuits, timing is provided by the City-supplied terminating equipment. With the City's consent, its equipment was disconnected in the T1 -- so the T1 circuits could be intrusively tested. Upon insertion of the technician's T1 test equipment into both CSUs, or Channel Service Units, all alarm indications immediately cleared. Head-to-head testing was performed on the circuits and 38 6/8/10 - PUBLIC SAFETY - RES. 100263 they were found to be error free. Throughout the testing process, no 4 trouble was found within the Verizon network on either circuit, and upon reconnection to the City's equipment, the circuit restored and remained alarm free. As indicated by the trouble diagnoses and corrective actions it took in resolving these two incidents, Verizon remains firmly committed to ensuring the City's radio system is as reliable and dependable as possible. We recognize the extreme importance of the critical support system which is vital to the efficient operation of the City's public safety personnel and the protection and well-being of its citizens. As part of the commitment, Verizon has worked previously with the Department of Public Property and more recently with the Department of Technology to provide recommendations to optimize the resiliency and redundancy of the network supporting the radio system. These 39 6/8/10 - PUBLIC SAFETY - RES. 100263 recommendations have included a fully redundant SONET solution that would include the elimination of carrier systems, which have been a common source of failure in the support network. Additionally, Verizon has taken numerous steps to optimize its network in support of our commitment to provide exceptional service and reliability. A complete audit of all T1s supporting the radio system was conducted to ensure adherence to industry standards. Electrical grounding verification was performed at each of the 13 sites in support of the network. Verizon terminating equipment was connected to existing Uninterruptible Power Sources, or UPS, equipment. Four redundant T1s, along with switching gear, were provisioned and installed between City Hall and Domino Lane to provide an additional level of diversity. And, lastly, alarming equipment was set up for the primary and secondary circuits between City Hall and Domino Lane. 40 6/8/10 - PUBLIC SAFETY - RES. 100263 In partnering with the City and Motorola and in an effort to continue to optimize the radio system support network, Verizon is moving forward with providing additional levels of diversity from the Locust K34 DACS equipment, along with providing assigned resources that will review the overall design of the support network to identify any additional opportunities for network hardening. I will now be happy to address any questions that the Committee may have. Thank you.
We'll hear from Motorola and then we'll have questions.
Good afternoon, Madam Chair and members of the Public Safety Committee. Once again, my name is Jackie Wasni. I'm the Territory Vice-President for Motorola Sales and Services in the Midatlantic region, with responsibility for all government and 41 6/8/10 - PUBLIC SAFETY - RES. 100263 commercial business. This includes the City of Philadelphia, which I've had the pleasure of working with since 2005. I was asked and have prepared a statement regarding the recent resolution 7 and radio system incidents which occurred on March 24th, April 12th and April 13th, and with your approval, I am prepared to read these into the testimony.
Motorola appreciates this opportunity to present the facts surrounding the recent radio system incidents. I will discuss what happened, Motorola's response, what we believe caused the issue and the impact to the radio system. I will also discuss performance and operational changes in our response, where warranted, to address these issues in the future. It's very important for me to tell you and all first responders in the City that Motorola has and continues to 42 6/8/10 - PUBLIC SAFETY - RES. 100263 remain committed to supporting the City of Philadelphia and to ensuring that we are doing everything possible to maintain your mission-critical radio system and to improving any aspects that are creating a lack of confidence in the system. As one of the largest cities in North America and one of our largest public safety customers, I want to assure you of our commitment and of Motorola's dedication, resources and support. As you know, Motorola is currently working per a 2009 contract to upgrade the City's radio system. These incidents, however, occurred on the City's Legacy existing system, which was installed in 2000. I'd like to proceed with the incident on March 24th first. On March 24th, 2010, Motorola responded to two alarms resulting from telephone circuit, or T1, interruptions to the radio system. On March 24th, the first impact occurred at 4:00 p.m. when 43 6/8/10 - PUBLIC SAFETY - RES. 100263 eight of the ten radio system remote sites went offline on both the A and the B systems due to a condition we call "site unknown." You have ten sites simulcasting at the same time throughout the City. The impact was reduced coverage in areas of the City. A radio user on the street impacted would see an "out of range" indication on their portable radio. Connectivity was restored after ten seconds and all sites were back online. We did not receive any indication that a user was impacted, and this event went largely unnoticed. Our service team responded to the alarm and worked cooperatively with Verizon to understand that there was a T1 connectivity issue. The same day the second alarm occurred at 10:00 p.m. when once again eight of ten radio system remote sites went offline on both the A and the B systems due to the "site unknown" condition, this time for 44 6/8/10 - PUBLIC SAFETY - RES. 100263 approximately four minutes. After four minutes, connectivity was restored and all sites were back online. Our service team again responded to the alarm and worked cooperatively with Verizon to understand that this was another telephone connectivity T1 issue. Our service team has a close working relationship with Verizon, remained in communication with Verizon and was apprised of their status and their corrective actions. The eight sites impacted were Police Academy, Southampton, Engine 9, Ford and Belmont, 31st and Grays Ferry, Victory Garden streets and highway, and Bustleton, and Hoffnagle. Are there any questions with the first incident, or would you like to save the questions for the end?
Okay. , 45 6/8/10 - PUBLIC SAFETY - RES. 100263 Motorola's Network Operation Center opened a case for primary and secondary T1 connectivity issues at dispatch. Our Network Operation Center is located in Schaumburg, Illinois, and that's at our headquarters where we monitor your system for alarms 24/7. Motorola Service was paged and en route to dispatch by approximately 12:10. At 12:20, we received a service case opened by the Fire Dispatch Supervisor that reported all consoles not working on the north fire band. An audible tone burst was being made over the north fire channel radios and consoles. m. With our service technicians from Motorola already en route, our service manager dialed into the system and determined that the tones could be stopped by resetting a card in the system. This card was the card that controls the T1 link timing. The tone 46 6/8/10 - PUBLIC SAFETY - RES. 100263 burst ended immediately upon resetting this card at 12:32, and the system then returned to normal operation. , a Motorola system technologist arrived on site and inspected the hardware with a focus on the T1 timing cards. He also called Verizon technicians to come to the site. , Verizon arrived and tested the T1 links and found no 12 issues. m. After monitoring the system, the primary card was replaced with a spare card and continued monitoring. All hardware tested is functioning. In summary, Motorola replaced a T1 timing card in the system. There was no impact to the radio system other than the tone burst on the fire consoles. We understand there was no fire in process at the time on the north fire band at the time of this incident and it was limited to the A system. 47 6/8/10 - PUBLIC SAFETY - RES. 100263 The third incident occurred on April 13th when the Police South and Southwest Units reported an "out of range" on their portable radios for approximately minutes. m. during the time Motorola 8 had planned a maintenance repair on the 9 site at 31st and Grays Ferry and planned 10 to take this site off the air for 11 approximately ten minutes. The nearby 12 site, Victory Garden near the airport, 13 had a broken antenna. With one site offline, there's not much impact to coverage due to the overlap of simulcast coverage designed into your system. , police supervisors reported that an "out of range" was 48 6/8/10 - PUBLIC SAFETY - RES. 100263 occurring near the airport. We could not back out of the repair at 31st and Grays Ferry until the maintenance work was finished. m. And at this time, all police portable units reconnected to the system and were operating normally. Upon leaving Grays Ferry, the Motorola maintenance team proceeded to check nearby sites and system alarm information for facts as to what occurred and went to Victory Garden within the hour. Upon inspection, all Victory Garden site equipment was working properly, with the exception of the fiberglass antenna. We found the antenna to be damaged and not working. Stress fractures were found toward the bottom of the antenna and water had collected in the base. The other two transmit and the receive antenna at Victory Garden were not impacted. 49 6/8/10 - PUBLIC SAFETY - RES. 100263 Motorola, as part of our maintenance contract since 2007, is required to check all antenna and lines in the City twice a year. It takes a team approximately four days to check one site. We call this sweeping the site. We last swept Victory Garden site in December 2009, at which time no issues were reported. Between December 2009 and April 2010, the antenna suffered damage. The good news, though, is that due to the overlap in simulcast coverage in the system, with ten sites there was no 15 reported coverage issue prior to the April 13th incident. So to summarize thus far, Motorola was scheduled to take one site off the air and the bad antenna at Victory Garden contributed to a broader coverage issue.
This coverage issue was felt by police for approximately 13 minutes, impacting coverage and presenting an "out of range" to the PD radios on the B system. There was also 50 6/8/10 - PUBLIC SAFETY - RES. 100263 an impact to six fire channels at that site on the fire system A, but due again to our system design and simulcast voting technology, no impact was reported by the Fire Department. m. on April 14th, Motorola's tower crew replaced the antenna at Victory Garden, checked all components and verified that all equipment was working properly. No other issues were found and Victory Garden site was back in optimal operating condition. I would like to speak to the Motorola lesson learned from this incident. To avoid this situation in the future, Motorola is amending its service and maintenance operational procedures to include a five-minute pre-maintenance work outage to test coverage prior to starting a repair. Today, Motorola and the City have an existing procedure established whereby we schedule planned outages by coordinating in advance with 51 6/8/10 - PUBLIC SAFETY - RES. 100263 the City. Also in our existing procedure, five minutes prior to the start of any scheduled maintenance, we verify that we can proceed. We do this to make certain that the activity level in the City has not increased due to a sudden emergency, causing us to postpone the work. m. As I stated, because maintenance had already begun at the time we became aware of the "out of range" issue, we could not fall back and turn the site on until the work was complete. However, with a five-minute pre-maintenance coverage check, this situation would have been quickly corrected two minutes into the check. m. when the "out of range" calls came to our attention, we could have reinstated 31st and Grays Ferry and eliminated the problem. Also in the future we will work 52 6/8/10 - PUBLIC SAFETY - RES. 100263 with the City to schedule these sorts of repairs at off-peak hours to eliminate major operational issues. And, finally, all antenna and lines are currently in the process of being inspected at this time. These inspections are being coordinated with the City. Since this event, we've already finished the Police Academy site, and that inspection showed no issues. We do regret that this incident occurred. We responded as quickly as possible to restore the 31st and Grays Ferry site and to replace the Victory Garden antenna once the issues were identified. And that concludes my testimony.
Thank you. Thank you very much. Councilman Jim Kenney.
Thank you, Madam Chair. Certainly any incident is 53 6/8/10 - PUBLIC SAFETY - RES. 100263 unacceptable, both to us and to you. I mean, you don't ever want to be in a situation where you have to come to hearings like this and respond to incidents, and so I guess the goal is 100 percent up time. I'm curious as to in other cities that you provide similar service to, comparable cities with comparable size or larger, police and fire departments. Do you have numbers on the number of incidents in a year that happen in other cities and the appropriate related response times to correcting it? So give me an example of a city that you provide service in, other than Philadelphia.
Say Baltimore. New York is another whole thing. Baltimore say for example. How many incidences were there last year in 54 6/8/10 - PUBLIC SAFETY - RES. 100263 Baltimore or in the last couple years?
Councilman, I don't have that information with me. I can certainly take that as a follow-up item.
It would be interesting to know what other cities experience. I mean, I'm sure that there's not one -- how many cities do you provide service in?
I actually have a responsibility for a limited area of the overall Verizon, but --
I support the Pennsylvania and Delaware from a government and education perspective, and then Enterprise customers all throughout North and South America, but your question --
I guess the request would be to provide through the Chair a number of similar cities both 55 6/8/10 - PUBLIC SAFETY - RES. 100263 Verizon and Motorola provide this service to, and over the past five years or so how many annual incidents are there and what are the corresponding response times, so to speak, to getting it back up and running. I'm just curious as to how it relates to our situation. Certainly the two that we've had recently are upsetting to Fire and Police personnel and to City government in general and we want to try to avoid those in the future, but I'm just curious as to, considering the fact that this is technology, some different levels of age. I understand the Legacy system is a ten-year-old system.
So we get a better idea as to how we can keep upgrading our stuff so that we don't -- I assume the antenna was a snow-related issue perhaps over the course of that winter?
We believe it was 56 6/8/10 - PUBLIC SAFETY - RES. 100263 weather related, yes.
There was a lot of snow. So maybe we need to look at protocols that require us to go after major snow events or major wind events, those kind of things, to keep up with it. Because I don't know of any system that's 100 percent perfect, although, again, the goal -- and we need to strive to be that, because those folks sitting here need to be safe and obviously we don't want to see any one incident.
Absolutely, Councilman. We remain committed to providing the utmost resilient and diverse network so that we can minimize or remove those. But to your point, equipment will fail at some point and we will encounter outages, and that's where I think having a structure as we've talked -- as has been identified previously. With a new system and with a back-up, the intention I believe is to help minimize even the few impacts. 57 6/8/10 - PUBLIC SAFETY - RES. 100263
I do think in addition to keeping up with our equipment, I mean, I do think the protocols on response are as important to continue to monitor and change, because if, for example, the issue with the antenna was weather related, then when weather happens, we need to take that into account and put that into the list of protocols we need to follow. If, again, the equipment maintenance or whatever you're doing in one spot -- I mean, you made that clear in your testimony that those changes have been made, but I guess it's an ongoing process to try to keep up with both the equipment age and with how we respond to down time.
But I would be interested in seeing other comparable cities' annual down time versus up time and the average response time to those down times. I also found the testimony very 58 6/8/10 - PUBLIC SAFETY - RES. 100263 interesting in how complicated the equipment and the technology is to respond to that. I mean, I'm not a techie by any stretch. I can get on the Internet. But I do recognize the critical nature of and the knowledge needed to respond to these things so that our folks are not going 10, 13, 10 minutes in the dark. 11 But I do appreciate your 12 testimony and any information you can 13 provide on that. 14
If I could just comment, please, in response. It was mentioned in Joe James' testimony previously that the system has been in place since the year 2000, and it is a highly sophisticated and intelligent radio system. Every radio system Motorola puts in is based on a certain platform of technology, and many throughout the United States are on different platforms. It's hard to compare apples and apples. 59 6/8/10 - PUBLIC SAFETY - RES. 100263 I will tell you that our service response time commitment to the City is two hours. We typically beat that on every occasion when we have a service issue or a case that's opened, and because we're doing so much preventative maintenance through our Network Operation Center, we have statistics that show that 90 percent of issues are solved prior to them being even aware by the user. The preventative maintenance, because of the tracking that we're doing in the back room at our center, really does help the preventative maintenance. I also just wanted to mention for the record, because I don't think it's been stated previously, is that your radio system is really a workhorse. You are processing over a million pushes to talk for Police and Fire and City activity on a monthly basis. In March, 1.2 million calls for service, and it's escalating as you ramp up to the summer, 60 6/8/10 - PUBLIC SAFETY - RES. 100263 1.3 million, 1.4 million pushes to talk, pushes and chances to talk. So we have statistics for all of that. We have statistics for every call for service, whether it was Motorola initiated or customer initiated, and we'd be happy to share that with you.
Is there some effort that could be made with our own personnel, additional training or additional expertise that you guys could impart to our folks who are City employees that are handling this equipment or dealing with this equipment? Is there any type of -- I'm just kind of throwing stuff out there -- as far as training that they could receive to maybe get ahead of the curve themselves or is it all monitored internally by machine?
Well, I think it's a partnership. It's a cooperative effort. There's always training that can be done, I think especially on the upgrade where there are new things 61 6/8/10 - PUBLIC SAFETY - RES. 100263 happening with the system, new features, new functionality. We will increase the training there, but there are zone watch and system watch terminals that guide us and give us all this information, and the City is shared -- we share that information with the City.
And this is just a kind of forward-looking question. We just had a recent hearing relative to a purchase of a WiFi setup. I don't know what the term of art is, but this WiFi company. And I guess the goal is down the road to have our mobile terminals in our police cars and fire apparatus to be able to hook up -- to be able to communicate through the WiFi. I know myself when the Internet goes down or your wireless goes down, I mean, there's reason -- is that less dependable or more difficult to determine the problems when it's a WiFi-related down time issue, or do you have information on -- 62 6/8/10 - PUBLIC SAFETY - RES. 100263
Without knowing exactly why and what type of exact system it is, it's hard to say, but in any private system, there is unique design. In a commercial off-the-shelf system like some WiFi systems are, they may not have the public safety rigor built into it with immediate access that's required like it would be required in a mission-critical system.
My question is, for example, on these particular two instances, you could identify the equipment that failed. In a WiFi situation, it could be weather related, it could be satellite crossing some other area cutting something off where you can't get to the bottom of what it is. So I guess the redundancy would be going back onto the -- I guess having a connection back onto the wired system as opposed to the WiFi. Would that be the solution to that or the backup to that? And, again, I know you're talking in 63 6/8/10 - PUBLIC SAFETY - RES. 100263 generalities and you're talking about different systems and a system that doesn't even exist yet here, but I'm just trying to think ahead. As far as having the WiFi is terrific, but if there's anticipated other unknown down time instances with that, how do we figure it out?
I think for the architecture that have been discussed just from the previous panel, there is an overlap there, so you would have both a primary service and a back-up service that would provide some of the resiliency that you'd be looking for to make sure that you have maximum availability of the service. The WiFi solution would be over and above I believe what you're talking about.
Sort of a third, if you would, because that's really not as much your microwave or your land-based 64 6/8/10 - PUBLIC SAFETY - RES. 100263 T1 issue.
Okay. And just if you get the other cities' information, that would be interesting.
Thank you. I actually wanted to know about the other cities' information also, so we'll look forward to getting that information. I'd also like to know -- I just want everybody to understand this. Motorola uses Verizon's lines, correct?
Can you explain that structure so we're real clear? And then you guys contract with the City to provide the services. The City owns the equipment or do you rent the equipment? And how much does it cost 65 6/8/10 - PUBLIC SAFETY - RES. 100263 for upgrade?
I'll try to address the Motorola piece of that. There is an interdependency between our equipment and having the connectivity to the sites, and the City actually chose back in 2000 when the City came in to use Verizon to provide that connectivity. The transport is also a decision of the City and they decided to use the SONET and the fiber and the things that were mentioned earlier for the connectivity to our sites. So that is a critical part of the system, and we have specifications on the type of reliability and up time that's required so that our radio equipment continues to have the heartbeat. I would have to defer to Verizon for the exact configuration of their SONET and their T1 configuration.
Thank you. From a Verizon perspective, you mentioned the upgrades and what could be deployed. 66 6/8/10 - PUBLIC SAFETY - RES. 100263 Really I think to answer the question most appropriately, we'd have to identify that there are multiple ways in which we could build out the network to make it more resilient, and I'll just give a couple examples, high-level examples. Today, the Verizon network is supported by two separate SONET rings. There is a solution which can remove much of the DACS or the connection between the two SONET rings by having one SONET infrastructure that would be deployed. So that would be a change upon which we've identified in the testimony. We could also deploy multiple nodes at some of the locations. We could diversify some of the circuits so that you minimize any potential impacts of it. You don't have as much commonality. So to the level and degree that you want full redundancy, diversity or even multiple entrances into each of the locations, it really depends upon the investment that would be necessary to do 67 6/8/10 - PUBLIC SAFETY - RES. 100263 that. Where we've done and some of the things we've identified in the deployment of additional T1s and looking at the Uninterruptible Power Supply, we've looked at points of failure that could cause issues and have tried to address those in collaboration both with Motorola and with the City. We've worked very closely. But there isn't one fix as to an upgrade where I could just provide you with one answer and say that this is the absolute right approach. I mean, we certainly have suggestions and we're working very collaboratively in doing that and identifying ways in which we can make the system robust and to continue to upgrade it and taking that into full consideration with the plan to move forward, I guess, with the microwave solution.
Okay. One thing I'd like to receive as Chair to distribute to members is a list of 68 6/8/10 - PUBLIC SAFETY - RES. 100263 lessons learned, not just from Motorola or Verizon but also the City of Philadelphia as it relates to prevention. And I just have one question for Motorola. I think you said push to talk?
When an officer, a police officer, or firefighter wants to gain access to the system, they take their portable radio and hit the side bar, and that's considered a push to talk.
Okay. And that can come from a firefighter, police officer or a customer?
Okay. And who are users on the system other than police and fire?
Well, you have the 69 6/8/10 - PUBLIC SAFETY - RES. 100263 jail. You have the schools. In the City of Philadelphia proper, those are the people that are mainly using the system.
And how many calls did you say we get a year? Or you said we have --
On average in a month, you're asking the system to process over a million calls a month, and it escalates as you get into the summer months. So March, for example, was 1.2, as I stated, and 1.3, 1.4 as you get into the summer. So it is a very busy city.
This is directed to Motorola. Did I hear Technology, Joe James, say that there was a device on that antenna that was monitoring --
I didn't hear you mention the fact that that 70 6/8/10 - PUBLIC SAFETY - RES. 100263 didn't work.
The fault monitor was in service and did not have any faults with it. This was, as Joe said, a unique failure of the antenna. The stress fractures allowed water into the antenna, and we believe the water in the antenna base -- this is a 22-foot antenna, to give you some idea of its size. And the base of the antenna had water collection, and we think that masked the ability for the fault monitoring to catch the reflective power. An antenna is a passive device. It doesn't tell you when it's not working. It just won't reflect the power appropriately, and the state of the art way to determine that is to have fault monitoring capture that power and signal that something is wrong. This was a unique fault and it did not trip the monitoring device, which was working and is still working today.
Should also 71 6/8/10 - PUBLIC SAFETY - RES. 100263 part of your list of things to do look at fault monitoring that may be better enhanced to identify the fault? You speculated that that antenna was out for many, many months because of the overlap --
We have no idea. We know that we checked it in December 2009, and we're not sure if it was something where it was a weakening of signal strength over time, but due to the overlap in coverage, it was not detected.
But I cannot imagine a fault detector not able to detect a total outage of an antenna. At some point, it was totally out of service. I mean, that's unacceptable, and I think that a fault monitor is supposed to tell you when you have a fault, no matter how unique it is. So I don't think -- I think that's something that you need to revisit to see if there's any way that those fault monitors can do a better job. 72 6/8/10 - PUBLIC SAFETY - RES. 100263
Any other questions for these witnesses? (No response.)
Seeing none, thank you. Thank you very much for attending.
Third panel: Eugene Blagmond and Bill Gault. Eugene Blagmond is from the FOP Lodge No. 5 and Bill Gault is the President IAFF Local 22. (Witnesses approached witness table.)
Hi. Good afternoon. We can just go in order that you were called, and don't forget the mike. Proceed with your testimony.
Good afternoon. 73 6/8/10 - PUBLIC SAFETY - RES. 100263 Thank you, Madam Chairwoman, members of the Committee. FOP President John McNesby and the 14,500 members would like to thank you for this public hearing on the opportunity to address the police radio system. It's very important regarding the seemingly endless problems and failures and total lack of consistency regarding the operation of the police emergency services radio system. This hearing is a direct result of the failures of the radio system in the Police Department on April 13th, 2010. On that date, officers reported that they could not hear dispatchers and could not communicate with dispatchers or any other patrol officers on the street. The Department implemented the all-too-familiar redeployment of officers into two-person units, and for a time officers only responded to priority assignments. These radio problems are 74 6/8/10 - PUBLIC SAFETY - RES. 100263 nothing new. Our officers, the men and women of the Police Department, have had little or no confidence in this radio system since the City of Philadelphia first contracted with Motorola in 1999. Since 1999, we have had two to three dozen radio failures, ranging from isolated incidents of officers on the street unable to communicate to widespread outages impacting the entire Police Department. These radio failures have placed police officers in danger numerous times. Many officers have turned to their radios when they were needed the most and in situations where a police officer's or civilian's life was in danger. Quite frankly, this emergency services radio system is a boondoggle of intense proportions. The radios have failed in Port Richmond in which two firefighters lost their lives in 2004. The radios failed when a police officer was shot during a drug raid in 2007. In 75 6/8/10 - PUBLIC SAFETY - RES. 100263 2008, the Police Commissioner was unable to communicate with police radio while in a volatile situation on the street in North Philadelphia. The radio system initially cost the taxpayers $62 million. Since the initial contract, the City has been forced to upgrade the system several times, increasing the original price of the radio system to numbers in excess of perhaps a hundred million dollars. After this, the radios still do not work. Our members are not communication technicians or engineers. However, we have become very familiar with the technology of the radio system, from tower sites, T1 lines, fixed repeaters, card racks and trunk lines to interoperability. We don't know what all these terms mean, but we do know that officers on the street have absolutely no 23 confidence in this system. We don't know what the answer to the problem is. We've been 76 6/8/10 - PUBLIC SAFETY - RES. 100263 shareholders and participants on the issue from the very beginning. We have proceeded to arbitration. We have participated in media and public education campaigns. We have testified in front of legislative bodies and public hearings, and we have had corporate representatives from both Motorola and Verizon attend our general membership meetings to address members directly regarding these issues. To illustrate that there may be no end in sight to this problem, Lodge 5 leadership was meeting last Wednesday regarding testimony for this very hearing when we were notified of a radio failure of the radio system once again. Our members are stuck on the streets without the vital communication lifeline that they need to protect the public. As we previously stated, we don't know what answers are to this longstanding problem, but we are glad to have a seat at the table. 77 6/8/10 - PUBLIC SAFETY - RES. 100263
Thank you, Councilwoman. I don't have a prepared speech written like that. Radios are our lifeline. If they don't work, we die. We've buried guys. Motorola has paid out. We've even gone so far as we're currently in arbitration. We still have a contract. They keep cutting us. Communication is our most important thing. Now we're communicating before we even get there, while we're driving. If the radios don't work, the job is not going to get done, civilians are going to die, we're going to get wounded, maybe die. This has been going on a long time, and we're talking about $165 million, of which we're all taxpayers too. And sitting here playing the blame game is just a waste of time. Fix it. 78 6/8/10 - PUBLIC SAFETY - RES. 100263 Our radio room is trying. Commissioner, he's trying. But what's going on here? They're blaming weather? We were in the weather. We were working. We're out there every day. All we have is that radio. Now, all the cops, this is what we're using now, our phones. You paid a lot of taxpayers' money and we didn't get a bang for the buck and now we get a lot of blame game. Please, we're begging you, just fix it. That's all. Thank you very much for listening. I could bring in a bunch of stories, a bunch of guys, a bunch of wounded people. We could parade it all day and bring the media in. It serves no 19 purpose. You's have the power. You's can get it done. Please. Thank you for your time, Council. I'll answer any questions you's have.
You know, I know you realize this is troubling to us as it is to you. I mean, I'm not a firefighter, I'm not a police officer, I'm not out needing that radio, but I know when an emergency happens and I need to be able to get in touch with someone, it's important that I'm able to do that. So I know that yours is triple, ten times the need as my personal needs in that regard. We would love to fix this. I've been here since 1996 and this has always been an issue at one point or another, the failure of the police radio. I would love to see it repaired. It's very complicated. All I know is that I was going to call my Internet provider and my telephone provider last week, because every other day at home when I went to make a phone call, I couldn't use my phone. I'm like, What is going on here? So finally I talked to my daughter 80 6/8/10 - PUBLIC SAFETY - RES. 100263 that's way more technical than I am, and she said you need to call your provider. And I just won't name the provider, but that was just a normal needing to use the phone to make a phone call. So, again, the fact that when you guys are pushing to talk -- that's a new term I just learned today. When you are pushing to talk, you expect it to work, and I empathize.
Councilwoman Miller, on the push to talk issue, while we were presenting our case on the radios, we were asking for repeaters put on our fire engines and ladder trucks. A repeater will just be an antenna or a disc to send a signal. I'm trying to get too geeky. But the signal is bouncing the satellites to the towers back and forth and all. So we had to ask in arbitration, using our tax dollars against us and our dues money on one side, for repeaters on fire trucks. And while the City was up in the 24th floor 81 6/8/10 - PUBLIC SAFETY - RES. 100263 of the Sheraton Hotel, 17th and Race, the radios bonked. Press to talk is when we push the button to speak. "Engine 1 to radio." It goes ehhhh, ehhhh. It bonks. You can't get through. That's what happened to the guys he's talking about on Belgrade Street. When it gets overloaded, nothing gets through. There's dead spots. Our radios don't work in the subway. They don't work in elevators. They don't work in high rises. Some floors around the City they don't work. They don't work in the Elevated. The system is very complicated, and it doesn't seem like a lot of thought was put into getting it.
Well, I would hope with the upgrades and I would guess -- I'll just ask the question and we can have Motorola, Verizon or the City answer. Will the upgrades help this 82 6/8/10 - PUBLIC SAFETY - RES. 100263 system work in elevators, a high rise and all the names that Mr. Gault -- or all the situations Mr. Gault named? We're spending a lot of money. To me it should work everywhere. (Witness approached witness table.) DEPUTY COMMISSIONER JAMES: Again, Deputy Commissioner Joe James with the Division of Technology. In respect to the question, the upgrade will not change the coverage of the radio system. The radio system was designed with a 95 percent reliability of coverage in 95 percent of the City. It was never designed to work in the subway, into areas below grade and in certain high-rise buildings where it's just not able to penetrate. That will not change with the upgrade. It didn't change with the system when we implemented it back in 2000. In many cases, it didn't work on the old system either. We have been -- since 1992 when 83 6/8/10 - PUBLIC SAFETY - RES. 100263 we first started the planning for the project, we looked at a solution for the upgrade -- to upgrade to a solution that would have dealt with the underground, and in the past, it's been primarily -- the cost impact has just been more than what the City can afford to build out the underground, which we determined in, let's say, '99 it was $23 million just to do the subways.
Well, we do have a lot of activity in our subways. Does SEPTA's system work there? DEPUTY COMMISSIONER JAMES: Well, specific to your question, the SEPTA has a 500 megahertz radio system that works in the subways. We have worked on a number of different things in particular to the question when you talk about repeaters. In the upgrade, we are getting 50 repeaters, specifically repeaters for the Fire Department to deploy as portable repeaters, and that's part of the upgrade. It's included in 84 6/8/10 - PUBLIC SAFETY - RES. 100263 what we got with the upgrade. Those repeaters, we actually are working with the Fire Department right now to get a test model to start testing up at the Academy. So we are going to have portable and mobile repeaters and fixed repeaters, and some of those fixed repeaters we may implement in the underground for specific coverage areas. We've also been in discussions with SEPTA for a number of years on a way to integrate our interoperability between their radio system and our system so that if there is an incident in the underground, somehow we can figure out a way to patch the two radio systems together so that the police and firefighter and first responders will be able to continue to communicate if they have to respond to the incident in the underground. But there are certain -- I mean, we don't try to match. There are 85 6/8/10 - PUBLIC SAFETY - RES. 100263 certain areas where you're not going to have coverage. What the Fire Department has been doing for years whenever they've done their inspections of high rise and something, they identify those particular issues, and in certain cases we've been in discussions with certain building providers to put the kind of repeating cable within the building. The new Comcast building has that in their new tower. So that these type of repeater solutions are being discussed and we are looking at them, but we do not have an answer that I can speak to what the gentleman raised about having a coverage that's going to be 100 percent everywhere at all times.
Well, you know, I would think that you wouldn't have 100 percent. DEPUTY COMMISSIONER JAMES: Right.
However, I really didn't realize that the systems 86 6/8/10 - PUBLIC SAFETY - RES. 100263 didn't work in the subways, because we have a whole lot of activity going on in subways and I would think that that would be an important part, so if you're working with SEPTA to figure out how the two systems can work together. I think when I had my first cell phone, it didn't work certain places, and I thought probably it wasn't enough towers, but as more and more towers were built, now my cell phone works. And I got to do this in a simple way. I'm not trying to be a technical expert here. I know that my cell phone didn't work in certain places like years ago that now it works because there's more towers, I assume, and it's easier. I mean, it works on the elevator. Years ago it didn't work on the elevators. So I'm trying to compare that with this system that costs more than a hundred million. And even with upgrades, will there ever be or should there be or 87 6/8/10 - PUBLIC SAFETY - RES. 100263 in any other cities does the system have the capability to work underground, to work on high rises, to work in elevators? DEPUTY COMMISSIONER JAMES: Well, I think when you try to compare a cell phone, I mean, there is places in the elevators where my Verizon phone will work, but my AT&T won't work. I mean, there's areas where if I have a T-Mobile phone, my phone won't work at all. I mean, there's certain areas where you will always have a coverage issue, because you're dealing with RF signals and they're trying to penetrate properties and the physics of a building or a space. So there are certain things that's always going to be a limitation no 19 matter how many towers you put out and how much power you turn up. We are always looking at trying to figure out a way to increase the coverage of the system. And to the underground, we have looked at solutions. Until a couple years ago, your cell phone 88 6/8/10 - PUBLIC SAFETY - RES. 100263 didn't work in the subway. Until they built an antenna system that works specifically in the subways, now you have the ability to use your cell phone down there. In the radio system, there's parts of the underground that your radio will work. If you're walking in the concourse underground, it will work. Just when you get down on the track level, you're so far down that the signal just won't penetrate. Do we have a solution? Like I said, we looked at repeaters. Other than spending $23 million or finding some other technical solution to do that, we don't have that answer at this point in time. But it's just like everything in evolution in technology, there's always something new coming out, there's always some new advancements and there's always something that we're going to try to explore to figure out if there is a way to deal with the question that's come up about the 89 6/8/10 - PUBLIC SAFETY - RES. 100263 radio coverage.
Thank you, Madam Chair. Councilwoman Miller just mentioned about other cities. I too didn't realize that the radios don't work in some of the areas Mr. Gault talked about. Have you looked at other cities? And maybe the union members know this. Do the radio systems work, say -- Councilman Kenney talked about other cities -- New York, Chicago? Have you looked at those? DEPUTY COMMISSIONER JAMES: Yes, we have. We've looked at other cities that have underground subways, like New York City, which has a mixed bag of just about every type of radio communication under the sun, every type of system, but even New York had a long legacy of a history in the New York Times 90 6/8/10 - PUBLIC SAFETY - RES. 100263 about their 800 megahertz digital radio system did not work in the underground or they had problems with their underground solution, to the point they stopped using it. There's Baltimore, there's DC, there's other areas, and you look at their solutions and most of it is, you have to build out a repeater, you have to build out an antenna system in the underground if you truly want to do it. You have to do above ground, what you've done, in the underground, and we have just not gotten to that point where we have come up with a cost-effective solution that deals with that that we can be able to make that expenditure, other than patching in other systems that are currently -- and one of the things -- it's an IP-based system, so we're hoping that there's some technology advancements, because the newer systems are IP based. Maybe there's a way that we can get the signal there that's in a 91 6/8/10 - PUBLIC SAFETY - RES. 100263 non-traditional method, but it's just a matter of trying to look at -- when you look at some of these other cities, it's kind of either they built it from ground up with an antenna system in it or they've augmented it over time.
And I understand everything costs money, but, I mean, we are talking -- not to make a speech here, but obviously public safety when firefighters and police can't communicate. And as Councilwoman said, the subways jumped out, I think, at both of us, because stuff does happen down there obviously, and that's concerning. And I know -- look, we certainly understand here about not having enough money for stuff, but it just does seem -- and, again, I might be one of the least technological people here, but it just does seem like there should be -- we got to keep looking into that, I guess is the simplest way to say it. Thank you. 92 6/8/10 - PUBLIC SAFETY - RES. 100263 DEPUTY COMMISSIONER JAMES: To answer your question, Councilman, we have not stopped looking at it, and one of the things about getting these digital repeaters is maybe that's a solution that we can implement in the subway that allows us to repeat the signal down there just like the cell phone carriers do and give the firefighters and the officers the coverage they need and it doesn't cost us $23 million to do that.
Not to beat a dead horse, but I made a recommendation, and the cell phone providers were open arms, that when they were building out the subway to provide their coverage, that the biggest expense building out the subway is trying to find the time when you can work down there 93 6/8/10 - PUBLIC SAFETY - RES. 100263 because of the trains, et cetera. So I believe there's only one at this moment, AT&T Wireless, and fortunately they're our vendor, and the Fire Department and the Police Department are fortunate, even though the radios don't work there, their ace in the hole is their cell phone, in many cases their company cell phone, City cell phone or their own personal cell phone. But also I think that we could help you if you think there's something legislatively we can do to require new construction or asking some of the high rises that are below grade, above grade to go back and retrofit and put repeaters in there, because, A, it benefits them. If they have a group of firefighters that go in to fight a fire or a police officer, it benefits them for that police or firefighter to be able to do what they have to do, communicate, flash information, whatever. And I know there are some cities that require repeaters in 94 6/8/10 - PUBLIC SAFETY - RES. 100263 buildings below grade, above grade so the phones work. DEPUTY COMMISSIONER JAMES: Yes.
So that's what I'd like to do. And I'd like you to possibly communicate to the next phase of a fit-out for the subway to join hands, Mr. Frank, to be able to save the taxpayers of this city some money by doing it jointly. Why should T-Mobile do it once and AT&T and Verizon and Nextel? Let's get down there and do it all at one shot and maybe save a few bucks.
This is Allan Frank, Chief Technology Officer. If I could just make a comment about the subway. I mean, I'm equally concerned about it, and I think it's really a matter of creating a real focused strategy so that we all know what the heck we really need to do. I mean, we are a homeland security region one location, as has been pointed out. The 95 6/8/10 - PUBLIC SAFETY - RES. 100263 subways and, frankly, I'm sure there's other areas of the City, including high rises, that are absolutely critical, and I, for one, if a high rise was burning, I want to make sure firefighters get up there, and certainly we don't want any lives lost. So let me just speak personally. I want to make a commitment that we are going to focus on what the heck the strategy is, and we owe it to ourselves and we certainly owe it to the citizens and to the fine professionals in both the Police Department and the Fire Department. So I just want to make a direct comment. And I think the key here is, I think as we pointed out, there is no perfect technology, but I don't think neither can we sit here and just talk about it. I think we have to say to ourselves, over a period of time how do we get there, and if repeaters or whatever it is are the way we do it today, that's great, but what are we 96 6/8/10 - PUBLIC SAFETY - RES. 100263 doing tomorrow and how are we going to get there. So I'm equally concerned, and I don't want to sit here a year from now without really being able to articulate and say, Hey, look, here's how we want to chip away at this, including working with SEPTA and others, just how the heck do we deal with this.
And that's really great. That's what we really want to hear. I mean, we stated the problem. We need to figure out a way to start moving towards the solution. How long will that take you to get back with us, about six, seven months, a year, giving us an update, keeping us informed?
Well, I would commit to an update in six months, but, again, we obviously have a lot of work to do.
We 97 6/8/10 - PUBLIC SAFETY - RES. 100263 understand that.
We understand, but it's good that we're in a hearing, we're talking about it, we set a date with a "to do," and we'd like to hear back of what's being done. So this is April and -- no; this is June.
And I want to make it clear I want to satisfy myself as to what other cities are also doing. There's no excuse.
Okay. So around December then we want to hear something. You can just send it to me as the Chair.
It's interesting what Councilwoman Miller just said, our Chair, that we're here in the hearing and things that you're doing 98 6/8/10 - PUBLIC SAFETY - RES. 100263 today, we discussed in the hearing, like joining hands with the School District to acquire their channels, save them money on not having to maintain their own system, using ours, Delaware River Port Authority, all things that were brought up in hearings. There's another agency besides what I just mentioned, the School District, that -- well, there's a couple others -- that are going to be utilizing this system and increasing those push to talks, but, again, saving money for them, all the new school buses. So, again, it started in a public forum and things are starting to happen. DEPUTY COMMISSIONER JAMES: We learned our lesson, and I think what we've taken away is some of those lessons learned that we're going to put back is those things that we are doing right now to deal with this issue and some of the things that we will find through investigating and looking at what other 99 6/8/10 - PUBLIC SAFETY - RES. 100263 cities are doing and other joint opportunities to try to leverage other people's infrastructure.
If I could make one more point I think going back to the original discussion around the hearings with some of the failures and so forth. I think it's going to be particularly critical. One of the questions was what did the City learn. So one of the things certainly that I learned through this experience is, again, looking at all the weak links, all the possible things, even just asking the question of, Okay, we replaced the radio, what's the life? Ten years. Well, what's the life of the batteries? What's the life of this and what's the life of that? So I really think one of the -- certainly the things that I've learned that we need to do is, as we move forward here and start the clock and effectively what Deputy Mayor Gillison and I have ordered is, we're going to refresh all 100 6/8/10 - PUBLIC SAFETY - RES. 100263 those little weak links as we speak, and then I want to make sure we write down somewhere what the life is so that five years from now or ten years from now, we're not sitting here and saying, Oops, we forgot the life of the next antenna is ten years. So it's remedial maintenance and ongoing making sure we understand that we have to continue to maintain this equipment or replace it after its life.
Right. I think that's one reason why I would like to get and the Committee members would like to have a statement or a list of lessons learned, prevention, moving forward. It seems like we keep going back. I've been in most of these hearings, and it just seems like we're just going around and around and around, and at some point, we need to figure out a way to address some of the issues. Like I really think it's good that you're going to be pursuing who can you partner with as it relates to the subway or the 101 6/8/10 - PUBLIC SAFETY - RES. 100263 underground issue, high rise issue, whatever. But we need to feel like we're making some kind of progress rather than spending a lot of money and police officers and firefighters still don't feel safe. So we got to figure that one out. All right. So that's on our "to do" list. DEPUTY COMMISSIONER JAMES: Yes.
Okay. Do we have any more questions for the unions, the FOP, the firefighters? (No response.)
All right. I want to thank everyone for coming in, and this Committee is recessed until the call of the Chair. Thank you. (Committee on Public Safety recessed at 3:20 p.m.) - - - 102 CERTIFICATE I HEREBY CERTIFY that the proceedings, evidence and objections are contained fully and accurately in the stenographic notes taken by me upon the foregoing matter on June 8, 2010, and that this is a true and correct transcript of same. -------------------- MICHELE L. MURPHY RPR-Notary Public (The foregoing certification of this transcript does not apply to any reproduction of the same by any means, unless under the direct control and/or supervision of the certifying reporter.)