IPRAN故障案例分析

基站闪断故障案例分析摘要：某IPRAN站点在运行过程中出现频繁的设备闪断故障，造成此站点下挂4个3G站、3个4G站业务频繁闪断。背景2016年8月24日上午在网优人员给局方维护人员打来电话，说有数个站点出现频繁断站，然后过几分钟就好；我方工程师在收到局方告知后，立即与维护人员取得联系，为了不影响局方的业务以及考核指标，决定先将此站点下挂的站点端口进行关闭处理。问题描述及排查1.登录到闪断的设备上，一般成环的节点很少会出现瞬断，但是不排除。首先检查瞬断节点的收发光功率；如果没问题就逐条检查环上的其他节点的光功率。一般情况下是光口实际收光超出临界值；zxr10#showperf-valueinterfacegei_5/1laser-basis-cur(mA):6.8160laser-basis-min(mA):6.8160laser-basis-max(mA):6.8480laser-temp-cur(deg):39.8430laser-temp-min(deg):39.5620laser-temp-max(deg):39.8430optical-send-cur(dBm):-6.1650optical-send-min(dBm):-6.1670optical-send-max(dBm):-6.1650optical-rcv-cur(dBm):-7.5030optical-rcv-min(dBm):-7.5100optical-rcv-max(dBm):-7.5000send-bandwidth-usage:0.0131rcv-bandwidth-usage:0.0213zxr10#showperf-valueinterfacegei_6/1laser-basis-cur(mA):17.7500laser-basis-min(mA):17.7500laser-basis-max(mA):17.8000laser-temp-cur(deg):33.2260laser-temp-min(deg):33.2260laser-temp-max(deg):33.2260optical-send-cur(dBm):-5.9190optical-send-min(dBm):-5.9190optical-send-max(dBm):-5.8850optical-rcv-cur(dBm):-7.2490optical-rcv-min(dBm):-7.3680optical-rcv-max(dBm):-7.2490send-bandwidth-usage:0.0016rcv-bandwidth-usage:0.00082.检查设备温度，看风扇状态：设备温度过高会导致板卡假死，当温度降到板卡可以承受的温度的时候，板卡又会起来，所以就会出现闪断。zxr10#showversionZXR10RouterOperatingSystemSoftware,ZTECorporationZXR10ROSVersionV4.08.34ZXCTN6220Software,VersionZXCTN620063006220V2.10.00P02B10,RELEASESOFTWARECopyright(c)2001-2014byZTECorporationCompiledApr242015,18:46:39Systemimagefilesareflash:/flash/img/zxr10.zarSystemuptimeis244day(s),16hour(s),53minute(s)[SCCU,panel1,master]MainProcessor:PowerPCMPC8544ProcessorSystemMemory:1024Mbytes,SystemFLASH:128MbytesSystemNVRAM:8Kbytes,SystemTemperature:41℃SystemSerial:16383,SystemBaudRate:9600SystemClock:PTPstratum3BoardType:RSCCP2SerialNumber:269776601152,PartNumber:123353851003MainCPLD1Version:V1.4,MainCPLD2Version:V1.4CPUCPLDVersion:N/A,PCICPLDVersion:N/AFPGAVersion:V6.1,MainPCBNumber:101001[SCCU,panel2,slave]MainProcessor:PowerPCMPC8544ProcessorSystemMemory:1024Mbytes,SystemFLASH:128MbytesSystemNVRAM:8Kbytes,SystemTemperature:43℃SystemSerial:16383,SystemBaudRate:9600SystemClock:PTPstratum3BoardType:RSCCP2SerialNumber:269776600138,PartNumber:123353851003MainCPLD1Version:V1.4,MainCPLD2Version:V1.4CPUCPLDVersion:N/A,PCICPLDVersion:N/AFPGAVersion:V6.1,MainPCBNumber:101001[NPCI,panel1]MainProcessor:W90N740ProcessorSystemMemory:16Mbytes,SystemTemperature:N/ABoardType:R4EGC,LogicBoardType:R4EGC_ETHPortNumber:4SerialNumber:269494602008,PartNumber:123352651015MainCPLDVersion:V1.8FPGAVersion:V0.8,MainPCBNumber:081001[NPCI,panel2]MainProcessor:MPC8321ProcessorSystemMemory:128Mbytes,SystemTemperature:43℃BoardType:R16E1F,LogicBoardType:(TDM+IMA)PortNumber:16SerialNumber:269539101105,PartNumber:123352651023MainCPLDVersion:V2.3FPGAVersion:V2.1,MainPCBNumber:080902[NPCI,panel5]MainProcessor:W90N740ProcessorSystemMemory:16Mbytes,SystemTemperature:43℃BoardType:R4EGC,LogicBoardType:R4EGC_ETHPortNumber:4SerialNumber:269494601680,PartNumber:123352651015MainCPLDVersion:V1.8FPGAVersion:V0.8,MainPCBNumber:081001[NPCI,panel6]MainProcessor:W90N740ProcessorSystemMemory:16Mbytes,SystemTemperature:43℃BoardType:R4EGC,LogicBoardType:R4EGC_ETHPortNumber:4SerialNumber:279887200696,PartNumber:123352651015MainCPLDVersion:V1.8FPGAVersion:V0.8,MainPCBNumber:081001zxr10#showfan-informationPCBVersion:100801LogicalVersion:V3.1Fan1:NormalWorkFan2:NormalWorkFan3:NormalWorkFan4:NormalWork3.查看端口是否有错包，当端口错包累计到一定程度后，很大可能性会出现设备闪断。zxr10#showinterfacegei_5/1gei_5/1isup,lineprotocolisup,detectstatusisOKBynameisnoneDescriptionisnoneTheportisopticalDuplexfullSmbipg12Smbmccbandwidth10MbitsMccvlanidis3024Smbmccpriority7NoloopbackBW1000000KbitsLastclearingofshowinterfacecountersnever120secondsinputrate:2428214Bps,8820pps120secondsoutputrate:1571473Bps,7147ppsInterfacepeakrate:input20166528Bpsoutput8806412BpsInterfaceutilization:input1%,output1%Input:Packets:149523563988Bytes:43053642220163Unicasts:149493477935Multicasts:30086051Broadcasts:2LittUndersize:0Oversize:0CRC-ERROR:259107Dropped:2869615Fragments:0Jabber:0MacRxErr:0Errframe:0Disframe:2869615UnicErrors:0MultiErrors:0BroErrors:0GoodJumb:259107460764B:065-127B:44437087826128-255B:9959552083256-511B:85801245887512-1023B:7465987381024-Max:8579079454AlignErrors:0Output:Packets:138147568647Bytes:31382769692997Unicasts:138117545657Multicasts:30022986Broadcasts:4Collisions:0LateCollision:0ColliTimes:0FlowContr:0DeExceFrame:0DecolliFra:0NonUniFrame:30022990ErrBytes:0ErrFrame:0MacControl:0JabberFrame:0GoodJumb:167604028LittUndersize:064B:065-127B:39707412701128-255B:11607380287256-511B:85696141688512-1023B:5325916111024-Max:604042360Oversize:0zxr10#showinterfacegei_6/1gei_6/1isup,lineprotocolisup,detectstatusisOKBynameisnoneDescriptionisnoneTheportisopticalDuplexfullSmbipg12Smbmccbandwidth10MbitsMccvlanidis0Smbmccpriority7NoloopbackBW1000000KbitsLastclearingofshowinterfacecountersnever120secondsinputrate:83450Bps,809pps120secondsoutputrate:150818Bps,484ppsInterfacepeakrate:input2781929Bps,output10989989BpsInterfaceutilization:input0%,output0%Input:Packet