📢 [Network Update – China Unicom Packet Loss in LAX]
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
đź”’ DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
đź”’ DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
/var/log/DMIT-NOC.log
[HK Maintenance] Date: June 27 - July 1, 2025 Content: - Replacement of edge router outdated 10Gbps line cards(2x); ---- CMI expected upto 1hr downgrade; ---- CN2 GIA has redundnacy circuits expected graceful switch over; - Replacement of main switch;…
APAC:
There are couple fiber cut inside APAC region;
GSL is now losing both short path connection between HK and TY.
RETN is also experience congestion.
NTT won't be good, as our experience.
DMIT is now configuring router to adapter the MPLS Ethernet connectivity.
After that, we can delivery the PNI and ASE connection.
* The TPE-TYO of GSL shows available but 0% IN/OUT; might need wait for GSL furtuer action.
There are couple fiber cut inside APAC region;
GSL is now losing both short path connection between HK and TY.
RETN is also experience congestion.
NTT won't be good, as our experience.
DMIT is now configuring router to adapter the MPLS Ethernet connectivity.
After that, we can delivery the PNI and ASE connection.
* The TPE-TYO of GSL shows available but 0% IN/OUT; might need wait for GSL furtuer action.
CoreSite Edge Router encounter an unexpected issue bring the rpd process crash; rollback and it was resovled.
There is an uncommon, and undocumented bug on JunOS that make unpredictable error in the routing and rdp.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
/var/log/DMIT-NOC.log
There is an uncommon, and undocumented bug on JunOS that make unpredictable error in the routing and rdp. We’ve ran many test past 2 months to ident the root cause. The issue on Los Angeles RIB/FIB desync has been resolved. Like some China Unicom prefer…
Configuration has been loaded.
This issue has been permanently fixed.
This issue has been permanently fixed.
Observed CTGnet AS23764-AS4809 withdrawal all AS4837 and AS9808 routes on both Tokyo and Hong Kong. We've emailed CTGnet NOC about this changes. It should be an error configuration on CTGnet end.
/var/log/DMIT-NOC.log
Observed CTGnet AS23764-AS4809 withdrawal all AS4837 and AS9808 routes on both Tokyo and Hong Kong. We've emailed CTGnet NOC about this changes. It should be an error configuration on CTGnet end.
Thank you for your troubleshooting report to Telecom Global Service Center. Telecom Global Call Center has opened a ticket [ _ ] for you, and you can use this ticket number to check with us on the progress of this issue. The fault will be investigated by our team of professional engineers and we will update you as soon as we have made progress. We apologize for any inconvenience this may cause.
/var/log/DMIT-NOC.log
Thank you for your troubleshooting report to Telecom Global Service Center. Telecom Global Call Center has opened a ticket [ _ ] for you, and you can use this ticket number to check with us on the progress of this issue. The fault will be investigated by our…
Restored 5 mins ago. Please contact with us if there is any more issues.
/var/log/DMIT-NOC.log
APAC: There are couple fiber cut inside APAC region; GSL is now losing both short path connection between HK and TY. RETN is also experience congestion. NTT won't be good, as our experience. DMIT is now configuring router to adapter the MPLS Ethernet…
Hong Kong GSL now has only 1/5 subsea alive within Hong Kong.
This could lead some packet loss and speed issue in recent.
DMIT current strategy is:
Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize.
Outbound: Balance between all ISPs, unless special optimize.
====
GSL is the primary inbound ISPs for DMIT as the current design. But the unfortunate things (fiber cut + massive DDoS) happened just after DMIT had connect with GSL. The timing is super unlucky.
Please wait us (DMIT+GSL) for a while for the network construction.
At the end, DMIT will primarily use our backbone to carry APAC<>US/EU access to lower latency and less jitter.
*Notes: the GSL has many DDoS protection customer but they are using "detect" method for DDoS mitigation in default which means once there is a huge attack; it might congested the subsea for few secs before the detection finish detouring.
With using DMIT's subsea/backbone, it minimize the impact; since the territory fiber(within continents) has grater capacity, and before the traffic leave continents it arrives nearest DMIT/GSL PoP. On the DMIT US/EU PoP, we can easily scale up to multiple 100G ports for against the peak traffic; at the same time we use port mirroring and DPDK capture for DDoS detection which gives less than secs DDoS mitigation/diversion. This cam make sure all the traffic going thru subsea will be mostly legit.
===DMIT===
DMIT has signed multiple subsea between Hong Kong, Tokyo, and Los Angeles.
It will be alive between Aug ~ Sep, 2025.
HK-TY: ASE
HK-US: FASTER + SJC, ASE + JUNO
US-TY: JUNO
===GSL===
GSL is going to bring SJC2 [HK-TY] alive in early August, 2025.
C2C[HK-TW] ETR is end of August, 2025
NCP[HK-KR] ETR is mid of August, 2025.
===LAST===
DMIT will disconnect with RETN in Tokyo after terms end.
The Hong Kong will be kept. But DMIT will use alternative strategy for EU-HK latency opts.
RETN removed Japan - EU territory fiber a month after we signed the contract; and they won't restore it.
RETN's subsea isn't good for Japan - EU, it even worse than HE/Cogent somehow. So this action won't make Japan network has any downside change..
This could lead some packet loss and speed issue in recent.
DMIT current strategy is:
Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize.
Outbound: Balance between all ISPs, unless special optimize.
====
GSL is the primary inbound ISPs for DMIT as the current design. But the unfortunate things (fiber cut + massive DDoS) happened just after DMIT had connect with GSL. The timing is super unlucky.
Please wait us (DMIT+GSL) for a while for the network construction.
At the end, DMIT will primarily use our backbone to carry APAC<>US/EU access to lower latency and less jitter.
*Notes: the GSL has many DDoS protection customer but they are using "detect" method for DDoS mitigation in default which means once there is a huge attack; it might congested the subsea for few secs before the detection finish detouring.
With using DMIT's subsea/backbone, it minimize the impact; since the territory fiber(within continents) has grater capacity, and before the traffic leave continents it arrives nearest DMIT/GSL PoP. On the DMIT US/EU PoP, we can easily scale up to multiple 100G ports for against the peak traffic; at the same time we use port mirroring and DPDK capture for DDoS detection which gives less than secs DDoS mitigation/diversion. This cam make sure all the traffic going thru subsea will be mostly legit.
===DMIT===
DMIT has signed multiple subsea between Hong Kong, Tokyo, and Los Angeles.
It will be alive between Aug ~ Sep, 2025.
HK-TY: ASE
HK-US: FASTER + SJC, ASE + JUNO
US-TY: JUNO
===GSL===
GSL is going to bring SJC2 [HK-TY] alive in early August, 2025.
C2C[HK-TW] ETR is end of August, 2025
NCP[HK-KR] ETR is mid of August, 2025.
===LAST===
DMIT will disconnect with RETN in Tokyo after terms end.
The Hong Kong will be kept. But DMIT will use alternative strategy for EU-HK latency opts.
RETN removed Japan - EU territory fiber a month after we signed the contract; and they won't restore it.
RETN's subsea isn't good for Japan - EU, it even worse than HE/Cogent somehow. So this action won't make Japan network has any downside change..
DMIT observed China statewide TCP 443 disconnection, reset or block.
The IP replacement won't fix the issue; due to we might receive over load ticket or inquiries; the related ticket might be auto reply, and the IP replacement request might be rejected directly during this time.
Thank you for your support and patience.
The IP replacement won't fix the issue; due to we might receive over load ticket or inquiries; the related ticket might be auto reply, and the IP replacement request might be rejected directly during this time.
Thank you for your support and patience.
We have observed that Alipay callbacks have been affected.
We will manually match the deposits after restoration.
We will manually match the deposits after restoration.
TYO Scheduled Maintenance
Aug 23, 3 PM JST (2PM CST)
Duration: <= 60mins per node
Hardware component modification.
Aug 23, 3 PM JST (2PM CST)
Duration: <= 60mins per node
Hardware component modification.
/var/log/DMIT-NOC.log pinned «TYO Scheduled Maintenance Aug 23, 3 PM JST (2PM CST) Duration: <= 60mins per node Hardware component modification.»
HKG Pro:
We have detected abnormal trickle traffic. DMIT has implemented firewall rules and deducted a certain amount from the affected customers' transfer billing.
We have detected abnormal trickle traffic. DMIT has implemented firewall rules and deducted a certain amount from the affected customers' transfer billing.
/var/log/DMIT-NOC.log
Hong Kong GSL now has only 1/5 subsea alive within Hong Kong. This could lead some packet loss and speed issue in recent. DMIT current strategy is: Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize. Outbound: Balance…
DMIT has launched the following circuits:
1. ASE: HKG-TYO
2. JUPITER: TYO-LAX
80% traffic from the U.S. towards to HKG and TYO is now way more better than months ago, and other GSL clients in APAC.
The return traffic from APAC towards to the U.S. is still direct thru GSL since there is less congestion, and also for less unexpected route detour.
The SEA and MIA has been excluded, due to route leaking and worse latency. SEA-LAX-TYO will increase the latency between SEA-TYO.
The following is pending installed:
1. JUNO: TYO-LAX
2. PC1/NCP: TYO-SEA
Once JUNO has been installed, there will be more capacity and additional backup.
PC-1 and NCP will bring backup; also lower latency to finish up our APAC-NA design.
=====
At the same time, DMIT observed the following things:
1. GSL still experience SIN-TYO fiber offline. The traffic between TYO and SIN need bypass HKG which is kind overloaded in recent.
2. GSL has increasing sudden packet loss rate; it might caused by increasing number of high risk target has also connected GSL for DDoS protection.
DMIT’s plan:
1. Having protected TYO-SIN connectivity.
2. Having protected HKG-SIN connectivity.
3. Finishing TYO-SEA and JUNO connectivity.
It helps us prevent the traffic congestion on the GSL backbone.
1. ASE: HKG-TYO
2. JUPITER: TYO-LAX
80% traffic from the U.S. towards to HKG and TYO is now way more better than months ago, and other GSL clients in APAC.
The return traffic from APAC towards to the U.S. is still direct thru GSL since there is less congestion, and also for less unexpected route detour.
The SEA and MIA has been excluded, due to route leaking and worse latency. SEA-LAX-TYO will increase the latency between SEA-TYO.
The following is pending installed:
1. JUNO: TYO-LAX
2. PC1/NCP: TYO-SEA
Once JUNO has been installed, there will be more capacity and additional backup.
PC-1 and NCP will bring backup; also lower latency to finish up our APAC-NA design.
=====
At the same time, DMIT observed the following things:
1. GSL still experience SIN-TYO fiber offline. The traffic between TYO and SIN need bypass HKG which is kind overloaded in recent.
2. GSL has increasing sudden packet loss rate; it might caused by increasing number of high risk target has also connected GSL for DDoS protection.
DMIT’s plan:
1. Having protected TYO-SIN connectivity.
2. Having protected HKG-SIN connectivity.
3. Finishing TYO-SEA and JUNO connectivity.
It helps us prevent the traffic congestion on the GSL backbone.