Observed CTGnet AS23764-AS4809 withdrawal all AS4837 and AS9808 routes on both Tokyo and Hong Kong. We've emailed CTGnet NOC about this changes. It should be an error configuration on CTGnet end.
/var/log/DMIT-NOC.log
Observed CTGnet AS23764-AS4809 withdrawal all AS4837 and AS9808 routes on both Tokyo and Hong Kong. We've emailed CTGnet NOC about this changes. It should be an error configuration on CTGnet end.
Thank you for your troubleshooting report to Telecom Global Service Center. Telecom Global Call Center has opened a ticket [ _ ] for you, and you can use this ticket number to check with us on the progress of this issue. The fault will be investigated by our team of professional engineers and we will update you as soon as we have made progress. We apologize for any inconvenience this may cause.
/var/log/DMIT-NOC.log
Thank you for your troubleshooting report to Telecom Global Service Center. Telecom Global Call Center has opened a ticket [ _ ] for you, and you can use this ticket number to check with us on the progress of this issue. The fault will be investigated by our…
Restored 5 mins ago. Please contact with us if there is any more issues.
/var/log/DMIT-NOC.log
APAC: There are couple fiber cut inside APAC region; GSL is now losing both short path connection between HK and TY. RETN is also experience congestion. NTT won't be good, as our experience. DMIT is now configuring router to adapter the MPLS Ethernet…
Hong Kong GSL now has only 1/5 subsea alive within Hong Kong.
This could lead some packet loss and speed issue in recent.
DMIT current strategy is:
Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize.
Outbound: Balance between all ISPs, unless special optimize.
====
GSL is the primary inbound ISPs for DMIT as the current design. But the unfortunate things (fiber cut + massive DDoS) happened just after DMIT had connect with GSL. The timing is super unlucky.
Please wait us (DMIT+GSL) for a while for the network construction.
At the end, DMIT will primarily use our backbone to carry APAC<>US/EU access to lower latency and less jitter.
*Notes: the GSL has many DDoS protection customer but they are using "detect" method for DDoS mitigation in default which means once there is a huge attack; it might congested the subsea for few secs before the detection finish detouring.
With using DMIT's subsea/backbone, it minimize the impact; since the territory fiber(within continents) has grater capacity, and before the traffic leave continents it arrives nearest DMIT/GSL PoP. On the DMIT US/EU PoP, we can easily scale up to multiple 100G ports for against the peak traffic; at the same time we use port mirroring and DPDK capture for DDoS detection which gives less than secs DDoS mitigation/diversion. This cam make sure all the traffic going thru subsea will be mostly legit.
===DMIT===
DMIT has signed multiple subsea between Hong Kong, Tokyo, and Los Angeles.
It will be alive between Aug ~ Sep, 2025.
HK-TY: ASE
HK-US: FASTER + SJC, ASE + JUNO
US-TY: JUNO
===GSL===
GSL is going to bring SJC2 [HK-TY] alive in early August, 2025.
C2C[HK-TW] ETR is end of August, 2025
NCP[HK-KR] ETR is mid of August, 2025.
===LAST===
DMIT will disconnect with RETN in Tokyo after terms end.
The Hong Kong will be kept. But DMIT will use alternative strategy for EU-HK latency opts.
RETN removed Japan - EU territory fiber a month after we signed the contract; and they won't restore it.
RETN's subsea isn't good for Japan - EU, it even worse than HE/Cogent somehow. So this action won't make Japan network has any downside change..
This could lead some packet loss and speed issue in recent.
DMIT current strategy is:
Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize.
Outbound: Balance between all ISPs, unless special optimize.
====
GSL is the primary inbound ISPs for DMIT as the current design. But the unfortunate things (fiber cut + massive DDoS) happened just after DMIT had connect with GSL. The timing is super unlucky.
Please wait us (DMIT+GSL) for a while for the network construction.
At the end, DMIT will primarily use our backbone to carry APAC<>US/EU access to lower latency and less jitter.
*Notes: the GSL has many DDoS protection customer but they are using "detect" method for DDoS mitigation in default which means once there is a huge attack; it might congested the subsea for few secs before the detection finish detouring.
With using DMIT's subsea/backbone, it minimize the impact; since the territory fiber(within continents) has grater capacity, and before the traffic leave continents it arrives nearest DMIT/GSL PoP. On the DMIT US/EU PoP, we can easily scale up to multiple 100G ports for against the peak traffic; at the same time we use port mirroring and DPDK capture for DDoS detection which gives less than secs DDoS mitigation/diversion. This cam make sure all the traffic going thru subsea will be mostly legit.
===DMIT===
DMIT has signed multiple subsea between Hong Kong, Tokyo, and Los Angeles.
It will be alive between Aug ~ Sep, 2025.
HK-TY: ASE
HK-US: FASTER + SJC, ASE + JUNO
US-TY: JUNO
===GSL===
GSL is going to bring SJC2 [HK-TY] alive in early August, 2025.
C2C[HK-TW] ETR is end of August, 2025
NCP[HK-KR] ETR is mid of August, 2025.
===LAST===
DMIT will disconnect with RETN in Tokyo after terms end.
The Hong Kong will be kept. But DMIT will use alternative strategy for EU-HK latency opts.
RETN removed Japan - EU territory fiber a month after we signed the contract; and they won't restore it.
RETN's subsea isn't good for Japan - EU, it even worse than HE/Cogent somehow. So this action won't make Japan network has any downside change..
DMIT observed China statewide TCP 443 disconnection, reset or block.
The IP replacement won't fix the issue; due to we might receive over load ticket or inquiries; the related ticket might be auto reply, and the IP replacement request might be rejected directly during this time.
Thank you for your support and patience.
The IP replacement won't fix the issue; due to we might receive over load ticket or inquiries; the related ticket might be auto reply, and the IP replacement request might be rejected directly during this time.
Thank you for your support and patience.
We have observed that Alipay callbacks have been affected.
We will manually match the deposits after restoration.
We will manually match the deposits after restoration.
TYO Scheduled Maintenance
Aug 23, 3 PM JST (2PM CST)
Duration: <= 60mins per node
Hardware component modification.
Aug 23, 3 PM JST (2PM CST)
Duration: <= 60mins per node
Hardware component modification.
/var/log/DMIT-NOC.log pinned «TYO Scheduled Maintenance Aug 23, 3 PM JST (2PM CST) Duration: <= 60mins per node Hardware component modification.»
HKG Pro:
We have detected abnormal trickle traffic. DMIT has implemented firewall rules and deducted a certain amount from the affected customers' transfer billing.
We have detected abnormal trickle traffic. DMIT has implemented firewall rules and deducted a certain amount from the affected customers' transfer billing.
/var/log/DMIT-NOC.log
Hong Kong GSL now has only 1/5 subsea alive within Hong Kong. This could lead some packet loss and speed issue in recent. DMIT current strategy is: Inbound: one primary at each location. Additional 1~3 ISPs as backup or regional optimize. Outbound: Balance…
DMIT has launched the following circuits:
1. ASE: HKG-TYO
2. JUPITER: TYO-LAX
80% traffic from the U.S. towards to HKG and TYO is now way more better than months ago, and other GSL clients in APAC.
The return traffic from APAC towards to the U.S. is still direct thru GSL since there is less congestion, and also for less unexpected route detour.
The SEA and MIA has been excluded, due to route leaking and worse latency. SEA-LAX-TYO will increase the latency between SEA-TYO.
The following is pending installed:
1. JUNO: TYO-LAX
2. PC1/NCP: TYO-SEA
Once JUNO has been installed, there will be more capacity and additional backup.
PC-1 and NCP will bring backup; also lower latency to finish up our APAC-NA design.
=====
At the same time, DMIT observed the following things:
1. GSL still experience SIN-TYO fiber offline. The traffic between TYO and SIN need bypass HKG which is kind overloaded in recent.
2. GSL has increasing sudden packet loss rate; it might caused by increasing number of high risk target has also connected GSL for DDoS protection.
DMIT’s plan:
1. Having protected TYO-SIN connectivity.
2. Having protected HKG-SIN connectivity.
3. Finishing TYO-SEA and JUNO connectivity.
It helps us prevent the traffic congestion on the GSL backbone.
1. ASE: HKG-TYO
2. JUPITER: TYO-LAX
80% traffic from the U.S. towards to HKG and TYO is now way more better than months ago, and other GSL clients in APAC.
The return traffic from APAC towards to the U.S. is still direct thru GSL since there is less congestion, and also for less unexpected route detour.
The SEA and MIA has been excluded, due to route leaking and worse latency. SEA-LAX-TYO will increase the latency between SEA-TYO.
The following is pending installed:
1. JUNO: TYO-LAX
2. PC1/NCP: TYO-SEA
Once JUNO has been installed, there will be more capacity and additional backup.
PC-1 and NCP will bring backup; also lower latency to finish up our APAC-NA design.
=====
At the same time, DMIT observed the following things:
1. GSL still experience SIN-TYO fiber offline. The traffic between TYO and SIN need bypass HKG which is kind overloaded in recent.
2. GSL has increasing sudden packet loss rate; it might caused by increasing number of high risk target has also connected GSL for DDoS protection.
DMIT’s plan:
1. Having protected TYO-SIN connectivity.
2. Having protected HKG-SIN connectivity.
3. Finishing TYO-SEA and JUNO connectivity.
It helps us prevent the traffic congestion on the GSL backbone.