The thing that broke the net yesterday

There was trouble on the internet yesterday.

Tuesday Morning, various networks experienced outages from 4-6am EDT (8-10am UTC). [It] appears the outage was the result of a somewhat anticipated problem with older routers and their inability to deal with the ever increasing size of the Internet’s routing table.

Older routers were designed to handle an impossibly huge, not possible to pass, 512K router table entries. So big. No chance this could ever be an issue. Except now we’re hovering right around that threshold and some older routers with that limit are being tested and found wanting.

From this Cisco document:

In March, 2014, the Classless Inter-Domain Routing (CIDR) Report, which provides statistics on the global Internet routing table, reported that the global Internet routing table had passed 500,000 routes for the first time in Internet history.

Most platforms have more than enough space to support larger routing tables, but the default configurations might require adjustment. As the Internet routing table approaches 512,000 routes, it can cause the Catalyst 6500 and 7600 3BXL-based modules to exceed the default routing TCAM allocations.

Bottom line, this is fixable, even without switching out your old routers. Presumably, yesterday’s outage has spurred IT folks to action.

It’s a growth spurt.



  • Austin Brower

    It’s not entirely correct to call this a growth spurt. Some analyses have pointed to the fact that most of the “extra” routes that were propagated originated as more specific routes for existing IP networks. ( See http://www.bgpmon.net/what-caused-todays-internet-hiccup/ for details.)

    This problem of deaggregation has been ongoing and the source of much debate within the internet operations community. There are legitimate reasons to deaggregate (traffic engineering) but it’s mostly just bad operational practice/overly busy internet engineers.

  • Bob Greschke

    They should have listened to Bill Gates. He said 640K, not 512K. :)