Like Tree4Likes
  • 3 Post By shri
  • 1 Post By shri

Network Instability

  1. #1

    Join Date
    Dec 2002
    Location
    ???
    Posts
    25,975

    Network Instability

    Looks like our cache / CDN provider Cloudflare (who also serve over half or more of HK's websites) is having some issues.

    No conspiracies until they publish their post-mortem report...

    But expect things to be unstable for a while.

    Will have to work on some alternatives - second time in 2-3 weeks that they've had some issues and we're now looking at rather big single point of failure.

    Have a GeoExpat related problem - please create a support ticket.

  2. #2

    Join Date
    Aug 2006
    Posts
    6,214

    Websites going down ...

    HKFP, LIHKG and even geoexpat all down for a few minutes around 10pm on 2 July. Coincidence, or a sign of things to come?


  3. #3

    Join Date
    Dec 2002
    Location
    ???
    Posts
    25,975

    @drumbrake - I suspect a sign of things to come. We'll have to reconfigure our stack to make sure Cloudflare outages can be detected and the site can be switched over easily.

    Have a GeoExpat related problem - please create a support ticket.

  4. #4

    Join Date
    Dec 2002
    Location
    ???
    Posts
    25,975
    Have a GeoExpat related problem - please create a support ticket.

  5. #5

    Join Date
    Jun 2010
    Location
    SoCal
    Posts
    6,427
    Quote Originally Posted by shri
    Looks like our cache / CDN provider Cloudflare (who also serve over half or more of HK's websites) is having some issues.

    No conspiracies until they publish their post-mortem report...

    But expect things to be unstable for a while.

    Will have to work on some alternatives - second time in 2-3 weeks that they've had some issues and we're now looking at rather big single point of failure.
    You could party like it's 1995 and email a weekly newsletter that is a redux of the happenings in HK and letters from readers.

  6. #6

    Join Date
    Jan 2011
    Posts
    2,144
    Cloudflare blames bad software deployment for todays outage
    This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred.

  7. #7

    Join Date
    Dec 2002
    Location
    ???
    Posts
    25,975

    Hanlon's Razor ...

    hullexile likes this.
    Have a GeoExpat related problem - please create a support ticket.