Incident: Partial disruption 2023-09-11 00:10 - 07:20 due to DDoS attack

Type	MaintenanceIncident
Summary	UpdateDDos ofattack multipleon server / infrastructure componentsclimatejustice.social
Impact	Multiple downtimes of impactedall services
Duration	2023-08-1209-11 19:0000:10 - 2023-08-13 01:0007:20 (CEST)
Status	SuccessInvestigating
Reporter(s)	@b2c@wien.rocksMonitoring, @damadmai, upstream provider
Responder(s)	@fossie@wien.rocks, @b2c@wien.rocks
Internal reference	n/a

IntroductionSummary

Maintenance hasDue to bea conductedDDos attack on multiplethe coreMastodon infrastructureinstance servers and components to apply necessary software, server, distribution, security and firmware updates. This will ensure continued stable and secure operation ofclimatejustice.global the fediverse.foundation environment.

services

Servicesuffered interruptionsa arepartial tooutage bein expectedthe dueindicated to multiple rebootsperiod of the impacted components.

We apologize for any inconvenience!time.

MaintenanceSymptoms

fediverse.foundation services responding slowly or becoming unavailable.

 

Investigation

Log analysis shows more than one million requests per minute on the public endpoints /public/local and /explore of climatejustice.social

 

Workaround

Upstream provider took action and blackholed offending traffic.

 

Resolution

n/a

 

Follow-up tasks

• RouterReport firmwareattacker upgradeIPs to upstream provider abuse contact(s)
• Discussion with hoster and provider on how to handle such situations in the future
• Upstream DDos protection systems

• Configuration changes
  • Configure NICs on various virtual machines for IPv6 multicast
  • Misc. VM configuration changes (CPU/Memory allocation etc.)

• Distribution upgrade on all Debian-based servers
  
  • Upgrade all physical servers to Debian 12 'Bookworm'
  • Upgrade all virtual machines to Debian 12 'Bookworm'

• Update Openshift/OKD to 4.13

• Introduction of the Mastodon "Translate" feature (based on libretranslate)

• Update to Mastodon 4.2-beta1 on instance fedi.at