Fault Injection Research

Existing Works

• 故障演练 Chaos
• 火山引擎混沌工程之云原生场景实现
• 字节跳动异构场景下的高可用建设实践
• AWS FIS
  • AWS Fault Injection Simulator
  • Can select the target _figures by type, tag, ARN, or by querying for specific attributes.
  • Can stop the experiment if one or more stop conditions (as defined by CloudWatch Alarms) are met.
• Alibaba
  • 只能针对单个实例设置故障
  • 提供的故障注入配置项较多，能够设置流量精确匹配，设置的一些内容和AWS在会场上说的都比较一致。
• Huawei
  • 配置项是简单的开启关闭，开启多久，故障概率，没有实现流量精确匹配，也不能制定故障行为（时延，包损坏等）
• awesome-chaos-engineering
• ChaosBlade：阿里开源的故障注入工具。
• Chaos Mesh
  • Github Repo: Implimented by Go.
  • Supported network failures

• NSDI 11, FATE and DESTINI: A Framework for Cloud Recovery Testing: Testing cloud services with system failure injection.
• NSDI 12, A NICE Way to Test OpenFlow Applications: OpenFlow applications.
• HotCloud 13, Towards a Fault-Resilient Cloud Management Stack: Openstack
• ASE 20, CoFI: Consistency-Guided Fault Injection for Cloud Systems: network paration fault
• OSDI 22, Automatic Reliability Testing for Cluster Management Controllers:k8s applications.
• NSDI 23, Push-Button Reliability Testing for Cloud-Backed Applications with Rainmaker : dot Net Applications, Rainmaker puts its focus on transient errors faced by cloud-backed applications
• VirtualWire: a fault injection and analysis tool for network protocols, ICDCS 2003: with a fault specific language
• ThorFI: A Novel Approach for Network Fault Injection as a Service: targeting OVS network failures, implemented with Openstack
• Observability and Chaos Engineering on System Calls for Containerized Applications in Docker: targeting system call in docker
• Maelstrom: Mitigating Datacenter-level Disasters by Draining Interdependent Traffic Safely and Efficiently
• Using Chaos Engineering to Improve The Resiliency of Transportation Cyber Physical Systems: A Novel Approach to the Test and Evaluation of Complex Systems

Network Faults

• SIGCOMM 11, Understanding Network Failures in Data Centers: Measurement, Analysis, and Implications
• OSDI 18, An Analysis of Network-Partitioning Failures in Cloud Systems
• TON 19, Understanding the Limits of Passive Realtime Datacenter Fault Detection and Localization: Moreover, failures are not always obvious; network components can fail partially, dropping or delaying only subsets of packets. Thus, traditional fault detection techniques involving end-host or router-based statistics can fall short in their ability to identify these errors.
• NSDI 21, Debugging Transient Faults in Data Centers using Synchronized Network-wide Packet Histories
• Understanding and Mitigating Packet Corruption in Data Center Networks

Testing

• Frisbee: automated testing of Cloud-native applications in Kubernetes
• Test Coverage for Network Configurations
• Norma: Towards Practical Network Load Testing

Summary — Dec 21, 2023