1 sur 23

🇬🇧 Introduction Mangle enables you to run chaos engineering experiments

The Mangle Documentation provides information about how to install, configure, and use Mangle™.

To navigate to the appropriate documentation, start with the Mangle GitHub IO Page.

Current Product version: 2.0.1

Intended Audiences

This information is intended for the following audiences:

VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304

Source : https://github.com/vmware/mangle/blob/master/docs/overview.md

Overview

Mangle enables you to run chaos engineering experiments seamlessly against applications and infrastructure components to assess resiliency and fault tolerance. It is designed to introduce faults with very little pre-configuration and can support any infrastructure that you might have including K8S, Docker, vCenter or any Remote Machine with ssh enabled. With it's powerful plugin model, you can define a custom fault of your choice and run it without actually building your code from scratch.

The solution is delivered both as an appliance and as containers. It has the following major components:

Mangle Cassandra Database Container forms the persistence layer for the application and is shared across all the nodes in a multi node instance of Mangle. It is preferable to use an external mount point/persistent volumes for the Cassandra DB so that there is no data loss if the container restarts.

Mangle Application Container runs the core fault injection and scheduling engine. It retrieves and stores information from the database, controls the connections to an endpoint and runs faults using a robust task framework.

Mangle vSphere Adapter is a separate container that manages connections and run faults against the vCenter endpoints. It is not included as part of the core application in-order to make it as light weight as possible.

VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304

www.vmware.com

Mangle Deployment and Administration Guide

Mangle Deployment and Administration Guide provides information about how to install and configure Mangle as an administrative user.

Product version: 2.0.1

Intended Audience

This information is intended for Mangle administrators who would be setting up Mangle, adding users, adding metric providers for monitoring faults, setting log levels and creating support bundles. Knowledge of and will be useful.

VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304

Mangle Users Guide

Mangle Users Guide provides information about how to add endpoints, run faults and view reports.

Product version: 2.0.1

Intended Audience

This information is intended for SRE, Developers and Chaos engineers who would like to run chaos experiments against infrastructure or applications to assess the resilience of their applications when subjected to unexpected failures.

VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304

Mangle Troubleshooting Guide

This guide will help you to diagnose common issues with Mangle deployment and operation and determine what information to collect for further debugging.

General Support Information

This information about the environment and events that occurred leading to the failure should be included in every support request to assist in debugging.

Basic Information

REQUIRED:

Endpoint type:
Details of Fault that was injected:
Type of Deployment: OVA, Container (Single Node), Container (Multi Node), Container (Multi Node with HA Proxy)
What stage of the Mangle lifecycle are you running into the issue?
Attach the Mangle support bundle

OPTIONAL, but helpful:

IP address of Mangle:
Hostname of Mangle:
IP address of Endpoint:
Hostname of Endpoint:

Detailed Information

What operation was being performed when the failure was noticed?
Provide information from the Support Information section of the appropriate Mangle Lifecycle stage
Provide additional detail as necessary

Mangle Support Bundle

Please run /etc/vmware/support/mangle-support.sh and provide the resulting file to support. This script gathers application state and log information and is the best tool for gathering comprehensive support information. Provide this output for all support requests along with the Support Information from the corresponding stage of the Appliance Lifecycle.

The location of the resulting log bundle is shown in the script output, similar to this example:

Provide this .tar.gz file to support.

Mangle Lifecycle

It is important to determine what stage in the appliance lifecycle you are at when encountering issues so that targeted troubleshooting steps can be followed. Please use the links below to identify what stage the failure is in, to apply the appropriate troubleshooting steps, and to provide the appropriate troubleshooting information for support.

Mangle Developers' Guide

Sub Modules

Mangle is a spring-boot application with implementation of web services to invoke Fault injection on Supported Endpoints.

The mangle Code is organised is as below sub modules using Maven build tool.

mangle-byteman-root:

Module for the Mangle java Agent, which is used to support application level fault injection against apps running on JVM.

mangle-models:

Module for the complete data model of Mangle Web Application.

mangle–utils:

Module for all the core logic shared across Mangle Application.

mangle-task-framework:

Module for command execution orchestration in Mangle. Every fault execution or request processing in Mangle is handled as an asynchronous task. This module also contains the code responsible for managing scheduler functionality of Mangle.

mangle-test-plugin:

Module for the test utilities required for testing the mangle–task-framework.

mangle-default-plugin:

Module for all the out of the box faults supported by Mangle as a plugin. This is developed using the pf4j-spring framework. Mangle can support fault execution only if this module is available as plugin.

mangle-ui

Module for the presentation layer of mangle.

mangle –services

Module for core web services exposed to user, corresponding persistence layer and business logic.

mangle-vcenter-adapter:

Module for the spring-boot application that has to be deployed as a container or hosted as an application reachable by Mangle for executing faults against VMware vCenter Server.

The order of execution of sub modules is:

Build Profiles

Different build profiles available in Mangle pom are:

default

This profile builds mangle with last known good configuration of mangle-byteman-agent and does not take latest changes to mangle-byteman-root module into consideration.

To build:

build-all

This profile builds mangle with latest changes of mangle-byteman-agent and is recommended for use if the Mangle java agent jar has to be updated.

To build using this profile:

byteman

This profile builds only the mangle-byteman-root module with the latest changes.

To build using profile byteman

vcenter-adapter

This profile builds only mangle-vcenter-adapter with the latest changes.

Building the code

Prerequisites

Java

Java 8 JDK installed on your OS of choice (Mac OSX, Linux variants, Windows are all supported hosts)
Git for source code management.

Maven

Maven is used to build and test the project. Install maven 3.5.X to your local system.

(Optional) Install Maven with your system's package manager (e.g. apt on Ubuntu/Debian, homebrew on OSX, ...).
Set your JAVA_HOME environment variable to be the home of the Java 8 JDK. On OSX, this lands in /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/.
Run mvn clean install to compile the code, run checkstyle and run unit tests.

Packaging a fat jar

Resulting JAR goes to mangle-services/target/mangle-services.jar.

./mvnw clean package -DskipTests (packages without running tests)

Code Style

License

Each source file has to specify the license at the very top of the file:

Line length

Set to 100

Import statements

The order of import statements is:

import static java.*
import static javax.*
blank line
import static all other imports
blank line
import static com.vmware.*
blank line
import java.*
import javax.*
blank line
import all other imports
blank line
import com.vmware.*

Comments are always placed at independent line. Do not append the comment at the end of the line.

Wrong

Correct

Commit message

Follow the widely used format:

Sample:

50 char in title
Wrap the body at 72 char or less

Checkstyle

Checkstyle runs as part of maven validate lifecycle.

You can call it manually like ./mvnw validate or ./mvnw checkstyle:checkstyle.

IDE Settings

Formatter

For both Eclipse and IntellJ, import contrib/eclipse-java-style.xml

IntelliJ

IntelliJ can import eclipse formatter file.

Preference - Editor - Code Style - Manage - Import

Import contrib/eclipse-java-style.xml.

IntelliJ Specific

Setting java package import order

Update .idea/codeStyleSettings.xml with contrib/idea-java-style.xml
Restart IntelliJ

Building Mangle Docker Images

Prerequisites

To build a docker image for Mangle

To build a docker image for Mangle vCenter Adapter

Contributing to Mangle

The Mangle project team welcomes contributions from the community. We are always thrilled to receive pull requests, and do our best to process them as fast as we can. If you wish to contribute code and you have not signed our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any questions about the CLA process, please refer to our FAQ.

Before you start to code, we recommend discussing your plans through a Github issue or discuss it first with the official project maintainers via Slack, especially for more ambitious contributions. This gives other contributors a chance to point you in the right direction, give you feedback on your design, and help you find out if someone else is working on the same thing.

Contribution Flow

This is a rough outline of what a contributor's workflow looks like:

Create a topic branch from where you want to base your work
Make commits of logical units
Make sure your commit messages are in the proper format (see below)
Push your changes to a topic branch in your fork of the repository
Submit a pull request

Example:

git remote add upstream https://github.com/vmware/mangle.gitgit checkout -b my-new-feature mastergit commit -agit push origin my-new-feature

Staying In Sync With Upstream

When your branch gets out of sync with the vmware/master branch, use the following to update:

git checkout my-new-featuregit fetch -agit pull --rebase upstream mastergit push --force-with-lease origin my-new-feature

Updating pull requests

If your PR needs changes based on code review, you'll most likely want to squash these changes into existing commits.

If your pull request contains a single commit or your changes are related to the most recent commit, you can simply amend the commit.

git add .git commit --amendgit push --force-with-lease origin my-new-feature

If you need to squash changes into an earlier commit, you can use:

git add .git commit --fixup git rebase -i --autosquash mastergit push --force-with-lease origin my-new-feature

Be sure to add a comment to the PR indicating your new changes are ready to review, as GitHub does not generate a notification when you git push.

Code Style

Formatting Commit Messages

We follow the conventions on How to Write a Git Commit Message.

Be sure to include any related GitHub issue references in the commit message. See GFM syntax for referencing issues and commits.

Reporting Bugs and Creating Issues

When opening a new issue through Github, try to roughly follow the commit message format conventions above.

Repository Structure

├──  assets/files├──  checkstyle├──  docker├──  docs├──  formatter├──  mangle-byteman-root│    ├──  agent│    ├──  bin│    ├──  download│    ├──  install│    ├──  licenses│    │    submit│    │    README│    │    pom.xml ├──  mangle-default-plugin├──  mangle-metric-reporter├──  mangle-models├──  mangle-services├──  mangle-support├──  mangle-task-framework├──  mangle-test-plugin├──  mangle-ui├──  mangle-utils├──  mangle-vcenter-adapter│    .gitbook.yaml│    .gitignore│    CONTRIBUTING.md│    LICENSE│    NOTICE│    pom.xml

mangle-administration

Supported Deployment Models

Single Node Deployments

Deploying the Mangle Virtual Appliance

For a quick POC we recommend that you deploy a single node instance of Mangle using the OVA file that we have made available here.

Using the OVA is a fast and easy way to create a Mangle VM on VMware vSphere.

After you have downloaded the OVA, log in to your vSphere environment and perform the following steps:

Start the Import Process
From the Actions pull-down menu for a datacenter, choose Deploy OVF Template.

Provide the location of the downloaded ova file.
Choose Next.
Specify the name and location of virtual machine
Enter a name for the virtual machine and select a location for the virtual machine.

Choose Next.
Specify the compute resource
Select a cluster, host or resource pool where the virtual machine needs to be deployed.

Choose Next.
Review details

Choose Next.
Accept License Agreement
Read through the Mangle License Agreement, and then choose I accept all license agreements.

Choose Next.
Select Storage
Mangle is provisioned with a maximum disk size. By default, Mangle uses only the portion of disk space that it needs, usually much less that the entire disk size ( Thin client). If you want to pre-allocate the entire disk size (reserving it entirely for Mangle instead), select Thick instead.

Choose Next.
Select Network
Provide a static or dhcp IP for Mangle after choosing an appropriate destination network.

Choose Next.
Verify Deployment Settings and click Finish to start creating the virtual machine. Depending on bandwidth, this operation might take a while. When finished, vSphere powers up the Mangle VM based on your selections.

After the VM is boots up successfully, open the command window. Press Alt+F2 to log into the VM.

The default account credentials are:

Username: root
Password: vmware

Because of limitations within OVA support on vSphere, it was necessary to specify a default password for the OVA option. However, for security reasons, we would recommend that you modify the password after importing the appliance.

It takes a couple of minutes for the containers to run. Once the Mangle application and DB containers are running, the Mangle application should be available at the following URL:

You will be prompted to change the admin password to continue.

Default Mangle Username: [email protected]
Password: admin

Export the VM as a Template (Optional)

Consider converting this imported VM into a template (from the Actions menu, choose Export ) so that you have a master Mangle instance that can be combined with vSphere Guest Customization to enable rapid provisioning of Mangle instances.

Now you can move on to the Mangle Users Guide.

Deploying the Mangle Containers

Prerequisites

Before creating the Mangle container a Cassandra DB container should be made available on a Docker host. You can choose to deploy the DB and the Application container on the same Docker host or on different Docker hosts. However, we recommend that you use a separate Docker host for each of these. You can setup a Docker host by following the instructions here.

To deploy Cassandra, you can either use the authentication enabled image tested and verified with Mangle available on the Mangle Docker repo or use the default public Cassandra image hosted on Dockerhub.

If you chose to use the Cassandra image from Mangle Docker Repo:

docker run --name mangle-cassandradb -v /cassandra/storage/:/var/lib/cassandra -p 9042:9042 -d -e CASSANDRA_CLUSTER_NAME="manglecassandracluster" -e CASSANDRA_DC="DC1" -e CASSANDRA_RACK="rack1" -e CASSANDRA_ENDPOINT_SNITCH="GossipingPropertyFileSnitch"  mangleuser/mangle_cassandradb:

If you chose to use the Cassandra image from Dockerhub:

docker run --name mangle-cassandradb -v /cassandra/storage/:/var/lib/cassandra -p 9042:9042 -d -e CASSANDRA_CLUSTER_NAME="manglecassandracluster" -e CASSANDRA_DC="DC1" -e CASSANDRA_RACK="rack1" -e CASSANDRA_ENDPOINT_SNITCH="GossipingPropertyFileSnitch" cassandra:3.11

Deploying the Mangle application container

To deploy the Mangle container using a Cassandra DB deployed using the image from Mangle Docker repo or with DB authentication and ssl enabled, run the docker command below on the docker host after substituting the values in angle braces <> with actual values.

docker run --name mangle -d -e DB_OPTIONS="-DcassandraContactPoints= -DcassandraSslEnabled=true -DcassandraUsername=cassandra -DcassandraPassword=cassandra" -e CLUSTER_OPTIONS="-DclusterValidationToken=mangle -DpublicAddress=" -p 8080:8080 -p 8443:8443 mangleuser/mangle:

To deploy the Mangle container using a Cassandra DB deployed using the image from Dockerhub or with DB authentication and ssl disabled, run the docker command below on the docker host after substituting the values in angle braces <> with actual values.

docker run --name mangle -d -e DB_OPTIONS="-DcassandraContactPoints= -DcassandraSslEnabled=false" -e CLUSTER_OPTIONS="-DclusterValidationToken=mangle -DpublicAddress=" -p 8080:8080 -p 8443:8443 mangleuser/mangle:

The Mangle docker container takes two environmental variables

"DB_OPTIONS", which can take a list of java arguments identifying the properties of the database cluster

"CLUSTER_OPTIONS", which can take a list of java arguments identifying the properties of the Mangle application cluster

Although the docker run commands above lists only a few DB_OPTIONS and CLUSTER_OPTIONS parameters, Mangle supports a lot more for further customization.

Supported DB_OPTIONS

-DcassandraContactPoints : IP Address of Cassandra DB (mandatory, default value is "localhost" and works only if the Mangle application and DB are on the same Docker host)-DcassandraClusterName : Cassandra cluster name (mandatory, value should be the one passed during cassandra db creation as an environmental variable CASSANDRA_CLUSTER_NAME)-DcassandraKeyspaceName : Cassandra keyspace name (optional, default value is "mangledb" if you are using the Cassandra image file from Mangle Docker repo)-DcassandraPorts : Cassandra DB port used (optional, default value is "9042")-DcassandraSslEnabled : Cassandra DB ssl configuration (optional, default value is "false"...mandatory and should be set to true if ssl is enabled)-DcassandraUsername : Cassandra DB username (mandatory only if the Cassandra DB is created with authentication enabled)-DcassandraPassword : Cassandra DB password (mandatory only if the Cassandra DB is created with authentication enabled)-DcassandraSchemaAction : Cassandra DB schema action (optional, default value is "create_if_not_exists")-DcassandraConsistencyLevel : Cassandra DB Consistency level (optional, default value is "local-quorum")-DcassandraSerialConsistencyLevel : Cassandra DB serial consistency level (optional, default value is "local-serial")-DcassandraDCName : Cassandra DB DC name (optional, value should be the one passed during cassandra db creation as an environmental variable CASSANDRA_DC or if it is set a value other than "DC1")-DcassandraNoOfReplicas : Cassandra DB replicas numbers (optional, default value is "1"...mandatory only if multiple nodes are available in the DB cluster)

Supported CLUSTER_OPTIONS

-DpublicAddress : IP address of the mangle node (mandatory)-DclusterValidationToken : Any string token name for mangle cluster (mandatory and should be kept in mind if more nodes need to be added to the cluster.)-DclusterName : Any string cluster name for mangle (optional, default value is "mangle")-DclusterMembers : Members in the mangle cluster (optional)

Deploying the Mangle vCenter adapter container

Mangle vCenter Adapter is a fault injection adapter for injecting vCenter specific faults. All the vCenter operations from the Mangle application will be carried out through this adapter.

To deploy the vCenter adapter container run the docker command below on the docker host.

docker run --name mangle-vc-adapter -v /var/opt/mangle-vc-adapter-tomcat/logs:/var/opt/mangle-vc-adapter-tomcat/logs -d -p 8080:8080 -p 8443:8443 mangleuser/mangle_vcenter_adapter:

The API documentation for the vCenter Adapter can be found at:

https://::/mangle-vc-adapter/swagger-ui.html

The vCenter adapter requires authentication against any API calls. It supports only one user, admin with password admin. All the post APIs that are supported by the adapter will take the vCenter information as a request body.

Deploying Mangle on Kubernetes

All the relevant YAML files are available under the Mangle git repository under location 'mangle\mangle-support\kubernetes'

Create a namespace called 'mangle' under the K8s cluster.
kubectl --kubeconfig= create namespace mangle
Create Cassandra pod and service.
kubectl --kubeconfig=-n mangle apply -f/cassandra.yaml
kubectl --kubeconfig=-n mangle apply -f/cassandra-service.yaml
Create Mangle pod and service
kubectl --kubeconfig=-n mangle apply -f/mangle.yaml
kubectl --kubeconfig=-n mangle apply -f/mangle-service.yaml

Multi Node Deployment

A multi-node setup for Mangle ensures availability in case of unexpected failures. We recommend that you use a 3 node Mangle setup.

Prerequisites

You need at least 4 docker hosts for setting up a multi node Mangle instance; 1 for the Cassandra DB and 3 for Mangle application containers . You can setup a docker host by following the instructions here.

A multi node setup of Mangle is implemented using Hazelcast. Mangle multi node setup uses TCP connection to communicate with each other. The configuration of the setup is handled by providing the right arguments to the docker container run command, which identifies the cluster.

The docker container takes an environmental variable "CLUSTER_OPTIONS", which can take a list of java arguments identifying the properties of the cluster. Following are the different arguments that should be part of "CLUSTER_OPTIONS": clusterName - A unique string that identifies the cluster to which the current mangle app will be joining. If not provided, the Mangle app will by default use string "mangle" as the clusterName, and if this doesn't match the one already configured with the cluster, the node is trying to join to, container start fails. clusterValidationToken - A unique string which will act similar to a password for a member to get validated against the existing cluster. If the validation token doesn't match with the one that is being used by the cluster, the container will fail to start.

publicAddress - IP of the docker host on which the mangle application will be deployed. This is the IP that mangle will use to establish a connection with the other members that are already part of the cluster, and hence it is necessary to provide the host IP and make sure the docker host is discoverable from other nodes clusterMembers - This is an optional property that takes a comma-separated list of IP addresses that are part of the cluster. If not provided, Mangle will query DB and find the members of the cluster that is using the DB and will try connecting to that automatically. It is enough for mangle to connect to at least one member to become part of the cluster.

deploymentMode - Takes either CLUSTER/STANDALONE value. Deployment Mode parameter is mandatory for the multi-node deployment, with the value set to CLUSTER on every node that will be part of the cluster, where as for the single node deployment, this field can be ignored, which by default will be STANDALONE. If DB was used by one type of deployment setup, one needs to update this parameter to support the other type, either though the mangle or by directly going in to DB.

NOTE:

All the nodes (docker hosts) participating in the cluster should be synchronized with a single time server.

If a different mangle app uses the same clusterValidationToken, clusterName and DB of existing cluster, the node will automatically joins that existing cluster.

All the mangle app participating in the cluster should use the same cassandra DB.

The properties clusterValidationToken and publicAddress are mandatory for any mangle container spin up, if not provided container will fail to start.

Deploy a Cassandra DB container by referring to the section here.

Deploy the Mangle cluster by bringing up the mangle container in each docker host.

For the first node in the cluster:

docker run --name mangle -d -v /var/opt/mangle-tomcat/logs:/var/opt/mangle-tomcat/logs -e DB_OPTIONS="-DcassandraContactPoints=" -e CLUSTER_OPTIONS="-DclusterName= -DclusterValidationToken= -DpublicAddress= -DdeploymentMode=CLUSTER" -p 8080:8080 -p 443:8443 -p 5701:5701 mangleuser/mangle:

For the subsequent nodes in the cluster:

docker run --name mangle -d -v /var/opt/mangle-tomcat/logs:/var/opt/mangle-tomcat/logs -e DB_OPTIONS="-DcassandraContactPoints=" -e CLUSTER_OPTIONS="-DclusterName= -DclusterValidationToken= -DpublicAddress= -DclusterMembers= -DdeploymentMode=CLUSTER" -p 8080:8080 -p 443:8443 -p 5701:5701 mangleuser/mangle:

docker run --name mangle -d -v /var/opt/mangle-tomcat/logs:/var/opt/mangle-tomcat/logs -e DB_OPTIONS="-DcassandraContactPoints=" -e CLUSTER_OPTIONS="-DclusterName= -DclusterValidationToken= -DpublicAddress= -DclusterMembers= -DdeploymentMode=CLUSTER" -p 8080:8080 -p 443:8443 -p 5701:5701 mangleuser/mangle:

Deployment Mode and Quorum

Mangle implements quorum strategy to for supporting HA and to avoid any split brain scenarios when deployed as a cluster.

Mangle can be deployed in two different deployment modes

STANDALONE: quorum value is 1, cannot be updated to any other value
CLUSTER: min quorum value is 2, cannot be set to value lesser than ceil((n + 1)/2), n being number of nodes currently in the cluster

If a node is deployed initially in the STANDALONE mode, and user changes the deployment mode to CLUSTER, the quorum will automatically be updated to 2. When user keeps adding new nodes to the cluster, the quorum value updates depending on the number of nodes in the cluster. eg: when user adds new node to 3 node cluster(quorum=2) making it 4 nodes in the cluster, the quorum value is updated to 3. Determined by the formula, ceil((n + 1)/2)

If a node is deployed initially in the CLUSTER mode, and user changes the deployment mode to STANDALONE mode, all other nodes except for the Hazelcast master will shutdown. They will lose the connection with the existing node, and won't take any post calls, and will entries will be removed from the cluster table.

NOTE:

Active members list of the active quorum will be maintained in DB under the table cluster. Similarly, master instance entry will also be maintained in the db under the same table.

Actions triggered when Mangle loses quorum

All the schedules will be paused
No post calls can be made against mangle
Task that were triggered on any node will continue to be executed until Failed/Completed
Removes its entry from the cluster table's members list
If the node was the last known master, it will remove it's entry as master from the table

Actions triggered when Mangle gets a quorum (from no quorum in any of the cluster member)

All the schedules that are in the scheduled state and all the schedules that were paused because of the quorum failure will be triggered
All the tasks that are in progress will be queue to executed after 5minutes of quorum created event (This is to avoid any simultaneous Execution)
Master (Oldest member in the cluster) will take care of adding the list of active members to the cluster table, and updating itself as master

Actions triggered when a Mangle instance joins one active cluster

Some of the schedules will be assigned to the new the node due to migration
Some of the tasks will assigned to the new node, and they will be queued for triggering after 5mins
Existing tasks that in-progress will continue to execute in the old node
Master (Oldest member in the cluster) will take care of adding new member's entry to the list of active members to the cluster table

Scenarios around quorum

Scenario 1: When a mangle instance in 3 node cluster, leaves a cluster due to network partition, and the quorum value is 2

Let us consider 3 instances a, b, c in the CLUSTER
a leaves the cluster due to network partition
a is not able to communicate with b, c
two cluster will be created, one having only a, and the other have b and c
since for the cluster having a doesn't have quorum(2), it loses the quorum, and removes itself as active member from cluster config
cluster having nodes b and c continue execute since they have enough nodes to satisfy the quorum

Scenario 2: When mangle instance in 2 node cluster leaves a cluster due to network partition, and the quorum value is 2

Let us consider 2 instances a, b in the CLUSTER
a leaves the cluster due to network partition, a cannot communicate with b and vice versa
two clusters are created with each having one node
both the cluster loses the quorum(2)

Advanced Cassandra Configuration

Open /etc/cassandra/cassandra.yaml and modify authenticator: from AllowAllAuthenticator to PasswordAuthenticator, so Cassandra will create a default user cassandra/cassandra.

To create own user : create dir /docker-entrypoint-initdb.d/ and create cql file init-query.cql with content (CREATE USER IF NOT EXISTS admin WITH PASSWORD 'vmware' SUPERUSER;) so it will create a user admin/vmware.

To execute the init-query.cql file on db startup, need to modify the docker-entrypoint.sh file, add the below content right before exec ""

for f in docker-entrypoint-initdb.d/*; do case "$f" in *.sh) echo "$0: running $f"; . "$f" ;; *.cql) echo "$0: running $f" && until cqlsh --ssl -u cassandra -p cassandra -f "$f"; do >&2 echo "Cassandra is unavailable - sleeping"; sleep 2; done & ;; *) echo "$0: ignoring $f" ;; esac echo done

Here, cqlsh --ssl -u cassandra -p cassandra used to run *.cql file (if ssl is not enabled then remove --ssl option)

Modify the start_rpc: true in /etc/cassandra/cassandra.yaml file.

To enable the SSL : generate the self sign certificate(Run generateDbCert.sh file inside container) and modify the /etc/cassandra/cassandra.yaml file with below content

To login cqlsh client : need to create a cqlshrc file and copy in /root/.cassandra/ and /home/cassandra/.cassandra/ folder

Exit from the running container and restart the container.

See logs : /var/log/cassandra .

Attaching the required files which help to enable the authentication and ssl in Cassandra base image.

To download the Cassandra client as DevCenter from .

To Create Multi-Node Cassandra cluster

Create seed Node :

Join the Other Node to Seed Node :

Admin Settings

Managing Authentication

Adding additional Authentication sources

Mangle supports using Active Directory as an additional authentication source.

Steps to follow:

Login as an admin user to Mangle.
Navigate to -----> Auth Management -----> Auth Source .
Click on .
Enter URL, Domain and click on Submit.
A success message is displayed and the table for Auth sources will be updated with the new entry.
Click on against a table entry to see the supported operations.

Relevant API List

For access to Swagger documentation, please traverse to link -----> API Documentation from the Mangle UI or access https:///mangle-services/swagger-ui.html#/auth-provider-controller

Adding/Importing Users

Mangle supports adding new local user or importing users from Active Directory sources added as additional authentication sources.

Steps to follow:

Login as an admin user to Mangle.
Enter User Name, Auth Source, Password if the Auth Source selected is "mangle.local", an appropriate role and click on Submit.
A success message is displayed and the table for Users will be updated with the new entry.

Relevant API List

Default and Custom Roles

Mangle has the following default Roles and Privileges.

Edit and Delete operations are supported only for custom roles. It is forbidden for default roles.

Mangle supports creation of custom roles from the default privileges that are available.

Steps to follow:

Login as an admin user to Mangle.
Enter Role Name, Privileges and click on Submit.
A success message is displayed and the table for Roles will be updated with the new entry.

Relevant API List

Loggers

Log Levels

Mangle supports modifying log levels for the application.

Steps to follow:

Login as an admin user to Mangle.
Enter Logger name, Configured Level, Effective Level and click on Submit.
A success message is displayed and the table for Log levels will be updated with the new entry.

Relevant API List

Integrations

Metric Providers

Mangle supports addition of either Wavefront or Datadog as metric providers. This enables the information about fault injection and remediation to be published to these tools as events thus making it easier to monitor them.

Steps to follow:

Login as an admin user to Mangle.
Choose Wavefront or Datadog, provide credentials and click on Submit.
A success message is displayed and the table for Monitoring tools will be updated with the new entry.

Relevant API List

sre-developers-and-users

Adding Endpoints

Supported Endpoints

Endpoint in Mangle refers to an infrastructure component that will be the primary target for your chaos engineering experiments.

For version 1.0, Mangle supported four types of endpoints:

Kubernetes
Docker
VMware vCenter
Remote Machine

From version 2.0, apart from the four endpoints listed above, support has been extended to the following new endpoint:

AWS (Amazon Web Services)

Kubernetes Endpoint

Mangle supports K8s clusters as endpoints or targets for injection. It needs a kubeconfig file to connect to the cluster and run the supported faults. If a kubeconfig file is not provided, Mangle assumes that it is running on a K8s cluster and targets the same cluster for fault injection.

Steps to follow:

Login as an user with read and write privileges to Mangle.
Navigate to Endpoint tab ---> Kubernetes Cluster.
Click on .
Enter a name, credential (kubeconfig file), namespace (mandatory...Please specify "default" if you are unsure of the namespace else provide the actual name), tags (refers to additional tags that should be send to the enabled metric provider to uniquely identify faults against that endpoint) and click on Test Connection.
If Test Connection succeeds click on Submit.
A success message is displayed and the table for Endpoints will be updated with the new entry.
Click on against a table entry to see the supported operations.

Docker Endpoint

Mangle supports docker hosts as endpoints or targets for injection. It needs the IP/Hostname, port details and certificate details (if TLS is enabled for the docker host with --tlsverify option specified) to connect to the docker host and run the supported faults.

Steps to follow:

Login as an user with read and write privileges to Mangle.
Navigate to Endpoint tab.
Enter a name, IP/Hostname, port details, tags (refers to additional tags that should be send to the enabled metric provider to uniquely identify faults against that endpoint), certificate details (if TLS is enabled for the docker host)and click on Test Connection.
If Test Connection succeeds click on Submit.
A success message is displayed and the table for Endpoints will be updated with the new entry.

VMware vCenter Endpoint

Mangle supports VMware vCenter as endpoints or targets for injection. It needs the IP/Hostname, credentials and a vCenter adapter URL to connect to the vCenter instance and run the supported faults.

Steps to follow:

Login as an user with read and write privileges to Mangle.
Navigate to Endpoint tab.
Enter a name, IP/Hostname, credentials, vCenter Adapter URL (format: "https://:"where the IP/hostname is the docker host where the adapter container runs appended with the port used), username, password, tags (refers to additional tags that should be send to the enabled metric provider to uniquely identify faults against that endpoint) and click on Test Connection.
If Test Connection succeeds click on Submit.
A success message is displayed and the table for Endpoints will be updated with the new entry.

When the vCenter adapter is deployed on the same machine on which Mangle is running, vCenter adapter IP used for adding vCenter endpoint can be either

A internal docker container IP OR
A docker host IP

To find out the internal docker container IP for mangle-vc-adapter run

docker inspect --format '{{.NetworkSettings.IPAddress}}' *mangle-vc-adapter

Remote Machine Endpoint

Mangle supports any remote machine with ssh enabled as endpoints or targets for injection. It needs the IP/Hostname, credentials (either password or private key), ssh details, OS type and tags to connect to the remote machine and run the supported faults.

Steps to follow:

Login as an user with read and write privileges to Mangle.
Navigate to Endpoint tab.
Enter a name, IP/Hostname, credentials (either password or private key), ssh port, ssh timeout, OS type, tags (refers to additional tags that should be send to the enabled metric provider to uniquely identify faults against that endpoint) and click on Test Connection.
If Test Connection succeeds click on Submit.
A success message is displayed and the table for Endpoints will be updated with the new entry.

AWS (Amazon Web Services)

Mangle supports AWS as endpoint or target for injection. It needs the Region, credentials (Access key ID and Secret key) and tags to connect to AWS and run the supported faults. Currently the only supported service is EC2. However, there are plans to extend this to other important services in AWS.

Steps to follow:

Login as an user with read and write privileges to Mangle.
Navigate to Endpoint tab.
Enter a name, Region, credentials (Access key ID and Secret key), tags (refers to additional tags that should be send to the enabled metric provider to uniquely identify faults against that endpoint) and click on Test Connection.
If Test Connection succeeds click on Submit.
A success message is displayed and the table for Endpoints will be updated with the new entry.

Relevant API Reference

For access to Swagger documentation:

Injecting Faults

Mangle supports two broad category of faults:

Infrastructure Faults
Application Faults

Infrastructure Faults are a set of faults that target IAAS components where developers host and run their applications. For eg: this might be a virtual machine or an AWS EC2 instance where the application runs as a service or a Docker host where the application containers are hosted or a K8s cluster where the pods host the application. These components are usually shared with multiple applications running on the same infrastructure and are referred to as endpoints in Mangle. So faults against these components will impact multiple applications unless they have different levels of fault tolerance.

Application Faults are a set of faults that target specific applications running within a given infrastructure component or endpoint. For eg: this could be a specific tomcat application running within a virtual machine or an AWS EC2 instance or JAVA applications running within containers on a Docker host or K8s pods. Faults against applications typically will impact just that application and ideally should not bring down any other applications running on the same infrastructure or is dependent on the affected service. If it does, your system is prone to cascading failures and should be examined in great detail to improve fault tolerance levels.

Infrastructure Faults

For version 1.0, Mangle supported the following types of infrastructure faults:

CPU Fault
Memory Fault
Disk IO Fault
Kill Process Fault
Docker State Change Faults
Kubernetes Delete Resource Fault
Kubernetes Resource Not Ready Fault
vCenter Disk Fault
vCenter NIC Fault
vCenter VM State Change Fault

From version 2.0, apart from the faults listed above, support has been extended to the following new faults:

File Handler Leak Fault
Disk Space Fault
Kernel Panic Fault
Network Faults: Packet Delay, Packet Duplication, Packet Loss, Packet Corruption
Kubernetes Service Unavailable Fault
AWS EC2 State Change Fault
AWS EC2 Network Fault

Minor improvements have also been included for Kill Process Fault in version 2 of Mangle.

CPU Fault

CPU fault enables spiking cpu usage values for a selected endpoint by a percentage specified by the user. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> CPU.
Select an Endpoint.
Provide a "CPU Load" value. For eg: 80 to simulate a CPU usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the CPU load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Memory Fault

Memory fault enables spiking memory usage values for a selected endpoint by a percentage specified by the user. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Memory.
Select an Endpoint.
Provide a "Memory Load" value. For eg: 80 to simulate a Memory usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Memory load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Disk IO Fault

Disk IO fault enables spiking disk IO operation for a selected endpoint by an IO size specified by the user in bytes. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Disk IO.
Select an Endpoint.
Provide a "IO Size" value in bytes. For eg: To write in blocks of 5 KB to the disk of the selected Endpoint specify the IO Size as 5120 (5 KB = 5120 bytes). With the specified block size of 5120 bytes, Mangle will not use more than 5 MB (5 MB = 5120 * 1024 bytes) of disk space during the simulation of fault. The space is cleared at the time of fault remediation.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the IO load of 8192000 to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kill Process Fault

Kill Process fault enables abrupt termination of any process that is running on the specified endpoint. Unlike other infrastructure faults like CPU, Memory and Disk IO this fault does not have a timeout field because the fault completes very quickly. Some processes/services may be configured for auto-start and some might require a manual start command to be executed. For the first case, auto-remediation through Mangle is not needed. For the second case, you can specify the remediation command that Mangle should use to start the process again. After the fault in completed and if remediation command was accurately specified, a manual remediation can be triggered from the Requests and Reports tab.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Kill Process.
Select an Endpoint.
Provide a "Process Identifier". This can either be a process id or process name. A process name is preferred if the fault is expected to be scheduled.
From version 2.0 onward, Kill Process fault can kill multiple processes with the same process descriptor. This can be done by setting the "Kill All" drop down to true. If set to false, it will fail if the process descriptor is not unique. Alternatively, you can also use the process id to uniquely identify and kill a process.
Provide a "Remediation Command". For eg: To start the sshd process that was killed on an Ubuntu 17 Server, specify the remediation command as "sudo service ssh start" .
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

File Handler Leak Fault

File Handler Leak fault enables you to simulate conditions where a program requests for a handle to a resource but does not release it when the resource is no longer in use. This condition if left over extended periods of time, will lead to "Too many open file handles" errors and will cause performance degradation or crashes. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> File Handler Leak.
Select an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the out of file handles error to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Disk Space Fault

Disk Space Fault enables you to simulate out of disk or low disk space conditions. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Disk Space.
Select an Endpoint.
Provide a "Target Directory" so Mangle can target a specific directory location or partition to write to for simulating the low disk space condition.
Provide a "Load" value. For eg: 80 to simulate a Disk usage of 80% of the total disk size or space allocated for a partition, on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the low disk or out of disk condition to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kernel Panic Fault

Kernel Panic Fault simulates conditions where the operating system abruptly stops to prevent further damages, security breaches or data corruption and facilitate diagnosis of a sudden hardware or software failure.

REMEDIATION OPTIONS FOR KERNEL PANIC

Remediation for Kernel Panic is controlled by the operating system itself. Typically on Linux systems, Kernel Panic is usually followed by a system reboot. But in some cases due to the settings specified under file /etc/sysctl.d/99-sysctl.conf the automatic system reboot may not occur. For such cases, a manual reboot needs to be triggered on the endpoint to bring it back to a usable state.

To modify this setting as a one-time option, please run the following command on the endpoint sysctl --system

To modify this setting permanently, remotely log in to the endpoint, modify file /etc/sysctl.d/99-sysctl.conf and add the following command

kernel.panic = 20

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Kernel Panic.
Select an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Network Faults

Network Faults enables you to simulate unfavorable conditions such as packet delay, packet duplication, packet loss and packet corruption. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Packet Delay

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Delay.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Latency" value in milliseconds. For eg: 1000 to simulate a packet delay of 1 second on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet delay to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Duplication

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Duplicate.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be duplicated. For eg: 10 to simulate a packet duplication of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet duplication to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Loss

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Loss.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be dropped. For eg: 10 to simulate a packet drop of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet drop to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Corruption

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Corruption.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be corrupted. For eg: 10 to simulate a packet corruption of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet corruption to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Docker State Change

Docker State Change faults enable you to abruptly stop or pause containers running on a Docker host. Unlike other infrastructure faults like CPU, Memory and Disk IO this fault is specific to the Docker endpoint and does not have a timeout field because the fault completes very quickly. Some containers may be configured for auto-start and some might require a manual start command to be executed. For the first case, auto-remediation through Mangle is not needed. For the second case, a manual remediation can be triggered from the Requests and Reports tab after the fault completes.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Docker ---> State Change.
Select an Endpoint (Only Docker Endpoints are listed).
Select the fault.
Provide a "Container Name".
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Delete Resource

Kubernetes (K8s) Delete Resource faults enable you to abruptly delete pods or nodes within a K8s cluster. Unlike other infrastructure faults like CPU, Memory and Disk IO this fault is specific to the K8s endpoint and does not have a timeout field because the fault completes very quickly. In most cases, K8s will automatically replace the deleted resource. This fault allows you see how the applications hosted on these pods behave in the event of rescheduling when a K8s resource is deleted and re-created.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Delete Resource.
Select an Endpoint (Only K8S endpoints are listed).
Select a Resource Type: POD or NODE.
Select a Resource identifier: Resource Name or Resource Labels.
If you choose Resource Name to identify a pod or a node, enter a string.
If you choose Resource Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Resource Not Ready

Kubernetes (K8s) Resource Not Ready faults enable you to abruptly put pods or nodes within a K8s cluster into a state that is not suitable for scheduling.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Delete Resource.
Select an Endpoint (Only K8S endpoints are listed).
Select a Resource Type: POD or NODE.
Select a Resource identifier: Resource Name or Resource Labels.
If you choose Resource Name to identify a pod or a node, enter a string.
If you choose Resource Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Provide an app container name. Please note that the application specified should have a readiness probe configured for this fault to be triggered successfully.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Service Not Available

Kubernetes (K8s) Service Not Available faults enable you to abruptly make a service resource in K8s cluster not available, although the pod will be healthy and running.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Service Unavailable.
Select an Endpoint (Only K8S endpoints are listed).
Choose the appropriate service identifier: Service Name or Service Labels.
If you choose Service Name, enter an appropriate string.
If you choose Service Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter Disk Fault

vCenter Disk faults enable you to abruptly disconnect disks from a virtual machine in its inventory. It requires the VM Disk ID and VM Name to trigger the fault. For all vCenter faults, Mangle talks to the mangle-vc-adapter to connect and perform the required action on VC. So it is mandatory that you install the mangle-vc-adapter container prior to adding vCenter Endpoints or running vCenter faults. To find how to install and configure the mangle-vc-adapter, please refer here.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> Disk.
Select an Endpoint (Only vCenter endpoints are listed).
Select the fault: Disconnect Disk.
Provide the VM Name and VM Disk ID. To identify the disk id, the VM moid is required. This information can be gathered from the vCenter MOB (Managed Object Browser). Refer to Looking up Managed Object Reference for vCenter for help on this. Once you have retrieved the VM moid, the disk id can be retrieved from the disk properties section in the link below after replacing the values in angle braces <>:
https:///mob/?moid=&doPath=layout
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter NIC Fault

vCenter NIC faults enable you to abruptly disconnect network interface cards from a virtual machine in its inventory. It requires the VM Nic ID and VM Name to trigger the fault. For all vCenter faults, Mangle talks to the mangle-vc-adapter to connect and perform the required action on VC. So it is mandatory that you install the mangle-vc-adapter container prior to adding vCenter Endpoints or running vCenter faults. To find how to install and configure the mangle-vc-adapter, please refer here.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> Nic.
Select an Endpoint (Only vCenter endpoints are listed).
Select the fault: Disconnect Nic.
Provide the VM Name and VM Nic ID. To identify the Nic id, the VM moid is required. This information can be gathered from the vCenter MOB (Managed Object Browser). Refer to Looking up Managed Object Reference for vCenter for help on this. Once you have retrieved the VM moid, the disk id can be retrieved from the deviceConfigId section in the link below after replacing the values in angle braces <>:
https:///mob/?moid=&doPath=guest%2enet
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter VM State Change Fault

vCenter VM State Change faults enable you to abruptly power-off, reset or suspend any virtual machine in its inventory. It requires just the VM Name to trigger the fault. For all vCenter faults, Mangle talks to the mangle-vc-adapter to connect and perform the required action on VC. So it is mandatory that you install the mangle-vc-adapter container prior to adding vCenter Endpoints or running vCenter faults. To find how to install and configure the mangle-vc-adapter, please refer here.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> State.
Select an Endpoint (Only vCenter endpoints are listed).
Select one of the faults: Poweroff, Suspend or Reset VM.
Provide the VM Name.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

AWS EC2 State Change Fault

AWS EC2 State Change fault enables you to abruptly terminate, stop or reboot any EC2 instance. It requires AWS tags to uniquely identify instances against which the fault should run.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> AWS ---> EC2 ---> State.
Select an Endpoint (Only AWS accounts are listed).
Select one of the faults: Terminate_Instances, Stop_Instances, Reboot_Instances.
Provide the AWS tag (key value pair to uniquely identify the instance(s). Since multiple instances can have the same tag, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one instance from a list of instances identified using the tag, for introducing the fault. If "Random Injection" is set to false, it will introduce the fault into all the instances identified using the tag.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

AWS EC2 Network Fault

AWS EC2 Network fault enable you to abruptly terminate, stop or reboot any EC2 instance. It requires AWS tags to uniquely identify instances against which the fault should run.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> AWS ---> EC2 ---> Network.
Select an Endpoint (Only AWS accounts are listed).
Select the faults: Block_All_Network_Traffic.
Provide the AWS tag (key value pair to uniquely identify the instance(s). Since multiple instances can have the same tag, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one instance from a list of instances identified using the tag, for introducing the fault. If "Random Injection" is set to false, it will introduce the fault into all the instances identified using the tag.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Relevant API Reference

For access to relevant API Swagger documentation:

Application Faults

For version 1.0, Mangle supported the following types of application faults:

CPU Fault
Memory Fault

From version 2.0, apart from the faults listed above, support has been extended to the following new faults:

File Handler Leak Fault
Thread Leak Fault
Java Method Latency Fault
Spring Service Latency Fault
Spring Service Exception Fault
Simulate Java Exception
Kill JVM Fault

CPU Fault

CPU fault enables spiking cpu usage values for a selected application within a specified endpoint by a percentage specified by the user. Mangle uses a modified Byteman agent to simulate this fault and supports only Java based applications at present. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure which includes a cleanup of the Byteman agent from the target endpoint.

This fault therefore takes additional arguments to identify the application under test.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> CPU.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide a "CPU Load" value. For eg: 80 to simulate a CPU usage of 80% on the selected application.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the CPU load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Memory Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Memory.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide a "Memory Load" value. For eg: 80 to simulate a Memory usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Memory load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

File Handler Leak Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Memory.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the File Handler leak to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Thread Leak Fault

Thread Leak fault enables you to simulate conditions where an open thread is not closed. This condition if left over extended periods of time, leads to too many open threads thus creating thread leaks and out of memory issues. Usually a thread dump is required to troubleshoot such issues. With the help of a timeout field the duration for the fault run can be specified after which Mangle triggers the automatic remediation procedure.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Thread Leak.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Set of Out of Memory required flag to true if you want the thread leak to eventually result in OOM errors.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Thread leak to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Java Method Latency Fault

Java Method Latency Fault helps you simulate a condition where calls to a specific Java method can be delayed by a specific time. Please note that you would have to be familiar with the application code; Java classes and methods in order to simulate this fault.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Java Method Latency.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Spring Service Latency Fault

Spring Service Latency Fault helps you simulate a condition where calls to a specific API can be delayed by a specific time. Please note that you would have to be familiar with the REST application URLs and calls in order to simulate this fault.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Spring Service Latency.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Service URI" as /rest/api/v1/plugin if the REST URL of interest is as follows https://xxx.vmware.com/mangle-services/rest/api/v1/plugins.
Provide "Http Method" as GET, POST, PUT, PATCH or DELETE as applicable.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Spring Service Exception Fault

Spring Service Exception Fault helps you simulate a condition where calls to a specific API can be simulated to throw an exception. Please note that you would have to be familiar with the REST application URLs and calls; application code, classes, methods and exceptions in order to simulate this fault.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Spring Service Exception.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Service URI" as /rest/api/v1/plugin if the REST URL of interest is as follows https://xxx.vmware.com/mangle-services/rest/api/v1/plugins.
Provide "Http Method" as GET, POST, PUT, PATCH or DELETE as applicable.
Provide "Exception Class" as for eg: java.lang.NullPointerException if you want a null pointer exception to be thrown.
Provide "Exception Message" as any sample message for testing purposes.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Simulate Java Exception

Java Method Exception Fault helps you simulate a condition where calls to a specific Java method can result in exceptions. Please note that you would have to be familiar with the application code; Java classes, methods and exceptions in order to simulate this fault.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Simulate Java Exception.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Provide "Exception Class" as for eg: java.lang.NullPointerException if you want a null pointer exception to be thrown.
Provide "Exception Message" as any sample message for testing purposes.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kill JVM Fault

Kill JVM Fault helps you simulate a condition where JVM crashes with specific exit codes when calls to a specific Java method are done. Please note that you would have to be familiar with the application code; Java classes, methods and exceptions in order to simulate this fault.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Kill JVM.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Select an appropriate "Exit Code" from the drop down menu.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Relevant API Reference

For access to relevant API Swagger documentation:

Custom Faults

Task-Extension: An example is available as HelloManglePluginTaskHelper at package com.vmware.mangle.test.plugin.helpers of mangle-plugin-skeleton. This task Helper is an implementation of AbstractRemoteCommandExecutionTaskHelper. The implementation of AbstractRemoteCommandExecutionTaskHelper is only expected to provide the implementation for below methods:

public Task init(T faultSpec) throws MangleException;

Should provide the Implementation to initialize the Task Helper for executing the Fault. And the commands required for injection/remediation of the Fault are expected to be provided here. More details on the model for providing the Command Information is explained later.

public Task init(T taskData, String injectedTaskId) throws MangleException;

Should provide the Implementation to initialize the Task Helper for executing the Fault, if the existing Task id also provided. This method will be used for executing the Remediation on a Task if the Remediation is available. This initialization is not used for task rerun or the Re-trigger.

public void executeTask(Task task) throws MangleException;

Provide the Implementation for execution steps required in addition to Implementation available in AbstractRemoteCommandExecutionTaskHelper. Plugin developer can use this interface to invoke his own implementation of Helpers for supporting his Fault across multiple endpoints supported in mangle.

protected ICommandExecutor getExecutor(Task task) throws MangleException;

Provide the Implementation for defining the Executor required for the Fault Execution. Mangle provide a default implementation of a executor for each Supported Endpoint. The Plugin user is free to use his own executor as long as he is implementing the resource as per the interface ICommandExecutor available at package com.vmware.mangle.utils;

protected void checkTaskSpecificPrerequisites(Task task) throws MangleException;

Provide the Implementation if the Fault being developed expect the test machine to be satisfying a condition for the execution. This step is separated from the Fault execution as Mangle wants to make sure the Fault execution or Remediation will not leave the user environment in a irrecoverable state due to execution of them in a non-perquisite satisfying machine.

protected void prepareEndpoint(Task task, List listOfFaultInjectionScripts) throws MangleException; Provide the Implementation if the Fault execution needs certain changes to the Test Machine before execution. Examples are Copying a binary file required to execute a fault. This step is optional for user as the predefined implementation already copies the files returned by listFaultInjectionScripts() to the remote machine.

public String getDescription(Task task);

Provide Implementation to generate description for Fault based on user inputs to help him to identify the task in future through the description. A generic implementation is already available at TaskDescriptionUtils.getDescription(task).

public List listFaultInjectionScripts(Task task);

Provide a implementation that return details of the support scrips to be copied to test machine required for executing the fault getting implemented.

Task-Extension Deep Dive: An example is available as HelloManglePluginTaskHelper at package com.vmware.mangle.plugin.tasks.impl of mangle-plugin-skeleton. This task Helper is an implementation of AbstractRemoteCommandExecutionTaskHelper. The implementation of AbstractRemoteCommandExecutionTaskHelper is only expected to provide the implementation for below methods:

public Task init(T faultSpec) throws MangleException ;

public Task init(T taskData, String injectedTaskId) throws MangleException;

public void executeTask(Task task) throws MangleException;

protected ICommandExecutor getExecutor(Task task) throws MangleException;

Provide the Implementation for defining the Executor required for the Fault Execution. Mangle provide a default implementation of a executor for each Supported Endpoint. The Plugin user should use appropriate executor as per the endpoint provided as the target. Below is the Mapping of Executors to their Endpoint Types.

REMOTE_MACHINE – SSHUtils
DOCKER - DockerCommandUtils
AWS - AWSCommandExecutor
K8s - KubernetesCommandLineClient
vCENTER - VCenterCommandExecutor

EndpointClientFactory class of mangle-task-framework can be used for initializing the appropriate Executor for Injecting the Fault as per user request.

All these executors expect the user to provide a command to be executed on the target machine with associated meta data to mark if it is executed successfully.

protected void checkTaskSpecificPrerequisites(Task task) throws MangleException;

public String getDescription(Task task);

public List listFaultInjectionScripts(Task task);

Provide an implementation that return details of the support scrips to be copied to test machine required for executing the fault getting implemented. The support files can be any file required to be placed in the target in order to execute the developed fault. All the out of the box executors is capable of copying files to the corresponding targeted endpoint and the process completes automatically by default implementation of the AbstractRemoteCommandExecutionTaskHelper, provide that the names of the files are returned through listFaultInjectionScripts() implementation.

private List getInjectionCommandInfoList(T faultSpec) {}

Provide the commands to be executed for the Fault to be Injected. The commands should be provided as List. The fields and descriptions for the CommandInfo Fields.

private String command; String value of the actual command with references to members in pool of variables will be available to executor during command execution. The types and the accessing mechanism are explained in below section.
private boolean ignoreExitValueCheck; Boolean value to find if a command execution result should consider the return value of the command execution. Can be given false where there can be possibility that the command execution need not be resulted in only success return value, but it will be based on the command output.
private List expectedCommandOutputList; List of patterns to be provided to validate a command execution output to consider if the execution is success. The relation among the patterns verification is defaulted to logical ‘or’.
private int noOfRetries; Retries to be attempted by the executor before marking the command execution as a Failure.
private int retryInterval; Interval in seconds between any two attempts of a command execution incase of execution failures and opted for retry attempts.
private int timeout; Timeout interval in milliseconds to consider a command execution failure if the response was not received by the executor from the target.
private Map knownFailureMap; Mapping of Patterns to be looked for in the command execution output, to provide easier troubleshooting messages to user by masking stack traces in the result.
private List commandOutputProcessingInfoList; Explained in detailed below.

public class CommandOutputProcessingInfo
Fields are
1. private String regExpression;
  Regular Expression Pattern to be used to collect an crucial information from current command’s execution to make it available throughout the Fault execution.
2. private String extractedPropertyName;
  Name should be given to the collected information using the pattern given as regExpression
Types of Variables and Their Usage:
The information provided by the user or collected during the runtime of Fault are made available to command executor as below types of Variables.
1. TaskTroubleShootingInfo of the Task holds the extracted information from the command execution Output.
2. args field of CommandExecutionFaultSpec available as taskData in Task holds the data received from the user as args.
3. $FI_ADD_INFO_FieldName can be used to refer to variables from TaskTroubleShootingInfo
4. $FI_ARG_Fieldname can be used to refer to variables from args.
5. $FI_STACK can be used to refer to the output of the previous command.
private List getRemediationCommandInfoList(T faultSpec) {}
Provide the commands to for remediating the fault already Injected. The semantics of CommandInfo is same as it described in the previous section. The args and TaskTroubleShootingInfo collected during the injection will be available during the execution of remediation as well. Hence the dependency data from injection task can be passed to remediation by using the References in the commands.

Requests and Reports

Request & Reports page provides insight to the tasks running during fault execution, fault remediation and triggering of scheduled jobs. Mangle creates tasks that transition to one of the stages : NOT STARTED, IN_PROGRESS, COMPLETED, FAILED.

Processed Requests

It provides details of the tasks executed by Mangle.

Important fields of Mangle tasks

Task Name: Name of the task created for any fault execution, remediation or schedule.
Status: Will reflect one of the Stages : NOT STARTED, IN_PROGRESS, COMPLETED, FAILED.
Endpoint Name: Name of the targeted endpoint during fault execution.
Task Type: Type of the task executed. For eg: INJECTION or REMEDIATION
Task Description: You can get more details about the fault, fault parameters, endpoint targeted, targeted component within an endpoint etc form this field.
Start Time: Task trigger time
End Time: Task end time

Supported operations for Mangle tasks

Click on to understand what operations are supported for a specific task.

Primarily, the operations supported are Delete, Remediate Fault and Report.

Remediate Fault option will be enabled only if the the task type is INJECTION and status is set to COMPLETED.

Delete is not supported for tasks created through scheduled jobs.

Refreshing the Mangle task data grid

Scheduled Jobs

Scheduled Jobs data grid lists all the schedules available on Mangle.

Important fields of schedules

ID: Contains id of the schedule.
Job Type: Type of the schedule. For eg: CRON, SIMPLE
Scheduled At: Recurrence and Time at which the schedule will be triggered. If job type is CRON, it shows a cron expression and if the job type is SIMPLE, it shows the epoch time in milliseconds.
Status: Status of the schedule. Will reflect one of the values: INITIALIZING, CANCELLED, SCHEDULED, FINISHED, PAUSED, SCHEDULE_FAILED

Triggers of each schedule

Click on the ID link of each schedule to view all the triggers of that schedule.

Supported operations for Mangle schedules

Primarily, the operations supported are Cancel, Pause, Resume, Reports, Delete, and Delete Schedule Only.

Refreshing the schedule data grid

Logs

Click on the Logs link to open up a browser window displaying the current Mangle application log.

Relevant API Reference

For access to relevant API Swagger documentation:

troubleshooting-guide

Deployment Stage

Deployment involves deploying the Mangle appliance OVA or the containers.

During Deployment, the user provides customization such as configuring the root password and other optional configurations such as providing TLS certificates.

Deployment Failures

We have not experienced many failures during Deployment. If any issues occur, provide the Support Information from below.

Support Information

Provide the following information to support if encountering Deployment Stage failures:

Hash (MD5, SHA-1, or SHA-256) of the OVA/container images you deployed
Deployment method:
Deployment environment
- Verify that the targeted datastore has enough space
- Provide details about the targeted vCenter compute, storage, and networking

Boot/Initialization Stage

Boot involves the Mangle appliance powering on and the containers being setup.

At the end of boot or initialization phase, the Mangle application should be available at URL: https:///mangle-services and the default admin user should be prompted to change password on login.

Boot Failures

Support Information

Provide the following information to support if encountering Boot Stage failures:

Is this a fresh deploy or an upgrade?
- If upgrading, what version is the old Mangle application?
Were any changes made to the Mangle configuration at the time of deployment?
- If yes, what changes?
Are you able to view the web page at http://?
- If no, did you provide custom TLS certificates during the Deployment Stage?
  - If yes and the version is 1.3.1 or less, verify the format is correct [Certificate
    Reference](#additional-information)
  - Run journalctl -u fileserverand provide the entire resulting output to support
- If no, attempt SSH debugging in the following step.
Are you able to SSH into the Mangle appliance? If no, continue with [Network
Troubleshooting](#network-troubleshooting)
- Please obtain a Mangle appliance support bundle

Network Troubleshooting

Does the VM console of the deployed Mangle appliance show an IP address? Is this address the expected
value based on DHCP or provided static IP settings?
Do you have a route to the deployed Mangle appliance's IP address?
- ping
  - If ping is successful, but SSH is not, check network firewall settings.
  - If ping is not successful, check network settings and continue with [Console
    Troubleshooting](#console-troubleshooting)

Console Troubleshooting

The goal of this step is to be able to SSH to the Mangle appliance to allow for better debugging information to be obtained from the appliance.

Access the vSphere console for the Mangle appliance. Press ALT + F2 to access the login prompt.
- Login with username root and the credentials you provided in the OVA deployment
  customization. If the deployment has failed to set your credentials, the default password is
  vmware.
- Are there any startup components that failed to start?
  - Run docker ps. It should list two or three containers in running state; mangle or mangleWEB, mangleDB and the mangle-vsphere-adapter.
    If no, continue with the next steps. If DB container is not running execute:
    docker start mangleDB. Wait for 10-20 seconds and run docker start mangleWEB. Wait for a couple of seconds and see if the portal below can be reached.
- Run ip addr show
  - Is the IP address the expected value based on DHCP or provided static IP settings?
- Run ip route show
  - Is the default route valid?
- Can you ping the default gateway? Run ping. Obtain the default gateway IP from the ip route show command output.
  - If no, check your network settings. Attach the Mangle appliance to a network that has a valid
    route between your client and the appliance.
  - If yes, verify the routing configuration between the client that is unable to SSH to the mangle appliance.
- If still unable to SSH to the Mangle appliance, provide the output of the following commands to
  support:
  - docker start mangleDB
  - docker start mangleWEB
  - ip addr show
  - ip route show
  - ping

Endpoint Addition Stage

Endpoint addition is the first step to starting your chaos engineering experiments. It helps you set up the targets for the experiment.

Usually you encounter failures in this stage if Mangle has some difficulty reaching the endpoint either due to network connectivity issues, blocked ports or restrictive firewalls, bad credentials or wrong IP/Hostname values.

Common Error Codes and Next Steps

FIRM01: Connection Refused

Usually affects a remote machine endpoint.
Ensure that the machine is remotely accessible by running the ping command.
Ensure that the ssh service is running and the credentials are correct.

FIRM03: Mangle requires file transfer access

Usually affects a remote machine endpoint.
Ensure that the sftp configuration on the remote machine is correct.
Ensure that the ssh service is running and the credentials are correct.

Cannot connect to adapter while adding a vCenter Endpoint

Verify if you are able to open up the vCenter adapter Swagger URL which is normally available at https://:8443/mangle-vc-adapter/swagger-ui.html
Verify if they can get the health of the vCenter adapter from the mangle container using:
curl -k https://:8443/mangle-vc-adapter/application/health
If the first verification succeeds and the second fails, inter-container communication is blocker. So ensure that the port used for the vCenter adapter is open. If you have followed the mangle documentation this port is usually 8443.
If the first and second verification fails, check if the vCenter adapter container is up and running.

Fault Injection Stage

Last updated 10 months ago

Infrastructure Faults

For version 1.0, Mangle supported the following types of infrastructure faults:

CPU Fault
Memory Fault
Disk IO Fault
Kill Process Fault
Docker State Change Faults
Kubernetes Delete Resource Fault
Kubernetes Resource Not Ready Fault
vCenter Disk Fault
vCenter NIC Fault
vCenter VM State Change Fault

From version 2.0, apart from the faults listed above, support has been extended to the following new faults:

File Handler Leak Fault
Disk Space Fault
Kernel Panic Fault
Network Faults: Packet Delay, Packet Duplication, Packet Loss, Packet Corruption
Kubernetes Service Unavailable Fault
AWS EC2 State Change Fault
AWS EC2 Network Fault

Minor improvements have also been included for Kill Process Fault in version 2 of Mangle.

CPU Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> CPU.
Select an Endpoint.
Provide a "CPU Load" value. For eg: 80 to simulate a CPU usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the CPU load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Memory Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Memory.
Select an Endpoint.
Provide a "Memory Load" value. For eg: 80 to simulate a Memory usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Memory load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Disk IO Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Disk IO.
Select an Endpoint.
Provide a "IO Size" value in bytes. For eg: To write in blocks of 5 KB to the disk of the selected Endpoint specify the IO Size as 5120 (5 KB = 5120 bytes). With the specified block size of 5120 bytes, Mangle will not use more than 5 MB (5 MB = 5120 * 1024 bytes) of disk space during the simulation of fault. The space is cleared at the time of fault remediation.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the IO load of 8192000 to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kill Process Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Kill Process.
Select an Endpoint.
Provide a "Process Identifier". This can either be a process id or process name. A process name is preferred if the fault is expected to be scheduled.
From version 2.0 onward, Kill Process fault can kill multiple processes with the same process descriptor. This can be done by setting the "Kill All" drop down to true. If set to false, it will fail if the process descriptor is not unique. Alternatively, you can also use the process id to uniquely identify and kill a process.
Provide a "Remediation Command". For eg: To start the sshd process that was killed on an Ubuntu 17 Server, specify the remediation command as "sudo service ssh start" .
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

File Handler Leak Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> File Handler Leak.
Select an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the out of file handles error to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Disk Space Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Disk Space.
Select an Endpoint.
Provide a "Target Directory" so Mangle can target a specific directory location or partition to write to for simulating the low disk space condition.
Provide a "Load" value. For eg: 80 to simulate a Disk usage of 80% of the total disk size or space allocated for a partition, on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the low disk or out of disk condition to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kernel Panic Fault

REMEDIATION OPTIONS FOR KERNEL PANIC

To modify this setting as a one-time option, please run the following command on the endpoint sysctl --system

To modify this setting permanently, remotely log in to the endpoint, modify file /etc/sysctl.d/99-sysctl.conf and add the following command

kernel.panic = 20

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Kernel Panic.
Select an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Network Faults

Packet Delay

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Delay.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Latency" value in milliseconds. For eg: 1000 to simulate a packet delay of 1 second on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet delay to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Duplication

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Duplicate.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be duplicated. For eg: 10 to simulate a packet duplication of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet duplication to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Loss

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Loss.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be dropped. For eg: 10 to simulate a packet drop of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet drop to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Packet Corruption

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Network ---> Packet Corruption.
Select an Endpoint.
Provide a "Nic Name". For eg: For a remote machine endpoint Nic name could be eth0, eth1 etc depending on what adapter you would want to target for the fault.
Provide a "Percentage" value to specify what percentage of the packets should be corrupted. For eg: 10 to simulate a packet corruption of 10 percentage on a particular network interface of an Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the packet corruption to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Docker State Change

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> Docker ---> State Change.
Select an Endpoint (Only Docker Endpoints are listed).
Select the fault.
Provide a "Container Name".
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Delete Resource

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Delete Resource.
Select an Endpoint (Only K8S endpoints are listed).
Select a Resource Type: POD or NODE.
Select a Resource identifier: Resource Name or Resource Labels.
If you choose Resource Name to identify a pod or a node, enter a string.
If you choose Resource Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Resource Not Ready

Kubernetes (K8s) Resource Not Ready faults enable you to abruptly put pods or nodes within a K8s cluster into a state that is not suitable for scheduling.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Delete Resource.
Select an Endpoint (Only K8S endpoints are listed).
Select a Resource Type: POD or NODE.
Select a Resource identifier: Resource Name or Resource Labels.
If you choose Resource Name to identify a pod or a node, enter a string.
If you choose Resource Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Provide an app container name. Please note that the application specified should have a readiness probe configured for this fault to be triggered successfully.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kubernetes Service Not Available

Kubernetes (K8s) Service Not Available faults enable you to abruptly make a service resource in K8s cluster not available, although the pod will be healthy and running.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> K8S ---> Service Unavailable.
Select an Endpoint (Only K8S endpoints are listed).
Choose the appropriate service identifier: Service Name or Service Labels.
If you choose Service Name, enter an appropriate string.
If you choose Service Labels provide a key value pair for eg: app=mangle. Since multiple resources can have the same label, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one resource in a list of resources identified using the label, for introducing the fault. If "Random Injection" is set to false, it will introduce fault into all resources identified using the resource label.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". Remediation requests are not supported for this fault.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter Disk Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> Disk.
Select an Endpoint (Only vCenter endpoints are listed).
Select the fault: Disconnect Disk.
Provide the VM Name and VM Disk ID. To identify the disk id, the VM moid is required. This information can be gathered from the vCenter MOB (Managed Object Browser). Refer to Looking up Managed Object Reference for vCenter for help on this. Once you have retrieved the VM moid, the disk id can be retrieved from the disk properties section in the link below after replacing the values in angle braces <>:
https:///mob/?moid=&doPath=layout
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter NIC Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> Nic.
Select an Endpoint (Only vCenter endpoints are listed).
Select the fault: Disconnect Nic.
Provide the VM Name and VM Nic ID. To identify the Nic id, the VM moid is required. This information can be gathered from the vCenter MOB (Managed Object Browser). Refer to Looking up Managed Object Reference for vCenter for help on this. Once you have retrieved the VM moid, the disk id can be retrieved from the deviceConfigId section in the link below after replacing the values in angle braces <>:
https:///mob/?moid=&doPath=guest%2enet
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

vCenter VM State Change Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> vCenter ---> State.
Select an Endpoint (Only vCenter endpoints are listed).
Select one of the faults: Poweroff, Suspend or Reset VM.
Provide the VM Name.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

AWS EC2 State Change Fault

AWS EC2 State Change fault enables you to abruptly terminate, stop or reboot any EC2 instance. It requires AWS tags to uniquely identify instances against which the fault should run.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> AWS ---> EC2 ---> State.
Select an Endpoint (Only AWS accounts are listed).
Select one of the faults: Terminate_Instances, Stop_Instances, Reboot_Instances.
Provide the AWS tag (key value pair to uniquely identify the instance(s). Since multiple instances can have the same tag, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one instance from a list of instances identified using the tag, for introducing the fault. If "Random Injection" is set to false, it will introduce the fault into all the instances identified using the tag.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

AWS EC2 Network Fault

AWS EC2 Network fault enable you to abruptly terminate, stop or reboot any EC2 instance. It requires AWS tags to uniquely identify instances against which the fault should run.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Infrastructure Faults ---> AWS ---> EC2 ---> Network.
Select an Endpoint (Only AWS accounts are listed).
Select the faults: Block_All_Network_Traffic.
Provide the AWS tag (key value pair to uniquely identify the instance(s). Since multiple instances can have the same tag, you also need to specify if you are interested in a Random Injection. If "Random Injection" is set to true, Mangle will randomly choose one instance from a list of instances identified using the tag, for introducing the fault. If "Random Injection" is set to false, it will introduce the fault into all the instances identified using the tag.
Schedule options are not available for this fault.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Relevant API Reference

For access to relevant API Swagger documentation:

Please traverse to link -----> API Documentation from the Mangle UI or access https:///mangle-services/swagger-ui.html#/fault-injection-controller

Application Faults

For version 1.0, Mangle supported the following types of application faults:

CPU Fault
Memory Fault

From version 2.0, apart from the faults listed above, support has been extended to the following new faults:

File Handler Leak Fault
Thread Leak Fault
Java Method Latency Fault
Spring Service Latency Fault
Spring Service Exception Fault
Simulate Java Exception
Kill JVM Fault

CPU Fault

This fault therefore takes additional arguments to identify the application under test.

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> CPU.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide a "CPU Load" value. For eg: 80 to simulate a CPU usage of 80% on the selected application.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the CPU load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Memory Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Memory.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide a "Memory Load" value. For eg: 80 to simulate a Memory usage of 80% on the selected Endpoint.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Memory load of 80% to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

File Handler Leak Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Memory.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the File Handler leak to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Thread Leak Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Thread Leak.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Set of Out of Memory required flag to true if you want the thread leak to eventually result in OOM errors.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide a "Timeout" value in milliseconds. For eg: if you need the Thread leak to be sustained for a duration of 1 hour then you should provide the timeout value as 3600000 (1 hour = 3600000 ms). After this duration, Mangle will ensure remediation of the fault without any manual intervention.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Java Method Latency Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Java Method Latency.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Spring Service Latency Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Spring Service Latency.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Service URI" as /rest/api/v1/plugin if the REST URL of interest is as follows https://xxx.vmware.com/mangle-services/rest/api/v1/plugins.
Provide "Http Method" as GET, POST, PUT, PATCH or DELETE as applicable.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Spring Service Exception Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Spring Service Exception.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Service URI" as /rest/api/v1/plugin if the REST URL of interest is as follows https://xxx.vmware.com/mangle-services/rest/api/v1/plugins.
Provide "Http Method" as GET, POST, PUT, PATCH or DELETE as applicable.
Provide "Exception Class" as for eg: java.lang.NullPointerException if you want a null pointer exception to be thrown.
Provide "Exception Message" as any sample message for testing purposes.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Simulate Java Exception

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Simulate Java Exception.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Provide "Exception Class" as for eg: java.lang.NullPointerException if you want a null pointer exception to be thrown.
Provide "Exception Message" as any sample message for testing purposes.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Kill JVM Fault

Steps to follow:

Login as a user with read and write privileges to Mangle.
Navigate to Fault Execution tab ---> Application Faults ---> Kill JVM.
Select an Endpoint.
If the Endpoint is of type Kubernetes:
Provide additional K8s arguments such as Container Name, Pod Labels and the Random Injection flag.
If the Endpoint is of type Docker:
Provide additional Docker argument such as Container Name.
Provide "Injection Home Dir" only if you would like Mangle to push the script files needed to simulate the fault to a specific location on the endpoint. Else the default temp location will be used.
Provide "Latency" value in milliseconds so that Mangle can delay calls to the method by that time.
Provide "Class Name" as PluginController if the class of interest is defined as public class PluginController {...}.
Provide "Method Name" as getPlugins if the method to be tested is defined as follows:
public ResponseEntity> getPlugins(
@RequestParam(value = "pluginId", required = false) String pluginId, @RequestParam(value = "extensionType", required = false) ExtensionType extensionType) {
log.info("PluginController getPlugins() Start.............");
if (StringUtils.hasLength(pluginId)) {
return new ResponseEntity<>(pluginService.getExtensions(pluginId, extensionType), HttpStatus.OK);
}
return new ResponseEntity<>(pluginService.getExtensions(), HttpStatus.OK);
}
Provide "Rule Event" as "AT ENTRY" OR "AT EXIT" to specify if the fault has to be introduced in the beginning or at the end of the method call.
Select an appropriate "Exit Code" from the drop down menu.
Provide additional JVM properties such as Java Home, JVM Process, Free Port and Logon User. For eg: If the application under test is a VMware application then the JRE for the application resides in a specific location so for Java Home enter string /usr/java/jre-vmware/bin/java. The JVM Process can be either the process id of the application or the JVM descriptor name. In cases where you schedule, application faults, it is preferable to specify the JVM descriptor name. The Free Port is for the Byteman agent to talk to the application, so provide one that is not in use. The Logon User should be a user who has permissions to access and run the application under test. If it is root specify that else specify the appropriate user id.
Schedule options are required only if the fault needs to be re-executed at regular intervals against an endpoint.
Tags are key value pairs that will be send to the active monitoring tool under Mangle Admin settings ---> Metric Providers at the time of publishing events for fault injection and remediation. They are not mandatory.
Click on Run Fault.
The user will be re-directed to the Processed Requests section under Requests & Reports tab.
If Mangle was able to successfully trigger the fault, the status of the task will change to "COMPLETED". The fault will continue to run at the endpoint until the timeout expires or a remediation request is triggered. The option to trigger a remediation request at anytime can be found on clicking the button against the task in the Processed Requests table.
For monitoring purposes, log into either Wavefront or Datadog once it is configured as an active Metric provider in Mangle and refer to the Events section. Events similar to the screenshots provided below will be available on the monitoring tool for tracking purposes.

Relevant API Reference

For access to relevant API Swagger documentation:

Please traverse to link -----> API Documentation from the Mangle UI or access https:///mangle-services/swagger-ui.html#/fault-injection-controller