Chaos Testing
Apply some chaos engineering by throwing in some HTTP errors or network delays. Understanding failure scenarios is a critical aspect of microservices architecture (aka distributed computing)
Before Start
You should have NO virtualservice nor destinationrule (in
|
HTTP Error 503
By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift
$ kubectl get pods -l app=recommendation -n tutorial
NAME READY STATUS RESTARTS AGE
recommendation-v1-3719512284-7mlzw 2/2 Running 6 18h
recommendation-v2-2815683430-vn77w 2/2 Running 0 3h
You can inject 503’s, for approximately 90% of the requests
kubectl create -f istiofiles/destination-rule-recommendation.yml -n tutorial
kubectl create -f istiofiles/virtual-service-recommendation-503.yml -n tutorial
./scripts/run.sh
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 138
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v1 from '6976858b48-dswhz': 139
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 139
customer => preference => recommendation v1 from '6976858b48-dswhz': 140
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 140
customer => preference => recommendation v1 from '6976858b48-dswhz': 141
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 141
Clean up
kubectl delete -f istiofiles/virtual-service-recommendation-503.yml -n tutorial
Delay
The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.
kubectl create -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial
kubectl replace -f istiofiles/destination-rule-recommendation.yml -n tutorial
And hit the customer endpoint
./scripts/run.sh